Optimal Pruning for Neural Architectures using Fisher Information Distances

A new approach to pruning neural‑network parameters relies on differential geometric distance within model space. When a weight is pruned, its value is set to zero, effectively moving the model toward the surface where that parameter vanishes. The shortest distance between the original model and that surface is measured using the geodesic defined by the Fisher information metric, allowing an accurate assessment of the true impact of pruning on model performance.

Building on successive approximations of this geodesic distance, the authors construct a hierarchy of pruning techniques, starting with the conventional magnitude‑based method and progressing to more sophisticated schemes that exploit Fisher information. The method is evaluated on fully connected networks and Vision Transformers using the MNIST and CIFAR‑10 datasets, covering the full pruning range from 0 % to 100 % and repeating experiments with five random seeds. Across all architecture‑data combinations, the proposed technique outperforms both magnitude pruning and Fisher‑only pruning in terms of accuracy and Matthews correlation coefficient.

Moreover, by considering intermediate levels of geodesic approximation, the authors obtain pruning methods that require less computation while retaining near‑optimal performance. This proposal not only provides a state‑of‑the‑art pruning tool but also offers a solid mathematical justification for parameter‑reduction methods, which is relevant for developing more efficient and sustainable AI models.

Read the original article on arXiv cs.AI

This summary is an informational synthesis produced by dataqbs.com. All rights to the original content belong to its author and the cited media outlet. We act solely as curators of technology news and claim no authorship.