WeightWatcher 0.7 Release: Introducing Advanced Feature – fix_fingers for DNN Analysis

WeightWatcher 0.7 has just been released, and it comes with an exciting new feature for analyzing Deep Neural Networks (DNN) called fix_fingers. This advanced feature aims to improve the reliability of alpha values for model layers and introduces a new metric called num_fingers to report the number of outliers found.

To activate the fix_fingers feature, simply use the following syntax:

details = watcher.analyze(…, fix_fingers=”clip_xmax”, …)

This new feature takes a bit longer to run but promises more consistent results for your model’s layers. It is important to note that other metrics such as alpha_weighted will not be affected by this update.

The motivation behind introducing the fix_fingers feature comes from previous observations made in a Nature paper (Nature Communications 2021) where large alpha values, greater than 8 or even 10, were identified as outliers in GPT2 models. These outliers were problematic as they deviated from the expected behavior of layer alphas in a DNN.

The fix_fingers feature aims to address these outliers by identifying and removing them during the analysis process. By doing so, it helps to improve the overall fit of the Power Law model to the Empirical Spectral Density of the layers, leading to more accurate and stable results.

The weightwatcher project is grounded in the theory of Self-Organized Criticality (SOC), which suggests that DNNs exhibit power law behavior similar to real neurons in biological systems. By leveraging this theory, weightwatcher aims to better understand the behavior of DNNs and unlock their full potential.

In addition to the fix_fingers feature, weightwatcher also offers new and improved metrics for models like GPT and GPT2. By running the analysis with the fix_fingers option, users can generate histograms of layer alphas to compare the performance of different models.

Overall, the new fix_fingers feature in WeightWatcher 0.7 introduces a more robust and reliable Power Law estimator for DNN analysis. Users are encouraged to try out this feature and provide feedback on its effectiveness. For more information on WeightWatcher and consulting services related to Data Science, Machine Learning, and AI, reach out to Calculation Consulting at Info@CalculationConsulting.com.