Introducing WW-PGD: A Cutting-Edge Add-On for Optimizer Enhancement 🚀

Discover the latest release of WW-PGD, a PyTorch add-on designed to supercharge your model training by integrating epoch-boundary spectral projections with standard optimizers. Unleash optimized performance and detailed spectral control in your deep learning workflows!

Announcing: 𝗪𝗪-𝗣𝗚𝗗 — 𝗪𝗲𝗶𝗴𝗵𝘁𝗪𝗮𝘁𝗰𝗵𝗲𝗿 𝗣𝗿𝗼𝗷𝗲𝗰𝘁𝗲𝗱 𝗚𝗿𝗮𝗱𝗶𝗲𝗻𝘁 𝗗𝗲𝘀𝗰𝗲𝗻𝘁 🚀

I’m thrilled to announce the release of WW-PGD—a novel PyTorch add-on designed to empower your deep learning optimization process. This small yet powerful tool wraps around standard optimizers like SGD, Adam, and AdamW, incorporating an epoch-boundary spectral projection powered by WeightWatcher diagnostics.

🚀 Elevator Pitch

WW-PGD doesn’t just optimize; it strategically nudges each layer towards the Exact Renormalization Group (ERG) critical manifold during training. This approach ensures that you’re aiming for the right optimization targets right from the get-go, rather than relying on post-hoc diagnostics.

📚 Theory in Short

• HTSR Critical Condition: α ≈ 2
• SETOL ERG Condition: trace-log(λ) over the spectral tail = 0

By making these conditions explicit optimization goals, WW-PGD brings a new level of precision to layer management during training.

⚙️ How It Works

Here’s a quick overview of the mechanics:

1. Runs WeightWatcher (ww) at Epoch Boundaries: At the end of each epoch, WW-PGD evaluates the model’s weight distribution.
2. Identifies the Spectral Tail: Utilizes layer quality metrics from ww to determine which portion of the weight distribution is the spectral tail.
3. Optimal Tail Guess Selection: It selects an optimal guess for the tail at each epoch.
4. Applies Projected Gradient Descent Update: Uses a stable, Cayley-like Proximal step to update the layer’s spectral density.
5. Retracts to Satisfy SETOL ERG Condition: Ensures that updates adhere to the spectral constraints.
6. Blends Projected Weights Back In: Incorporates a "warmup" + ramping process to avoid instability early on.

In essence, WW-PGD provides a mechanism to project the optimizer’s results onto the ERG critical manifold, enhancing efficiency in spectral constraint optimization.

🔍 Scope (Important)

This initial public release is tailored for training small models from scratch, and is not yet optimized for large-scale fine-tuning tasks. Consider it a proof of concept, with ongoing tests extending to:

• 3-layer MLPs (MNIST / FashionMNIST)
• nano-GPT-style small Transformer models

Future work is dedicated to adapting larger architectures and fine-tuning workflows.

📊 Early Results (FashionMNIST, 35 Epochs, Mean ± Std)

The initial tests yield intriguing results:

• Plain Test: Baseline 98.05% ± 0.13 vs WW-PGD 97.99% ± 0.17
• Augmented Test: Baseline 96.24% ± 0.17 vs WW-PGD 96.23% ± 0.20

This indicates that while accuracy is nearly neutral at this scale, WW-PGD offers a significant advantage with a spectral control knob and comprehensive per-epoch tuning.

📥 Repo & QuickStart

• 🧩 Repo: GitHub Repository
• 📓 QuickStart (with MLP3+FashionMNIST example): QuickStart Guide
• 🔍 More Info: WeightWatcher

If you’re experimenting with training and optimization on your models, or looking for a data-free spectral health monitor + projection step, your feedback is invaluable. Join us in exploring other optimizers or small Transformer setups!

💬 Community Engagement

Join the WeightWatcher Community on Discord to share insights and learn from fellow developers: Discord Invitation

A special thanks to Hari Kishan Prakash for his invaluable contributions to this project!

If you have any questions or need assistance with AI, feel free to reach out. Let’s talk! #talkToChuck