Spectral decomposition methods and metrics used to quantify representation space complexity in LLMs.
📐We measured representation complexity using the #eigenspectrum of the final layer representations. We used 2 spectral metrics:
- Spectral Decay Rate, αReQ: Fraction of variance in non-dominant directions.
- RankMe: Effective Rank; #dims truly active.
⬇️αReQ ⇒ ⬆️RankMe ⇒ More complex!
🧵1/9