Spectral decomposition methods and metrics used to quantify representation space complexity in LLMs.

📐We measured representation complexity using the #eigenspectrum of the final layer representations. We used 2 spectral metrics:

- Spectral Decay Rate, αReQ: Fraction of variance in non-dominant directions.

- RankMe: Effective Rank; #dims truly active.

⬇️αReQ ⇒ ⬆️RankMe ⇒ More complex!

🧵1/9