Long Context In-Context Compression by Getting to the Gist of Gisting Long context processing is critical for the adoption of LLMs, but existing methods often introduce architectural complexity that hinders their practical adoption. Gisting, an in-context compression me...

It’s incredible how much mileage you get out of simple ideas, once you understand where the original method breaks!

📄 Read the full paper here: arxiv.org/abs/2504.08934

This work is led by our brilliant intern Aleks Petrov, with Mark Sandler, Andrey Zhmoginov, and Nolan Miller as co-authors.

The result is GistPool:

✅ Matches or beats average pooling;
✅ Fixes the issues with Gisting;
✅ Achieves compression using in-context learning with no complex model modifications, so it’s easy to implement and deploy at scale.

It’s fast, simple, and works across datasets and compression rates.

So we added an inductive bias:

👉 Spread gist tokens across the context
👉 Restrict each gist token to attend only to its own pooling window

This tiny masking tweak nudges the model toward pooling—and suddenly performance shoots up.

Even fixing these issues, Gisting still couldn’t match average pooling. Why? Because standard attention can't learn average pooling well. We show both experimentally and theoretically: attention struggles with simple operations like copying and pooling! 😫

Second issue: 𝐜𝐨𝐧𝐟𝐥𝐢𝐜𝐭𝐢𝐧𝐠 𝐨𝐛𝐣𝐞𝐜𝐭𝐢𝐯𝐞𝐬. You're asking the same parameters to summarize and to do inference. These clash. We split the two: one set of parameters for compressing context, another for predicting answers. That helped a lot. But still not enough. 🖖

First culprit: 𝐢𝐧𝐟𝐨𝐫𝐦𝐚𝐭𝐢𝐨𝐧 𝐟𝐥𝐨𝐰 𝐝𝐞𝐥𝐚𝐲. Gist tokens summarizing layer i only get the summary to the model at layer i+2. That’s too late—the model expects the information a layer earlier. When we shifted activations down, performance improved immediately! 🔧

Surprisingly, even a simple 𝘢𝘷𝘦𝘳𝘢𝘨𝘦 𝘱𝘰𝘰𝘭𝘪𝘯𝘨—just mean over token activations—beats Gisting by a mile. That shouldn’t happen. Why can't Gisting at least learn to emulate average pooling? 🤔

Gisting (Mu et al., 2023) offers a starting point: just add "gist" tokens and adjust the attention mask to funnel context through them. This creates an attention bottleneck, forcing summarization into these tokens.

But it turns out that this method when compressing more than just a few tokens.

Handling long contexts efficiently is a major hurdle for LLMs! While models support longer windows, cost & effectiveness remain challenging.

Excited to share our paper on in-context compression for long contexts.

Check out his thread and the paper bellow! 👇
arxiv.org/abs/2504.08934

It’s incredible how much mileage you get out of simple ideas, once you understand where the original method breaks!

📄 Read the full paper here: arxiv.org/abs/2504.08934

This work is led by our brilliant intern Aleks Petrov, with Mark Sandler, Andrey Zhmoginov, and Nolan Miller as co-authors.

The result is GistPool:

✅ Fixes the issues with Gisting;
✅ Matches or beats AvgPool baseline;
✅ Achieves compression using in-context learning, meaning no complex model modifications are needed.

It’s fast, simple, and works across datasets and compression rates.

So we added an inductive bias:

👉 Spread gist tokens across the context
👉 Restrict each gist token to attend only to its own pooling window

This tiny masking tweak nudges the model toward pooling—and suddenly performance shoots up.

Even fixing these issues, Gisting still couldn’t match average pooling. Why? Because standard attention can't learn average pooling well. We show both experimentally and theoretically: attention struggles with simple operations like copying and pooling! 😫

Second issue: 𝐜𝐨𝐧𝐟𝐥𝐢𝐜𝐭𝐢𝐧𝐠 𝐨𝐛𝐣𝐞𝐜𝐭𝐢𝐯𝐞𝐬. You're asking the same parameters to summarize and to do inference. These clash. We split the two: one set of parameters for compressing context, another for predicting answers. That helped a lot. But still not enough. 🖖

First culprit: 𝐢𝐧𝐟𝐨𝐫𝐦𝐚𝐭𝐢𝐨𝐧 𝐟𝐥𝐨𝐰 𝐝𝐞𝐥𝐚𝐲. Gist tokens summarizing layer i only get the summary to the model at layer i+2. That’s too late—the model expects the information a layer earlier. When we shifted activations down, performance improved immediately! 🔧

Surprisingly, even a simple 𝘢𝘷𝘦𝘳𝘢𝘨𝘦 𝘱𝘰𝘰𝘭𝘪𝘯𝘨—just mean over token activations—beats Gisting by a mile. That shouldn’t happen. Why can't Gisting at least learn to emulate average pooling? 🤔

Gisting (Mu et al., 2023) offers a starting point: just add "gist" tokens and adjust the attention mask to funnel context through them. This creates an attention bottleneck, forcing summarization into these tokens.

But it turns out that this method when compressing more than just a few tokens.

Long Context In-Context Compression by Getting to the Gist of Gisting Long context processing is critical for the adoption of LLMs, but existing methods often introduce architectural complexity that hinders their practical adoption. Gisting, an in-context compression me...

It’s incredible how much mileage you get out of simple ideas, once you understand where the original method breaks!

📄 Read the full paper here: arxiv.org/abs/2504.08934

This work is led by our brilliant intern Aleks Petrov, with Mark Sandler, Andrey Zhmoginov, and Nolan Miller as co-authors.

The result is GistPool:

✅ Matches or beats average pooling;
✅ Fixes the issues with Gisting;
✅ Makes only minimal modifications to the classic transformer so it’s easy to implement and deploy at scale.

It’s fast, simple, and works across datasets and compression rates.

So we added an inductive bias:

👉 Spread gist tokens across the context
👉 Restrict each gist token to attend only to its own pooling window

This tiny masking tweak nudges the model toward pooling—and suddenly performance shoots up.

Even fixing these issues, Gisting still couldn’t match average pooling. Why? Because standard attention can't learn average pooling well. We show both experimentally and theoretically: attention struggles with simple operations like copying and pooling! 😫

Second issue: 𝐜𝐨𝐧𝐟𝐥𝐢𝐜𝐭𝐢𝐧𝐠 𝐨𝐛𝐣𝐞𝐜𝐭𝐢𝐯𝐞𝐬. You're asking the same parameters to summarize and to do inference. These clash. We split the two: one set of parameters for compressing context, another for predicting answers. That helped a lot. But still not enough. 🖖

First culprit: 𝐢𝐧𝐟𝐨𝐫𝐦𝐚𝐭𝐢𝐨𝐧 𝐟𝐥𝐨𝐰 𝐝𝐞𝐥𝐚𝐲. Gist tokens summarizing layer i only get the summary to the model at layer i+2. That’s too late—the model expects the information a layer earlier. When we shifted activations down, performance improved immediately! 🔧

Surprisingly, even a simple 𝘢𝘷𝘦𝘳𝘢𝘨𝘦 𝘱𝘰𝘰𝘭𝘪𝘯𝘨—just mean over token activations—beats Gisting by a mile. That shouldn’t happen. Why can't Gisting at least learn to emulate average pooling? 🤔

Gisting (Mu et al., 2023) offers a starting point: just add "gist" tokens and adjust the attention mask to funnel context through them. This creates an attention bottleneck, forcing summarization into these tokens.

But turns out that this method when compressing more than just a few tokens.

Don't lie to your friends: Learning what you know from collaborative self-play To be helpful assistants, AI agents must be aware of their own capabilities and limitations. This includes knowing when to answer from parametric knowledge versus using tools, when to trust tool outpu...

We all want LLMs to collaborate with humans to help them achieve their goals. But LLMs are not trained to collaborate, they are trained to imitate. Can we teach LM agents to help humans by first making them help each other?

arxiv.org/abs/2503.14481

Amazing work! Do you think it would be possible to express a general learning algorithms using ALTA, e.g. gradient descent?