read the full paper: https://arxiv.org/abs/2502.12996
@huggingface page: https://huggingface.co/papers/2502.12996
congrats to my collaborators @SatyenKale who led that work and Yani Donchev
required bandwidth reduction is massive, for a 100B params model:
DP requires 471 Gbits/s
Streaming DiLoCo with inner com. overlap: 1.4 Gbits/s
Streaming DiLoCo with eager outer com. overlap: 400Mbits/s, more than 1000x reduction
400Mbits/s is consumer-grade bandwidth FYI
scaling up to 1B params, and we see that our eager method can reach no loss of performance when synchronizing every 30 steps (thus overlapping 30 computation steps!), and follow closely when overlapping 100 steps
thus we propose an *eager* version:
the update is made of the average of the *local up-to-date* update of the self replica and the *remote stale* update from the other replicas
but its performance are dramatically bad.
we can recover a bit by lowering the outer learning by 4x, but this is still unsatisfying
in this work, we explore if we can overlap an entire outer step, made of dozen to hundred of computation steps!
we first try a naive "delayed" version
Streaming DiLoCo's second contribution is to overlap communication with computation, massively increasing the tolerable latency
we can safely overlap up to 5 steps, but more than that and performance drops rapidly!
https://x.com/Ar_Douillard/status/1885292127678021751
DiLoCo allows us to distributed data-parallel across the world by only synchronizing once in a while, thus amortizing the communication cost
however, when syncing, this is a blocking operation!
https://x.com/Ar_Douillard/status/1724732329740976187
arxiv is here: https://arxiv.org/abs/2502.12996
read more below!
one more step towards decentralized learning: Eager Updates
can we overlap communication with computation over hundred of steps?
-- yes we can
in this work led by @SatyenKale, we improve DiLoCo and use x1177 less bandwidth than data-parallel
from Jeff Dean at The Dwarkesh podcast:
"asynchronous training where each copy of the model does local computation [...] it makes people uncomfortable [...] but it actually works"
yep, i can confirm, it does work for real
see arxiv.org/abs/2501.18512
We received an outstanding interest in our #ICLR2025 @iclr-conf.bsky.social workshop on modularity! Please sign up to serve as a reviewer if you are interested in Model Merging, MoEs, and Routing, for Decentralized and Collaborative Learning t.co/HIsZKWNaOx
I'll be in SF in two weeks to talk at the AlgoPerf workshop, and i have a bunch of stickers to give, so let me know if you want to meet!
Big thanks to all my collaborators!
We finished this last spring, and it was one of the coolest project i've been on.
The future will be distributed 🫡
https://arxiv.org/abs/2501.18512v1
All of this is why we say Streaming DiLoCo is a good step towards distributed free lunch 🥪
So so many ideas we try just work on top of DiLoCo. And it can scale too! Look at the cracked folks of @PrimeIntellect who scaled their version to 10B
What if each replica overlap a different num of steps (\tau) because they run a different speeds?
Can we break away from the lockstep synchronization? yes!
Workers can have a few delay steps, and it just work, w/o any special handling.
There are tons of plots, tables, and charts in our paper; but let me share two more exciting plots:
Over how many steps can you overlap safely communication?
At least 5 without any significant loss of perf! That's a massive increase of tolerated latency.
Likewise, with a Llama with 405B parameters.
Speaking about DeepSeek, how to distribute its pretraining across the world with low-bandwidth?
It has only 35B activated params, but you need to sync 671B params in total! Hard to do across continents with data-parallel...
However, with our method? ❤️🔥
Indeed, post-training RL reasoning is easier to distribute (good post here primeintellect.ai/blog/intellect-math ) than pretraining
but we need to scale more our pretraining, this is still a relevant axis!
DeepSeek-R1 also notes it:
Of course the number displayed in that table are from a "simulation", but it's a pretty good indicator to what we find in practice.
Abolish the tyranny of requiring huge bandwidth! ✊
The good part of overlapping communication with computation?
As @m_ryabinin noted in Swarm Parallelism: larger networks spent more time doing computation O(n^3) vs doing communication O(n^2).
We have much more time to sync at larger scales!
My co-author, Yani, built a simulator: a DAG with fwd, bwd, and gradient reduction nodes.
It estimates how much time is spent in the costly com. between non-colocated devices and how much is spent crunching flops.
Put everything together, scale it to 4B, and reach similar performance than data-parallel.
It's even better when overtraining with a larger token budget? remember the bitter lesson? just put more data and flops in your model, Streaming DiLoCo enables that.
[3] You don't need full precision for your communication.
Quantize your update with 4 bits is enough -- you can barely see any changes on the performance.
And that's the free dessert 🍦
[2] Instead of blocking computation to receive the communication, we do several compute steps asynchronously.
See L9-12: the longer your model takes to do fwd/bwd, the more time you have to do communication!
That's the free main course 🍱
[1] Instead of syncing the whole model every hundreds of steps, sync a subset of it!
We split the model in fragments of three layers, it reduces massively the peak bandwidth w/o hurting ML performance.
That's the free appetizer 🥗
With Streaming DiLoCo, we improve over the successful recipe of DiLoCo in three ways:
1. partial synchronization
2. communication overlapping with computation
3. quantized communication
This is a distributed free lunch 🥪
Problem: in data-parallel's every step synchronization, communication is costly!
DiLoCo (arxiv.org/abs/2311.08105 ) synchronizes less often --> amortizing the cost
Later, @PrimeIntellect released Intellect-1, a repro of DiLoCo, with a 10B model! arxiv.org/abs/2412.01152
PrimeIntellect's OpenDiLoCo, an open-source reproduction of DiLoCo, where compute is distributed across the world.
The bitter lesson is that neural networks *really* want more flops and data. This has been the key success behind LLMs. It also means you need tons of compute.
however it's hard to have all that compute available in a single place, can we distribute it across the world?
