I must also add that I’m assuming there’s no breakthrough architecture/pre-training/post-training method that pushes us to start everything from scratch. I’m simply asking about the decision factors in greenlighting such a full restart in the current status quo.

Are there any good pointers on when/why one would decide to run pre-training from scratch (and follow it with post-training ofc) to create a fresh LLM? Is it simply about shifting the knowledge cutoff or more than that? Do we know how/if that happens nowadays? What are the deciding factors?

i was annoyed at having many chrome tabs with PDF papers having uninformative titles, so i created a small chrome extension to fix it.

i'm using it for a while now, works well.

today i put it on github. enjoy.

github.com/yoavg/pdf-ta...

Given how bad I am at it, it’s out of my league too; still fun though 😅

Were you doing the NYT’s crossword? That’s how it happened for me. Also, if you want a bonus one, “doe” :)

f’ as in fine-tuned from f, not the derivative of f 😅

I got confused there yoo. Maybe something like “further condition the model’s output” (instead of update the model)?
So if the model is f(x), before the dashed line it’s f’(x), and after that it’s f(x|prompt/context).

USC NLP folks are on Bluesky!
Follow my amazing colleagues here

go.bsky.app/KUwSZ6W