I really enjoyed my MLST chat with Tim @neuripsconf.bsky.social about the research we've been doing on reasoning, robustness and human feedback. If you have an hour to spare and are interested in AI robustness, it may be worth a listen 🎧
Check it out at youtu.be/DL7qwmWWk88?...
"Rather than being animals that *think*, we are *animals* that think"; the last sentence of Tom Griffiths's characterisation of human intelligence through limited time, compute, and communication hits different today than it did 4 years ago.
leave parrots alone!!
Sometimes o1's thinking time almost feels like a slight. o1 is like "oh I thought about this uninvolved question of yours for 7 seconds and here is my 20 page essay on it"
What counts as in-context learning (ICL)? Typically, you might think of it as learning a task from a few examples. However, we’ve just written a perspective (arxiv.org/abs/2412.03782) suggesting interpreting a much broader spectrum of behaviors as ICL! Quick summary thread: 1/7
Big congratulations to Dr. @jumelet.bsky.social for obtaining his PhD today and crafting a beautiful thesis full of original and insightful work!! 🎉 arxiv.org/pdf/2411.16433?
I'll be at NeurIPS tues-sun, send me a message if you'd like to chat!
Cool that those experiments changed your mind :) The referenced appendix was important to convince myself of what we eventually concluded (that the correlations indicate procedural knowledge). And thank you for the praise!! What kind of ideas did you get?
If that's how you define retrieval, then they are doing retrieval under your definition. The heavy lifting is of course done by the word "synthesize", how do they do that? That's what we're characterising in the paper
This is an incredible paper that I've longed to do for a long time. However the engineering challenges were far too daunting, so my collaborators and I settled for indirect evidence for this hypothesis instead (or did other things).
It should be much less computationally expensive to do for fine tuning data
Just want to add to Stella's responses that the reason we went with procedural knowledge very much came from the correlation results; documents influence each query with the same underlying task similarly, even though the task is applied to different numbers for different queries.
Definitely! Next time will be Christmas so I presume that's not ideal, but I can reach out when I know the next time I will be there?
What did you think
Yeah. I do think as you become more senior you become better at determining from the intro whether a paper is likely to be good or bad. The point is just that we should still actively keep an open mind when reading the rest of the paper
I learn so much from reviewing, it’s the papers I review that I keep coming back to for my own ideas and citations. They broaden and deepen my view on the field. Let’s give it the time it deserves.
It’s actually pretty cool if you as a reviewer get to make papers better by suggesting improvements. This cycle, I’ve given an 8 where all other reviewers gave a rejecting rating. Now, the scores are 8, 5, 8, 6, 8. Pretty exciting, if you ask me.
You don’t have to add these to the review (unless it’s TMLR). But hold yourself accountable when you are rejecting it. What could the authors do to lift your scores? If the answer is nothing, be sure to have a good reason for this. If there is something, tell the authors.
There’s an easy way to hold yourself accountable (thanks TMLR guidelines ✌️): "make a list of proposed adjustments to the submission, specifying for each whether they are critical to securing your recommendation for acceptance or would simply strengthen the work in your view."
The art of rebuttal is to learn how to stick firmly to the points you believe are important, while at the same time allowing yourself to be wrong. Admitting when you might be misunderstanding (after all, the authors probably spent about ~1000x more time thinking about it).
The hardest part is to keep an open mind all the way down 🐢. The rebuttal phase is the kicker. If you don’t spend enough time in this phase, just don’t sign up to be a reviewer, because it’s incredibly demoralising to people who work months to years on a submission.
I’ve heard people say they know whether they will accept or reject a paper after reading the abstract/intro. That’s great, but what is even greater is when you realise that is *just presentation*, and the soundness and contribution are *not* going to be determined by that part.
Reviewing requires constant questioning of the motive behind your responses, every step of the way. Which btw, according to chatty, will help you become a better scientist yourself.
The art is to lift up the best bits of the paper together with the authors, not to call missing baseline and be done with it.
Personally, reviewing for NeurIPS a couple years back changed me as a reviewer. For one paper I rejected, I kept citing it throughout the year to people for a finding it had. This made me realise it was a good paper, it just had some easy targets for rejection.
To paraphrase Dennett (rip 💔), the goal of reviewing is to determine truth, not to conquer your opponent.
Too many reviewers seem to not have internalised this. In my opinion, this is the hardest lesson a reviewer has to learn, and I want to share some thoughts.
Do you know what rating you’ll give after reading the intro? Are your confidence scores 4 or higher? Do you not respond in rebuttal phases? Are you worried how it will look if your rating is the only 8 among 3’s? This thread is for you.
didnt mean to ask chatgpt how to do it, but rather ask chatgpt for pointers to papers on the topic ;)
