Blog post: “Some takes on UV & cancer”
• Part 1: In which I use my optical physics background to share some hopefully-uncontroversial observations
• Part 2: In which I boldly defy Public Health Orthodoxy on the whole UV situation
www.lesswrong.com/posts/t7GeZn...
There’s a bunch of discussion at www.lesswrong.com/posts/E9xfgJ... & www.lesswrong.com/posts/NEFNs4... , including the comments sections where company staff has been answering questions
New blog post: “‘Act-based approval-directed agents’, for IDA skeptics” www.alignmentforum.org/posts/RKtTi8...
screenshot of the title and first couple paragraphs of the article at the link
New blog post: “You can’t imitation-learn how to continual-learn” www.lesswrong.com/posts/9rCTjb...
I linkposted a podcast with some excerpts. Title: “Jeremy Howard is bearish on LLMs” [I mean “bearish” compared to most people in my professional circles] www.lesswrong.com/posts/hvun2m...
I.e., this kind of “self-play” will make them dumber and dumber, gradually at first but inexorably.
(Not sure there’s no point in arguing about this—presumably we can just wait and see. ¯\_(ツ)_/¯ )
…by proposing new ideas and “training themselves” when those ideas “seem right” to them, then I don’t think they’ll invent Ricci flow, rather I think they’ll have bad ideas that “seem right” to them, then lock them into the training data, and mistakes will compound in a spiral into nonsense.
If math is a human enterprise that LLMs are helping with, it’s basically OK, because the LLM’s “reflexes” are all honed on good (human-provided) data. Whereas if we pretrain LLMs on exclusively pre-1970 math, put them in a sealed box for 100 years, and ask them to discover new-to-them math concepts…
I disagree. I think LLMs lack a general ability to notice that something doesn’t make sense, a sense that humans have, but this deficiency is not too obvious because human-provided training data can substitute in-distribution. Cf. Litt’s last blog post ↓
We’re talking about this kind of self-contained system that searches for proofs, and adds Lean-verified ones to the training data perpetually, right? …I would call that system “RL”.
Terminology aside, I claim it shares with RL the property of asymptoting to ruthlessness as self-play proceeds.
So the latter (no-ground-truth) version might not be ruthless, but I don’t think you can ASI that way.
(Sorry if I’m misunderstanding.)
Another version would lack any ground truth—it would just be LLMs judging each other. What I actually expect here is that the system would be incompetent. If the LLM judges make mistakes, the “self-play” would make the system ever more confident about those mistakes. It would spiral into nonsense.
OK so one version of this would have ground truth (e.g. proof assistant) gatekeeping the training data. In this case, as you self-play more and more, I claim you’ll dilute away any human kindness from pretraining, and gradually turn the LLM into a ruthless pursuer of “satisfy the proof assistant”.
There’s one human brain design, barely changed since Pleistocene Africa. Many copies of it, over centuries, built language, science, technology, & the whole global economy from scratch.
If an AI design can’t do that, I’d vote against even calling it “human-level” let alone “ASI”.
So I guess “yes”
This requires a kind of open-ended continual autonomous learning and figuring-things-out and putting those things in the weights (not context window). Nobody has yet invented that for LLMs (though they’re sure trying). See also §1 of www.lesswrong.com/posts/2yLyT6...
This part ↓ (including the link www.lesswrong.com/posts/xJWBof...) might help explain what I have in mind by “ASI”. There isn’t a quadrillion-dollar market for humans with linear-time SAT solvers. Whereas ASIs could run an ever-growing global economy themselves, including self-reproducing etc.
(3) Even if we got past those two hurdles, I don’t think the results would be great for reasons here ↓
[& sorry if I’m missing your point :) ]
(2) If people tried that, I expect they would explore a space of RL reward functions in which EVERY possibility leads to ruthless sociopaths. (I claim evolution does weird unorthodox things with reward functions, beyond the imaginings of RL researchers, see alignmentforum.org/posts/xw8P8H... )
(1) That’s very unlikely to happen even if it’s a good idea. (Note that almost nobody does that in RL today. Generally, outer-loop searches in ML are super expensive.)
Social instincts are part of the RL reward function, not the trained model. So in theory, an RL programmer could do an outer-loop search over RL reward functions, as evolution did. This is true! But, some problems are:
In this post, I make my case, including why we should expect superintelligence to be MUCH MORE ruthless than either humans or LLMs. (2/2) www.lesswrong.com/posts/ZJZZEu...
New post: “Why we should expect ruthless sociopath ASI”
A rift between super-pessimists like me, vs the “merely” AI-concerned, is an intuition that future AI will be kinda like a ruthless sociopath. I claim it’s a sound intuition, but it might seem to come from left field… 1/2
I dunno, feels pretty major to me. Here’s the changelog.
My post from last week, “The nature of LLM algorithmic progress”, is now a heavily-rewritten version 2! Thanks commenters for setting me straight on a number of points :) www.lesswrong.com/posts/sGNFtW...
Blog post: “In (highly contingent!) defense of interpretability-in-the-loop ML training”.
Using interpretability as input into a loss function / reward function has a bad rap (and deservedly so). But there’s a specific version of it that might work. www.alignmentforum.org/posts/ArXAyz...
Blog post: “The nature of LLM algorithmic progress”
bit of Cunningham’s Law energy with this one: spicy hot takes, far outside my area of expertise. Feedback welcome! www.lesswrong.com/posts/sGNFtW...
The part where I agree: we need to get there, somehow or other—right now we don’t have the deep understanding required for high-reliability engineering, so we’d better get it! Link again: www.alignmentforum.org/posts/hiigux... (3/3)
The part where I disagree: some people say that if we “just” apply known best practices, everything will be fine. I summarize what those best practices are, and argue that applying those best practices to AGI, in our current state of understanding, is impossible. (2/3)
New blog post: “Are there lessons from high-reliability engineering for AGI safety?” People sometimes suggest that high-reliability engineering is a model for how AGI safety could or should work. I agree in some ways and disagree in other ways. (1/3) www.alignmentforum.org/posts/hiigux...
