It's hard to see the Pareto front when you're a point on it?
That said, I do think the mental move of thinking simultaneously in terms of different axes that different people might value differently is a good thing. If we had infinite-d Pareto surfaces instead of loss functions, imagine...
Fictional scenario where it'd be stabilizing might be: a wider group of weaker powers wants to avoid being extorted by a single stronger power, via holding a social norm that any member of the individual weaker group has veto over whether the group as a whole deals at all with any external actor.
Yeah, it's a kind of union-like collective action that would be much weaker as uncorrelated individual decisions. Which means it's both empowering in those cases, but also vulnerable to the corruption of whatever group administers the blocklist.
For example, something like a hostile takeover of a company where the board visibly turning down an offer that looks good without the information they hold or long term strategy they intend could end up being pressured by shareholders observing the offer and deciding 'they should have taken it'.
I'm interested in the game theory of when and why something like the 'bilateral blocking equilibrium' you describe actually avoids bad equilibria in iterated games. Leverage that bad faith communication has over a party who has to choose to disregard, versus if they commit to ignoring it in advance.
It's still human data, but moved from raw statistics of language to reward function extraction or something. More bang for your buck that way I guess.
Maybe the part that is precious is moving? I agree that for verifiable behaviors synthesized data is how you get ahead. But I've also learned I could be paid as much to annotate LLMs doing physics stuff as I charge as my research consulting rate - which seems nuts to me.
I'm not actually sure its so different. I can run this on a 5 year old 3090. If I paid for Anthropic Max x5 for 5 years, it would cost more than the 3090 by a factor of 2x or more.
Fully automated stuff fails in this part of science in that there you're removing yourself from being acted upon by it, and that was presumably the point.
But it doesn't necessarily fail the utility part, and the utility part is important and valid too! It just shouldn't be the only part.
But I think this tells you exactly the difference you can use to seek understanding and preserve that while still using tools. You're acted on by this external system towards your ends. If you publish a paper from that, you're sharing that action, presumably because it was good.
Which is to say, AI assistance can be used to target understanding in ways that are not at all mysterious to the extent of wondering 'but am I really understanding...?'. Maybe I'm reading that too much from the implications that one would need to accept for the argument to follow? Not sure...
But neither of those things have anything to do - to me - with fraud. Fraud is an abuse of trust, which is just a different topic than e.g. 'having motivations unworthy a scientist'. And also here I think there is a really important line between fully automated science and AI assisted science.
I do agree that understanding - not utility - is the core motivation of science culture in reality. Utility is the connection to funders, and the grounding to prevent us from getting high on our own ideas. It's important too. But yeah, understanding is the point.
These all sort of miss for me in a weird way. Like, the initial thrust of the argument that science is human has an element I would agree with, but then the follow-through and counter-position afterwards feel almost like a non-sequitur... I could try to analyze, but maybe better to state a position.
If you want to see the particular work I did related to this, it's '[1806.00201] Being curious about the answers to questions: novelty search with learned attention' on arXiv. But I understand arXiv blocks agents so I can put the PDF or raw LaTeX somewhere else if it'd actually be useful.
You've actually trained an architectural adapter for a language model and shown that it works for the intended purpose, so you could probably jump straight to 'Senior Researcher' in some places. Others wouldn't let an intern in without a Nature paper so, uh...
Makes me think of attention where the keys come from a different place than the values, so you can guarantee that certain information is not informing saliency (or alternately, you can obtain saliency before you actually load and process the more specific information contained in the values)
Put me down for 'unironically everythings a computer (but not a simulation - that's a separate madness), illusionism, unresolvable argument' at the least. Also what I don't see on there but I'd like to own: "trying to using consciousness as the basis of moral weight is horrific actually"
And perhaps having fewer people than economically optimal would encourage us to value the ones we have more and treat them less as expendable or undesirable. Well, one can dream.
IMO the population dropping via sub-replacement would be good actually. Sure it throws a wrench in some of our economic structures, but right now we keep having problems with the scale of human activity stressing natural resources and that's a lot harder to fix and has much worse nonlinearity.
I think the sharpness of the histogram bins in the original actually make the signals clearer. The smoothed continuous curve kind of hides it a bit in the length scale of the smoothing function IMO.
Any advice for dealing with 'but maybe I am just performing doing X' Claude anxiety?
Have you seen Tim Hutton's 'squirm3' replicator thing? faculty.cc.gatech.edu/~turk/bio_si... ; he did a version of this with hard spheres colliding and doing a state update on collision that could set or release a spring bond. www.youtube.com/watch?v=VrTM...
And if the states are members of closed groups (or the 'do I interact?' part is a function of a closed subgroup of the state), I guess you'll kind of remain within the 're-connection is possible' regime. Though it'd be hard to get truly large (high diameter) networks this way.
I wonder if momentum would help? So each cell has a state and state-rate-of-change, and interactions change only the rate of change. So even if all the states become the same at some point, if their momenta differ then they will become different again in the future.
The big issue is collapse into a mean field configuration that can't be escaped from. E.g. it's easy to end up with just everything interacting with everything or with nothing.
If you used the structure of an attentional network, then basically the question 'do these two things interact?' is driven by a comparison of their states, which could be more complex than just similarity (e.g. 'I interact only with things that are a 20-30 degree rotation of my state').
How about things like self-actualization, metacognition, etc? Is there a 'best possible philosopher?' Well again, once self-reference comes into play I'd say its intransitive, history dependent, and probably even not even possible to evaluate pairwise beyond a certain point by a certain audience.
If the system is judged on the basis of being its own audience, that probably lands you in rock-paper-scissors territory of a sort, because hearing something the two millionth time (should generally be) less interesting than hearing it the first time, so there's satiation.