There's a lot more on the way. Stay tuned.

We hope this eval gives governing agencies and model developers a useful starting point in measuring and responding to one of the many, many potentially disruptive aspects of AI.

I and my colleagues at @securebio.org had (and continue to have) tons of discussions about whether it is a net positive even to release or publicize this. Ultimately, we come down on the side of "it's really important that society knows about this."

There are some concerning implications about literally anyone having access to what is, at some level, an expert virologist in their pocket.

While it will probably help lots of well-meaning researchers make faster progress, it also lowers the barriers to those who would do harm with biology.

For the last little while, I've been helping to create a tool to measure the capacity of LLM AI models to answer meaningful, challenging, practical questions about virology lab work.

Short answer: as best we can measure, they're way better at it than actual virologists. www.virologytest.ai