Evals for AI Engineers Stop using guesswork to find out how your AI applications are performing. Evals for AI Engineers equips you with the proven tools and processes required to systematically test,... - Selection from Eva...

Relevant links

- Our course: evals.info

- Early release ( just has the TOC & intro now learning.oreilly.com/library/view...

👀 Animals have been assigned.

Scheduled to print fall 2026!

We have iterated on this with over 3k students (and continue to do so). We give our students access to the full draft as part of our evals course (link in bio)

Hi

Love that @hamel.bsky.social is putting on a hackathon where the goal is for your agent to score the highest on evaluations, not just do something flashy.

click.convertkit-mail2.com/gkumlz753lc5...

Building resilient prompts using an evaluation flywheel | OpenAI Cookbook This cookbook provides a practical guide on how to use the OpenAI Platform to easily build resilience into your prompts. A resilient prom...

If your looking to get started with evals check out this cookbook from @hamel.bsky.social
cookbook.openai.com/examples/eva...

"Can I just get an LLM to do my error analysis?"

We get this question constantly. The answer is no, and trying is the fastest way to miss critical bugs.

Full podcast: youtu.be/BsWxPI9UM4c?...

Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar Hamel Husain and Shreya Shankar teach the world’s most popular course on AI evals and have trained over 2,000 PMs and engineers (including many teams at OpenAI and Anthropic). In this conversation,…

Full podcast. youtu.be/BsWxPI9UM4c?...

We are teaching our final course of the year on evals starting this Monday. You can enroll with this link to get 35% off: maven.com/parlance-lab...

I recently sat down with Lenny Rachitsky to discuss why AI Evals are becoming the most sought after skill for product builders.

As a bonus, we step through an end-to-end example of building an eval in a spreadsheet so everyone can understand. See reply for links.

I think it’s more like 99% 🤣 the 1% worked super hard on data modeling

You Don't Need a Graph DB Many teams adopt graph databases believing they need specialized tools for relationship data, adding unnecessary complexity to their stack. This session reveals that for most use cases, the…

This one is going to be spicy. 80% of the time I've seen a graph DB in production, it's been an overcomplicated mess (especially in AI applications).

In this talk, Jo and I will discuss when GraphDBs are overkill and when they actually make sense.

Sign up here:

For technical domains especially, getting non-DSes involved in analyzing outputs is vital. It’s hard to build anything good without it bc v1s almost always have major fail modes. Finding the appropriate system design—let alone optimizing—requires a tight coupling of output analysis and system design

Can you screenshot it and tell me what it is so I can troubleshoot it

AI Evals Email Course A free 17-part email series on the principles of application-centric LLM evals.

If you're wanted to learn applied AI evals but not sure if its for you, @sh-reya.bsky.social and I put together something that might help.

This free email course compiles what we've learned from teaching 2k+ students. It’s 17 emails plus 2 free e-books.

Here's the link: ai.hamel.dev/eval-course

I've been eval pilled by @hamel.bsky.social . Everyone is all let's build some MCP servers and ship a minimal AISRE as quickly as possible. And I'm writing a design doc about how to build evals with @runme.dev so we can iterate rapidly on the AI

Last chance to signup for this free lesson with OpenAI on evals, Including a sneak peek of their new eval products!

Link: maven.com/p/d2dc30/how...

AI Evals For Engineers: Book Review YouTube video by Hamel Husain

Link to full talk youtube.com/live/jJMYWfQ...

AI Evals For Engineers & PMs by Hamel Husain and Shreya Shankar on Maven Learn proven approaches for quickly improving AI applications. Build AI that works better than the competition, regardless of the use-case.

We'll be discussing this in our upcoming course on May 19th - AI Evals For Engineers & PMs (This link has a 35% discount code):

maven.com/parlance-lab...

Can non-data scientists write AI Evals? The answer is nuanced and not just "Yes". @eugeneyan.com and I discuss this in the context of the "analyze-measure-improve" cycle from our course.

Links to more resources in the reply

If you are writing evals without error analysis, our course AI Evals for Engineers & PMs is for you. Begins monday next week. Full syllabus in this link: maven.com/parlance-lab...

LLM Evals: Common Mistakes YouTube video by Hamel Husain

It is very easy to make mistakes when creating evals for your AI product. @sh-reya.bsky.social and I run through the most common errors in this talk.

35% discount code to our upcoming course in the video notes

youtu.be/GL0XhAj5LPE?...

I thought this was a meme 🤣 … but it’s real

GitHub CoPilot is one of the first commercially successful LLM products (predating ChatGPT). What was the secret? A robust eval suite!

In this lightning lesson, John Berryman will reveal the eval techniques (and mistakes) from working on this product

maven.com/p/da8264/how...

Effective Evals for AI products

@hamel.bsky.social & @sh-reya.bsky.social are two of the world's best on evals. They've built evals for 35+ AI apps & helped teams ship confidently. Now they'll teach everything they know on building evals that work.

Enrollment closes in 4 days.

Secret 35% discount code: maven.com/parlance-lab...

GitHub CoPilot is one of the first commercially successful LLM products (predating ChatGPT). What was the secret? A robust eval suite!

In this lightning lesson, John Berryman will reveal the eval techniques (and mistakes) from working on this product

maven.com/p/da8264/how...

I keep hearing about the emerging role of AI PM. How is this any different than a normal PM? Is it hype? We are gonna find out in this free lightning lesson. I will ask difficult questions. With @schof.bsky.social and Aman Khan

maven.com/p/544677/wha...

I keep hearing about the emerging role of AI PM. How is this any different than a normal PM? Is it hype? We are gonna find out in this free lightning lesson. I will ask difficult questions. With @schof.bsky.social and Aman Khan

maven.com/p/544677/wha...

AI Evals For Engineers & PMs by Hamel Husain and Shreya Shankar on Maven Learn proven approaches for quickly improving AI applications. Build AI that works better than the competition, regardless of the use-case.

As genAI projects mature, proper evals are becoming table stakes for production deployment. But how do we evaluate probabilistic machines?

Looking forward to learning about the latest techniques and best practices from @hamel.bsky.social and Shreya Shankar next month!

Last chance to sign up for this. Recording sent to everyone who signs up. maven.com/p/29a33a/hyb...

@hamel.bsky.social & his wisdom on evals, error analysis, looking at your data is what we need. Here are his 10 Don'ts:
• Don't skip error analysis
• Don't skip looking at your data
• Don't gatekeep who can write prompts
• Don't let zero users be a roadblock
• Don't be blindsided by criteria drift

If you are building RAG applications, you don't want to miss this. Doug Turnbull is going to show you his tricks he's learned from a decade of optimizing retrieval in search systems, and how that transfers to RAG.

Link: maven.com/p/29a33a/hyb...