The best part? It's all open source, warehouse-native, and transparent - you can audit exactly how we calculate everything.
Want to see the implementation details? Check out our deep dive: blog.growthbook.io/holdouts-dee...
We built this because our community of 5,000+ engineers kept asking for better ways to prove impact. One customer told us: "Holdouts helped us discover our 'successful' feature was actually causing a 2% revenue decline after 30 days."
🪝 Catch delayed effects - Some changes take weeks to show impact
⚖️ Measure cumulative impact - Stack multiple features and still understand individual contribution
🤝 Build trust in your data - Show stakeholders exactly what drove that revenue increase
Think of them as your experiment's control group that persists even after launch. While 95% of users get your shiny new feature, 5% stay in the original experience. This gives you a clean baseline to measure against.
Ever wonder if your feature actually moved the needle, or if that metric bump was just seasonal variance?
Most teams struggle with this because they can't isolate the real impact of their changes.
That's where holdouts come in - and why we just shipped them in GrowthBook.
What are holdouts?
Yesterday's comic was, of course, in reference to Twyman's Law.
Applicable not only to experiment analysis, but also life. If it looks too good to be true, then it probably is 😅
A black-and-white comic strip with two stick figures. The first figure says, "When a metric becomes a target, it ceases to be a good metric." The second figure, wearing a hat, responds, "Sounds bad. Let's offer a bonus to anyone who identifies a metric that has become a target." The comic illustrates Goodhart's Law, which states that when a measure is used as a goal, it will no longer serve as a good measure.
This fantastic comic (xkcd 2899) is about which law?
The kicker? ChatGPT already drives 10% of new signups for companies like Vercel. This isn't future speculation - it's happening now.
Here's our full guide: blog.growthbook.io/guide-to-geo/
Question: Is anyone A/B testing GEO yet? How is it working?
• There's no page 2 in AI search - you're either in the answer or you're invisible
• Statistics boost AI visibility by 33.9% - AI can't create data, so it gravitates toward sources that provide it
• Most AI traffic is invisible - it shows up as "Direct" in analytics
As a company, we do things companies do:
• Run A/B tests (obviously)
• SEO
• And now, GEO - optimizing for visibility in AI searches like ChatGPT
What's GEO (Generative Engine Optimization)? We were wondering that too, so we did some research.
Here are 3 surprising findings:
This week we've been sharing everything included in our latest 4.1 release, which includes:
→ 150+ commits
→ 8 major features (holdouts, dashboards, #AI, Vercel, Pre-computed dimensions, MCP, FerretDB, Sanity)
→ 0 vendor lock-in
Curious? Full release notes: blog.growthbook.io/growthbook-v...
We're just going to go ahead and say it: GrowthBook is the most comprehensive open-source experimentation platform.
Now with #FerretDB support.
Because open source means ALL the way down.
Learn more: docs.growthbook.io/self-host/fe...
We provide an in-depth guide and a video tutorial to help you set it up efficiently. Watch the tutorial here: www.youtube.com/watch?v=tg85...
Sanity offers a powerful content management solution that effortlessly integrates with GrowthBook, enabling you to conduct experiments directly from your content editor without altering any code. This approach allows you to benefit from feature-flag experiments seamlessly.
If you're building on Vercel and NOT running experiments... what are you even doing? 😅
🔗 Install from Vercel Marketplace: vercel.com/marketplace/...
#Vercel #NextJS #WebPerformance #EdgeComputing
The technical details you care about:
→ SDK size: <15KB gzipped
→ Evaluation time: <1ms at edge
→ Zero cold starts
→ Works with App Router, Pages Router, and Edge Runtime
Plus, your feature flags now live where your code deploys. No more environment mismatches.
GrowthBook is now native on Vercel Marketplace 🎯
What this means for your Next.js apps:
→ One-click installation
→ Unified billing (one less vendor to manage)
→ Edge-ready performance (we're talking microseconds)
→ Feature flags that actually work at the edge
We've introduced practical AI features in GrowthBook 4.1:
🤖 Auto-summarize experiment results
📊 Generate SQL queries from natural language
💡 Enhance experiment hypotheses with AI
🔍 Automatically find similar past experiments
➕ Customize prompts for your org
🙅 Prefer no AI? Disable it entirely.
custom dashboard in GrowthBook
Your PM: "What happened with that experiment?"
You: Opens 17 tabs, 3 spreadsheets, and a Slack thread
Sound familiar? We fixed that.
Introducing Experiment Dashboards in GrowthBook 4.1:
✅ Create custom views
✅ Highlight metrics
✅ Add context
✅ Share a single link instead of a dissertation
The issue with A/B tests? They conclude. But what about tracking the combined effect of 50 updates over six months? Enter Holdouts: the perpetual experiment. Keep 5% of users on the initial experience, while others enjoy new features. Gauge the true long-term impact of your product changes.
Most holdouts measure only shipped features. Ours measure everything—including failed experiments.
Because that's what your users actually experienced.
Technical breakdown → blog.growthbook.io/holdouts-dee...
Experiments showed +5%, +3%, +7% wins, but do they really add up to +15%? Likely not, as Airbnb found naive sums can overstate impact by 2x. GrowthBook now offers ✨ Holdouts ✨ for accurate metric measurement:
Another big week for the experimentation industry: OpenAI has acquired Statsig for 1.1B.
Graham shares his thoughts on the acquisition and our new Statsig to GrowthBook migration kit:
www.linkedin.com/posts/graham...
It's possible to build beautiful sites with Framer. But how do you know if they actually convert?
You don't unless you test! Our official plugin for Framer lets you do enterprise-level experimentation on your components.
Our tutorial helps you get set up in minutes. youtu.be/XqHDLbYCfTM?...
It's a hard truth—but we're not all SQL geniuses.
Get a little (or lot of help) with your next query by using our new natural language to SQL generation feature. Ask, for example, to list the top 10 pages by view count in the past 3 months and get back a crisp SELECT statement.
See Graham's demo
Teams often struggle with an ever-growing list of metrics, ballooning from 20 to 200. Finding a specific metric during a new experiment can be tough, leading to even more creation ☠️. AI can automatically summarize metrics, making them easier to find and preventing duplicates. Watch Graham's demo 👇
Hypothesis enforcement just dropped ✨
No, you won't get a ticket for sloppy hypotheses—but this AI-powered feature ensures your team writes them right.
YOU set the quality bar. Academic rigor or lighter touch, your choice.
Don't just run experiments, run them right.
Automatic experiment result summary generation is here 🪄 Get a summary of your experiment results instantly, including key findings based on your metrics and variants, that's totally customizable to your organization's style and requirements.
Here's the tutorial you've been looking for: #Sanity + #Next.js + GrowthBook
Ryan provides a comprehensive walkthrough of everything you need to know to set up an enterprise-grade experimentation system that everyone on the team can use!
youtu.be/tg85K9K_8-g
Graham and Ryan demo all the new AI-powered features recently released in GrowthBook.
The best part? Every AI prompt is fully customizable. Want results in haikus? Different language? Your company's tone of voice? You control it all.
youtu.be/jhyGxknbGk0
