This is just a specific example of a more general issue: we're treating hallucinations like a big new AI problem when quite often humans would have the same issue if they didn't know how to look things up. And we're barely investing at all in giving AIs the same level of "just look it up" power.
And the second you *do* give an AI a tool to do the search, it quite prefers to use it over making something up, especially if you provide the most minor nudge in the project instructions or system prompt.
The problem isn't that we need to make LLMs better at knowing all possible package names a priori. The problem is that we haven't given them a tool to do the search.
But: if you asked any developer to come up with a package name out of their head, exactly the same problem would happen. But actual developers don't make up packages out of their heads. They go to npmjs.com , search for packages, and look at signals like popularity, maintenance, etc.
An example I give is: there's *tons* of papers trying to figure out how to get AIs to stop hallucinating packages and endless hand wringing about the supply chain implications of package hallucinations + typosquatting.
I've said it before, but I really think we're under investing, as a broad AI ecosystem, in tools that help AIs get the right answer by asking a tool the question, and overinvesting in trying to fine tune away "hallucinations" that should never be happening in the first place.
So we're collectively spending a lot of time coming up with new ways of "specifying" software (not a bad thing on its own) and imo not spending nearly enough time on refining our existing inner loop tools for AI interactions.
Watch this space :)
A lot of what makes AI coding assistance feel like it doesn't scale past a certain point comes from a lack of quality control tooling that fits into the natural core loop of software.
And tools like husky/lefthook are really not designed for AI styles of interaction.
I've been working on a personal AI coding workflow for vscode for a few months and I'm really impressed by how much vscode offers in the LM APIs.
I'm trying to get it to a place where I can share it (and it will be usable by people other than me) and am really close!
I have a couple of announcements to make next week. Stay tuned!
Fun fact: the internal codename for Heroku Vibes was "Heroku Garden".
PS: If you're in the mood for a deep dive about the original Heroku vision and the history of Heroku Garden, the Salesforce acquisition and the pivot to platform as a service, I could not recommend this
@changelog.com podcast with @adamwiggins.bsky.social enough: changelog.com/podcast/513
We have a lot of work to do to really deliver on that vision, but I'm extremely excited about the prospect of creating something that can finally deliver on the earliest Heroku vision of making software development accessible to everyone.
But the crucial thing is: it's a starting point, not an ending point. It's a way of getting something that works (like Rails scaffolding, iykyk) and solves a real problem, and building it on a platform with enough sophistication to support your software development journey.
Not only a way to prototype and build apps without having to know anything about setting up a local development environment or how to deploy to production, but an accessible onramp to building applications that doesn't require learning to code.
But with the advent of AI tools, we finally have the ability to fully deliver on that vision.
That vision was ahead of its time (we were still living in a pre-Atom, pre-vscode, IE-driven world), and Heroku pivoted to stripping away the deployment papercuts. That was the right move at the time and it was world-changing.
The Heroku Garden vision stripped away nearly all of the papercuts that stopped people from getting an idea out of their head and into production. Not just deployment pieces, but even getting a local development environment set up so you could build the app.
Over a decade ago, Heroku started out with "Heroku Garden", the idea that we could smooth out the onramp for programmers by creating an environment for building out their app in an in-browser editor and immediately deploying it for real.
We can make the gap between "can generate an app that works well enough to be useful" and "can edit the copy in the code myself" much, much smaller.
And once you're editing copy in the code, you're off to the races.
But there's a huge space in between! Everyone who's tried to iterate with an AI on copy changes has experienced the feeling of "Can I just making this freaking change myself?!".
I think people tend to think of vibe coding as the new no-code. Either you're a programmer and you're in an IDE, or you're not a programmer and you're making every edit by using natural language to talk to an AI assistant.
I think a good way to think about Heroku Vibes is that it's an accessible onramp for people to get something real working on a real platform. And then, if they want, they can start to evolve the code that they generated, and have the full power of Heroku to help them get there.
That's very much an _aspiration_ for this project, but the first release was focused on the meat-and-potatoes of getting everything wired up and working well, and I'm hoping to have a lot more to say about this as we implement features that make that vision a lot more real.
I don't just mean "we should collaborate on specs and then have the AI autonomously do the work, and then you write another spec and have the AI autonomously break half of what you liked in the first place."
I mean actual back-and-forth iteration as you go.
3. The word "collaborative" in the marketing tweet isn't an accident. We still have a lot of work to do to realize this vision, but I personally believe strongly that the status quo with AI generation is _far_ too one-shot and not nearly collaborative enough.
2. When you're happy with your app in Vibes, you can transfer it to a normal Heroku app and get the full power of the Heroku software development lifecycle (GitHub integration, review apps, pipelines, CI/CD, etc.)
Even if something isn't already supported, there's a good chance Vibes can figure it out by leaning on a decade's worth of people during nearly anything you can think of on Heroku.
A few things worth pointing out:
1. Vibes has the ability to do most things that a normal Heroku app can do (build backends, create databases, accept config vars etc.), which is a real superpower.
This is what I've been working on! This tweet is of course the marketing way of describing it ("game-changing"!) and this is very much a first release, but I'm so excited about the overall vision for this thing.