We recently chatted with Priya Sarathy of @informs.bsky.social Analytics Society about our last year's IAAA finalist work on optimizing donor milk pooling at the Rogers Hixon Milk Bank!
www.youtube.com/watch?v=VP5k...
You can read more in our paper here:
arxiv.org/abs/2411.02661
We will also be presenting this work at Neurips in a few weeks. Feel free to swing by our poster on Friday Dec 13!
16/16
Finally, we assume everyone knows how good all models are. This is defensible since many models are available on the Chatbot Arena, but the problem becomes much harder if you have to price while forecasting how good competitors will get in the near future.
15/n
Moreover, AI technology is subject to the well-known AI flywheel effect where companies are incentivized to release a product sooner in order to collect data and reap performance rewards later. These time dependent dynamics may require rethinking pricing decisions.
14/n
A few caveats: Our analysis is theoretical and focuses only on pricing rather than the costs of building the models. If your model performance is limited, then the optimal price may be so low that you cannot net any profit from the product.
13/n
Note that what matters is relative competitive ratios. Your model could be worse than competitors, but if it is a lot worse on translation & slightly worse on coding, then you can still get revenue by setting a low price & take the coding market.
12/n
If competitive ratios are different, competitors cant outprice everything. If a model is very good at coding and okay for translation versus competitors, then you can set a price to incentivize competitors to focus on translation, since underpricing on coding cuts their revenue
11/n
This brings us back to the issue of rising costs for gen AI development when optimizing for every benchmark.
However, there may be an alternative: specializing gen AI models to a few applications.
10/n
The most dangerous regime is if the ratio of competitive ratios is close to 1, i.e., if all AI models perform similarly on all tasks. Here, competitors can always underprice their models to capture the entire market, leading to commoditization.
9/n
k2/k1 is the relative competitive ratios on the 2 tasks and a1/(a1+a2) is the fraction of market demand occupied by task 1
Due to competition, Gen AI models must continue to improve or face being underpriced.
Given 2 tasks, we show pricing reduces to 3 regimes based on the ratio of competitive ratios versus the ratio of user demands:
1) high prices,
2) low prices,
3) 0 revenue!
8/n
Competing companies go the other direction by pricing for the tasks where your model is least competitive. However, if your model is too pricey, then the competitors could be more cost-effective to users for all tasks.
7/n
To set token prices, companies can rank tasks on the relative # of tokens needed for a satisfactory output between different models.
Given user demand, companies can maximize their revenue if they focus on only the most competitive tasks and ignore the ones where they are uncompetitive.
6/n
We focus on per-token pricing. Here, when choosing between gen AI models w/ different performances, users should prefer to minimize their cost
= price/token × # of tokens needed for a satisfactory output.
Users can choose the best model for each task.
5/n
3️⃣ Users rarely stop after a bad response—they can keep prompting it with more info until they get a good output. Some tasks need many prompts!
With these 3 factors, we see two popular pricing models today: subscription-based (for chatbot users) and per-token (for API users)
4/n
2️⃣ Models are best evaluated by user satisfaction to responses (e.g., Chatbot Arena).
For instance on Github Copilot, the ‘acceptance rate’ of AI suggestions is the largest predictor of user productivity.
dl.acm.org/doi/pdf/10.1...
3/n
Pricing gen AI is harder than other ML products for a few reasons.
1️⃣ A single model can handle many tasks, e.g., coding, translation, image gen, but users interact differently with each.
2/n
Building frontier gen AI models is incredibly expensive & limited to a few players. So how do we expect the pricing of gen AI to play out?
Our #neurips2024 paper shows why the future may be specialized models focusing on fewer applications
Link: arxiv.org/abs/2411.02661
1/n (a long thread)