Quick answer
The founder litmus test asks one question: does a better base model make your product stronger — or obsolete? Vijaye Raji’s rule from Nate B Jones’s interview: if each model drop worries you, you are probably wrapping the model; if each drop excites you, you are building on top of it. Use that check before you pour months into a thin GPT wrapper.
Key takeaways
- Pass: new models improve quality, cost or coverage without a rewrite.
- Fail: a better public model can replace your core demo tomorrow.
- Assets that help you pass: proprietary data, workflow depth, distribution, compliance and integrations.
- Re-run the test whenever a frontier lab ships a major model.
- Failing is useful if it pushes you to deepen the product instead of polishing a chat UI.
📘 Main article: This guide supports OpenAI Outgrew Git: What Happens When AI Agents Write Code Too Fast. It expands Raji’s litmus test into a practical founder checklist.
In this guide
- What is the founder litmus test?
- What does pass vs fail look like?
- How do you run the test on your product?
- What are concrete pass and fail examples?
- What should you change if you fail?
- How do you run the test with a team, not just a founder?
- What mistakes fake a “pass”?
- Frequently asked questions
- The bottom line
- Related posts
Model leaps are getting cheaper for customers and harder for thin products. This guide answers one question: how do you apply the founder litmus test for new AI models? It supports our summary of OpenAI outgrowing Git and the source interview.
What is the founder litmus test?
Raji’s framing: models keep improving. The smallest model next year may beat last year’s flagship. Your job is to build so that those gains flow into customer value — the same way OpenAI’s ads stack, in his telling, got smarter as LLMs improved without rebuilding every stage from scratch.
What does pass vs fail look like?
| Signal | Pass | Fail |
|---|---|---|
| Model drop day | You schedule evals to adopt it | You panic about competitors cloning you |
| Switching cost | Customers stay for your workflow/data | Customers stay only for the chat quality |
| Margins | Better models cut cost or raise win rate | Your only edge was access to yesterday’s model |
| Roadmap | You plan features around durable jobs | You plan features that are “prompts with a logo” |
How do you run the test on your product?
- Write your product’s core job in one sentence.
- List what you own that a raw model does not (data, integrations, brand trust, compliance, distribution).
- Imagine the best public model in six months is free and multimodal.
- Ask: what still makes a buyer choose you?
- If the list is empty, you failed — by design. That is the point.
- Pick one asset to deepen this quarter.
- Book a calendar reminder for the next major model launch to re-run the test.
What are concrete pass and fail examples?
- Pass example: a vertical tool that files claims into insurer systems with audit trails — better models draft faster, but the workflow and credentials remain yours.
- Fail example: a generic “ChatGPT for X” landing page whose demo is a single prompt anyone can paste into the base model.
- Borderline: a fine-tune with no proprietary data and no workflow — often a slow-motion fail.
- Pass-shaped platform play: agents embedded in a monorepo with your tests, policies and deploy gates — models get better, your control plane still matters.
💡 Tip: OpenAI’s ads story in the interview is a pass-shaped example: LLM stages inside a larger system that improves when models improve. Your version will look less glamorous and more domain-specific.
What should you change if you fail?
- Attach the model to a painful multi-step workflow, not a blank chat box.
- Capture proprietary feedback loops (outcomes, labels, corrections).
- Sell implementation, compliance or distribution if the software edge is thin.
- Cut features that only exist to hide a prompt.
- Write evals that prove your workflow still wins when the base model is swapped.
How do you run the test with a team, not just a founder?
Founders should not run this alone in a vacuum. Ask engineering, sales and support the same question: “If GPT-next is free and better, why do customers still need us?” Collect the answers in one page. Disagreement is useful — it shows where the story is weak.
What mistakes fake a “pass”?
- Calling a system prompt “IP”.
- Relying on temporary API scarcity or waitlists.
- Confusing demo wow with retention.
- Skipping evals so you never notice a base model overtaking you.
- Renaming a thin wrapper and calling it a platform.
Next: read what OpenAI changed when agents outpaced Git, and tighten reviews and testing so faster models do not just create faster mess.
Frequently asked questions
Who said this litmus test?
Vijaye Raji, in Nate B Jones’s interview, as a question he asks founders.
Does failing mean shut down?
It means rethink the edge. Many products can pass by deepening workflow and data.
How often should we re-run it?
At every major model launch, and quarterly at minimum.
Is building on OpenAI automatically a fail?
No. Building only a thin UI on OpenAI often is. Dependencies can still pass the test.
Can agencies pass the test?
Yes, if clients pay for process, accountability and domain judgement — not only for “access to a model”.
The bottom line
The litmus test is a kindness: it tells you early whether you are building a product or renting a model’s aura. Aim for excitement on model-drop day, not dread.
Related posts
- OpenAI Outgrew Git: What Happens When AI Agents Write Code Too Fast
- How to Fix Code Reviews and Testing When AI Writes More Code
- AI Agent Project Readiness Brief

[…] The Founder Litmus Test for New AI Models — check whether your product gets better when frontier models improve […]
[…] your normal tasks and compare it with what you use today before you switch anything important. Our founder litmus test for new AI models gives you a quick way to do […]