Your product team has shipped an AI feature that nobody knows how to evaluate.
It works in demo. It mostly works in production. Whether it is getting better or worse with each model release is anybody's guess.
AI product development
When AI is the product. Build it to outlast the model.
For Australian product teams shipping AI as a feature or as the whole proposition. Architecture, evaluation, vendor liquidity. The boring engineering that turns a demo into something you can release every Tuesday.
AI products without evals fork into anecdote-driven roadmaps.
Customer success heard one bad story this week. Sales heard a great one. The team patches around both. The product slowly turns into a list of overrides.
Architecture, eval harness, vendor liquidity. Then the prompt.
The interesting work happens before the first prompt is written. Get those right and the model becomes a swappable component, not the product.
How we approach AI products
Four principles that survive the next model release.
Architecture before prompt
Most AI products that fail in their second year fail because the architecture was the prompt. We design the data flows, evaluation harness and observability before any user-facing copy.
Evaluations that survive a release
If you cannot tell whether the next model release made your product better or worse, you do not have an AI product. You have a chat interface. Eval harnesses are the difference.
Vendor liquidity by design
Today the strongest model for your task is one. In six months it will be another. Your product should not care. We build the abstraction so it does not.
Honest evaluation of when AI is not the answer
Some product features genuinely benefit from AI. Some are better as deterministic logic with a UX that admits it. We will say which is which on your specific feature list.
Got a feature on the roadmap that needs the engineering thinking through?
Two-hour architecture session with a senior engineer. Bring the spec. We bring the questions that matter.