What I'm Learning While Building With AI
Notes from shipping AI features: evaluation matters more than prompts, and costs hide in places you do not expect.
The first lesson is that prompts are the easy part. Getting a good answer once is straightforward; getting a good answer consistently across a hundred real inputs is the actual engineering problem, and it is solved with evaluation rather than better wording.
So I keep a small set of stored inputs with expected characteristics for the output, and I run it on every change. Without that, improving a prompt is guesswork — you fix one case and silently break three others.
The second lesson is about cost and latency. Model calls are slow and priced per token, which pushes you towards caching, shorter context, and doing as much as possible with ordinary code before reaching for a model.
The third is architectural. Treat the model as an unreliable external service: validate its output against a schema, handle failures explicitly, and never let it be the only thing standing between a user and your database.