AI output evaluation

With AI work we change model versions and prompts often, and each change can alter the results. The agent didn't give consistent output across different preferences, and a change could quietly make things worse. Initially there were no tests for the AI at all.

I pushed for evaluation tests, and I built an evaluation agent on the ReAct framework. It runs predefined inputs through the generation, checks the results against evaluation criteria, and reasons about their quality. It covers both itineraries and review summaries. It runs in GitHub Actions on every AI change, with a quality gate: if the outputs get worse, the change is blocked, and only a change that passes is deployed.

There was hesitation, because every change would now cost money to evaluate. My case was that it gives us confidence that we are delivering something that works against our metrics, and a confirmation that existing behavior still works after every change.

Shipping without evaluation tests, which is where the project started.

Designed and developed an AI evaluation mechanism for validating generated itinerary results before deployment.

Implemented an evaluation agent using the ReAct framework to test AI-generated itineraries against predefined inputs and evaluation criteria, analyze the quality of generated results, and support iterative refinement of the AI workflow.

Extended AI evaluation testing to cover both itinerary generation and review-summary functionality.

• It costs money on every change.
• The cost isn't optimized. I planned a cheaper way using the same model but didn't get to it, so it is costlier than it could be.

AI changes are tested like code before they ship, and the team can change prompts and models with confidence. It catches real regressions: when we change a model or a prompt, it shows which results are affected, and we rework the change before it ships.