AI evaluation tests

With AI work we change model versions and prompts often, and each change can alter the results. A change can quietly make the generated descriptions worse, and without tests nobody would notice. The client also wanted the check in the pipeline, so that before every merge they can see the actual result.

I built evaluation tests with predefined scenarios for the description generator. They run as a GitHub Actions workflow before every merge, not inside the live pipeline, so checking quality never slows it down, and results are recorded in LangSmith. A quality gate blocks a change if the generated descriptions get worse, and only a change that passes is deployed. When we change a model, a prompt or the generation logic, the tests show what that change did to the results.

Shipping without evaluation tests.

Implemented an AI evaluation test suite for the seed-description generation functionality to validate output quality when changing LLM models, prompts, or generation logic.

Created predefined evaluation scenarios for generated seed descriptions, allowing changes to the AI implementation to be tested against consistent expectations before being incorporated into the application.

The evaluation provided a repeatable way to assess description-generation behavior and identify regressions or quality differences introduced by changes to prompts or underlying AI models.

This evaluation functionality was developed specifically for the seed-description generation component and operated independently from the main seed-data enrichment and approval workflow.

Every change costs money to evaluate.

Regressions are caught before they reach the live pipeline, and the client can see the result before every merge.