CI/CD for AI Applications

Git Push to Production

The simplest CI/CD pipeline for AI apps: push to main, auto-deploy. Vercel does this out of the box. Your GitHub repository connects to Vercel, and every merge to main triggers a production deployment. Preview deployments happen on every pull request.

For edge functions (Supabase), deployment is a CLI command: supabase functions deploy function-name. Automate this with a GitHub Action that triggers on changes to your functions directory.

The critical addition for AI apps: your pipeline needs to verify that API keys are set, rate limits are configured, and your AI providers are reachable — before traffic hits the new deployment.

Testing Non-Deterministic Systems

Traditional tests assert exact outputs: "given input X, expect output Y." AI systems don't work that way. Ask the same question twice and you'll get different responses. So how do you test?

Contract testing: Don't test the exact response — test the shape. Does the response have the expected fields? Is it within the expected length? Does it contain required information?

Eval suites: Maintain a set of known questions with acceptable answer ranges. Run them against your AI pipeline on every deploy. Flag regressions when answers drift outside acceptable bounds.

Mock in CI, test live in staging: Use recorded API responses for unit tests (fast, free, deterministic). Use real API calls in staging tests (slow, costs money, but catches real issues).

Smoke tests post-deploy: After every production deployment, automatically hit your key AI endpoints and verify they respond correctly. This catches configuration issues that unit tests miss.