CI/CD for AI Applications

Git Push to Production

The simplest CI/CD pipeline for AI apps: push to main, auto-deploy. Vercel does this out of the box. Your GitHub repository connects to Vercel, and every merge to main triggers a production deployment. Preview deployments happen on every pull request.

For edge functions (Supabase), deployment is a CLI command: supabase functions deploy function-name. Automate this with a GitHub Action that triggers on changes to your functions directory.

The critical addition for AI apps: your pipeline needs to verify that API keys are set, rate limits are configured, and your AI providers are reachable — before traffic hits the new deployment.

Testing Non-Deterministic Systems

Traditional tests assert exact outputs: "given input X, expect output Y." AI systems don't work that way. Ask the same question twice and you'll get different responses. So how do you test?

Contract testing: Don't test the exact response — test the shape. Does the response have the expected fields? Is it within the expected length? Does it contain required information?

Eval suites: Maintain a set of known questions with acceptable answer ranges. Run them against your AI pipeline on every deploy. Flag regressions when answers drift outside acceptable bounds.

Mock in CI, test live in staging: Use recorded API responses for unit tests (fast, free, deterministic). Use real API calls in staging tests (slow, costs money, but catches real issues).

Smoke tests post-deploy: After every production deployment, automatically hit your key AI endpoints and verify they respond correctly. This catches configuration issues that unit tests miss.

Blue-Green and Canary Deployments

Blue-green deployment: Run two identical environments. Deploy to the inactive one, verify it works, then switch traffic. If something breaks, switch back instantly. Vercel handles this automatically — every deployment is atomic and instantly rollbackable.

Canary deployment: Route 5-10% of traffic to the new version. Monitor error rates, latency, and costs. If everything looks good, gradually increase to 100%. This is especially valuable for AI features where a bad prompt template could generate harmful or incorrect content.

For AI-specific rollouts, consider feature flags. Ship the new AI feature behind a flag, enable it for internal users first, then gradually roll out. This decouples deployment from release.