Measuring and Iterating

AI Metrics That Actually Matter

Output acceptance rate: What percentage of AI outputs do users accept without editing? If it's below 60%, your AI isn't good enough yet. If it's above 90%, your users might be blindly accepting everything — which is a different problem.

Edit depth: When users do edit AI output, how much do they change? Light edits (fixing a word, adjusting tone) mean the AI is close. Heavy rewrites mean the AI is fundamentally missing the mark.

Return rate: Do users come back for a second, third, tenth time? First-use "wow" is easy. Repeated use means the product delivers consistent value. Track day-1, day-7, and day-30 retention separately.

Cost per successful output: Not cost per query — cost per output the user actually kept. If users need 3 regenerations to get something usable, your true cost is 3x what you think.

Building the Feedback Loop

Every AI product needs a closed feedback loop: output goes to user, user reacts (accept, edit, reject), reaction feeds back into the system. This loop is your competitive moat. Over time, you accumulate data that makes your product better in ways competitors can't replicate.

Collect implicit feedback (acceptance, edits, regenerations) alongside explicit feedback (thumbs up/down, ratings). Implicit feedback is more reliable because users give it without thinking. Store every piece of feedback alongside the prompt and output that generated it — this is your training data for future improvements.

Optimize Prompts vs. Change Approach

Optimize prompts when: The output is in the right ballpark but lacks precision. Users edit lightly. The structure is correct but the content needs refinement. Prompt optimization is cheap — iterate daily.

Change approach when: Users consistently reject outputs entirely. The output format doesn't match the workflow. No amount of prompt tweaking fixes the core issue. This might mean switching models, adding RAG, restructuring the pipeline, or even changing the product's scope.

Finding the Aha Moment

Every successful product has an "aha moment" — the action that correlates with long-term retention. For Facebook it was adding 7 friends in 10 days. For your AI product, it might be "users who get a successful output on their first try retain 4x better."

Dig into your data to find this moment. Compare retained users vs. churned users. What did the retained users do differently in their first session? Once you find it, engineer your onboarding to push every user toward that moment as fast as possible.

The AI Product Iteration Cycle

Traditional product iteration follows a build-measure-learn loop. AI products need a more specific cycle that accounts for the unique ways AI output quality affects everything.

Week 1 — Observe: Don't change anything. Just watch. Read every piece of user feedback. Review the outputs users rejected. Track the queries that produced the worst results. Build a "worst outputs" list — this is your improvement roadmap.

Week 2 — Hypothesize: For each category of bad output, form a hypothesis. "Users rejecting summaries because they're too long" → "If I constrain output to 150 words, acceptance rate will increase." Be specific. Vague hypotheses ("make it better") lead to vague improvements.

Week 3 — Test: Change one thing at a time. If you change the prompt, the model, and the temperature simultaneously, you won't know which change helped (or hurt). Run the new version on your test suite first. Then A/B test with 10% of live traffic.

Week 4 — Measure and decide: Did acceptance rate go up? Did edit depth go down? Did retention improve? If yes, roll out to 100%. If no, revert and try a different hypothesis. If the data is ambiguous, extend the test for another week.

This four-week cycle should run continuously. At any given time, you should have one experiment in observation, one in hypothesis, one in testing, and one in measurement. Parallel cycles accelerate learning without introducing chaos.