← Back

Downstream Evals for LLMs

·Bryan Lai
TechnicalMLEvals

Downstream Evals for LLMs

If you cannot measure quality, you are not improving a product.

You are watching a demo.

Evals turn guesses into evidence.

They help you answer three questions:

  1. Did the system get better?
  2. Did it break something that used to work?
  3. Is it good enough for the next user?

Eval Types

Unit Tests

Use code checks for facts that should never vary.

They are fast. They catch regressions. They are weak for taste, nuance, and judgment.

def test_adder():
    result = llm_add("2 + 3")
    assert "5" in result

LLM Judges

A stronger model grades your model's output.

Give it the task, the answer, and the criteria. Ask for a score and an explanation.

This is useful for shape and direction. It is not truth. Judges can be biased, expensive, and inconsistent.

Human Evals

Real people still catch what models miss.

Use humans for high-stakes outputs, final checks, user value, and cases where taste matters.

The cost is speed. Humans are slower, expensive, and inconsistent unless the rubric is clear.

Common Eval Mistakes

  1. No baseline.
  2. Starting too complex.
  3. Confusing clicks with quality.
  4. Treating an LLM judge like a human.
  5. Testing a clone instead of the live path.
  6. Optimizing the score while users still hate the product.

Clickbait can raise clicks and lower quality. A perfect score on a bad metric is still bad.

Conclusion

Know your inputs, outputs, and traces.

Start with a dumb eval that catches real mistakes.

Then improve it.

No eval is perfect. The useful one is the one that changes what you ship.