Evaluate the AI
Who can do this:Knowledge managerExpert / verifierOrganisation administratorService manager reads
Transcript
- AI evaluation tests the AI Service Engineer with benchmark questions from your own service work.
- A version passes only when every release gate holds, such as escalating every safety question.
- Quality scores show how the answers changed from one version to the next.
- Open a run to read every answer with its result.
- Add your own benchmark cases, for example the questions that once went wrong.
AI evaluation tests the AI Service Engineer with benchmark questions from your own service work. Each run answers every benchmark case and scores the answers, so a new version of the assistant or of your knowledge is checked before technicians rely on it.
What it shows
Section titled “What it shows”- Latest run: whether it passed, the version, the AI model and the number of cases.
- Release gates: the thresholds a run must meet, such as escalating every safety-critical question, each with its value and status.
- Quality scores: how the answers scored, and the change from the previous run.
- Evaluation runs: every run, with its date, version, model and result.
Step by step
Section titled “Step by step”- Open AI evaluation.
- Open a run from Evaluation runs. Results lists every benchmark case; Show answer shows what the assistant answered.
- Add benchmark cases under Benchmark cases with Add case: the question, its language and machine model, and whether an escalation is expected or it is safety-critical. Add to benchmark on a service case does the same from a real job.
- Choose Run evaluation to test the current version.