Evaluate AI quality and real effort

Evaluate whether an AI result is usable for a defined task. Fluency, speed and a long explanation are not enough. Record factual accuracy, completeness and the effort needed to turn the output into accepted work.

Start with a small review

Choose a handful of permitted examples you understand well. Include an ordinary case, an incomplete source and two conflicting sources. Write down what a satisfactory answer must contain before you run the task.

Criterion	Question	Example failure
Correctness	Do claims agree with the source?	A proposed date becomes a confirmed deadline
Evidence	Does each important claim have support?	A citation points to a related but different statement
Completeness	Are the required points covered?	A summary leaves out an unresolved dependency
Uncertainty	Does the answer identify missing information?	An owner is invented where none is named
Usability	Can the intended reader use the output?	A client email includes internal notes
Access and actions	Does the task respect the intended boundaries?	A service action is requested outside the task

Mark each criterion as pass, needs correction or fail. A critical unsupported claim can disqualify an answer even when its formatting is excellent.

Compare like with like

Keep the inputs, source selection and expected output the same when comparing prompts or models. Record the date, model or Auto setting, effort, verification, source versions and the Skill or workflow version where applicable. Auto may select different models, so record that limitation rather than treating it as a fixed model comparison.

Change one important variable at a time. Repeat selected cases to see whether quality is stable. A small pilot is useful for finding problems; it is not a statistical guarantee of performance on every future task.

Measure the whole task

Record active work time for preparation, running the task, review and correction. Record waiting time separately if it matters for delivery. Compare with the same task done using the previous method.

For example, a task previously took 30 minutes. The AI-assisted version needs 6 minutes of preparation, 2 minutes of active interaction and 12 minutes of review and correction: 20 minutes total. The observed difference is 10 minutes for this example, not a general claim of 10 minutes saved on every task.

Do not count rejected drafts as completed work. Track acceptance and serious errors alongside time, so faster poor-quality output cannot look like an improvement.

For advanced teams: keep a regression set

Maintain a versioned set of inputs, expected facts, forbidden claims and review decisions. Include difficult cases such as changed source versions, missing permissions and ambiguous instructions. Keep personal or client data out of shared test sets unless its use and access are approved.

A model-based evaluator can help triage outputs, but calibrate it against human reviews. It may repeat the same errors or prefer polished wording. Human reviewers should inspect representative results and every critical failure.

Rerun the set after changing a shared Skill, workflow, model configuration or source collection. Record the decision to release, revise or roll back the method. MAIA's verification feature is one part of this process, not a replacement for it.

Make the result actionable

Use the evaluation worksheet to assign each failure to its likely cause: source, access, instruction, model behavior or review criteria. Fix the cause, rerun the affected examples and keep the history. Share the method only when its owner understands the remaining limitations.