We quantised a model to 31ms and lost almost nothing

The requirement was blunt: detect a defect fast enough to divert the part before it left the station. That gave us about 40ms end to end on hardware that could survive a factory floor, which ruled out a round trip to anything.

The measurement that mattered

Quantisation discussions usually happen in aggregate — "we lost 0.4% accuracy" — and aggregate accuracy is the least useful number available. What we needed to know was which classes degraded, because the cost of missing a hairline crack is not the cost of missing a surface scuff.

for cls in CLASSES:
    fp32 = evaluate(model_fp32, holdout.filter(cls))
    int8 = evaluate(model_int8, holdout.filter(cls))
    delta = int8.recall - fp32.recall
    print(f"{cls:24} {fp32.recall:.3f} -> {int8.recall:.3f}  ({delta:+.3f})")

Six of eight classes were unchanged within noise. Two were not: hairline cracks dropped 2.1 points of recall, and low-contrast discolouration dropped 3.4. The aggregate number was 0.4%, which would have told us nothing.

The regression harness

The most valuable artefact from this project was not the model. It was the evaluation suite that runs on every model change and fails the build on a per-class regression, the same way a broken test would. Model changes stopped being a judgement call and became a pull request with a diff.

Final numbers: 31ms end to end on the edge device, 74% smaller model, and a written record of exactly what that cost. The record is the part the client can act on.