Zero errors in testing: how many parts make a defensible validation sample?

A camera inspects 100 known defective parts and accepts none of them. The team sees a false-accept rate of 0% and wants to release the system. Under a simple binomial model, this is not enough to support a target of at most 1%: the one-sided upper 95% confidence bound is about 2.95%. This article works through the calculation and turns it into a practical test plan. All numbers are illustrative calculations, not RobuTrace measurements.

1. Define the error being counted first

Here, a false accept means accepting a genuinely defective part as good. The denominator is the number of defective parts tested. A false reject means rejecting a genuinely good part; its denominator is the number of good parts tested. Other reports may use different denominators. Always report the counts alongside the percentage.

Example: a test contains 9,900 good parts and 100 defective parts. Zero missed defects means 0 out of 100 false accepts, not 0 out of 10,000. Adding good parts does not strengthen the evidence for detecting defective parts. A claim about defective shipments per produced part additionally requires the defect prevalence and the actual process flow.

2. The calculation for zero observed errors

The model assumes a fixed, predeclared number of independent binary decisions with the same error probability p. The binomial distribution gives P(X = 0) = (1 − p)ⁿ. Setting this probability equal to α yields the exact one-sided upper bound. NIST documents the underlying exact binomial method.

Our calculations from these equations, each assuming zero errors:

• 100 independent decisions: upper bound about 2.951%.
• 300 independent decisions: upper bound about 0.994%.
• 1% target: at least 299 decisions.
• 0.1% target: at least 2,995 decisions.
• 0.01%, or 100 ppm, target: at least 29,956 decisions.

These minimum counts are not universal requirements for every installation. They answer one specific question: how large must a fixed test be so that, with zero errors, the one-sided upper 95% bound does not exceed the chosen target? If errors occur, the zero-error formula no longer applies.

3. What “95%” actually means

Confidence describes the long-run coverage of the statistical procedure under repeated sampling and the model assumptions. It is neither a model confidence score nor a guarantee for the next shift. It also does not mean that every individual future inspection decision now has a 95% probability of being correct.

A two-sided 95% interval allocates the error probability to both tails and produces a different upper bound here. Document the one-sided question in advance when the requirement is an upper limit on an error rate. Do not choose the interval after seeing which version happens to meet the target.

4. 300 images do not necessarily represent 300 independent parts

Thirty captures of the same part under nearly identical conditions provide different evidence from thirty independently selected parts. Repeated captures can investigate repeatability, but do not automatically increase the independent sample size. A shared batch, tool, surface or acquisition sequence may correlate the errors.

Define the experimental unit: part, assembly or complete inspection cycle. Keep multiple views of a part together when the production decision is made at part level. Images of the same part should not be randomly spread between training and the final test. The scikit-learn documentation on grouped evaluation explains why dependent groups need separate treatment. Applying that principle to parts and batches is an engineering recommendation here.

5. A practical plan for the 1% example

1. Define the scope: Which defect types, variants, sizes, surfaces and operating conditions should the claim cover? Confirm reference labels independently of the inspection algorithm.
2. Freeze the system: Version the camera, lens, lighting, software, model, ROI and thresholds. Keep tuning images separate from the final test.
3. Write the plan first: In this example, 299 independently selected defective parts, zero false accepts and a one-sided upper 95% bound no greater than 1%. Good parts need their own false-reject plan.
4. Account for the complete test: Record all planned decisions, acquisition failures and exclusions traceably. Do not make an observed error disappear by restarting the counter.
5. Limit the conclusion: Report counts, bound, system version and covered population. Explicitly leave untested variants unresolved.

If an error occurs, recalculate using the actual error count. SciPy binomtest with proportion_ci provides an implementation reference; direction and confidence level must match the question. Repeatedly continuing until the result passes requires a suitable sequential design, rather than reusing the fixed-sample calculation unchanged.

6. A large total does not replace coverage

Testing 299 clearly visible scratches does not establish a bound for barely visible cracks. An aggregate result concerns the distribution actually examined. If every defect type must meet its own target, each needs an appropriate evaluation. Several individual 95% statements also do not automatically form a joint 95% claim covering every group.

Disturbance tests and boundary samples complement the statistical plan. They are deliberately selected and must not be treated as a random production sample without justification. Our article on systematic robustness testing helps plan this coverage; the false-accept and false-reject overview explains the different consequences.

7. Where RobuTrace fits

The inspected development version calculates observed false-accept and false-reject rates from imported, labelled measurement datasets and separates them from simulated projections. The exact confidence bound derived here is an additional analytical method presented in this article; it does not imply an existing automatic RobuTrace acceptance feature. A green local status replaces neither an appropriate sampling plan nor industrial validation.

A useful result statement is: “0 false accepts among 299 independent defective parts; exact one-sided upper 95% bound approximately 0.997%; applicable to the documented system version and tested defect population.”

Sources and calculation basis

• NIST: Binomial Distribution
• NIST Dataplot: Exact Binomial
• SciPy: binomtest
• scikit-learn: Cross-validation

Sources checked on 10 October 2026. Numerical examples calculated from the stated equations.