Decide whether you are choosing a product or learning a cause
A comparison can answer “Which complete version works better?” without answering “Why is it better?” Both questions are legitimate, but they need different interpretation. If one prototype changes material, geometry, and surface finish, an improved result cannot be assigned confidently to any one change.
Write the intended conclusion before testing. If you need to select a complete concept, compare representative complete concepts and record the tradeoffs. If you need to understand a parameter, hold other relevant factors stable or use a structured experiment that estimates their effects.
NIST's experimental-design material discusses interactions, where the effect of one factor depends on another. This is common in product work: a geometry may perform well in one material but poorly in another. A sequence of isolated changes can miss that relationship.
Control the task and the measurement
Use the same reference objects, loading arrangement, environment, and measurement method unless a difference is deliberately part of the study. Label samples with neutral identifiers where practical so expectations do not influence observations. Preserve failures and awkward attempts, not only the best run for each design.
Consider order effects. Users may learn the task, become tired, or adapt to the first design. Testing every participant on A and then B can make B appear better through practice. Vary order in a planned way and record it. The appropriate design depends on the task and available participants.
For bench tests, fixture settling, equipment drift, or sample conditioning can also follow time order. Randomizing or balancing run order can reduce some biases, but it does not fix a poorly defined test. Start with a method that measures the intended response consistently.
A hypothetical latch comparison
An inventor compares two latches. Version A uses a short flexible arm in material X. Version B uses a longer arm in material Y. B opens more easily. The immediate conclusion is that the B configuration requires less effort under the tested conditions; the test does not show whether length or material caused the difference.
If the next decision is simply which configuration to refine, that may be enough. If the inventor wants to choose a production material while retaining A's geometry, the comparison is insufficient. A small two-factor study could include short and long arms in both materials, provided the combinations are feasible and safely testable.
Suppose the longer arm reduces effort greatly in X but only slightly in Y. That suggests an interaction worth investigating. The response should include retention and damage as well as opening effort, because minimizing effort alone could produce a latch that releases accidentally.
These observations remain exploratory unless the sample size and analysis support stronger inference. A table of four single measurements can reveal useful patterns, but it does not establish precise effect estimates or broad production reliability.
Use a scorecard without hiding disqualifying failures
List essential requirements separately from preferences. A design that fails a required retention condition should not win because its color and low cost increase an average score. First determine whether each candidate meets mandatory conditions; then compare tradeoffs among acceptable candidates.
Keep raw measures beside any normalized score. Converting force, time, and subjective preference into a single number requires choices that can dominate the result. State the weighting rationale and test whether a small change in weights reverses the winner. If it does, the decision may be sensitive rather than decisive.
Include uncertainty and missing evidence. Mark an untested durability property as unknown, not equal to a passing result. A candidate with fewer tests should not appear superior merely because fewer weaknesses have been sought.
Turn the comparison into a useful design decision
The report should list configurations, controlled conditions, sequence, observations, anomalies, and the conclusion appropriate to the study. State whether the result selects a complete design, suggests a causal factor, or merely identifies a question for the next experiment.
If the result is close relative to measurement variation, do not force a winner. Improve the method, repeat a relevant comparison, or decide using another important criterion. More decimal places will not make a noisy distinction meaningful.
Your deliverable is a decision with a visible basis and limits. It should tell the next designer what to retain, what remains uncertain, and which changed feature needs investigation. That is more useful than declaring a winner after two uncontrolled demonstrations.