Effective product testing provides valuable and actionable insights. Typically, we aim to determine whether our test product outperforms the benchmark product or falls short. Armed with this knowledge, marketing decisions can be made with greater ease and confidence.
While qualitative research can be useful to understand product drivers, a product test usually requires a quantitative approach. The core requirement is to test whether our product is preferred to a benchmark product or not. This requires hard quantitative data using a sample that is large enough to get a statistically accurate result.
1. Choose a market leader competitive product to benchmark against
When testing a new product the benchmark product should ideally be a competitive product that is a market leader in the category or market segment. When the aim is to test a new design or formulation of an existing brand, the product should ideally be tested against the existing product design or formulation as well as a competitor product.
2. The respondents should ideally test 2 products
The best approach is to get each respondent to test two of the products, for example, the new product against the existing product, or the new product against the competitor.
It is a good idea to test more than one new formulation, in order to identify the best alternative and also to understand what product attributes drive preference.
3. Make use of pairwise test cells in your survey design
A good survey design will have several pairwise test cells, with each test cell having two products. This works best for in-home product tests, where the requirement is to place the products with the respondents to use over a period of time, usually about a week. The interviewer will do an initial interview at the placement stage and return a week later for the post-use interview. The post-use interview can also be done telephonically, which is more cost-effective.
4. A popular design is a BIB design.
In statistical parlance, this refers to a Balanced Incomplete Block design. It just means that each test product is tested against each other in a pairwise “round robin” design.
For the BIB design, the number of test cells is given by the formula N(N-1)/2 where N is the number of test products. If there are 4 products the number of test cells would be 4(4-1)/2 = 6. For 3 products the number of cells needed is 3. As can be seen, the number of test cells increases quite rapidly as the number of test products increases, for 5 products it is 10 and for 6 products it is 15.
The BIB design has some technical advantages. Because it is a balanced design, with each product tested against the other one. This controls for biases. We are able to use the full sample for certain analyses, in addition to the direct preference between pairs of products in each test cell.
5. In some cases a two-cell would be a better choice.
In some cases, we might decide not to use a BIB or round-robin design. For example, where we have a new product, an existing product and a competitor, we might decide on a two-cell test, with the new product being tested against the existing product and against the competitor product. The existing vs competitor test cell is excluded in this case.
6. Use a minimum sample of 130 per test cell/per pair of products.
Bigger sample sizes are always better in order to detect statistically significant differences between products, but this must be weighed against the cost. A rule of thumb is to use a minimum sample of 130 per test cell / per pair of products.
7. Where possible, make use of central location testing for faster turnaround.
In some instances a CLT (Central Location Test) is appropriate, for example, a taste test where new flavours are being tested, where it might not be necessary for the respondents to use the products over an extended period in-home, to be able to evaluate the products. In these instances, each respondent can test more than two alternative flavours, products or brands. These projects can be turned around quite quickly, using venues in the main centres of the country.
8. Sample design should match the defined target market.
As with all quantitative projects the sample design is important. We need to put in some controls to make the sample representative of the population. As a minimum, it should match the defined target market on main demographics. This could be the profile of category users or segment users. The demographic profiles can be accessed from tracking studies or external data such as the MAPS survey.
9. Sample design should match the defined target market.
As with all quantitative projects the sample design is important. We need to put in some controls to make the sample representative of the population. As a minimum, it should match the defined target market on main demographics. This could be the profile of category users or segment users. The demographic profiles can be accessed from tracking studies or external data such as the MAPS survey.
In some projects, we also need to ensure that the sample of the users of the main brands in the market is large enough for separate analysis.
The product test on the product liking overall, usually an overall rating, likes and dislikes, the preference or ratings on a range of functional attributes applicable to the category, as well as key attributes drivers. There are a number of statistical techniques that can be used to understand the key drivers in the market.
Product Testing can be an expensive investment. We can help advise on cost-effective solutions to deliver your product tests fast and accurately while providing you with clear insights and direction in your business decisions.