What you get from unmoderated and moderated #testing of #prototypes
Posted on October 22nd, 2015
10/22/2015 @Ishi Systems, 185 Hudson St, Plaza 5, STE 1400, Jersey City , NJ
Steven Cohen @Validately spoke about the goals and advantages of prototype testing.
He started by reviewing his history in startups: one successful start and one failure. After these experiences he asked why does innovation fail?
- False positives – estimated that 50% of a typical product’s features are not used (see http://versionone.com)
He then asked what you can learn from users
- UI usability – can user complete a task by clicking on the appropriate links? – the inventor is too close to the product to know this
- Real life usability – get a lot of false positives here. “It’s not a big enough problem.”
- Are there blockers – usable in real life, such as privacy/security issues.
- Better alternatives – something I’m using now. What is it like?
- Does it solve a need? Needs to be big to change inertia. Need to get to “no” to show you know the issues. For instance introduce a price and ask if they would buy it (time, reputation, money)
Steven then presented two ways to test prototypes
- Unmoderated – asynchronous testing as the user explores the functionality – good for UI usability
- Less work, subjects pace themselves and give verbal/written feedback
- Can’t learn real life issues
- Moderated – live test interacting with the user and asking for opinions – better for real life usability
- Deep user learning – usability
- More time commitment
Steven did a simulated interview to show how the moderator would probe the user his/her actual usage of the product. One of the most important aspects is to get the user to make decisions on feature tradeoffs. The main tradeoff is whether a feature is work a specific amount of money and if not, why not and would there be a lower price point.
He wants to put up road blocks and see how people react – e.g. put up a page and ask for a credit card (even if you don’t keep the number) – see how much commitment. This is an attempt to get the feedback that users might not way to give you since they do not want to hurt your feelings or be confrontational.
Other observations were
- If you need to pay your own existing customers to participate in a test, then the problem is not that important. Power users will probably always test for free.
- Validately will start video recording the user, but does not believe eye tracking adds much since mouse movements correlate highly with eye movements
- For unmoderated tests, if 5 of 6 users can do it. it’s probably adequate
- For moderated test, start with 6 to 10 people. If there is a consensus, you can get by with fewer. If there is no consensus, then your persona (a straw man demo) is probably not well defined.
- Once you have made a change, go back to the initial set of users and see if it fixes the problem – did we understand the comments?
The bottom line is, don’t build stuff that users don’t need