Mobile A/B testing and more
Posted on October 23rd, 2014
10/23/2014 @ Pivotal, 625 6th Ave, NY
Matt Weber presented a template for surveying user experience and Zac Aghion @Splitforce talked about his work on mobile A/B #testing
Matt introduced a template questionnaire on #UX he has developed for his students. To retain the flavor of the experience, the questionnaire concentrates on the most recent time an action was performed. It also considers the experience before, during and after the action/decision. He also spoke about his analysis of the results which develops themes for each of the three time periods: before, during and after.
Template is available at http://www.qualitative.io and is licensed under creative commons.
The logic behind the experimental design and testing is the same, but these issues affect the complexity of implementing the test and the data collected
- The app may need to collect and store user interaction data while offline. Only later is it transmitted once an online connection is available
- Greater variability in the environment in which the user experiences the interaction. This may be either a positive (location data could eventually provide additional control variables) or negative (distractions and time constraints could add extraneous noise which makes it harder to see the signal in the data)
Zac presented two examples of A/B testing on mobile devices: 1. Two different format of buttons to access more detailed recipes 2. Whether eliminating a message box increases the number of in-app purchases.
Splitforce collects hardware and system setting information – operating system, language setting, etc.
In the two studies presented they did no collect the click path or other context information.