New York Tech Journal
Tech news from the Big Apple

Mobile A/B testing and more

Posted on October 23rd, 2014

UX + Data

10/23/2014 @ Pivotal, 625 6th Ave, NY

Matt Weber presented a template for surveying user experience and Zac Aghion @Splitforce talked about his work on mobile A/B #testing


Matt introduced a template questionnaire on #UX he has developed for his students. To retain the flavor of the experience, the questionnaire concentrates on the most recent time an action was performed. It also considers the experience before, during and after the action/decision. He also spoke about his analysis of the results which develops themes for each of the three time periods: before, during and after.

Template is available at and is licensed under creative commons.

Zac talked about how A/B testing (testing which of two applications is more successful) on mobile devices is different from testing web pages. Issues include languages: javaScript vs Objective-C/Java and the uncertainty of connection during a mobile session.

The logic behind the experimental design and testing is the same, but these issues affect the complexity of implementing the test and the data collected

  1. The app may need to collect and store user interaction data while offline. Only later is it transmitted once an online connection is available
  2. Fewer off-the-shelf tools for mobile A/B testing and the different character of Objective-C/Java vs. JavaScript means more programming effort to implement a test
  3. Greater variability in the environment in which the user experiences the interaction. This may be either a positive (location data could eventually provide additional control variables) or negative (distractions and time constraints could add extraneous noise which makes it harder to see the signal in the data)


Zac presented two examples of A/B testing on mobile devices: 1. Two different format of buttons to access more detailed recipes 2. Whether eliminating a message box increases the number of in-app purchases.

Splitforce collects hardware and system setting information – operating system, language setting, etc.

In the two studies presented they did no collect the click path or other context information.

posted in:  Android, data, iOS, UX    / leave comments:   No comments yet