New York Tech Journal
Tech news from the Big Apple

What you get from unmoderated and moderated #testing of #prototypes

Posted on October 22nd, 2015

Jersey City Technology Startups

10/22/2015 @Ishi Systems,  185 Hudson St, Plaza 5, STE 1400, Jersey City , NJ


Steven Cohen @Validately spoke about the goals and advantages of prototype testing.

He started by reviewing his history in startups: one successful start and one failure. After these experiences he asked why does innovation fail?

  1. False positives – estimated that 50% of a typical product’s features are not used (see

He then asked what you can learn from users

  1. UI usability – can user complete a task by clicking on the appropriate links? – the inventor is too close to the product to know this
  2. Real life usability – get a lot of false positives here. “It’s not a big enough problem.”
    1. Are there blockers – usable in real life, such as privacy/security issues.
    2. Better alternatives – something I’m using now. What is it like?
    3. Does it solve a need? Needs to be big to change inertia. Need to get to “no” to show you know the issues. For instance introduce a price and ask if they would buy it (time, reputation, money)

Steven then presented two ways to test prototypes

  1. Unmoderated – asynchronous testing as the user explores the functionality – good for UI usability
    1. Less work, subjects pace themselves and give verbal/written feedback
    2. Can’t learn real life issues
  2. Moderated – live test interacting with the user and asking for opinions – better for real life usability
    1. Deep user learning – usability
    2. More time commitment

Steven did a simulated interview to show how the moderator would probe the user his/her actual usage of the product. One of the most important aspects is to get the user to make decisions on feature tradeoffs. The main tradeoff is whether a feature is work a specific amount of money and if not, why not and would there be a lower price point.

He wants to put up road blocks and see how people react – e.g.  put up a page and ask for a credit card (even if you don’t keep the number) – see how much commitment. This is an attempt to get the feedback that users might not way to give you since they do not want to hurt your feelings or be confrontational.

Other observations were

  1. If you need to pay your own existing customers to participate in a test, then the problem is not that important. Power users will probably always test for free.
  2. Validately will start video recording the user, but does not believe eye tracking adds much since mouse movements correlate highly with eye movements
  3. For unmoderated tests, if 5 of 6 users can do it. it’s probably adequate
  4. For moderated test, start with 6 to 10 people. If there is a consensus, you can get by with fewer. If there is no consensus, then your persona (a straw man demo) is probably not well defined.
  5. Once you have made a change, go back to the initial set of users and see if it fixes the problem – did we understand the comments?

The bottom line is, don’t build stuff that users don’t need

posted in:  Jersey City Technology Startups, psychology, startup, UI, UX    / leave comments:   No comments yet

Design Driven NYC: Assembling great #design #teams

Posted on February 14th, 2016


02/10/2016 @NewSchool, W12th Street, NY

20160210_182548[1] 20160210_185220[1] 20160210_191104[1] 20160210_195918[1]

Four speakers spoke about team building as it relates to design:

The first speaker, Katy Kasmai @TeamExponent talked about a method of brainstorming for ideas to solve large, impactful, long-term, impossible projects, a.k.a. #moonshots or 10x moonshots. She conducts workshops that employ a #DesignSprintMethodology. Some of the key touch concepts are:

  1. Team commitment
  2. “Yes and !”
  3. Do it!

In the second presentation, Ben and Ben spoke about many aspects of the company emphasizing their team approach to improving the product. Jet, like Amazon, sells online. Since they are smaller and newer than Amazon, they look for competitive advantages, such as options to forego returns, etc. Their company is based on full availability of data to all in the company (this includes salaries). Real-time sales data are displayed prominently in the work area. All know the results of A/B testing in real time.

They Structure their teams to act. They build small vertical teams that are autonomous and self-select projects. All teams have unfettered access to data and research and are empowered to push ideas into the live sales environment. Anyone can hold a group brainstorming session.

The use a home grown tool, Juice, to design ideas. The tool is design so one can fail fast and better. It is a css based design system which creates a small screen first and is highly responsive. Better prototypes make better research.

The conduct in house research every Thursday in which, get customers do a live shop-in in which all members of staff can observe. They use an eye tracker to determine whether certain design elements really matter to the customer.

They also conduct impromptu “research on a budget” (a.k.a “sip and study”) where they go to a local coffee shop and ask customers to spend 15 minutes looking at a site.

They also conduct remote testing in the customer’s home, in both moderated and unmoderated sessions (see my previous notes on this topic)

Next, Jen @18F talked about working on a distributed design team. They are a civic consultancy for the U.S. government within the U.S. government. They help other agencies improve their web site and create tracking tools to monitor usage of those sites.

They have 159 people located throughout the U.S. with only 44% in D.C. Their design team has 30 people who are linked by weekly video calls and Slack. They have a team culture including weekly 15 minute updates by each person. Members are encouraged to give and receive backup and create self-organized critique groups.

They find it important to know the real-world context in which everyone works. So people post what their workspaces look like.

They do almost all communications online, but realize that it is important to meet, so they have an annual physical get together.

They try to over-communicate so everyone if comfortable when stuff gets hard people can talk to colleagues. They encourage mutual support (similar to moonshots, “Yes and !”) and have an appreciation bot in Slack which contains a channel for all message of appreciation.

They also have online Design studio exercises in which problems are presented and individuals present and upload their ideas to

The last session was an interview with Khoi conducted by Dan Kozikowski.

Khoi talked about a wide range of topics ranging from design organizations, his daily routines, etc.

He views the organization as being secondary to creating a team and having the team work together well. However, he feels that great products can also come from individuals.

In the design process he feels quieter is better so mornings or evenings are best. He talks with people during the day.

He starts with paper and pencil combined with research online using Pinterest. He feels there are few original ideas and wants to see what has worked or not worked

He uses compCC as a bridge between pencil and paper and Photoshop

He uses Basecamp or Slack for feedback. The best designs come from lots of healthy iterations including time spent thinking about the problem and away from the problem.

For his group, he looks for people who want to learn things especially at startups since they change rapidly.

He is methodology agnostic and is driven to solve problem with new tools and methods.

He notes that learning specific tools will not make a career and communication (especially writing) makes your career.

Mentoring requires development of a personal relationship to know how the other person defines their world.

On web design, current there is a lot of sameness, especially with flat design breaking the site into blocks with big photos and block letter. He sees the next step as development of designing rules (rather than layouts), especially content aware designs. Theses will consider key colors reflected in the layout and have tools using machine learning and computer vision.

He foresees tools moving from desktop to mobile. This will give all designers more flexibility to work in their best environment. Some of this process is already happening with cloud based apps such as Google docs, Dropbox, etc. The move from box to subscription software also moves in this direction as the ecosystem becomes an essential part of the infrastructure.

posted in:  DesignDrivenNYC, UX    / leave comments:   No comments yet