Please note that as of October 24, 2014, the Nokia Developer Wiki will no longer be accepting user contributions, including new entries, edits and comments, as we begin transitioning to our new home, in the Windows Phone Development Wiki. We plan to move over the majority of the existing entries. Thanks for all your past and future contributions.

Revision as of 08:57, 20 December 2012 by Krebbix (Talk | contribs)

Series 40 UI design usability testing webinar - companion article

From Wiki
Jump to: navigation, search

This article is companion for the Debug your design for Series 40 Full Touch webinars held in December 2012. It covers both sessions: 11th and 12th December 2012.

Article Metadata
Created: Krebbix (10 Dec 2012)
Last edited: Krebbix (20 Dec 2012)



You know it's important to test your app before you publish it, but if you’re testing only your code, you’re testing only part of your app. It’s important to find bugs in your design as well. Otherwise, the app may behave exactly the way your code intends, but users may still find it frustrating and give it low ratings in Nokia Store. Join UX expert Jan Krebber of Digia as he presents simple methods to find UX bugs and shows you how to correct them. These techniques can help you increase your app quality whether you work with a large organization or completely on your own. As usual for Nokia Developer UX webinars, the presentation will feature exercises that will receive follow-up treatment in this Nokia Developer Wiki article.

The webinar and the wiki article are a great starting point if you’re not a UI design expert. Check out Series 40 UI design library for more advanced information and resources.

This wiki page is the "companion" to the webinar. It includes:

  • Webinar exercises and proposals for how to solve them
  • Checklist example from the webinar
  • Time plan example from the webinar
  • Task examples (pending)
  • Open issues from Q/A
  • Further links and references
  • Slide deck link
  • Recording link (will be added )
  • Webinar announcement

Proposals and solutions to webinar problems

This section contains problems raised in the webinar exercises. In addition, it contains proposals and possible solutions to solve the problems.

Watch the videos below. What went wrong? How do you fix it?

Shopping list


The media player is loading...


  • She does not find the intended way of creating a new item via the list item.
    • She sees the text
    • But she is reluctant pressing it
  • She states that she would have expected an "add new" button in Action Button 1 and not as list item (0:50)
  • She goes via the Options menu to create a new list, which is more work than intended
  • As long as "create new list" is not becoming it's own button, people may miss its functionality


  • Change the list item "Create new list" to Action button #1.

S40UI testing 01.png

Purchase map


The media player is loading...


  1. More information is not anticipated
    • She is wondering from where to get more information before she buys the item
    • Not evident from the layout that there is more information after the next step
  2. Expects to buy the item after the 2nd screen, but she does not know how much will cost
    • It says "buy" even the next step still shows some information
    • The price information is missing
  3. Buying process broken at the end
    • She thinks the process got broken since she did not see/know where her item went
    • Wants to start the purchase process again


  1. Inform people that there is more information after this view before they actually buy
  2. Do not show "buy" here, but mention the price at this point
  3. Buy should be reserved for the last screen before the shop UI takes over
  4. Change the purchase flow so that the user sees the map download. This could be even an "artificial download indicator" just long enough for the user to see where it went, that it got downloaded.

current implementation

S40UI testing 02.png

Update proposal

S40UI testing 03.png

When are 2-3 participants sufficient for usability testing? Why?

  • If you test multiple times, e.g. in a 8 week mobile implementation phase with 4 sprints could have 5 tests (one before and 4 after each sprint) and 2,3,2,2,3 participants, that yields to 12 participants
  • You will not find all usability problems in the first test round, but with 12 participants you will spot the majority of your problems
  • Frequent tests will allow you to re-test some of your redesigns or tweaks
  • With more participants (4-6) you will not learn much more during one round

You are not sure what a participant is thinking. What do you say?

  • What are you thinking? :-)
  • What are you doing right now?
  • Is this what you expect to happen?

A participant mentions that she is not giving you what you expect from her. What do you say?

  • Even if you think this is not the right information you were looking for, it can be valuable information. You must collect it and analyze later, after you have a clear mind.
  • It is important to keep people talking and not to scare them away but make them feel comfortable.
  • Even if the current information is completely useless for you, you must keep the participant in a comfortable mood. Maybe she will will point out some important stuff later on.
  • Value participants' input, value their help, value that they give you their time.
  • You must have a positive attitude about all the information you are getting from a test. The information is there to improve your application and to help you in the end to improve the Return-of-Investment (no matter if this is real money or credits).
  • You say:
    • No worries, this is exactly what I am looking for.
    • I haven't looked at it from that point of view before, please tell me more about it.

A participant thinks she is doing everything wrong. What do you say to her?

  • You are testing the application, not the participant, therefore the participant is not doing anything wrong.
  • If you still think she is doing something wrong, you must question your preparation first - especially if this happening more often to you
  • You say:
    • No you are doing everything correct, the application is not working correctly.
    • You are doing a great job. It shows where we have to improve our application.
    • You are not the first one running into troubles here, we have to change that.
  • However, it is very important that she continues so that you check your app as much as possible and improve it wherever possible.

Why not give the participant all the tasks on paper

  • It is unlikely that you skip one task if you have a pile of cards and take one card after another from the pile
  • On paper you may skip a task more easy
  • Do not let the participant sneak into the next task
    • She should concentrate on the current task
    • The next task might reveal the answer to the current task

How would you classify the errors you find. Which errors do you fix first?

One possibility is to classify into 3 classes:

  1. Immediate (must fix, if you do not fix it people will give you bad ratings and it might happen that people won't open your app a second time because of bad user experience)
  2. When time (should fix, many will benefit and it still affects the rating of your app)
  3. When drive by (minor)
  • The most urgent error must be fixed as early as possible and it must get all your attention.
  • There is no use in continuing with anything else before, because if you do not fix it your app will fail
  • The biggest issue is if people get stuck in your app in a way that you (as an app author/publisher)
    • you do not make money
    • waste people's time
    • waste people's money

Here would be an additional list for classifying errors:

  • If the majority cannot reach a goal at all, it is a must fix
  • If the minority cannot reach a goal at all, it is a should fix
  • If the majority cannot reach a primary goal conveniently, it is a must fix
  • If the minority cannot reach a primary goal conveniently, it is a should fix
  • If the majority cannot reach a secondary goal conveniently, it is a should fix
  • If the minority cannot reach a secondary goal conveniently it is a minor error
  • Typos, broken layout (in a way that you still can read everything correctly), this is a minor error

However, this requires some feeling if the mistake shown was really an exception or if it might happen with most of the people, since you are testing with only few people in each round.

Checklist example

  • Get participants (for 28.11.)
  • Make schedule
  • Get helper to bring the next participant
  • Task list & scenarios on paper
  • Prototype for all test cases
  • Pencil, eraser, paper, scissors, post-its
  • Rooms
  • Refreshments
  • Rewards
  • Agreements printed
  • Task list & scenarios on index cards
  • Call participants day before and remind (27.11.)
  • Run pilot

Time plan example

Minutes Tasks
00 Welcome
01 Introduction
  • My name is...
  • The project is about booking tickets from your mobile phone for 1st and second 2nd league soccer events
02 Read and sign the agreement about the test and NDA is necessary
05 Warm-up questions
  • What are you doing for living?
  • What are your hobbies?
  • How old are you?
  • What is your current mobile phone?
  • Which phones did you use before that?
  • Have you ever bought any goods (real hardware) or event tickets from your mobile phone?
10 Tasks
  1. Give story 1 (start).
  2. Book 2 tickets for the game "FC Dudes vs. Dudes United" in March.
  3. Give story 2 (more info).
  4. Find information about the event place.
  5. How do you get there?
  6. Find more about FC Dudes team.
  7. How well performed Dudes United in comparison to FC Dudes this year?
  8. Give story 3 (you go there).
  9. Drive to the event place.
  10. Show the e-tickets at the entrance
  11. Find your location in the stadium
40 Closure
  • Open issues
    • What was unclear to me?
    • Any task which should be done again?
    • Any new task I would like to see performed?
  • Any questions from the participant?
  • I wrap up
    • What went well 1-2-3
    • What needs improvement so that it becomes easier for the participant 1-2-3
  • Give reward
  • Thank you & Good bye
50 Transition
  • Take additional notes
  • Reset prototype
  • Refill coffee
  • Toilet break

Task examples

Currently under review, I hope can put the material before the end of the year.

Open issues from Q/A

No open issues left from Q/A sessions.

Further links and references

179 page views in the last 30 days.