Crowdsourced Usability Testing Rnd 1 – Investigating Information Architecture


So, today is supposed to be our first day of Crowd Sourced Usability Testing – you might have noticed that we’re running a little behind on this, so to make up for the delay and lack of notice – and also because this is our very first go at such an activity, we’ve decided to try to make it a fairly simple task. You should be able to conduct this test in around 15-20 minutes. Please test as many or as few people as you can find the time (remember that you’ll need to allow some time for each person to share your results, which might take as much time again as your test!)

What are we testing:
If you have looked at some of the sketches that we have been working on over the past few weeks (some posted on this site and a lot on Flickr), you may have noticed that a ‘header’ has been quite a persistent feature. This is a concept that we are pursuing. We have a first draft of an ‘Information Architecture’ or navigation for that header. We would like you to help us test how well that navigation is working and where we need to make amendments. Based on these findings we will be able to further refine the navigation.

I have created a PDF document that includes the Header design we’ll be testing as well as a copy of the suggested script for your interview – you can download that here.

When are we testing:
I’m going out to do my first round of testing this afternoon, but if you can find any time this week to do a test and submit your findings that would be great (that is by end of Friday 10 April) – we’ll have another task ready for you shortly after then!

Who are we testing:
For this round of testing your interview participants should have some familarity with Web Content Publishing Systems. They do not have to be familiar with Drupal, nor do they have to be Drupal developers. Anyone who publishes content to the web using some kind of content management system is appropriate for this round of testing.

Sample script for your interview:

Your interview should be divided into five parts:

  1. Your introduction
    – Start by thanking the interview participant for being involved in the project and telling them how much we appreciate their time.
    – Explain to them that we are sharing the results of testing publicly, ask them for permission to record the session on video and to publish the video to the web. (Allow them to be unidentifiable/off camera if they prefer)
    – Give them a brief overview of the project (we are trying to find a way to make Drupal a better user experience, especially for people who are not developers). Tell them it is very early in the process and that their feedback will play a big role in helping us get the design right.
    – Reassure them that it is the design that is being tested, not them – if they don’t understand something, if something is unclear, if they don’t know the answer or feel they have made a mistake that is a problem with the design that we need to fix, not a problem with them as users!
    – Encourage your participant to ‘think aloud’ – if they are considering the answer to one of your questions, to talk you through the how they are making the decision (this is what is most interesting to us – *why* people get to the decision they do, not necessarily which decision they make).
  2. Background information of your participant – start with a simple question about your participant (if you know these already you don’t have to ask them, although it is a nice way to ease into the interview).
    – What is your experience with managing content on the web?
    – Do you have any experience with Drupal? (if no, what do you know about Drupal, if yes, describe experience)
    – (If appropriate, Are you a Drupal developer?)
  3. ‘What If’ questions
    We are going to work through the top line of the header navigation one item at a time and ask:
    ‘If you were to click on ‘Content’, what do you think would happen? What would you see?’
    ‘If you were to click on ‘Information Architecture’ what do you think would happen? What would you see?’
    and so on.
    For each of the items, if your participant seems unclear as to what the navigation item means or refers to, be sure to ask them what the *think* it might mean, and why they think that. For example, we suspect the vast majority of people have never heard of ‘Information Architecture’ but we wonder whether they can make a reasonable guess as to what that section contains.
  4. Tasks
    Now we are going to run through six quick tasks. Again, pay as much attention to *why* your participant is doing / saying what they are saying as what they actually do and ask questions along the way, encourage them to think aloud. These tasks ask the participant to imagine that they have a company website that they are administering.
    – Task 1.  – You need to add a new staff biography to your website
    – Task 2. – You need to remove an old staff biography from your website
    – Task 3. – You want to add a Contact Us form to your website
    – Task 4. – You want to give your new staff member access to publish content on your website.
    – Task 5. – You need to make revisions to an article you published on the website this time last year.
    – Task 6. – You need to make a new product category for the products listed on your website
  5. Final feedback and thank you
    Finish by asking your participant to give you and further feedback they would like to on what they have seen today, then thank them again for their assistance with the project.

Some Tips for Interviewing

The best way to get good feedback from your participant is if they are relaxed and focused. It is always worth spending some time chatting at the beginning of the interview to build some rapport with the participant (that is, if you’re not already friends!), so that they feel comfortable. Even if they *are* your friend, you should make sure that they know that it is not them being tested, it is the designnever allow your participant to feel ‘stupid’ during an interview. If they are having trouble, ask them to explain what is troubling them, then help them out or move on.

Remember that the most important information we can get from these sessions is *why* people do and think the way they do, not necessarily what they do – continue to prompt them to think aloud and ask questions whenever you see them do something interesting or unusual, or if they are stuck or unsure – ask them *why* they aren’t sure what to do, and to explain what is troubling them and how they are trying to work it out. This information is gold!

Sharing and reporting your findings:

I’ve created a webform where you can submit your findings for each interview you do. You can find it here. (I had hoped to use Google Forms for this, but it turned out easier and faster to do using Survey Monkey. Hopefully in the future we can switch to Google so that the data can be shared with you all more easily.)

You will need to complete one form for each participant, although you’re welcome to just email me your findings if that is easier.

If you are able to video your interview, it would be great if you can post it to our YouTube group (or otherwise, let me know where you have posted it!). Note that YouTube only allows videos of 10mins or less, so you may have to edit your video or, easier still, break it in two as you record it.

This is experimental – your feedback welcome!

As mentioned above, this is the first time we’ve run an exercise quite like this, so there are probably a million ways we can do it better. Rather than labour over trying to get it perfect the first time (which we would never be able to do anyway!) we’re going to go for it, have a go, and see what we learn. You are our guineapigs and we appreciate no end your participation, but as a result, there may be something that don’t work as well as they should. To that end, please let us know what works well and what could be better, and how we might do things differently, and we’ll iterate our process so that the next time and the next time after that are continually improved!

Thank you again for your participation, and do let me know if you have any questions!

9 thoughts on “Crowdsourced Usability Testing Rnd 1 – Investigating Information Architecture”

  1. Please do not ask participants “what if” questions before you do the tasks, as you will obviously influence the participants performance.

    E.g. if you point at an “Add new staff biography” link, and then ask the participant “What do you think this does”, when you ask the user to do Task 1. – “You need to add a new staff biography to your website”, they will immediately know where to find the answer, which would not be their natural behaviour.

  2. I don’t know if I’m missing something or not as I’m new in this initiative, but the testing is done only with this header in pdf?

    Are there any other materials in this first round?


  3. @Alina, yes, for this round we are only testing the header which you can find in the PDF file I’ve linked to. Just a simple exercise to get us started and to iron out any bumps in the process. As we move further through the project the prototypes will get a lot more complicated and move from paper/pdf to interactive/HTML.

    let me know if you have any questions

  4. re: the script order and whether you do the ‘tasks’ or the ‘what ifs’ first… I don’t really mind and I don’t think it makes a huge difference either way at this stage.

    The point of this round of research is to understand how the navigation items are working (or not) whether people can make sense of what would be beneath them and where they would accomplish different sets of tasks. The reason I put the script in this order is because I’m actually more interested in the ‘What If’ questions that the tasks for the moment, so I’m more interested in people trying to interpret them than in seeing whether they can complete the tasks.

    Certainly, if the prototype and project was more evolved I would agree 100% that the tasks should come first…

    Since David left this comment I’ve actually run two more interviews switching between the two script structures and I didn’t find a huge difference in the way the interview played out or the findings… so, feel free to experiment a little if you like.

    Just make sure you’re getting a good sense of people’s expectations and how they’re trying to understand what certain terms on the navigation mean and represent (and perhaps, if they are existing Drupal users) how they are trying to map this navigation back to their knowledge of how Drupal works.

    thanks again for your interest and feedback!

  5. Are we testing the very top line (content … help) only or also the second (add … dashboard)? In my first two test I did test the second line. I certainly did not harm, and the response to Find content and Dashboard was interesting.

  6. […] Crowd-sourced Usability Testing: help us get the D7UX tested all over the world and get some usability testing experience under your belt – the first testing ’session’ is happening right now! […]

Comments are closed.