So, today is supposed to be our first day of Crowd Sourced Usability Testing – you might have noticed that we’re running a little behind on this, so to make up for the delay and lack of notice – and also because this is our very first go at such an activity, we’ve decided to try to make it a fairly simple task. You should be able to conduct this test in around 15-20 minutes. Please test as many or as few people as you can find the time (remember that you’ll need to allow some time for each person to share your results, which might take as much time again as your test!)
What are we testing:
If you have looked at some of the sketches that we have been working on over the past few weeks (some posted on this site and a lot on Flickr), you may have noticed that a ‘header’ has been quite a persistent feature. This is a concept that we are pursuing. We have a first draft of an ‘Information Architecture’ or navigation for that header. We would like you to help us test how well that navigation is working and where we need to make amendments. Based on these findings we will be able to further refine the navigation.
I have created a PDF document that includes the Header design we’ll be testing as well as a copy of the suggested script for your interview – you can download that here.
When are we testing:
I’m going out to do my first round of testing this afternoon, but if you can find any time this week to do a test and submit your findings that would be great (that is by end of Friday 10 April) – we’ll have another task ready for you shortly after then!
Who are we testing:
For this round of testing your interview participants should have some familarity with Web Content Publishing Systems. They do not have to be familiar with Drupal, nor do they have to be Drupal developers. Anyone who publishes content to the web using some kind of content management system is appropriate for this round of testing.
Sample script for your interview:
Your interview should be divided into five parts:
- Your introduction
– Start by thanking the interview participant for being involved in the project and telling them how much we appreciate their time.
– Explain to them that we are sharing the results of testing publicly, ask them for permission to record the session on video and to publish the video to the web. (Allow them to be unidentifiable/off camera if they prefer)
– Give them a brief overview of the project (we are trying to find a way to make Drupal a better user experience, especially for people who are not developers). Tell them it is very early in the process and that their feedback will play a big role in helping us get the design right.
– Reassure them that it is the design that is being tested, not them – if they don’t understand something, if something is unclear, if they don’t know the answer or feel they have made a mistake that is a problem with the design that we need to fix, not a problem with them as users!
– Encourage your participant to ‘think aloud’ – if they are considering the answer to one of your questions, to talk you through the how they are making the decision (this is what is most interesting to us – *why* people get to the decision they do, not necessarily which decision they make).
- Background information of your participant – start with a simple question about your participant (if you know these already you don’t have to ask them, although it is a nice way to ease into the interview).
– What is your experience with managing content on the web?
– Do you have any experience with Drupal? (if no, what do you know about Drupal, if yes, describe experience)
– (If appropriate, Are you a Drupal developer?)
- ‘What If’ questions
We are going to work through the top line of the header navigation one item at a time and ask:
‘If you were to click on ‘Content’, what do you think would happen? What would you see?’
‘If you were to click on ‘Information Architecture’ what do you think would happen? What would you see?’
and so on.
For each of the items, if your participant seems unclear as to what the navigation item means or refers to, be sure to ask them what the *think* it might mean, and why they think that. For example, we suspect the vast majority of people have never heard of ‘Information Architecture’ but we wonder whether they can make a reasonable guess as to what that section contains.
Now we are going to run through six quick tasks. Again, pay as much attention to *why* your participant is doing / saying what they are saying as what they actually do and ask questions along the way, encourage them to think aloud. These tasks ask the participant to imagine that they have a company website that they are administering.
– Task 1. – You need to add a new staff biography to your website
– Task 2. – You need to remove an old staff biography from your website
– Task 3. – You want to add a Contact Us form to your website
– Task 4. – You want to give your new staff member access to publish content on your website.
– Task 5. – You need to make revisions to an article you published on the website this time last year.
– Task 6. – You need to make a new product category for the products listed on your website
- Final feedback and thank you
Finish by asking your participant to give you and further feedback they would like to on what they have seen today, then thank them again for their assistance with the project.
Some Tips for Interviewing
The best way to get good feedback from your participant is if they are relaxed and focused. It is always worth spending some time chatting at the beginning of the interview to build some rapport with the participant (that is, if you’re not already friends!), so that they feel comfortable. Even if they *are* your friend, you should make sure that they know that it is not them being tested, it is the design – never allow your participant to feel ‘stupid’ during an interview. If they are having trouble, ask them to explain what is troubling them, then help them out or move on.
Remember that the most important information we can get from these sessions is *why* people do and think the way they do, not necessarily what they do – continue to prompt them to think aloud and ask questions whenever you see them do something interesting or unusual, or if they are stuck or unsure – ask them *why* they aren’t sure what to do, and to explain what is troubling them and how they are trying to work it out. This information is gold!
Sharing and reporting your findings:
I’ve created a webform where you can submit your findings for each interview you do. You can find it here. (I had hoped to use Google Forms for this, but it turned out easier and faster to do using Survey Monkey. Hopefully in the future we can switch to Google so that the data can be shared with you all more easily.)
You will need to complete one form for each participant, although you’re welcome to just email me your findings if that is easier.
If you are able to video your interview, it would be great if you can post it to our YouTube group (or otherwise, let me know where you have posted it!). Note that YouTube only allows videos of 10mins or less, so you may have to edit your video or, easier still, break it in two as you record it.
This is experimental – your feedback welcome!
As mentioned above, this is the first time we’ve run an exercise quite like this, so there are probably a million ways we can do it better. Rather than labour over trying to get it perfect the first time (which we would never be able to do anyway!) we’re going to go for it, have a go, and see what we learn. You are our guineapigs and we appreciate no end your participation, but as a result, there may be something that don’t work as well as they should. To that end, please let us know what works well and what could be better, and how we might do things differently, and we’ll iterate our process so that the next time and the next time after that are continually improved!
Thank you again for your participation, and do let me know if you have any questions!