A Think-Aloud Interview
Written in 2016/11/22, Pittsburgh
BEFORE OBSERVATION
Participant information
A CMU student who has never used SIO before. She is a PhD student at Carnegie Mellon University and is quite familiar with computer.
Script
“Hello, I'm Yunting. I’m a master student at Carnegie Mellon University, and I’m conducting a study that looks at how people interact with computers. Do you mind if I talk with you for a few minutes?”
If they are interested, sit down.
“I’m here today because we’re trying to improve Carnegie Mellon University course registration website, also called SIO, that I have on my laptop now, and I need your help with using the website while I take down notes. It’s going to take about half an hour. So are you interested?”
If they are still interested, continue.
“Great! You will be using this laptop, and it will automatically record your voice and your actions on the screen as you use the website. The recordings will be used later to identify where we can make the website better. They will be kept confidential. Your name will not appear in the recordings. In fact, your name will not be associated with the recordings in any way.”
“ I’m going to ask you to perform some tasks with SIO. I’m testing the software, not testing you. If you can’t finish some of the tasks, please don’t feel bad. That’s exactly what we are looking for.”
“And remember, this is completely voluntary. Although I don’t know why this would happen, if you feel uncomfortable in any way, feel free to stop.”
“Here is the consent form, take your time to read through it and sign at the bottom of the page. If you have any questions about it, please feel free to ask.”
Give them the consent form, give them time to read it quietly, and ask for their signature. (Meantime, make sure the computer system is set up correctly.)
“In this observation, I’m interested in what you think about as you perform the tasks. I’m going to ask that you ‘think aloud’ while you are using the website. What I mean by ‘think aloud’ is that I want you to tell me everything that you are thinking from the first time that you see the statement of the task until you finish the task. I would like you to talk aloud constantly from the time I give you the task until you have completed it. I don’t want you to try to plan out what you say or try to explain to me what you are saying. Just act as if you are alone, speaking to yourself — just a little louder.”
“Let me demonstrate that for you as I try to multiply 42 x 22 in my head.”
Demonstrate thinking aloud.
“Do you have any questions about think aloud now?”
Answer questions and continue.
“As you’re doing the tasks, I won’t be able to answer any questions. But, if you do have questions, go ahead and ask them anyway so that I can learn more about what kinds of questions the SIO website brings up. I’ll answer your questions after the session. Also, if you forget to think aloud, I’ll say, ‘please keep talking’.”
“Now, I have some tasks printed out for you. I’m going to go over them with you and see if you have any questions before we start.”
Hand them the tasks and ask if they have any questions.
“Here is the tasks you will be working on. Why don’t you read it aloud, just so you can get comfortable with speaking your thoughts?
“Do you have any questions about the tasks?”
Tell them they may begin.
“You may begin.”
Start recording.
If seconds of silence happens, remind them “please keep talking”.
Prompt
Task 1: Register course Interaction Design Studio 2
Task 2: Register a course that is related to drawing
Task 3: Register a course that is taught by Zywica Matthew
Task 4: Register a course that is on Monday or Wednesday morning
DURING OBSERVATION
Task 1: Register course Interaction Design Studio 2
Confused when she saw the popup that says: “Have you spoken to your advisor?”
She chose “yes” with doubt
Couldn’t find the registration page at first
Started to feel anxious when she couldn’t find the place to add courses
Took about 1 and a half minute to find the “Browse for a course to register” button
Task 2: Register a course that is related to drawing
Mad about that she needed to choose section twice in order to register/add courses to the waitlist
Easier than task 1
Task 3: Register a course that is taught by Zywica Matthew
Confused when she put the name into search box and nothing was found
Tried a different way by changing the lowercase into capital, still didn’t work
Found the “advanced” button (randomly), she seemed not sure about what this button is for, but still gave it a try
Put the name into “instructor” box but still found nothing
Changed the lowercase into capital again, didn’t work
Spent about 3 minutes trying to find a course that is taught by Zywica
Nearly gave up, really frustrated
She acted like she was going to kill the computer
In desperation, she deleted the last name of the instructor, tried again, finally succeeded
Task 4: Register a course that is on Monday or Wednesday morning
Confused about the interface for choosing course time under the “advanced” button
Didn’t take a long time, much easier than the previous ones, pretty smooth
CRITICAL INCIDENT 1
No clear indication for users to browse and add courses
Nielsen’s heuristic(s) violated
Recognition rather than recall
Explanation of the problem
There’s no clear indication for users to browse and add courses, so users (especially first-time users) are confused about where to start. This violates the recognition rather than recall heuristic because the interface fails to give instruction to users from the very beginning. Fisrt-time users will probably abandon the task or delay their registration just because the difficulty to start, which may lead to them ending up in waitlists for the courses they are desperate to get in.
Severity and justification
3 = Major usability problem: important to fix, so should be given high priority
This is the very first step for users to start registering courses, which is the main function of this website. First-time users will probably abandon the task or delay their registration just because they fail to find the start button at the first glance, which may lead to them ending up in waitlists for the courses they are desperate to get in.
Ways to remedy and tradeoffs
The easiest way to fix this problem is to add an eye-catching button for browsing and adding courses. Place the button on the left or center column instead of the right because the left or center area of the webpage has a higher reading priority for most of the people. This is mainly an improvement with no real tradeoff for making this change.
CRITICAL INCIDENT 2
Unable to search courses by instructor’s full name
Nielsen’s heuristic(s) violated
User control and freedom
Explanation of the problem
The user is unable to search courses by instructor’s full name. So they have to put in only the first name in order to get the result successfully. This is very confusing for both first-time and expert users. And there might be two or more instructors sharing the same first name. It violates user control and freedom heuristic for this limits the freedom of the user by restricting them having to put in the first name in order to search.
Severity and justification
3 = Major usability problem: important to fix, so should be given high priority
This can lead to frustration in users. And more importantly, it might give users the wrong information that the course which is taught by a particular instructor doesn’t exist. Users then will probably develop trust issues with this system: this website doesn’t contain all the course information or it has some mismatches with the course information. This can lead to anxiety in users and distrust in this website, and also users missing the course that they really want to take.
Ways to remedy and tradeoffs
The easiest and best fix is to allow users to search courses by instructor’s full name and present all the possible results for users. Could be fuzzy search. This is mainly an improvement with no real tradeoff for making this change.
REFLECTION
Interesting discovery
- The participant found the “advanced” button easily, which is not supposed to be that easy for first-time users.
- Unable to search courses by instructor’s full name.
Improvements to script
Be more specific about the task. For example, make it clear to the participant that the tasks are specifically for courses in spring semester, 2017.
Improvements to observation style
During the observation of the think aloud interview, I found myself hard to resist giving hints to the participant. When the participant started to get really anxious, I started to focus on how to solve the problem instead of observing. I should focus on observing the participant instead of getting too involved in accomplishing the task myself.
ACKNOWLEDGEMENTS
Thanks to Jiajia who helped me with this study by kindly being the participant in this think aloud interview.
