The user evaluation involved stepping the user through a set of common tasks. Possible evaluation methods include the cognitive walk through and heuristic evaluation.
Using the cognitive approach we explained the scenario, showed the user the prototype and explained that it was split into several sections (actions on the left, screen, buttons on the ride hand side and below the screen). The user was asked to talk about each of the actions that they were doing. The details of both approaches are available in a book written by Russell Beale et al.
Using the cognitive approach we explained the scenario, showed the user the prototype and explained that it was split into several sections (actions on the left, screen, buttons on the ride hand side and below the screen). The user was asked to talk about each of the actions that they were doing. The details of both approaches are available in a book written by Russell Beale et al.
The user was given the following list of actions:
- Forward
- Backwards
- Left
- Right
- Add
- Settings
- Flip
- Time
And the following set tasks:
- View a person on the device.
- Find the particular person "Christina Angel."
- Find "Christina Angel's" age.
- Add Her To Facebook
- Find out what course Kate Hulme does.
- Find out an event Christine is attending
- Change the setting "Give Limited Info"
- Find a Person We Met Last Month
- Show the time
As the prototype required our intervention we had to process the task in stages.
This was then presented to each user and they were evaluated against the following 4 criteria:
This was then presented to each user and they were evaluated against the following 4 criteria:
- Will the users be trying to produce whatever effect the action has?
- Will users be able to notice that the correct action is available?
- Once users find the correct action at the interface, will they know that it is the right one for the effect they are trying to produce?
- After the action is taken, will users understand the feedback they get?
User evaluations
We performed four evaluations in total, two practice runs that we have not displayed the data for, as we felt that, firstly they were already too familiar with our prototype having explored it before, and inevitably the actual process of asking questions and recording results we didn't really get right on these attempts.
We then asked two people from computer science that we knew to a lesser extent; Jing Tang, & Ben (from Joe's robotics class) and they were both kind enough to submit to evaluations. The raw data of which is available to download here.
Findings:
On the whole we are pleased with how our users responded to our device prototype, yet we found deficiencies in a number of areas. Although in fairness we were harsh in our evaluations by giving the users the bare minimum of information and explanation to complete the task. The users we chose were completely cold to our concept device, as they were not people we had surveyed for paper prototypes.
On the first task we noted that it was not always clear to the user how to 'activate' the device in a sense to display a person. Ben needed some prompting, while Jing was more happy to press buttons to discover functionality.
On the second task to find a user, having activated the device both users found it easy to navigate to a particular person using up or down, but Jing noted that he had the expectation of alphabetical index, so we could have made the chronological nature clearer.
The third and fifth tasks highlighted quite a big deficiency in our design, as both users immediately went for the left and right buttons as oppose to pressing flip, it was only after having pressed left and right that they thought to press flip. On essentially being asked the same question in a different format Ben again got confused and selected left or right buttons, before hitting flip. Whereas Jing recalled the behaviour and got it correct.
Adding to facebook was simple for both users, to this is obviously very intuitive.
Finding the event Christina was attending was surprisingly simple for Ben, he selected Christina, hit right, then flipped the event. Whereas Jing navigated to the RISA card and assumed this was the event she was attending, and never bothered to flip card without prompting. So this is an obvious deficiency.
Changing the setting was a deliberately difficult question, as the setting mentioned is the odd one out on the particular display, but it is the button that I expect people to use most often. It became obvious to Ben that he needed to press the add button, after he had pressed one of the directional buttons and seen the reaction. Jing exhibited similar behaviour. On this particular setting some extra feedback or an icon could perhaps be helpful however.
The eighth question about finding a person met some time ago was only asked to Jing, but the behaviour was not obvious to him. Unfortunately I think this question is unanswerable with this prototype. This is because the user needs the mouse to navigate the prototype, and to press and hold an on screen button is completely alien to the majority of users.
Fortunately the ninth question of displaying the time was completely apparent at almost all times for both users, although Jing did press this button when asked to find a person he met some time ago, and commented that he expected the button to be a toggle as he had pressed settings and flip and found them both to be toggle. This was a surprise result, but from his point of view it was a logical thing for the device to do.
We performed four evaluations in total, two practice runs that we have not displayed the data for, as we felt that, firstly they were already too familiar with our prototype having explored it before, and inevitably the actual process of asking questions and recording results we didn't really get right on these attempts.
We then asked two people from computer science that we knew to a lesser extent; Jing Tang, & Ben (from Joe's robotics class) and they were both kind enough to submit to evaluations. The raw data of which is available to download here.
Findings:
On the whole we are pleased with how our users responded to our device prototype, yet we found deficiencies in a number of areas. Although in fairness we were harsh in our evaluations by giving the users the bare minimum of information and explanation to complete the task. The users we chose were completely cold to our concept device, as they were not people we had surveyed for paper prototypes.
On the first task we noted that it was not always clear to the user how to 'activate' the device in a sense to display a person. Ben needed some prompting, while Jing was more happy to press buttons to discover functionality.
On the second task to find a user, having activated the device both users found it easy to navigate to a particular person using up or down, but Jing noted that he had the expectation of alphabetical index, so we could have made the chronological nature clearer.
The third and fifth tasks highlighted quite a big deficiency in our design, as both users immediately went for the left and right buttons as oppose to pressing flip, it was only after having pressed left and right that they thought to press flip. On essentially being asked the same question in a different format Ben again got confused and selected left or right buttons, before hitting flip. Whereas Jing recalled the behaviour and got it correct.
Adding to facebook was simple for both users, to this is obviously very intuitive.
Finding the event Christina was attending was surprisingly simple for Ben, he selected Christina, hit right, then flipped the event. Whereas Jing navigated to the RISA card and assumed this was the event she was attending, and never bothered to flip card without prompting. So this is an obvious deficiency.
Changing the setting was a deliberately difficult question, as the setting mentioned is the odd one out on the particular display, but it is the button that I expect people to use most often. It became obvious to Ben that he needed to press the add button, after he had pressed one of the directional buttons and seen the reaction. Jing exhibited similar behaviour. On this particular setting some extra feedback or an icon could perhaps be helpful however.
The eighth question about finding a person met some time ago was only asked to Jing, but the behaviour was not obvious to him. Unfortunately I think this question is unanswerable with this prototype. This is because the user needs the mouse to navigate the prototype, and to press and hold an on screen button is completely alien to the majority of users.
Fortunately the ninth question of displaying the time was completely apparent at almost all times for both users, although Jing did press this button when asked to find a person he met some time ago, and commented that he expected the button to be a toggle as he had pressed settings and flip and found them both to be toggle. This was a surprise result, but from his point of view it was a logical thing for the device to do.
No comments:
Post a Comment