In the present phase of the course, we are looking at methods for analyzing in depth particular aspects of a design. In Class 8 (rescheduled to June 15th), we will discuss the GOMS model. We will use to this notation to describe with considerable precision the way in which a user U performs a particular task.

Since you probably don't know anything about GOMS yet, your job now is simply to (a) choose an aspect of your S that you would like to analyze in detail and (b) propose an improvement to S that you think will enhance its usability significantly.

If you would like to do some reading before the rescheduled Class 8, you can read Section 6.7, which introduces the GOMS model. But it isn't necessary to read this section in order to be able to do this assignment.

The following instructions include some examples involving (a) a digital alarm clock and (b) an email system. Since the alarm clock example will be discussed in Class 8, you may not understand all details here; but the point of the example should still be clear.

Part 1: Choose a Possible Improvement

Think of and describe briefly a relatively simple improvement that could be made in the design of your S.

Example from the alarm clock domain: "A 'backward' button should be added such that, when U holds this button down while pressing other buttons, the digits move backward rather than forward".

We will use the symbol S+ to refer to the modified version of S which results after you have made your improvement.

Here is an example of a reasonable improvement for an email client: In S+ there is a "Delete" button in the window for reading emails (whereas in S messages can only be deleted from the window in which all emails are listed).

When choosing a possible improvement to analyze, make sure that the change from S to S+ results in some clear difference in the observable steps that U needs to execute to perform some task. For example, don't choose an improvement that involves only the labeling of a button to make it more comprehensible.

Also, make sure that the improvement you choose is one that you find really significant - and not so obvious that it would be boring to study it in detail. You will probably be studying the same aspect of your S in the empirical study that you will perform later in the course.

Part 2: Describe a Simple Task Related to the Improvement

Describe in normal English a task T such that

• U can perform T with either S or S+

• you believe that the performance or learning of T will be better in some respect(s) with S+ than with S
  The improvement can concern various different evaluation criteria: speed of execution, mental load, ease of learning, ...

Alarm clock example: "Task T = set the alarm for a given time of day. Initial situation: U has not yet pressed any button, and the alarm time that is currently set is different from the time that U desires."

Note that the description of the task should not be so specific that it specifies particular values (e.g., a particular alarm time or folder name).

It's important to specify the initial situation. For example, with an email client, an initial situation might be: "U has just finished reading a message and has not yet closed the message-reading window."

Part 3: Explain the Expected Advantage(s) of Your Improvement

Explain why you think that S+ is better than S.

For the alarm clock example, you might write the following: "With S+, the 'backward' button should save U a good deal of time in cases where the backwards distance to the desired time is much shorter than the forwards distance (e.g., when U wants to change the alarm time from 7:10 am to 7:00 am)."

Write a GOMS analysis of the method(s) that U can apply to perform the task T.

For the alarm clock domain, an example analysis is shown on Slide 188. This analysis describes the actual alarm clock that we looked at in class.

Part 2: Write a GOMS Analysis for S+

Write a corresponding GOMS analysis that presupposes that U is using S+ rather than S.

We discussed briefly in class how the alarm clock analysis would have to be changed to describe the improved alarm clock with a 'backward' button.

If your analysis for S+ differs in only a couple of places from your analysis for S, you don't have to repeat all of the parts that are the same. Instead, you can write just those parts of the analysis that are different for S+, as long as it remains clear how the analysis as a whole looks.

For the alarm clock example, you would only have to write for S+ the second part, for the goal "Set hour / minute", since the first part remains unchanged. The only difference in the second part is that an additional optional step is introduced at the beginning: "Press and hold button 'backwards'".

Part 3: Explain the Advantage(s) of Your Improvement

Referring to your GOMS analyses, explain why you think that S+ is better than S.

Note: You already gave a similar explanation for the previous assignment. The difference now is that you must refer explicitly to specific parts of your GOMS analysis.

For the alarm clock example, you might write the following:

"With S+, the step in which U presses the 'backward' button is optional. If U chooses not to execute this step, the performance of the task will be exactly the same as with S. But I expect that many Us will choose to perform this step when the backwards distance to their goal is much shorter than the forwards distance (e.g., when they want to change the alarm time from 7:10 am to 7:00 am)."

If U does execute this step in such a situation, the performance of the remaining steps in the method will take less time, because there will be fewer digits to pass through before the goal is reached. In the most extreme case (e.g., moving from 12:00 pm to 11.59 am), U can save 22 button presses while changing the hour and 58 button presses while changing the minute - or a corresponding amount of time if he chooses to hold the button down and wait for the digits to change."

Part 4: Mention Possible Disadvantages of Your Improvement

Look at your GOMS analysis and see if S+ seems to have any disadvantages relative to S. Explain any such disadvantage in terms of the GOMS analysis, and say how serious you think it is. If you think that there are no disadvantages, explain why you think so.

Alarm clock example: "With S+, U has one additional choice to make, because there's one additional optional step. Making such a choice takes a certain amount of time. So in the cases where U decides not to use the 'backward' button, the total time to perform the task will be slightly longer than with S.

But this disadvantage is quite small compared to the potential time savings mentioned above."

General Hints

1. Pay close attention to the examples on Slides 193 - 198 and the hints on Slides 201 - 206, so as to make your GOMS analysis as correct as possible. (As was noted in class, the example on Slide 197 - 198 contains an error: the word "select" in the 8th line must be deleted, since what is involved is an objective condition, rather than a free selection.)

In this and the subsequent step you'll be carrying out an empirical evaluation of the same general sort as the one discussed in (Class 10). But your evaluation will be on a much smaller scale. In particular, instead of comparing two different systems, you will be looking only at how a user works with your system S. (For most of the systems, it would be impractical, in the limited time we have available, to realize two different variants of the system.)

To see how this work might fit into a real software development project, imagine the following situation: You are working as a usability consultant on the next version of your S. You have presented to the other team members the arguments in your previous report (with the GOMS analysis) as to why a particular aspect of your S should be improved in the next version. But the other team members are unconvinced; they say "We don't think this aspect of S is really that bad. Give us some hard, systematically collected evidence!"

In this step you'll work out the preparations for the evaluation; in the following (and final) step of the design project, you'll actually perform the evaluation and write up the results.

The evaluation and its results will be written in the form that is standard for reports on psychological experiments. This form is often used in the literature on human-computer interaction, so it's useful to have a good grasp of it. Also, having a predetermined format makes it easier to write, by allowing you to concentrate on the content.

For Class 11, you will write the Introduction and Method sections of the report. (For Class 12, you'll write the Results and Discussion.) Write these first two sections as if you had already done the evaluation. That is, write in the past tense (e.g., "Each subject performed three tasks ...") even though the events described haven't actually taken place yet. This way, you will need to make only minor modifications to the first two sections when submitting the final report. (This practice of writing the first two sections before the study has been conducted is frequently applied.)

For almost everything you need to write, you can find a relevant example in the long evaluation report that we discussed in Class 10. But since that report doesn't entirely follow the standard format, the relevant parts will usually be found under different headings than the ones used here.

Part 1. Write the Introduction

The Introduction should address the following questions, in turn:

1. What aspect of your S will you be evaluating?

2. What question(s) are you trying to answer?

Here, you can probably just cut and paste (and edit) some text from your previous report. But if you want to focus on a different aspect of S in this study, feel free to do so.

An example answer to the second question might look as follows:

"On the basis of my GOMS analysis, I predict that this aspect of S's design will mean that U spends a large proportion of his time waiting for a system response when U is performing tasks like [description of a type of task]. The purposes of this study are"

• to see objectively if this is the case and
• to find out how users subjectively evaluate the waiting times."

Part 2. Write the Method Section

The Method section comprises the following standard subsections:

• Subjects

  Which person(s) served as subjects? Don't give their names, but describe them, e.g., as follows: "Two members of the ABC Club, one male and one female, aged 20 and 22, respectively, served as subjects." (Reminder: You are writing in the past tense, even though these persons obviously haven't really already served as your subjects.) If they had any special knowledge, experience, or other relevant characteristics, describe these. What reward (if any) did they receive for their participation?

  If possible, try to find two subjects so that you'll have an interesting amount of data to work with. To increase efficiency, consider performing your study back-to-back with that of another course participant, so that the same subjects can serve both of you.

• Apparatus

  Describe briefly the type of computer used by the subjects, mentioning any properties of it that might affect the outcome - e.g., screen size and resolution, what kind of online connection (if any) was available), ....

• Materials

  Generally speaking, the Materials section of an empirical report describes the stimulus material presented to the subjects. In the present case, this material consists mainly (or exclusively) of the tasks that the subjects had to perform and the questions that they answered.

  First, write down a specification of the tasks in a form that you can present directly to the subjects; include this specification as an appendix to your report. Then, in this section of the report, describe each task more briefly, indicating the main idea behind it.

  How many tasks should you define? This depends on how long they will take. You'll want to give each subject at least 10 or 15 minutes of actual work to do, not counting the presentation of instructions. (Otherwise, you won't have much data to go on, and the subjects will feel as if it was a waste of their time to come and participate.) So if each individual task can be completed in much less time, define a number of instances of each task.

  In addition to the main tasks, you will probably want to define at least a few practice tasks that the subject can use to become acquainted with the system. You should omit practice tasks only if

  • your user will be a person who is already an experienced user of S;
  • you are mainly interested in how a new user deals with the system when first encountering it.

  Th other aspect of the materials is the questions that you asked the subjects after they were finished with each system. Summarize the content briefly in the main text, and present the exact questions in an appendix.

• Design

  Generally speaking, the "Design" section describes the structure of an experiment: what independent variables were manipulated, what dependent variables were measured, how the subjects were divided into different groups, and how different tasks (and/or task presentation orders) were assigned to different groups.

  Since your study is much simpler than a typical experiment, all you need to describe here is the objective measures that you will be looking at. Here are some examples of such measures:

  • How much time did U spend waiting for a system response, as a proportion of the total time spent working with S?
  • How often per minute did U have to switch from one web page to another?
  • How many errors of each of the types X and Y did U make?

• Procedure

  The Procedure section describes, basically in chronological order, what happened to the subjects from the time they started their participation until they left. It should deal with the following points:

  • General introduction

    What did you tell the subject at the beginning of the experiment? (You needn't include the exact words.)

  • Introduction to the system and the tasks

    How did you explain the system to U, if U was not already familiar with S? Summarize what screens you showed the subject and what you said about each one.

    How did you say that the subject should approach the tasks? For example, were they instructed to work as fast as possible, as accurately as possible, or with equal priority for speed and accuracy?

  • State here that the subject performed a set of tasks with the system in question and say how you recorded the objective measure(s) described in the "Design" section (e.g., using a stopwatch).

  • Asking of questions about the system

    Note here that you asked the questions described in the Materials subsection about the system after the subject completed the tasks with it.

Overview

The main goals here are to report the results of your evaluation and discuss their significance. It's customary to separate these two steps. That is, the Results section just describes objectively what happened, including at most a bit of comment on each aspect of the results. The Discussion section then draws conclusions on the basis of all of the results taken together.

Part 1. Write the Results Section

Comments on How the Evaluation Went

Describe here anything that didn't go quite as expected (e.g., problems with the apparatus, failure of a subject to follow instructions).

Objective Results

Present the results concerning the quantitative, objective measures (e.g., percentage of task achievement). The method of presentation used in our example evaluation of library information systems (Class 10) is probably most appropriate.

In an Appendix, make one or more tables with the raw data for each dependent variable (see, e.g., Tables 9 and 10 on Slide 285). Then in the main part of the report, present a shorter table showing the means (averages) for each subject, as well as the overall means (cf. Table 3 on Slide 259). One difference between your tables and those of the longer evaluation study will be due to the fact that you are not comparing two different systems. Therefore, you won't need separate columns (or tables) with titles like "VT100 Interface" and "WWW Interface".

Note: In published experimental reports, raw data like these are seldom presented. But it's always worthwhile for the investigator to examine the raw data. For example, extremely high or low values require explanation, and it may be misleading just to average them in with the other values. Moreover, since we are dealing with small samples, we will mostly be interpreting the data of individual users rather than just the means of the whole set of users.

After each such table in the main report, you may want to add a comment summarizing the results (e.g., "As can be seen in Table 3, the Subject 2 spent most of her time looking at Page A"). But leave deeper discussion of the results until the next section.

Questionnaire Results

Here again, the format in the library evaluation can be recommended:

In an Appendix, give the full details of the questionnaire results. For each question you asked and for each subject, note which category the subject checked and what comments (if any) he made.

In the main text, summarize the questionnaire results, once again without much interpretation in this section.

Other Observations

Describe here any other interesting information that you acquired. For example, did you notice anything in the subjects' behavior that isn't reflected in the results reported so far? Did subjects make spontaneous comments that revealed something that wasn't reflected in their answers to the questionnaire questions?

Part 2. Write the Discussion Section

Discussion of Main Issues

Look back at the "Main Issues" that you formulated in the Introduction: Try to answer each question in turn (not necessarily in the order in which it was asked). When answering a question, refer back to specific results from the Results section. Try to integrate the different types of results (objective measures, questionnaire results, other observations) so as to arrive at a coherent picture. Example: "The need to switch back and forth between Pages A and B results in a high proportion of frustrating waiting time, as is shown by the data concerning the sequence of pages visited and the waiting times, in combination with the subjects' questionnaire responses and spontaneous comments concerning system responsiveness."

If possible, refer back to the results of previous design project steps (e.g., your contextual analysis or your knowledge-based analysis) to help make sense of the results.

Discussion of Other Interesting Points

Maybe you gained some new insights about questions that hadn't originally occurred to you and so weren't mentioned in the Introduction. Discuss these points in a similar way.

Part 3. Write the Conclusions Section

The Conclusions section is brief, perhaps comprising just one paragraph. It should contain the "bottom line" that the reader should "take home" as the main thing that was learned from the study. (See, for example, the final sentence on Slide 271.)

Another example of a possible conclusion is the following:

"Expert users appear to be satisfied with S on the whole, but the need for them to switch back and forth between Pages A and B while performing Task T is an important source of frustration and time-wasting. If the full satisfaction of expert users is considered to have high priority, it will be worthwhile to introduce a single page or frameset in which the information from both of these two pages is available."