|Human-Computer Interface Design, Advanced Course|
|MICT||Course 216, second trimester 2000, 2 credits|
Contents of this page (last modified: Friday, 7 July 2000)
Design Project Assignments
The file with the Table of Contents has been upgraded so that it provides direct access to individual slides from the other files. See the instructions in the note on the first page of that file.
These slides are unusually easy to view on-line, as is explained on the following page:
(Reading assignments are only posted here if for some reason they were not included completely in the slides.)
This assignment was reduced relative to the initial assignment presented on the slides for Class 6, to take Exam Week into account. (The electronic slides for Class 6 contain this reduced assignment.)
The introductory course "Human-Computer Interface Design" focused mainly on the analysis of the problems that arise in the use of computing systems and devices. In this advanced course, we will turn our attention to the specific activities that are involved in user-oriented design. After successful completion of the course, you should be able to participate effectively in the design process of an interactive system (or application or device) - for example, in the context of an internship - as the person responsible for ensuring the system's usability. (Software engineering and implementation techniques that are dealt with in other courses will be discussed only briefly.)
About 40% of the time required for the course will be devoted to the application of a number of analytical and empirical methods to a particular system. In part, these methods will involve interviews, observations, and testing with users. Part of each class will be devoted to the comparison and discussion of participants' experiences with these methods.
A secondary focus of the course will be on usability issues that concern interactions among people rather than interactions between a single user and a system.
The literature will consist of Chapters 5-7 and 11-14 of the textbook that was used in the introductory course:
Students who wish to take the advanced course now and the introductory course later will be given an opportunity to do so, although the normal order is obviously preferable.
The following two groups of topics will be dealt with in parallel:
Any details about such organizational matters that is not included on this web page will probably be found in the electronic versions of the slides. You can view the overall Table of Contents to find the needed information quickly.
This quiz was given in Class 3 of the 1999 version of this course. The students had read Chapter 5, as we have this year; but instead of reading the sections on CSCW and Social Issues from Chapter 14, they had read a chapter on the World Wide Web. That's why there's a reference to hypertext but not to CSCW or social issues.
Here is the text of the quiz:
In Class 2 [last year] we discussed some guidelines concerning command names and related issues. The following questions relate this discussion to two other topics in the reading for today.
Information on such matters will be given in class or sent to course participants by email when it is required. Should questions or problems arise, the instructor can be contacted by email or by phone.
This is probably the best-known set of guidelines in the HCI literature. It is unusually exhaustive, and it includes references to the research on which the guidelines are based. You will want to consult it while reading about guidelines in Chapter 5 and working on the first step of the design project.
This is an amusing, instructive, and extensive collection of examples of poor interface design.
(Note: This assignment is unusually short, so that the new students will be able to catch up. The three students who began this week should see if they can finish this assignment in addition to the one for Class 2; but if it turns out to be very difficult to do so, you can submit it a few days later.)
On Slides 49-50, we discussed a few questions that might appear on a survey of potential users of an email client. These questions of course don't cover all important issues. Your job now is to write two questions that would be appropriate for getting information about the following additional matters:
1. How much email correspondence will the respondent engage in if they start using S?
2. In what situations will the respondent use email: only for private use, for professional use, or both? At the office, at home, on the road, or some combination of these? (You may or may not choose to write two questions to get the needed information.)
Here are some tips for formulating the questions:
Please submit your two or three questions at the beginning of class in electronic form (e.g., as an ASCII or Word file), so that we can compare and discuss the various formulations.
For Class 4 we will make the basic decisions required for the conducting of a contextual task analysis: decisions concerning the choice of the user and the circumstances of the interview. The more detailed preparation of the conduct of the interview itself will be done for Class 5.
A second goal of this assignment is to gain some experience with the (active or passive) use of the textual IBIS notation that we discussed in Class 2 (Slides 57-62; cf. also Slide 78 from Class 3). Since these decisions aren't very complex, this procedure will be a good way of getting to know the notation (if you aren't one of the participants who know it already from a previous course).
The following properties of your interview are more or less inevitable, so you don't need to decide about them:
The goal of your interview is to find out how typical potential users of S handle the task of processing email and what contextual factors (cf. Slide 56) influence their performance of this task. The goal is not to discover specific design flaws in the current version of S.
The overall task of U will be to process email in his or her usual way, in an environment that is typical of the ones in which he processes email.
A Word file has been prepared that contains an incomplete IBIS structure with the questions that you should address for this assignment.
Please download this file, which is called hcida4.rtf, and load it into Word. Choose outline mode in order to be able to view it effectively. (The exact way of switching to outline mode depends on your version of Word; use the help package if necessary to find out how to enter and use outline mode.)
It's up to you to decide whether you want to write your answers directly in the IBIS notation: If you think it would take too long to get used to this notation at this point, simply use this file to read the questions and get an idea of the possible answers, and write your answers in any format you like.
If you do use the template for answering the questions, try to write at least one argument for or against each possible answer listed in the template; and at least one possible answer for each question asked. If you don't use the template, answer the questions with about the same degree of detail that you would use if you did use the template.
The first two questions concern the properties that your chosen U should have. No answer to these questions is clearly best, and in fact it's best if different students choose different answers. But still, each possible answer has some advantages and disadvantages with regard to the goals of the interview.
The other three questions involve mainly common sense. But they are typical of the type of question that one has to deal with in cases like this.
Please bring your answers on a floppy disk (e.g., in a Word or HTML file) so that we can project them onto the screen and discuss them during Class 4.
The general instructions are given on Slide 130.
The further materials mentioned there are contained in the following Word RTF file: hcida5.rtf.
To complement the feedback given in Class 5, a separate email with suggestions is being sent to each course participant.
It was suggested in Class 5 that we should agree on some minimal information that should be obtained from all interviewees, so that we can more easily compare our results. Since this suggestion is reasonable, please make sure to find out the following things from your user (using any questions that seem appropriate to elicit the information):
You should of course feel free to ask any other questions that you have prepared as well.
As soon as possible after the interview, go through the notes and fill in the gaps, while you still have a clear recollection of what happened: If you were unable to write quickly enough to make notes on particular matters, write in the missing material now. If some notes would be hard for other people to understand, clarify them.
If you take notes on a laptop, we will be viewing them together in Class 6. So edit them enough to ensure that we can find our way around them and understand them without too much difficulty. Since you will be in class to help us, the notes don't have to be self-explanatory.
If you take notes on paper during the interview, please make sure that we can somehow look at them in class, either by
If you choose this second option, please come to Class 6 about 15 minutes early, so that we can make the overhead transparencies; you needn't bring the empty transparencies yourself.
Your notes should contain all of the specific results:
It is not necessary before Class 6 to perform systematic analyses of your results, but if you want to do so, that's fine.
It is necessary to add some general comments at the end, addressing the following questions:
As we discussed in connection with Slide 56, make a comment on each of the following aspects of the context in which U used email during the interview/observation:
Note: The revised version of this report was graded together with the report for Class 9.
In Class 6, we put together several types of analysis. The results are available here as Word RTF files that can be viewed in outline mode.
Although I've cleaned up the analyses a bit, they are still far from complete. So the main point of the first two analyses is to give you an idea of what an affinity diagram and a task scenario, respectively, are like and why they are useful. The third type of analysis, knowledge-based analysis, is the subject of this week's assignment.
The main question addressed in this step is:
Section 7.4 gives several examples of how to present an overview of the objects that are involved in a task. It's best here to use the format used in the first two examples on p. 270 (i.e., a hierarchy with the operators AND, XOR, and OR but without the special characters "/", "__", etc. (These characters don't add new information, and they make formating much more difficult.) This format was also used in Slide 177 and in the analysis we started with in Class 6. It's probably easiest to format the hierarchy using outline mode of Word.
As is done in these examples, consider the objects that are more or less inherent in the task itself and that are not specific to a particular computer system. These are the concepts that a user might use when answering the question "What did you do during the last 15 minutes?". For example, a user might say:
"I first deleted the junk mail. Then I sorted the rest of the messages into folders containing messages from my friends, my classmates, and other people, respectively. Then I added a new address to my address book and wrote 3 new messages, storing them so that I could send them later."
A user would presumably not say:
"I clicked on the button near the upper-right corner of the screen. Then a new window called "Schreiben" popped up, and I moved the cursor into that window ..."
Remember, the immediate goal here is to understand better what concepts people use in thinking about email. Ultimately, the goal is to design email systems so that they take into account the concepts that users find natural, which may differ from the concepts that the designers invent.
In particular, look out especially for concepts that (a) appeared in your interview and (b) are not represented in the email system that your U was using (or perhaps in any existing email system).
To think of relevant objects, consider everything that U had to do when performing the tasks that you observed and what objects U had to be dealt with. Also look at your notes to see what U said.
When you're describing specific objects at the lowest level of the hierarchy, you may find that there are too many to list (e.g., each email folder that U maintains is an object). In that case, just list a couple of examples and use ellipses ("...") to indicate that there are many more (e.g., "Friends folder" "Classmates folder, ...").
Section 7.4 contains a number of useful hints as to how to go about making the hierarchy of concepts. You can decide for yourself whether it makes sense to apply the "uniqueness rule".
This part is very similar to the previous part, but only the actions that U can perform are listed. Once again, interface-specific actions like "close the reading window" should be left out.
The two analyses together will probably take up about 1 printed page - in addition to whatever material you choose to adopt from the initial analyses developed in Class 6 - if you use outline mode with a normal-sized font. If your additions are together less than 1 page long, try to think of some more objects and actions!
Formulate three qualitative usability goals for a future version of C's email client. Take the examples on Slides 190-192 as a starting point, but make sure that your goals are different in the following respects:
As on the slides, for each usability goal give both a statement of the goal and a justification. Note that the justification can be an important means of persuading others (e.g., management) to commit themselves to this usability goal.
As a third element for each goal (not shown on the slides), state what you think is the priority of the goal. Choose from the following three priority levels:
It may be hard to come up with a firm decision on the priority without specific knowledge of business goals, budgets, etc.; but make your best guess.
If you come up with original ideas for goals, the goals may not be of especially high priority for a broad class of users; but that's not a problem.
Formulate three quantitative usability goals like the ones illustrated in Table 5.2 (p. 200 of the book) and Slide 195. Note that Table 5.3 and Slide 194 give some additional ideas on how you can define precise usability goals and later assess their achievement.
Each goal will correspond to one line in the table in Slide 195, with the following differences:
Weighting: The original plan was to weight the report for Class 7 with 4 points and to give a new, 5-point assignment for Class 9. Since all of the reports for Class 7 called for considerable improvement, the originally submitted versions were not graded. Instead, a total of 8 points will be assigned to the report for Class 9, which will include a revision of the original analysis.
Time allocation: Probably at least 3 hours will be required for this assignment. Although it may seem possible to get through it quickly, this strategy is not desirable, as is noted below.
Take as a starting point the version submitted for Class 7. Revise it, taking into account the hints presented in Class 8 (Slides 207-209). The main goal is to ensure that the analysis is formally correct and well structured. So you don't necessarily have to add any new material. On the other hand, while working on Part 2 of the assignment, you may think of new ideas to add to this analysis; in that case, feel free to come back and add them.
For concreteness, first think of a particular email system that you will discuss in terms of our analysis. If you feel sufficiently familiar with C's email client S, you should probably choose this system. But if you prefer, you can also choose an email system that you are more familiar with. In either case, in the rest of this assignment the symbol S will refer to whichever system you chose to evaluate. For clarity, write in your report: "I chose as S the following email system: ...".
First, find three conceptual distinctions in your analysis that are taken into account adequately in the design of your S. For each one, explain how it is taken into account. Here's a simple example (which you should of course not use yourself):
"My hierarchy of objects includes a distinction between messages that have been read and messages that have not yet been read. S takes this distinction into account by printing the subject line of unread messages in green, while the subject lines of read messages are printed in black. Moreover, messages that have been read are automatically copied to the "organizer", whereas this does not happen with unread messages."
Find three conceptual distinctions in your analysis that are not taken into account well in the design of S. Name each such distinction, and answer the following questions:
Here's an example answer:
"My user distinguished between messages that were sent specifically to his email address and those that reached him via a mailing list. The email system S does not support this distinction: In order to see if a message was sent specifically to his address, U has to read the message itself.
S could take this distinction into account in one or more of the following ways:
This command would be helpful to the U whom I interviewed, since he deals very differently with these two types of messages. He likes to process the specifically addressed messages first, coming back later to deal with the others. I expect that other users could benefit as well, since this distinction has similar implications for a lot of users. On the other hand, users who receive only a few messages each week will have less need to prioritize messages in this way. For them, the additional icons or commands would just be a distraction."
1. How can a design "take into account" a conceptual distinction? There are various ways, so you'll have to use your own common sense and experience. Here are some examples:
For distinctions among types of objects, S can provide:
For distinctions among tasks, S can provide:
2. It would be easy to complete Part 2b of the assignment by simply discussing concepts in your analysis that correspond to features that are missing in S, although they are present in other email systems. For example, if your analysis includes the concept of a folder and your S doesn't offer folders, you could suggest that support for folders should be added to S. But this type of feature-comparison would not be very interesting. Instead, focus as much as possible on concepts in your knowledge-based analysis that are not very well supported by any email system that you are aware of. I will be extracting the best ideas from these reports to discuss them with C, so tell them something that they don't already know!
Think of and describe briefly a relatively simple improvement that could be made in the design of C's email client S. (Even if your interview/observation involved another system, the present analysis should refer to C's S.)
Example from the alarm clock domain: "A 'backward' button should be added such that, when U holds this button down while pressing other buttons, the digits move backward rather than forward".
We will use the symbol S+ to refer to the modified version of S which results after you have made your improvement.
Here is an example of a reasonable improvement for the email client: (Please don't use this particular example, however, since C is already aware of this idea and its advantages.) In S+ there is a "Delete" button in the window for reading emails (whereas in S messages can only be deleted from the window in which all emails are listed). The task you could analyse is "Delete the message that you are now reading".
Describe in normal English a task T such that
Alarm clock example: "Task T = set the alarm for a given time of day. Initial situation: U has yet pressed any button, and the alarm time that is currently set is different from the time that U desires."
Note that the description of the task should not be so specific that it specifies particular values (e.g., a particular alarm time or folder name).
It's important to specify the initial situation - for example, with the email client, an initial situation might be: "U has just finished reading a message and has not yet closed the message-reading window."
Write a GOMS analysis of the method(s) that U can apply to perform the task T.
For the alarm clock domain, an example analysis is shown on Slide 225. This analysis describes the actual alarm clock that we looked at in class.
Write a corresponding GOMS analysis that presupposes that U is using S+ rather than S.
We discussed in class how the alarm clock analysis would have to be changed to describe the improved alarm clock with a 'backward' button.
If your analysis for S+ differs in only a couple of places from your analysis for S, you don't have to repeat all of the parts that are the same. Instead, you can write just those parts of the analysis that are different for S+, as long as it remains clear how the analysis as a whole looks.
For the alarm clock example, you would only have to write for S+ the second part, for the goal "Set hour / minute", since the first part remains unchanged. The only difference in the second part is that an additional optional step is introduced at the beginning: "Press and hold button 'backwards'".
Referring to your GOMS analyses, explain why you think that S+ is better than S.
For the alarm clock example, you might write the following:
"With S+, the step in which U presses the 'backward' button is optional. If U chooses not to execute this step, the performance of the task will be exactly the same as with S. But I expect that many Us will choose to perform this step when the backwards distance to their goal is much shorter than the forwards distance (e.g., when they want to change the alarm time from 7:10 am to 7:00 am)."
If U does execute this step in such a situation, the performance of the remaining steps in the method will take less time, because there will be fewer digits to pass through before the goal is reached. In the most extreme case (e.g., moving from 12:00 pm to 11.59 am), U can save 22 button presses while changing the hour and 58 button presses while changing the minute - or a corresponding amount of time if he chooses to hold the button down and wait for the digits to change."
Look at your GOMS analysis and see if S+ seems to have any disadvantages relative to S. Explain any such disadvantage in terms of the GOMS analysis, and say how serious you think it is. If you think that there are no disadvantages, explain why you think so.
Alarm clock example: "With S+, U has one additional choice to make, because there's one additional optional step. Making such a choice takes a certain amount of time. So in the cases where U decides not to use the 'backward' button, the total time to perform the task will be slightly longer than with S.
But this disadvantage is quite small compared to the potential time savings mentioned above."
1. Pay close attention to the examples on Slides 229-236 and the hints on Slides 237-242, so as to make your GOMS analysis as correct as possible. (As was noted in class, the example on Slide 233-234 contains an error: the word "select" in the 8th line must be deleted, since what is involved is an objective condition, rather than a free selection.)
2. When choosing a possible improvement to analyze, make sure that the change from S to S+ results in some clear difference in the steps that U needs to execute to perform your task T. For example, don't choose an improvement that involves only the labeling of a button to make it more comprehensible.
3. Try to choose an improvement which is not so obvious that C is probably already well aware of it. Think first of the insights that you gained from your own interview/observation.
4. The alarm clock example illustrates a degree of complexity which is appropriate for your own analyses. That is, you don't need to write analyses which are as complex as the ones on Slides 229-234.
In this and the subsequent step you'll be carrying out an empirical evaluation much like the one described in Slides 257-298 (Class 10) - though it will be on a much smaller scale. In this step you'll work out the preparations for the evaluation; in the following (and final) step of the design project, you'll actually perform the evaluation and write up the results.
By performing this evaluation, you will gain some experience with a number of activities that are often applied in empirical evaluations.
The evaluation and its results will be written in the form that is standard for reports on psychological experiments. This form is often used in the literature on human-computer interaction, so it's useful to have a good grasp of it. Also, having a predetermined format makes it easier to write, by allowing you to concentrate on the content.
For Class 11, you will write the Introduction and Method sections of the report. (For Class 12, you'll write the Results and Discussion.) Write these first two sections as if you had already done the evaluation. That is, write in the past tense (e.g., "Each subject performed three tasks ...") even though the events described haven't actually taken place yet. This way, you will need to make only minor modifications to the first two sections when submitting the final report. (This practice of writing the first two sections before the study has been conducted is frequently applied.)
For almost everything you need to write, you can find a relevant example in the long evaluation report that we discussed in Class 10. But since that report doesn't entirely follow the standard format, the relevant parts will usually be found under different headings than the ones used here.
As a further source of examples, an excellent report from the previous edition of this course is available here as an RTF file. This study may be useful in that it shows what can be done given the relatively limited time that you have available in this course. The above-mentioned study is more indicative of what you would do in an applied setting with about 10 times as much time available. When using this example study, make sure always to think about the aspects of your study that make it different. In particular, don't copy or paraphrase any specific texts from the student report, but rather use the report on a higher level as an illustration of an appropriate structure, length, and style which realizes the instructions given below. If you notice any discrepancy between this report and the instructions below, follow the current instructions, which have been adapted in a few ways since last year. (Note: Although the second example study earned an "A", an unimaginative imitation of it will definitely not earn the same grade.)
The Introduction should address the following questions, in turn:
You will compare S, the email client of C, with a hypothetical new system S+, which is a variant of S. The two systems will perhaps differ only with respect to a few details. (We discussed two examples of S+ in Class 10.) S+ should include some sort of intended improvement over S. The goal of the present study is to determine empirically to what extent, and in what ways, S+ really is an improvement.
You shouldn't say here exactly how the two systems will be implemented; these details will come in the next section.
Although the evaluation may yield new insights on any aspect of S, there are probably one or two questions that you are especially interested in answering. For example, maybe you expect that the learning time for the new aspect of system S+ will be considerably shorter than for the corresponding aspect of S. Your choice of tasks and variables to measure will be largely determined by these key issues.
For each issue, formulate it as a question (i.e., ending with a question mark). Then state what answer you expect to find, and say why. This explanation of your expectation can refer back to the results of your previous design project steps (e.g., your interview/observation, your knowledge-based analysis, or your GOMS analysis).
The Method section comprises the following standard subsections:
Which persons served as subjects? Don't give their names, but describe them, e.g., as follows: "Two members of the ABC Club, one male and one female, aged 20 and 22, respectively, served as subjects." (Reminder: You are writing in the past tense, even though these persons obviously haven't really already served as your subjects.) If they had any special knowledge, experience, or other relevant characteristics, describe these. What reward (if any) did they receive for their participation?
If possible, try to find at least two subjects. If only one subject is available, design a relatively large number of tasks for that subject to perform (see below), so that you'll have enough data to work with. To increase efficiency, consider performing your study back-to-back with that of another course participant, so that the same subjects can serve both of you.
Describe the type of computer used by the subjects, mentioning any properties of it that might affect the outcome - e.g., screen size and resolution, what kind of online connection (if any) was available), ....
Say how you realized the two system variants S and S+ - in particular, whether you used any sort of simulation of the real system (e.g., substituting previously prepared screen shots for real system output).
Also describe briefly the video apparatus used (e.g., whether the camera was stationary; what tape format was used).
Generally speaking, the Materials section of an empirical report describes the stimulus material presented to the subjects. In the present case, this material consists mainly (or exclusively) of the tasks that the subjects had to perform and the questions that they answered. If you provided some already existing emails for U to process, these should also be described as part of the material.
First, write down a specification of the tasks in a form that you can present directly to the subjects; include this specification as an appendix to your report. Then, in this section of the report, describe each task more briefly, indicating the main idea behind it.
How many tasks should you define? This depends on how long they will take. You'll want to give each subject at least 10 or 15 minutes of actual work to do, not counting the presentation of instructions. (Otherwise, you won't have much data to go on, and the subjects will feel as if it was a waste of their time to come and participate.) So if each individual task can be completed in much less time, define a number of instances of each task.
Divide the tasks into two sets, so that each subject can peform Task Set 1 with one system and Task Set 2 with the other system (see below).
In addition to the main tasks, you will probably want to define at least a few practice tasks that the subject can use to become acquainted with the system. You should omit practice tasks only if you are mainly interested in how a new user deals with the system when first encountering it.
Th other aspect of the materials is the questions that you asked the subjects after they were finished with each system. Summarize the content briefly in the main text, and present the exact questions in an appendix.
This subsection describes the overall structure of the study. While many possible structures exist, the following one is recommended in this case:
"A within-subjects design was used:
Subject 1 first performed task set 1 with System S, then task set 2 with System S+.
Subject 2 first performed task set 1 with System S+, then task set 2 with System S."
Also describe here the objective measures of the subjects' task performance that you used (e.g., percentage of tasks completed successfully).
The Procedure section describes, basically in chronological order, what happened to the subjects from the time they started their participation until they left. It should deal with the following points:
What did you tell the subject at the beginning of the experiment? Here again, it's best to include the exact words as an appendix and to summarize the content in the main text.
Note: The following steps are actually performed twice for each subject, once for each system. Since the steps are presumably similar or identical for the two systems, describe each one just once, noting any differences between the handling of the two systems.
How did you explain the system to the subject? Describe what screens you showed the subject and what you said about each one.
How did you say that the subject should approach the tasks? For example, were they instructed to work as fast as possible, as accurately as possible, or with equal priority for speed and accuracy? If these instructions are brief, you can include them in the main text; otherwise, put them in an appendix and summarize them in the main text.
Note here that you asked the questions described in the Materials subsection about the system after the subject completed the tasks with it.
The main goals here are to report the results of your evaluation and discuss their significance. It's customary to separate these two steps. That is, the Results section just describes objectively what happened, including at most a bit of comment on each aspect of the results. The Discussion section then draws conclusions on the basis of all of the results taken together.
On the basis of the class discussion and/or email feedback concerning your Introduction and Method sections, adapt your plan for the study to take into account the suggested improvements. You will submit the revised version of these first two sections along with the new sections described below. To the extent to which your revision of the first two sections results in an improvement, the grade for the assignment for Class 11 will be adjusted accordingly.
Describe here anything that didn't go quite as expected (e.g., problems with the apparatus, failure of a subject to follow instructions).
Present the results concerning the quantitative, objective measures (e.g., percentage of task achievement). The method of presentation used in our example evaluation of library information systems (Class 10, Slides 257-298) is probably most appropriate:
In an Appendix, make one or more tables with the raw data for each dependent variable (see, e.g., Tables 9 and 10 on Slide 291). Then in the main part of the report, present a shorter table showing the means (averages) for each subject, as well as the overall means (cf. Table 3 on Slide 265).
Note: In published experimental reports, raw data like these are seldom presented. But it's always worthwhile for the investigator to examine the raw data. For example, extremely high or low values require explanation, and it may be misleading just to average them in with the other values. Moreover, since we are dealing with small samples, we will mostly be interpreting the data of individual users rather than just the means of the whole set of users.
After each such table in the main report, you may want to add a comment summarizing the results (e.g., "As can be seen in Table 3, task achievement was slightly better with S+"). But leave deeper discussion of the results until the next section.
Here again, the format in the library evaluation can be recommended:
In an Appendix, give the full details of the questionnaire results. For each question you asked and for each subject, note which category the subject checked and what comments (if any) he made.
In the main text, summarize the questionnaire results, once again without much interpretation in this section.
Describe here any other interesting information that you acquired. For example, did you notice anything in the subjects' behavior (perhaps while studying the video tape) that isn't reflected in the results reported so far? Did subjects make spontaneous comments that revealed something that wasn't reflected in their answers to the questionnaire questions?
Look back at the "Main Issues" that you formulated in the Introduction: Try to answer each question in turn (not necessarily in the order in which it was asked). When answering a question, refer back to specific results from the Results section. Try to integrate the different types of results (objective measures, questionnaire results, other observations) so as to arrive at a coherent picture. Example: "The meanings of the menu options in S were often hard to understand, as was shown by the questionnaire answers and the many cases in which an incorrect option was chosen. This problem may be an explanation for the slower task execution times with S." If possible, refer back to the results of previous design project steps to help make sense of the results.
Maybe you gained some new insights about questions that hadn't originally occurred to you and so weren't mentioned in the Introduction. Discuss these points in a similar way.
The Conclusions section is brief, perhaps comprising just one paragraph. It should contain the "bottom line" that the reader should "take home" as the main thing that was learned from the study. (See, for example, the final sentence on Slide 275/276.)
Another example of a possible conclusion is the following:
"S+ was on the whole clearly more usable than S. I therefore recommend that C consider adopting the improvement realized in S+ into a future version of its system. But a remaining general problem with S+ is the unnecessarily long learning time, which is mainly due to the partly unnatural structure of the new feature that distinguishes it from S+."