|Kognitive Ergonomie: Vertiefung|
Universität des Saarlandes
|Vorlesung im Sommersemester 2000|
Anthony Jameson (firstname.lastname@example.org); Letzte Änderung: 11.7.00
Die Folien aus Vorlesungen 9 bis 11 sowie die Übungsaufgabe für Vorlesung 12 stehen
Während Vorlesung 11 wurden Themen behandelt, zu denen die Folien bereits
frühere verteilt worden waren. Deshalb umfassen die Folien für Vorlesung 11
vor allem die Lektüre-Empfehlung.
Inhalt dieser Seite
Electronische Versionen der Folien
If you install this file in the same subdirectory as the following files, you can use it to access individual slides directly.
Anweisungen zu den Elektronischen Versionen der Folien
Die Folien aus Vorlesungen 9 bis 11 sowie die Übungsaufgabe für Vorlesung 12 stehen zur Verfügung.
Während Vorlesung 11 wurden Themen behandelt, zu denen die Folien bereits frühere verteilt worden waren. Deshalb umfassen die Folien für Vorlesung 11 vor allem die Lektüre-Empfehlung.
(Vgl. den Hinweis zur Sprachverwendung unten.)
Paper copies of the slides for each class will be distributed at the beginning of the class, so that notes can be made on them during class.
In addition, each week's slides will be available from this web site in PDF (Portable Document Format) sometime during the day after the class (so that last-minute changes can be included).
Documents in this format can be viewed and printed conveniently on almost any platform with the free Adobe Acrobat Reader (see the notes below).
You may find it useful to view the course materials with the Reader even though you will have paper copies: Among other things, the electronic versions include a hierarchy of bookmarks that allow you to navigate among the slides. The Reader also allows you to zoom in and out, to search for particular words in the text, and to copy parts of the text into your own notes.
Hints on how best to use the electronic versions of the slides are available on a separate page.
Obwohl diese Vorlesung im Deutschen gehalten wird, ist ein Großteil des schriftlichen Materials auf Englisch verfaßt. Das gilt auch für Teile dieser WWW-Seite. Grund: Der Dozent hält zu diesem Thema auch Tutorien auf internationalen Tagungen sowie eine Vorlesung auf einer international ausgerichteten Universität. Eine Hin- und Herübersetzung schriftlichen Materials zwischen Deutsch und Englisch wäre weniger zweckmäßig als die stattdessen vorgenommene ständige inhaltliche Aktualisierung.
Es wurden während der zweiten Vorlesung zwei Übungstreffen vereinbart:
Man kann jeweils das Treffen besuchen, das terminlich am besten paßt. Dauer: ca. 1 Stunde.
In der Handbücherei der Informatik-Bibliothek steht ein Präsenz-Exemplar zur Verfügung, das während der Öffnungszeiten der Bibliothek gelesen werden kann.
Hier sind mehrere Exemplare zu DM 88 erhältlich.
(Die folgenden Hinweise können durch Änderungen in der WWW-Seite von Amazon ihre Aktualität verlieren.)
Dix, A., Finlay, J., Abowd, G., & Beale, R. (1998). Human-computer interaction (2nd edition). New York: Prentice-Hall. (638 pages) ISBN: 0-13-239864-8.
Achtung: Auch die erste Ausgabe wird noch verkauft, und dabei wird teilweise 1997 als Jahr angegeben. In Wirklichkeit stammt diese Ausgabe aus 1992; vom Kauf wird also abgeraten.
Wer die Einführungsvorlesung im WS1999/2000 nicht besucht hat, kann sich mithilfe von deren WWW-Seite orientieren. (Der Besuch der Einführung ist für die Teilnahme an der Vertiefungsvorlesung keine Voraussetzung.)
Ausführliche Informationen über das Buch stehen über eine eigene WWW-Seite zur Verfügung.
Notes: This assignment is relatively short, because of the two holidays that intervened between Classes 2 and 3.
Reports will usually be written in German, but English is also possible (e.g., for non-German students).
Each report should be submitted in platform-independent electronic form (PDF, PostScript, or HTML; no Microsoft-specific formats please), so that all participants can look at it together during the meetings. You can send it in advance by email (email@example.com) or bring it to the meeting on a diskette.
You can assume that, when looking at your report, we will be able to access the WWW to look directly at S. So you don't have to spend time including screen shots or details that are easily found in the system itself, though URLs will be helpful.
The first task now is to choose a system S that you will analyze during this course.
As we discussed in Class 2, at least most participants will be analyzing a certain type of system: web-based systems that interactively present information about products offered for sale. This type of system is of great current practical importance, but there is relatively little scientific literature available on the usability problems raised by such systems. So it seems especially valuable to acquire some first-hand experience by analyzing these usability problems ourselves.
If some participant is eager to analyze some other type of system, they should adapt the instructions given here accordingly.
On the basis of your previous experience - and/or by surfing now in the world-wide web - select a system (S) that you will analyze throughout this course.
What conditions should your choice of S fulfill?
First, you should be able to write meaningful answers to the questions listed in Part 2 of this assignment. If you have to write "Not applicable" in one or more cases, then your choice of S is probably not appropriate.
Second, you should find S to be interesting and motivating: You think that it's at least pretty good and that it fulfills a potentially significant function. (Don't worry if you think that S is so good that there is no aspect of its usability that can be improved; this case never arises in reality.) You will be spending many hours dealing with issues related to S; so be sure that this prospect is attractive to you before you finally settle on your choice of S.
Here are just a few examples of this type of system:
Write answers to the following questions:
Give one or more URLs and the name of S.
If you must choose a system that is not available in the WWW, please contact the instructor before the meeting to arrange some way in which we can look at it under Linux or Windows98.
Note: S should do more than simply present information on static web pages (as is done, e.g., on the example screens shown in Slides 20 and 21). It should
Possible example: "U may often want to compare two products P1 and P2, but it's impossible to have information on both of them displayed on the screen at the same time."
As was discussed in the meetings, the assignment for this week is basically the same as the one for the previous week, mainly because of all of the holidays.
A new web site has been added to the list of examples of sites that might be studied: the bus and train information system of VGS Verkehrsgemeinschaft Saar. Since this system is being developed here in Saarbrücken, constructive ideas and results concerning it would be especially likely to be taken into account.
In addition to answering the questions that were listed for the previous week, please answer the following one:
Suppose you are about to conduct a requirements analysis as the first step in a complete redesign of the system S that you have chosen. Suppose you have to choose just two types of "links" to potential users (cf. Slides 38-48 from Class 2).
The first major activity in the design project is to conduct a contextual analysis/interview.
The following sections refer to the general advice from the slides of Class 3 that are specifically relevant for your preparation of this interview.
At the end of each section you will find a specific assignment, perhaps preceded by some further comments.
For each "assignment", please write down your answer.
If you can't complete all of the assignments, at least get as far as you can. You will have an opportunity to improve your plan before conducting the actual analysis/interview.
See Slide 64.
Choose a person whom you can interview for the purposes of this analysis.
Describe this person, without giving their name.
See Slide 67.
State how you will explain the interview to U, both while scheduling it and at the beginning of the interview itself.
See Slide 68.
Some of these questions will not be applicable in your case, so they must be replaced with similar questions.
Formulate at least 5 questions that are specifically appropriate for your interview.
See Slides 69 - 70.
You will be asking questions like this spontaneously, as relevant situations occur.
In order to prepare for this process, for each question listed above, write a specific example of a question that it might make sense to ask.
Example, for Question Type 1: (U has just gone through the inbox of an email system, deleting unneeded messages) "When do you delete messages that you no longer need?"
See Slides 71.
During the interview/observation, you will be trying to answer these questions yourself by observing U.
Still, at the end, you will want to ask these questions explicitly so as to uncover problems that you haven't observed.
When asking these questions at the end, you may want to give concrete examples so as to make it clear what the question means.
Note: A bottleneck is a part of a task that is especially difficult to execute efficiently. Example: Getting rid of all of the junk mail that comes in, if it makes up 80% of the incoming mail and each message has to be examined and deleted individually.
Write down two examples of errors, bottlenecks, or other problems that you can use as examples when asking U these questions - ones which will be understandable to your U.
See Slide 72.
Write a paragraph that indicates what you will say at the beginning of the interview/observation in order to establish the proper relationship and to have U start working in a way that will yield useful information.
(You won't have to read this paragraph to U verbatim, but it's important to think about what you will say - both at the beginning and at any time during the interview/observation when you think things aren't going as well as they should be)
See Slide 74.
You don't need to use video now unless you especially want to. (We'll be using video in a later phase.)
The form that your notes should take is discussed in the next section.
Regarding task flows and scenarios, it's probably best just to show U your notes on the observed actions (see below) and ask him/her to comment on the extent to which the sequence observed is typical.
Make a list summarizing which of the methods mentioned on the slide you will use to acquire information during the interview/observation.
See Slides 79-81.
The slides give examples of the type of information you should write down and how it can be organized.
You can decide exactly how you will take notes and organize them.
Prepare the computer files or sheets of paper on which you will make your notes.
Put in some categories like "Trigger" so as to provide some advance organization.
Submit an example file or sheet of paper as part of the assignment.
As was mentioned during the meetings, the assignment for this class is quite simple: Conduct the interview/observation for which had have worked out a plan for Class 5.
You probably won't have time to write up the results in any detail. Instead, bring your notes, so that we can discuss them during the meetings and think of specific suggestions for how you might write up the results.
During Class 6 (Thursday, 18 May), we will also be discussing further general ways of analyzing the results of a contextual analysis.
Formulate two qualitative usability goals for a new system that serves the same general purpose as your system S. Take the examples on Slides 128-130 as a starting point, and keep the following points in mind:
As on the slides, for each usability goal give both a statement of the goal and a justification. Note that the justification can be an important means of persuading others (e.g., management) to commit themselves to this usability goal.
As a third element for each goal (not shown on the slides), state what you think is the priority of the goal. Choose from the following three priority levels:
It may be hard to come up with a firm decision on the priority without specific knowledge of business goals, budgets, etc.; but make your best guess.
If you come up with original ideas for goals, the goals may not be of especially high priority for a broad class of users; but that's not a problem.
Formulate two quantitative usability goals like the ones illustrated in Table 5.2 (p. 200 of the book) and Slides 87-88. Note that Table 5.3 and Slide 86 give some additional ideas on how you can define precise usability goals and later assess their achievement.
Each goal will correspond to one line in the table in Slides 87-88, with the following differences:
The main question addressed in this step is:
Some concrete examples are given below that refer to email systems; but of course you are to write about your own chosen system.
This assignment is more time-consuming than the previous ones, since there was no class on the June 1st holiday or on June 2nd.
Section 7.4 of the textbook gives several examples of how to present an overview of the objects that are involved in a task. It's best here to use the format used in the first two examples on p. 270 (i.e., a hierarchy with the operators AND, XOR, and OR but without the special characters "/", "__", etc. (These characters don't add new information, and they make formating much more difficult.) This format was also used in Slides 163 and 164 and in the analysis we started with in Class 7. It's probably easiest to format the hierarchy using the outline mode of your text-processing system. If you're not familiar with any such mode, you can improvise with tabs, etc.
As is done in these examples, consider the objects that are more or less inherent in the task itself and that are not specific to a particular computer system. These are the concepts that a user might use when answering the question "What did you do during the last 15 minutes?". For example, a email user might say:
"I first deleted the junk mail. Then I sorted the rest of the messages into folders containing messages from my friends, my classmates, and other people, respectively. Then I added a new address to my address book and wrote 3 new messages, storing them so that I could send them later."
A user would presumably not say:
"I clicked on the button near the upper-right corner of the screen. Then a new window called "Schreiben" popped up, and I moved the cursor into that window ..."
Remember, the immediate goal here is to understand better what concepts people use when performing the type of task they can perform with S. Ultimately, the goal is to design a system like S that it takes into account the concepts that users find natural, which may differ from the concepts that the designers invent.
In particular, look out especially for concepts that (a) appeared in your interview and (b) are not represented in the particular system that your U was using (or perhaps in any existing system).
To think of relevant objects, consider everything that U had to do when performing the tasks that you observed and what objects U had to be dealt with. Also look at your notes to see what U said.
When you're describing specific objects at the lowest level of the hierarchy, you may find that there are too many to list (e.g., each email folder that U maintains is an object). In that case, just list a couple of examples and use ellipses ("...") to indicate that there are many more (e.g., "Friends folder" "Classmates folder, ...").
Section 7.4 contains a number of useful hints as to how to go about making the hierarchy of concepts. You can decide for yourself whether it makes sense to apply the "uniqueness rule".
This part is very similar to the previous part, but only the actions that U can perform are listed. Once again, interface-specific actions like "close the reading window" should be left out.
The two analyses together will probably take up about 1 printed page, if you use outline mode with a normal-sized font. If your additions are together less than 1 page long, try to think of some more objects and actions!
First, find three conceptual distinctions in your analysis that are taken into account adequately in the design of your S. For each one, explain how it is taken into account. Here's a simple example:
"My hierarchy of objects includes a distinction between messages that have been read and messages that have not yet been read. S takes this distinction into account by printing the subject line of unread messages in green, while the subject lines of read messages are printed in black. Moreover, messages that have been read are automatically copied to the "organizer", whereas this does not happen with unread messages."
Find three conceptual distinctions in your analysis that are not taken into account well in the design of S. Name each such distinction, and answer the following questions:
Here's an example answer:
"My user distinguished between messages that were sent specifically to his email address and those that reached him via a mailing list. The email system S does not support this distinction: In order to see if a message was sent specifically to his address, U has to read the message itself.
S could take this distinction into account in one or more of the following ways:
This command would be helpful to the U whom I interviewed, since he deals very differently with these two types of messages. He likes to process the specifically addressed messages first, coming back later to deal with the others. I expect that other users could benefit as well, since this distinction has similar implications for a lot of users. On the other hand, users who receive only a few messages each week will have less need to prioritize messages in this way. For them, the additional icons or commands would just be a distraction."
1. How can a design "take into account" a conceptual distinction? There are various ways, so you'll have to use your own common sense and experience. Here are some examples:
For distinctions among types of objects, S can provide:
For distinctions among tasks, S can provide:
2. It would be easy to complete Part 2b of the assignment by simply discussing concepts in your analysis that correspond to features that are missing in S, although they are present in other systems like S. For example, if your analysis includes the concept of a folder and your S doesn't offer folders, you could suggest that support for folders should be added to S. But this type of feature-comparison would not be very interesting. Instead, focus as much as possible on concepts in your knowledge-based analysis that are not very well supported by any system that you are aware of.
Note: This assignment is being kept short, because of the holiday and because of the rescheduling of Class 8.
In the present phase of the course, we are looking at methods for analyzing in depth particular aspects of a design. In Class 8 (rescheduled to June 15th), we will discuss the GOMS model. We will use to this notation to describe with considerable precision the way in which a user U performs a particular task.
Since you probably don't know anything about GOMS yet, your job now is simply to (a) choose an aspect of your S that you would like to analyze in detail and (b) propose an improvement to S that you think will enhance its usability significantly.
If you would like to do some reading before the rescheduled Class 8, you can read Section 6.7, which introduces the GOMS model. But it isn't necessary to read this section in order to be able to do this assignment.
The following instructions include some examples involving (a) a digital alarm clock and (b) an email system. Since the alarm clock example will be discussed in Class 8, you may not understand all details here; but the point of the example should still be clear.
Think of and describe briefly a relatively simple improvement that could be made in the design of your S.
Example from the alarm clock domain: "A 'backward' button should be added such that, when U holds this button down while pressing other buttons, the digits move backward rather than forward".
We will use the symbol S+ to refer to the modified version of S which results after you have made your improvement.
Here is an example of a reasonable improvement for an email client: In S+ there is a "Delete" button in the window for reading emails (whereas in S messages can only be deleted from the window in which all emails are listed).
When choosing a possible improvement to analyze, make sure that the change from S to S+ results in some clear difference in the observable steps that U needs to execute to perform some task. For example, don't choose an improvement that involves only the labeling of a button to make it more comprehensible.
Also, make sure that the improvement you choose is one that you find really significant - and not so obvious that it would be boring to study it in detail. You will probably be studying the same aspect of your S in the empirical study that you will perform later in the course.
Describe in normal English a task T such that
Alarm clock example: "Task T = set the alarm for a given time of day. Initial situation: U has not yet pressed any button, and the alarm time that is currently set is different from the time that U desires."
Note that the description of the task should not be so specific that it specifies particular values (e.g., a particular alarm time or folder name).
It's important to specify the initial situation. For example, with an email client, an initial situation might be: "U has just finished reading a message and has not yet closed the message-reading window."
Explain why you think that S+ is better than S.
For the alarm clock example, you might write the following: "With S+, the 'backward' button should save U a good deal of time in cases where the backwards distance to the desired time is much shorter than the forwards distance (e.g., when U wants to change the alarm time from 7:10 am to 7:00 am)."
In the next assignment, you will be analyzing the advantages and disadvantages of this improvement in more detail.
This assignment follows up on the previous one: You will write a GOMS analysis for the S and S+ that you defined, and you will discuss these system variants with reference to the GOMS analysis.
Write a GOMS analysis of the method(s) that U can apply to perform the task T.
For the alarm clock domain, an example analysis is shown on Slide 188. This analysis describes the actual alarm clock that we looked at in class.
Write a corresponding GOMS analysis that presupposes that U is using S+ rather than S.
We discussed briefly in class how the alarm clock analysis would have to be changed to describe the improved alarm clock with a 'backward' button.
If your analysis for S+ differs in only a couple of places from your analysis for S, you don't have to repeat all of the parts that are the same. Instead, you can write just those parts of the analysis that are different for S+, as long as it remains clear how the analysis as a whole looks.
For the alarm clock example, you would only have to write for S+ the second part, for the goal "Set hour / minute", since the first part remains unchanged. The only difference in the second part is that an additional optional step is introduced at the beginning: "Press and hold button 'backwards'".
Referring to your GOMS analyses, explain why you think that S+ is better than S.
Note: You already gave a similar explanation for the previous assignment. The difference now is that you must refer explicitly to specific parts of your GOMS analysis.
For the alarm clock example, you might write the following:
"With S+, the step in which U presses the 'backward' button is optional. If U chooses not to execute this step, the performance of the task will be exactly the same as with S. But I expect that many Us will choose to perform this step when the backwards distance to their goal is much shorter than the forwards distance (e.g., when they want to change the alarm time from 7:10 am to 7:00 am)."
If U does execute this step in such a situation, the performance of the remaining steps in the method will take less time, because there will be fewer digits to pass through before the goal is reached. In the most extreme case (e.g., moving from 12:00 pm to 11.59 am), U can save 22 button presses while changing the hour and 58 button presses while changing the minute - or a corresponding amount of time if he chooses to hold the button down and wait for the digits to change."
Look at your GOMS analysis and see if S+ seems to have any disadvantages relative to S. Explain any such disadvantage in terms of the GOMS analysis, and say how serious you think it is. If you think that there are no disadvantages, explain why you think so.
Alarm clock example: "With S+, U has one additional choice to make, because there's one additional optional step. Making such a choice takes a certain amount of time. So in the cases where U decides not to use the 'backward' button, the total time to perform the task will be slightly longer than with S.
But this disadvantage is quite small compared to the potential time savings mentioned above."
1. Pay close attention to the examples on Slides 193 - 198 and the hints on Slides 201 - 206, so as to make your GOMS analysis as correct as possible. (As was noted in class, the example on Slide 197 - 198 contains an error: the word "select" in the 8th line must be deleted, since what is involved is an objective condition, rather than a free selection.)
2. The alarm clock example illustrates a degree of complexity which is appropriate for your own analyses. That is, you don't need to write analyses which are as complex as the ones on Slides 193 - 198.
In this and the subsequent step you'll be carrying out an empirical evaluation of the same general sort as the one discussed in (Class 10). But your evaluation will be on a much smaller scale. In particular, instead of comparing two different systems, you will be looking only at how a user works with your system S. (For most of the systems, it would be impractical, in the limited time we have available, to realize two different variants of the system.)
To see how this work might fit into a real software development project, imagine the following situation: You are working as a usability consultant on the next version of your S. You have presented to the other team members the arguments in your previous report (with the GOMS analysis) as to why a particular aspect of your S should be improved in the next version. But the other team members are unconvinced; they say "We don't think this aspect of S is really that bad. Give us some hard, systematically collected evidence!"
In this step you'll work out the preparations for the evaluation; in the following (and final) step of the design project, you'll actually perform the evaluation and write up the results.
The evaluation and its results will be written in the form that is standard for reports on psychological experiments. This form is often used in the literature on human-computer interaction, so it's useful to have a good grasp of it. Also, having a predetermined format makes it easier to write, by allowing you to concentrate on the content.
For Class 11, you will write the Introduction and Method sections of the report. (For Class 12, you'll write the Results and Discussion.) Write these first two sections as if you had already done the evaluation. That is, write in the past tense (e.g., "Each subject performed three tasks ...") even though the events described haven't actually taken place yet. This way, you will need to make only minor modifications to the first two sections when submitting the final report. (This practice of writing the first two sections before the study has been conducted is frequently applied.)
For almost everything you need to write, you can find a relevant example in the long evaluation report that we discussed in Class 10. But since that report doesn't entirely follow the standard format, the relevant parts will usually be found under different headings than the ones used here.
The Introduction should address the following questions, in turn:
Here, you can probably just cut and paste (and edit) some text from your previous report. But if you want to focus on a different aspect of S in this study, feel free to do so.
An example answer to the second question might look as follows:
"On the basis of my GOMS analysis, I predict that this aspect of S's design will mean that U spends a large proportion of his time waiting for a system response when U is performing tasks like [description of a type of task]. The purposes of this study are"
The Method section comprises the following standard subsections:
Which person(s) served as subjects? Don't give their names, but describe them, e.g., as follows: "Two members of the ABC Club, one male and one female, aged 20 and 22, respectively, served as subjects." (Reminder: You are writing in the past tense, even though these persons obviously haven't really already served as your subjects.) If they had any special knowledge, experience, or other relevant characteristics, describe these. What reward (if any) did they receive for their participation?
If possible, try to find two subjects so that you'll have an interesting amount of data to work with. To increase efficiency, consider performing your study back-to-back with that of another course participant, so that the same subjects can serve both of you.
Describe briefly the type of computer used by the subjects, mentioning any properties of it that might affect the outcome - e.g., screen size and resolution, what kind of online connection (if any) was available), ....
Generally speaking, the Materials section of an empirical report describes the stimulus material presented to the subjects. In the present case, this material consists mainly (or exclusively) of the tasks that the subjects had to perform and the questions that they answered.
First, write down a specification of the tasks in a form that you can present directly to the subjects; include this specification as an appendix to your report. Then, in this section of the report, describe each task more briefly, indicating the main idea behind it.
How many tasks should you define? This depends on how long they will take. You'll want to give each subject at least 10 or 15 minutes of actual work to do, not counting the presentation of instructions. (Otherwise, you won't have much data to go on, and the subjects will feel as if it was a waste of their time to come and participate.) So if each individual task can be completed in much less time, define a number of instances of each task.
In addition to the main tasks, you will probably want to define at least a few practice tasks that the subject can use to become acquainted with the system. You should omit practice tasks only if
Th other aspect of the materials is the questions that you asked the subjects after they were finished with each system. Summarize the content briefly in the main text, and present the exact questions in an appendix.
Generally speaking, the "Design" section describes the structure of an experiment: what independent variables were manipulated, what dependent variables were measured, how the subjects were divided into different groups, and how different tasks (and/or task presentation orders) were assigned to different groups.
Since your study is much simpler than a typical experiment, all you need to describe here is the objective measures that you will be looking at. Here are some examples of such measures:
The Procedure section describes, basically in chronological order, what happened to the subjects from the time they started their participation until they left. It should deal with the following points:
What did you tell the subject at the beginning of the experiment? (You needn't include the exact words.)
How did you explain the system to U, if U was not already familiar with S? Summarize what screens you showed the subject and what you said about each one.
How did you say that the subject should approach the tasks? For example, were they instructed to work as fast as possible, as accurately as possible, or with equal priority for speed and accuracy?
Note here that you asked the questions described in the Materials subsection about the system after the subject completed the tasks with it.
The main goals here are to report the results of your evaluation and discuss their significance. It's customary to separate these two steps. That is, the Results section just describes objectively what happened, including at most a bit of comment on each aspect of the results. The Discussion section then draws conclusions on the basis of all of the results taken together.
Describe here anything that didn't go quite as expected (e.g., problems with the apparatus, failure of a subject to follow instructions).
Present the results concerning the quantitative, objective measures (e.g., percentage of task achievement). The method of presentation used in our example evaluation of library information systems (Class 10) is probably most appropriate.
In an Appendix, make one or more tables with the raw data for each dependent variable (see, e.g., Tables 9 and 10 on Slide 285). Then in the main part of the report, present a shorter table showing the means (averages) for each subject, as well as the overall means (cf. Table 3 on Slide 259). One difference between your tables and those of the longer evaluation study will be due to the fact that you are not comparing two different systems. Therefore, you won't need separate columns (or tables) with titles like "VT100 Interface" and "WWW Interface".
Note: In published experimental reports, raw data like these are seldom presented. But it's always worthwhile for the investigator to examine the raw data. For example, extremely high or low values require explanation, and it may be misleading just to average them in with the other values. Moreover, since we are dealing with small samples, we will mostly be interpreting the data of individual users rather than just the means of the whole set of users.
After each such table in the main report, you may want to add a comment summarizing the results (e.g., "As can be seen in Table 3, the Subject 2 spent most of her time looking at Page A"). But leave deeper discussion of the results until the next section.
Here again, the format in the library evaluation can be recommended:
In an Appendix, give the full details of the questionnaire results. For each question you asked and for each subject, note which category the subject checked and what comments (if any) he made.
In the main text, summarize the questionnaire results, once again without much interpretation in this section.
Describe here any other interesting information that you acquired. For example, did you notice anything in the subjects' behavior that isn't reflected in the results reported so far? Did subjects make spontaneous comments that revealed something that wasn't reflected in their answers to the questionnaire questions?
Look back at the "Main Issues" that you formulated in the Introduction: Try to answer each question in turn (not necessarily in the order in which it was asked). When answering a question, refer back to specific results from the Results section. Try to integrate the different types of results (objective measures, questionnaire results, other observations) so as to arrive at a coherent picture. Example: "The need to switch back and forth between Pages A and B results in a high proportion of frustrating waiting time, as is shown by the data concerning the sequence of pages visited and the waiting times, in combination with the subjects' questionnaire responses and spontaneous comments concerning system responsiveness."
If possible, refer back to the results of previous design project steps (e.g., your contextual analysis or your knowledge-based analysis) to help make sense of the results.
Maybe you gained some new insights about questions that hadn't originally occurred to you and so weren't mentioned in the Introduction. Discuss these points in a similar way.
The Conclusions section is brief, perhaps comprising just one paragraph. It should contain the "bottom line" that the reader should "take home" as the main thing that was learned from the study. (See, for example, the final sentence on Slide 271.)
Another example of a possible conclusion is the following:
"Expert users appear to be satisfied with S on the whole, but the need for them to switch back and forth between Pages A and B while performing Task T is an important source of frustration and time-wasting. If the full satisfaction of expert users is considered to have high priority, it will be worthwhile to introduce a single page or frameset in which the information from both of these two pages is available."