[Eule-Logo] Kognitive Ergonomie: Vertiefung
Fachbereich Informatik
Universität des Saarlandes
Vorlesung im Sommersemester 2000

Anthony Jameson (jameson@cs.uni-sb.de); Letzte Änderung: 11.7.00

Inhalt dieser Seite


Neuigkeiten

Die Folien aus Vorlesungen 9 bis 11 sowie die Übungsaufgabe für Vorlesung 12 stehen zur Verfügung.

Während Vorlesung 11 wurden Themen behandelt, zu denen die Folien bereits frühere verteilt worden waren. Deshalb umfassen die Folien für Vorlesung 11 vor allem die Lektüre-Empfehlung.

[To Top]
Electronische Versionen der Folien

Die PDF-Dateien

  1. Table of Contents for all of the classes so far
    If you install this file in the same subdirectory as the following files, you can use it to access individual slides directly.
  2. Class 1, 13 April 2000
  3. Class 2, 20 April 2000
  4. Class 3, 27 April 2000
  5. Class 4, 4 May 2000
  6. Class 5, 11 May 2000
  7. Class 6, 18 May 2000
  8. Class 7, 25 May 2000
  9. Class 8, 15 June 2000
  10. Class 9, 29 June 2000
  11. Class 10, 29 June 2000
  12. Class 11, 6 July 2000

[To Top]
Anweisungen zu den Elektronischen Versionen der Folien

(Vgl. den Hinweis zur Sprachverwendung unten.)

Paper copies of the slides for each class will be distributed at the beginning of the class, so that notes can be made on them during class.

In addition, each week's slides will be available from this web site in PDF (Portable Document Format) sometime during the day after the class (so that last-minute changes can be included).

Documents in this format can be viewed and printed conveniently on almost any platform with the free Adobe Acrobat Reader (see the notes below).

You may find it useful to view the course materials with the Reader even though you will have paper copies: Among other things, the electronic versions include a hierarchy of bookmarks that allow you to navigate among the slides. The Reader also allows you to zoom in and out, to search for particular words in the text, and to copy parts of the text into your own notes.

Hints on how best to use the electronic versions of the slides are available on a separate page.

[To Top]
Hinweis zur Sprachverwendung

Obwohl diese Vorlesung im Deutschen gehalten wird, ist ein Großteil des schriftlichen Materials auf Englisch verfaßt. Das gilt auch für Teile dieser WWW-Seite. Grund: Der Dozent hält zu diesem Thema auch Tutorien auf internationalen Tagungen sowie eine Vorlesung auf einer international ausgerichteten Universität. Eine Hin- und Herübersetzung schriftlichen Materials zwischen Deutsch und Englisch wäre weniger zweckmäßig als die stattdessen vorgenommene ständige inhaltliche Aktualisierung.

[To Top]
Übungstreffen

Es wurden während der zweiten Vorlesung zwei Übungstreffen vereinbart:

Man kann jeweils das Treffen besuchen, das terminlich am besten paßt. Dauer: ca. 1 Stunde.

[To Top]
Verfügbarkeit des Buches

Lesen in der Bibliothek

In der Handbücherei der Informatik-Bibliothek steht ein Präsenz-Exemplar zur Verfügung, das während der Öffnungszeiten der Bibliothek gelesen werden kann.

Anschaffung

Angaben zum Buch

Dix, A., Finlay, J., Abowd, G., & Beale, R. (1998). Human-computer interaction (2nd edition). New York: Prentice-Hall. (638 pages) ISBN: 0-13-239864-8.

Achtung: Auch die erste Ausgabe wird noch verkauft, und dabei wird teilweise 1997 als Jahr angegeben. In Wirklichkeit stammt diese Ausgabe aus 1992; vom Kauf wird also abgeraten.

[To Top]
Andere einschlägige WWW-Seiten

Wer die Einführungsvorlesung im WS1999/2000 nicht besucht hat, kann sich mithilfe von deren WWW-Seite orientieren. (Der Besuch der Einführung ist für die Teilnahme an der Vertiefungsvorlesung keine Voraussetzung.)

Ausführliche Informationen über das Buch stehen über eine eigene WWW-Seite zur Verfügung.

[To Top]
Design Project Assignment for Class 3 (27 April 2000)

Notes: This assignment is relatively short, because of the two holidays that intervened between Classes 2 and 3.

Form of Reports

Reports will usually be written in German, but English is also possible (e.g., for non-German students).

Each report should be submitted in platform-independent electronic form (PDF, PostScript, or HTML; no Microsoft-specific formats please), so that all participants can look at it together during the meetings. You can send it in advance by email (jameson@cs.uni-sb.de) or bring it to the meeting on a diskette.

You can assume that, when looking at your report, we will be able to access the WWW to look directly at S. So you don't have to spend time including screen shots or details that are easily found in the system itself, though URLs will be helpful.

Introduction

The first task now is to choose a system S that you will analyze during this course.

As we discussed in Class 2, at least most participants will be analyzing a certain type of system: web-based systems that interactively present information about products offered for sale. This type of system is of great current practical importance, but there is relatively little scientific literature available on the usability problems raised by such systems. So it seems especially valuable to acquire some first-hand experience by analyzing these usability problems ourselves.

If some participant is eager to analyze some other type of system, they should adapt the instructions given here accordingly.

Part 1: Choose a System

On the basis of your previous experience - and/or by surfing now in the world-wide web - select a system (S) that you will analyze throughout this course.

What conditions should your choice of S fulfill?

First, you should be able to write meaningful answers to the questions listed in Part 2 of this assignment. If you have to write "Not applicable" in one or more cases, then your choice of S is probably not appropriate.

Second, you should find S to be interesting and motivating: You think that it's at least pretty good and that it fulfills a potentially significant function. (Don't worry if you think that S is so good that there is no aspect of its usability that can be improved; this case never arises in reality.) You will be spending many hours dealing with issues related to S; so be sure that this prospect is attractive to you before you finally settle on your choice of S.

Here are just a few examples of this type of system:

Part 2: Provide a Brief Initial Description of S

Write answers to the following questions:


[To Top]
Design Project Assignment for Class 4 (4 May 2000)

As was discussed in the meetings, the assignment for this week is basically the same as the one for the previous week, mainly because of all of the holidays.

A new web site has been added to the list of examples of sites that might be studied: the bus and train information system of VGS Verkehrsgemeinschaft Saar. Since this system is being developed here in Saarbrücken, constructive ideas and results concerning it would be especially likely to be taken into account.

In addition to answering the questions that were listed for the previous week, please answer the following one:

Part 3: Consider Methods for Requirements Analysis

Suppose you are about to conduct a requirements analysis as the first step in a complete redesign of the system S that you have chosen. Suppose you have to choose just two types of "links" to potential users (cf. Slides 38-48 from Class 2).


[To Top]
Design Project Assignment for Class 5 (11 May 2000)

General instructions

The first major activity in the design project is to conduct a contextual analysis/interview.

The following sections refer to the general advice from the slides of Class 3 that are specifically relevant for your preparation of this interview.

At the end of each section you will find a specific assignment, perhaps preceded by some further comments.

For each "assignment", please write down your answer.

If you can't complete all of the assignments, at least get as far as you can. You will have an opportunity to improve your plan before conducting the actual analysis/interview.

1. Whom will you interview?

See Slide 64.

Assignment

Choose a person whom you can interview for the purposes of this analysis.

Describe this person, without giving their name.

2. Setting up the interview

See Slide 67.

Assignment

State how you will explain the interview to U, both while scheduling it and at the beginning of the interview itself.

3. Questions about U's background

See Slide 68.

Comments

Some of these questions will not be applicable in your case, so they must be replaced with similar questions.

Assignment

Formulate at least 5 questions that are specifically appropriate for your interview.

4. Questions about task performance

See Slides 69 - 70.

Comments

You will be asking questions like this spontaneously, as relevant situations occur.

Assignment

In order to prepare for this process, for each question listed above, write a specific example of a question that it might make sense to ask.

Example, for Question Type 1: (U has just gone through the inbox of an email system, deleting unneeded messages) "When do you delete messages that you no longer need?"

5. Questions about errors and problems

See Slides 71.

Comments

During the interview/observation, you will be trying to answer these questions yourself by observing U.

Still, at the end, you will want to ask these questions explicitly so as to uncover problems that you haven't observed.

When asking these questions at the end, you may want to give concrete examples so as to make it clear what the question means.

Note: A bottleneck is a part of a task that is especially difficult to execute efficiently. Example: Getting rid of all of the junk mail that comes in, if it makes up 80% of the incoming mail and each message has to be examined and deleted individually.

Assignment

Write down two examples of errors, bottlenecks, or other problems that you can use as examples when asking U these questions - ones which will be understandable to your U.

6. The interview itself

See Slide 72.

Assignment

Write a paragraph that indicates what you will say at the beginning of the interview/observation in order to establish the proper relationship and to have U start working in a way that will yield useful information.

(You won't have to read this paragraph to U verbatim, but it's important to think about what you will say - both at the beginning and at any time during the interview/observation when you think things aren't going as well as they should be)

7. Collecting data

See Slide 74.

Comments

You don't need to use video now unless you especially want to. (We'll be using video in a later phase.)

The form that your notes should take is discussed in the next section.

Regarding task flows and scenarios, it's probably best just to show U your notes on the observed actions (see below) and ask him/her to comment on the extent to which the sequence observed is typical.

Assignment

Make a list summarizing which of the methods mentioned on the slide you will use to acquire information during the interview/observation.

8. Taking notes

See Slides 79-81.

Comments

The slides give examples of the type of information you should write down and how it can be organized.

You can decide exactly how you will take notes and organize them.

Assignment

Prepare the computer files or sheets of paper on which you will make your notes.

Put in some categories like "Trigger" so as to provide some advance organization.

Submit an example file or sheet of paper as part of the assignment.

[To Top]
Design Project Assignment for Class 6 (18 May 2000)

As was mentioned during the meetings, the assignment for this class is quite simple: Conduct the interview/observation for which had have worked out a plan for Class 5.

You probably won't have time to write up the results in any detail. Instead, bring your notes, so that we can discuss them during the meetings and think of specific suggestions for how you might write up the results.

During Class 6 (Thursday, 18 May), we will also be discussing further general ways of analyzing the results of a contextual analysis.

[To Top]
Design Project Assignment for Class 7 (25 May 2000)

Part 1: Formulate Qualitative Usability Goals

Formulate two qualitative usability goals for a new system that serves the same general purpose as your system S. Take the examples on Slides 128-130 as a starting point, and keep the following points in mind:

As on the slides, for each usability goal give both a statement of the goal and a justification. Note that the justification can be an important means of persuading others (e.g., management) to commit themselves to this usability goal.

As a third element for each goal (not shown on the slides), state what you think is the priority of the goal. Choose from the following three priority levels:

  1. Required for release (i.e., the new system should not be released unless this goal has been achieved.)
  2. Important if not excessively expensive or time-consuming to achieve
  3. Desirable, but only if achievable at low cost

It may be hard to come up with a firm decision on the priority without specific knowledge of business goals, budgets, etc.; but make your best guess.

If you come up with original ideas for goals, the goals may not be of especially high priority for a broad class of users; but that's not a problem.

Part 2: Formulate Quantitative Usability Goals

Formulate two quantitative usability goals like the ones illustrated in Table 5.2 (p. 200 of the book) and Slides 87-88. Note that Table 5.3 and Slide 86 give some additional ideas on how you can define precise usability goals and later assess their achievement.

Each goal will correspond to one line in the table in Slides 87-88, with the following differences:


[To Top]
Design Project Assignment for Class 8 (June 8th)

Introduction

The main question addressed in this step is:

Some concrete examples are given below that refer to email systems; but of course you are to write about your own chosen system.

This assignment is more time-consuming than the previous ones, since there was no class on the June 1st holiday or on June 2nd.

Part 1. Systematically List the Relevant Objects

Section 7.4 of the textbook gives several examples of how to present an overview of the objects that are involved in a task. It's best here to use the format used in the first two examples on p. 270 (i.e., a hierarchy with the operators AND, XOR, and OR but without the special characters "/", "__", etc. (These characters don't add new information, and they make formating much more difficult.) This format was also used in Slides 163 and 164 and in the analysis we started with in Class 7. It's probably easiest to format the hierarchy using the outline mode of your text-processing system. If you're not familiar with any such mode, you can improvise with tabs, etc.

As is done in these examples, consider the objects that are more or less inherent in the task itself and that are not specific to a particular computer system. These are the concepts that a user might use when answering the question "What did you do during the last 15 minutes?". For example, a email user might say:

"I first deleted the junk mail. Then I sorted the rest of the messages into folders containing messages from my friends, my classmates, and other people, respectively. Then I added a new address to my address book and wrote 3 new messages, storing them so that I could send them later."

A user would presumably not say:

"I clicked on the button near the upper-right corner of the screen. Then a new window called "Schreiben" popped up, and I moved the cursor into that window ..."

Remember, the immediate goal here is to understand better what concepts people use when performing the type of task they can perform with S. Ultimately, the goal is to design a system like S that it takes into account the concepts that users find natural, which may differ from the concepts that the designers invent.

In particular, look out especially for concepts that (a) appeared in your interview and (b) are not represented in the particular system that your U was using (or perhaps in any existing system).

To think of relevant objects, consider everything that U had to do when performing the tasks that you observed and what objects U had to be dealt with. Also look at your notes to see what U said.

When you're describing specific objects at the lowest level of the hierarchy, you may find that there are too many to list (e.g., each email folder that U maintains is an object). In that case, just list a couple of examples and use ellipses ("...") to indicate that there are many more (e.g., "Friends folder" "Classmates folder, ...").

Section 7.4 contains a number of useful hints as to how to go about making the hierarchy of concepts. You can decide for yourself whether it makes sense to apply the "uniqueness rule".

Part 2. Systematically List the Relevant Actions

This part is very similar to the previous part, but only the actions that U can perform are listed. Once again, interface-specific actions like "close the reading window" should be left out.

The two analyses together will probably take up about 1 printed page, if you use outline mode with a normal-sized font. If your additions are together less than 1 page long, try to think of some more objects and actions!

Part 3: Discuss the Implications of Your Knowledge-Based Analysis

Part 3a: Discuss Positive Examples

First, find three conceptual distinctions in your analysis that are taken into account adequately in the design of your S. For each one, explain how it is taken into account. Here's a simple example:

"My hierarchy of objects includes a distinction between messages that have been read and messages that have not yet been read. S takes this distinction into account by printing the subject line of unread messages in green, while the subject lines of read messages are printed in black. Moreover, messages that have been read are automatically copied to the "organizer", whereas this does not happen with unread messages."

Part 3b: Discuss Negative Examples

Find three conceptual distinctions in your analysis that are not taken into account well in the design of S. Name each such distinction, and answer the following questions:

  1. Why do you say that S does not take this distinction into account?
  2. How could the design of S be modified so that it takes the distinction into account?
  3. How could the user whom you interviewed - and perhaps users more generally - benefit from this modification?

Here's an example answer:

"My user distinguished between messages that were sent specifically to his email address and those that reached him via a mailing list. The email system S does not support this distinction: In order to see if a message was sent specifically to his address, U has to read the message itself.

S could take this distinction into account in one or more of the following ways:

This command would be helpful to the U whom I interviewed, since he deals very differently with these two types of messages. He likes to process the specifically addressed messages first, coming back later to deal with the others. I expect that other users could benefit as well, since this distinction has similar implications for a lot of users. On the other hand, users who receive only a few messages each week will have less need to prioritize messages in this way. For them, the additional icons or commands would just be a distraction."

General Notes on This Assignment

1. How can a design "take into account" a conceptual distinction? There are various ways, so you'll have to use your own common sense and experience. Here are some examples:

For distinctions among types of objects, S can provide:

For distinctions among tasks, S can provide:

2. It would be easy to complete Part 2b of the assignment by simply discussing concepts in your analysis that correspond to features that are missing in S, although they are present in other systems like S. For example, if your analysis includes the concept of a folder and your S doesn't offer folders, you could suggest that support for folders should be added to S. But this type of feature-comparison would not be very interesting. Instead, focus as much as possible on concepts in your knowledge-based analysis that are not very well supported by any system that you are aware of.

[To Top]
Design Project Assignment for Class 8 (after rescheduling) (June 15th)

Note: This assignment is being kept short, because of the holiday and because of the rescheduling of Class 8.

Introduction

In the present phase of the course, we are looking at methods for analyzing in depth particular aspects of a design. In Class 8 (rescheduled to June 15th), we will discuss the GOMS model. We will use to this notation to describe with considerable precision the way in which a user U performs a particular task.

Since you probably don't know anything about GOMS yet, your job now is simply to (a) choose an aspect of your S that you would like to analyze in detail and (b) propose an improvement to S that you think will enhance its usability significantly.

If you would like to do some reading before the rescheduled Class 8, you can read Section 6.7, which introduces the GOMS model. But it isn't necessary to read this section in order to be able to do this assignment.

The following instructions include some examples involving (a) a digital alarm clock and (b) an email system. Since the alarm clock example will be discussed in Class 8, you may not understand all details here; but the point of the example should still be clear.

Part 1: Choose a Possible Improvement

Think of and describe briefly a relatively simple improvement that could be made in the design of your S.

Example from the alarm clock domain: "A 'backward' button should be added such that, when U holds this button down while pressing other buttons, the digits move backward rather than forward".

We will use the symbol S+ to refer to the modified version of S which results after you have made your improvement.

Here is an example of a reasonable improvement for an email client: In S+ there is a "Delete" button in the window for reading emails (whereas in S messages can only be deleted from the window in which all emails are listed).

When choosing a possible improvement to analyze, make sure that the change from S to S+ results in some clear difference in the observable steps that U needs to execute to perform some task. For example, don't choose an improvement that involves only the labeling of a button to make it more comprehensible.

Also, make sure that the improvement you choose is one that you find really significant - and not so obvious that it would be boring to study it in detail. You will probably be studying the same aspect of your S in the empirical study that you will perform later in the course.

Part 2: Describe a Simple Task Related to the Improvement

Describe in normal English a task T such that

Alarm clock example: "Task T = set the alarm for a given time of day. Initial situation: U has not yet pressed any button, and the alarm time that is currently set is different from the time that U desires."

Note that the description of the task should not be so specific that it specifies particular values (e.g., a particular alarm time or folder name).

It's important to specify the initial situation. For example, with an email client, an initial situation might be: "U has just finished reading a message and has not yet closed the message-reading window."

Part 3: Explain the Expected Advantage(s) of Your Improvement

Explain why you think that S+ is better than S.

For the alarm clock example, you might write the following: "With S+, the 'backward' button should save U a good deal of time in cases where the backwards distance to the desired time is much shorter than the forwards distance (e.g., when U wants to change the alarm time from 7:10 am to 7:00 am)."

In the next assignment, you will be analyzing the advantages and disadvantages of this improvement in more detail.

[To Top]
Design Project Assignment for Class 9 (June 29th)

This assignment follows up on the previous one: You will write a GOMS analysis for the S and S+ that you defined, and you will discuss these system variants with reference to the GOMS analysis.

Part 1: Write a GOMS Analysis for S

Write a GOMS analysis of the method(s) that U can apply to perform the task T.

For the alarm clock domain, an example analysis is shown on Slide 188. This analysis describes the actual alarm clock that we looked at in class.

Part 2: Write a GOMS Analysis for S+

Write a corresponding GOMS analysis that presupposes that U is using S+ rather than S.

We discussed briefly in class how the alarm clock analysis would have to be changed to describe the improved alarm clock with a 'backward' button.

If your analysis for S+ differs in only a couple of places from your analysis for S, you don't have to repeat all of the parts that are the same. Instead, you can write just those parts of the analysis that are different for S+, as long as it remains clear how the analysis as a whole looks.

For the alarm clock example, you would only have to write for S+ the second part, for the goal "Set hour / minute", since the first part remains unchanged. The only difference in the second part is that an additional optional step is introduced at the beginning: "Press and hold button 'backwards'".

Part 3: Explain the Advantage(s) of Your Improvement

Referring to your GOMS analyses, explain why you think that S+ is better than S.

Note: You already gave a similar explanation for the previous assignment. The difference now is that you must refer explicitly to specific parts of your GOMS analysis.

For the alarm clock example, you might write the following:

"With S+, the step in which U presses the 'backward' button is optional. If U chooses not to execute this step, the performance of the task will be exactly the same as with S. But I expect that many Us will choose to perform this step when the backwards distance to their goal is much shorter than the forwards distance (e.g., when they want to change the alarm time from 7:10 am to 7:00 am)."

If U does execute this step in such a situation, the performance of the remaining steps in the method will take less time, because there will be fewer digits to pass through before the goal is reached. In the most extreme case (e.g., moving from 12:00 pm to 11.59 am), U can save 22 button presses while changing the hour and 58 button presses while changing the minute - or a corresponding amount of time if he chooses to hold the button down and wait for the digits to change."

Part 4: Mention Possible Disadvantages of Your Improvement

Look at your GOMS analysis and see if S+ seems to have any disadvantages relative to S. Explain any such disadvantage in terms of the GOMS analysis, and say how serious you think it is. If you think that there are no disadvantages, explain why you think so.

Alarm clock example: "With S+, U has one additional choice to make, because there's one additional optional step. Making such a choice takes a certain amount of time. So in the cases where U decides not to use the 'backward' button, the total time to perform the task will be slightly longer than with S.

But this disadvantage is quite small compared to the potential time savings mentioned above."

General Hints

1. Pay close attention to the examples on Slides 193 - 198 and the hints on Slides 201 - 206, so as to make your GOMS analysis as correct as possible. (As was noted in class, the example on Slide 197 - 198 contains an error: the word "select" in the 8th line must be deleted, since what is involved is an objective condition, rather than a free selection.)

2. The alarm clock example illustrates a degree of complexity which is appropriate for your own analyses. That is, you don't need to write analyses which are as complex as the ones on Slides 193 - 198.

[To Top]
Design Project Assignment for Class 11 (July 6th)

Overview

In this and the subsequent step you'll be carrying out an empirical evaluation of the same general sort as the one discussed in (Class 10). But your evaluation will be on a much smaller scale. In particular, instead of comparing two different systems, you will be looking only at how a user works with your system S. (For most of the systems, it would be impractical, in the limited time we have available, to realize two different variants of the system.)

To see how this work might fit into a real software development project, imagine the following situation: You are working as a usability consultant on the next version of your S. You have presented to the other team members the arguments in your previous report (with the GOMS analysis) as to why a particular aspect of your S should be improved in the next version. But the other team members are unconvinced; they say "We don't think this aspect of S is really that bad. Give us some hard, systematically collected evidence!"

In this step you'll work out the preparations for the evaluation; in the following (and final) step of the design project, you'll actually perform the evaluation and write up the results.

The evaluation and its results will be written in the form that is standard for reports on psychological experiments. This form is often used in the literature on human-computer interaction, so it's useful to have a good grasp of it. Also, having a predetermined format makes it easier to write, by allowing you to concentrate on the content.

For Class 11, you will write the Introduction and Method sections of the report. (For Class 12, you'll write the Results and Discussion.) Write these first two sections as if you had already done the evaluation. That is, write in the past tense (e.g., "Each subject performed three tasks ...") even though the events described haven't actually taken place yet. This way, you will need to make only minor modifications to the first two sections when submitting the final report. (This practice of writing the first two sections before the study has been conducted is frequently applied.)

For almost everything you need to write, you can find a relevant example in the long evaluation report that we discussed in Class 10. But since that report doesn't entirely follow the standard format, the relevant parts will usually be found under different headings than the ones used here.

Part 1. Write the Introduction

The Introduction should address the following questions, in turn:

  1. What aspect of your S will you be evaluating?

  2. What question(s) are you trying to answer?

Here, you can probably just cut and paste (and edit) some text from your previous report. But if you want to focus on a different aspect of S in this study, feel free to do so.

An example answer to the second question might look as follows:

"On the basis of my GOMS analysis, I predict that this aspect of S's design will mean that U spends a large proportion of his time waiting for a system response when U is performing tasks like [description of a type of task]. The purposes of this study are"

Part 2. Write the Method Section

The Method section comprises the following standard subsections:


[To Top]
Design Project Assignment for Class 12 (July 13th)

Overview

The main goals here are to report the results of your evaluation and discuss their significance. It's customary to separate these two steps. That is, the Results section just describes objectively what happened, including at most a bit of comment on each aspect of the results. The Discussion section then draws conclusions on the basis of all of the results taken together.

Part 1. Write the Results Section

Comments on How the Evaluation Went

Describe here anything that didn't go quite as expected (e.g., problems with the apparatus, failure of a subject to follow instructions).

Objective Results

Present the results concerning the quantitative, objective measures (e.g., percentage of task achievement). The method of presentation used in our example evaluation of library information systems (Class 10) is probably most appropriate.

In an Appendix, make one or more tables with the raw data for each dependent variable (see, e.g., Tables 9 and 10 on Slide 285). Then in the main part of the report, present a shorter table showing the means (averages) for each subject, as well as the overall means (cf. Table 3 on Slide 259). One difference between your tables and those of the longer evaluation study will be due to the fact that you are not comparing two different systems. Therefore, you won't need separate columns (or tables) with titles like "VT100 Interface" and "WWW Interface".

Note: In published experimental reports, raw data like these are seldom presented. But it's always worthwhile for the investigator to examine the raw data. For example, extremely high or low values require explanation, and it may be misleading just to average them in with the other values. Moreover, since we are dealing with small samples, we will mostly be interpreting the data of individual users rather than just the means of the whole set of users.

After each such table in the main report, you may want to add a comment summarizing the results (e.g., "As can be seen in Table 3, the Subject 2 spent most of her time looking at Page A"). But leave deeper discussion of the results until the next section.

Questionnaire Results

Here again, the format in the library evaluation can be recommended:

In an Appendix, give the full details of the questionnaire results. For each question you asked and for each subject, note which category the subject checked and what comments (if any) he made.

In the main text, summarize the questionnaire results, once again without much interpretation in this section.

Other Observations

Describe here any other interesting information that you acquired. For example, did you notice anything in the subjects' behavior that isn't reflected in the results reported so far? Did subjects make spontaneous comments that revealed something that wasn't reflected in their answers to the questionnaire questions?

Part 2. Write the Discussion Section

Discussion of Main Issues

Look back at the "Main Issues" that you formulated in the Introduction: Try to answer each question in turn (not necessarily in the order in which it was asked). When answering a question, refer back to specific results from the Results section. Try to integrate the different types of results (objective measures, questionnaire results, other observations) so as to arrive at a coherent picture. Example: "The need to switch back and forth between Pages A and B results in a high proportion of frustrating waiting time, as is shown by the data concerning the sequence of pages visited and the waiting times, in combination with the subjects' questionnaire responses and spontaneous comments concerning system responsiveness."

If possible, refer back to the results of previous design project steps (e.g., your contextual analysis or your knowledge-based analysis) to help make sense of the results.

Discussion of Other Interesting Points

Maybe you gained some new insights about questions that hadn't originally occurred to you and so weren't mentioned in the Introduction. Discuss these points in a similar way.

Part 3. Write the Conclusions Section

The Conclusions section is brief, perhaps comprising just one paragraph. It should contain the "bottom line" that the reader should "take home" as the main thing that was learned from the study. (See, for example, the final sentence on Slide 271.)

Another example of a possible conclusion is the following:

"Expert users appear to be satisfied with S on the whole, but the need for them to switch back and forth between Pages A and B while performing Task T is an important source of frustration and time-wasting. If the full satisfaction of expert users is considered to have high priority, it will be worthwhile to introduce a single page or frameset in which the information from both of these two pages is available."

[To Top]