How to Read an IUI Research Paper
(Original Text by Bill Griswold, revised by Gail Murphy, Christina Conati, and Charles Rich)
(Modified by Daniel Sonntag, October 2013)
Reading IUI research papers effectively is challenging. These papers
are often written in a very condensed style because of page limitations
and the intended audience, which is assumed to already know the area
well. Moreover, the reasons for writing the paper may be different than
the reasons the paper has been assigned, meaning you have to work
harder to find the content that you are interested in.
The first thing to understand is that the research papers you will read
have been published in different kinds of places. Some papers have been
published in the proceedings of a conference, for example at IUI, a SIGART / SIGCHI conference submission.
These papers have been peer-reviewed but are typically strictly limited
in length to about 10-12 pages. Other papers have been published in
archival journals, such as TiiS.
These papers have also been peer-reviewed but there are typically not
any length restrictions, so they generally include more detail in every
dimension. Some papers are technical reports. These are not
peer-reviewed. In areas related to Computer Science, you may find that
there is first a technical report version which is then later published
in a conference or a journal. If a paper appears both in conference and
journal form, the journal version is typically a later, expanded and
revised version of the work.
To develop an effective reading style for research papers, it can help
to know what you should get out of the paper, and where that
information is located in the paper. Typically, the introduction will
state not only the motivations behind the work, but also outline the
solution. Often this may be all the expert requires from the paper. The
body of the paper states the authors' solution to the problem in
detail, and should also describe a detailed evaluation of the solution
in terms of arugments or an empirical evaluation (case study,
experiment, etc.). Finally, the paper will conclude with a recap,
including a discussion of the primary contributions. A paper will also
discuss related work to some degree.
The questions you want to have answered by reading a paper are the following:
What are motivations for this work? For a research paper, there is an
expectation that a solution to a problem has been found that no one
else has published in the literature. The paper should describe why the
problem is important and why it does not have a trivial solution; that
is, why a new solution may be required. Implicitly there is implication
that previous solutions to the problem are inadequate. Occasionally an
author will fail to state either point, making your job much more
difficult.
What is the proposed solution? This is also called the hypothesis or
idea. There should also be an argument about why the solution solves
the problem better than previous solutions. There should also be a
discussion about how the solution is achieved (designed and
implemented) or is at least achievable.
How is the proposed solution evaluated? An idea alone is usually not
adequate for publication of a research paper. What argument and/or
experiment is made to make a case for the value of the ideas? What
benefits or problems are identified? Are they convincing?
How does the proposed solution relate to other proposed solutions? If
this is an important problem, other researchers have probably also
tried to solve it. How do the alternative approaches compare to this
one? Has this work already been done? Don't limit yourself to the
related work references cited by the authors.
What are the contributions? The contributions in a paper may be many
and varied. Ideas, software, experimental techniques, and area survey
are a few key possibilities.
IUI for example states that: "Unlike traditional AI, our focus is not
so much to make the computer smart all by itself, but to make the
interaction between computers and people smarter. Unlike traditional
HCI, we do not require in-depth studies with users to motivate or prove
the effectiveness of the research. Evaluations need only be appropriate
to the type of contribution claimed."
What are future directions for this research? Not only what future
directions do the authors identify, but what ideas did you come up with
while reading the paper?
As you read or skim a paper, you should actively attempt to answer the
above questions. Presumably, the introduction should provide
motivation. The introduction and conclusion may discuss the solutions
and evaluation at a high level. Future work is likely in the concluding
part of the paper. The details of the solution and the evaluation
should be in the body of the paper. You may find it productive to try
to answer each question in turn, writing your answer down. In practice,
you are not done reading a paper until you can answer all the questions.