Whether a system like SPECTER will
be appreciated by its users is in part a question of understanding
and trust. It is therefore indispensable to provide an appropriate
means for the user to learn about the records and models that the
system has built up about her. Techniques such as textual, tabular,
or graphical presentation seem unlikely to be adequate in themselves:
First, the amount of data captured by SPECTER and
the complexity of its user model will make exhaustive or unguided
inspection infeasible. Second, it can seem unnatural or inappropriate
for a machine to comment to a human about the humanís (sometimes
emotional) reactions to events and to actions of others.
So on the one hand, processes of reflection and
introspection need to be investigated that allow the user to access
and adjust SPECTERís records and models. But at the
same time, factors like the manifestation of empathy, emotional
argumentation, and the systemís apparent sincerity need to be addressed.
The emerging techniques will let SPECTER present
observed emotional and behavior patterns in an affective way, taking
into account an evolving longer-term user-Specter relationship.
In turn, the user will be allowed to annotate past situations in
terms of her own emotional responses, supplementing and perhaps
correcting the automatically generated records of affective states.
Here we give a short overview of our concept for
reflection and introspection and have a closer look on a special
case how a user interacts with the system to determine automated
2. Desktop and PDA
The setting for reflection and introspection is
that the user collects data with SPECTER in her daily
life. Then SPECTER provides a interface on the handheld
which enables the user to make restricted interactions like browsing
or ratings. She can use this for example when sitting in the train
or having spare time. The full functionality of reflection and introspection
can be accessed by the user at a desktop PC, for example when she's
at home, reviewing her day and helping the system to improve its
3. Types of Reflection and Introspection
Reflection and introspection can either be started
by the user or SPECTER.
3.1 User Has the Initiative
- Navigation and retrieval in personal journal and user model
- Views: textual, timelines, cluster, ...
- Filter: location, time, importance, categories, emotions,
- Services: summary, presentation, statistics, query, ...
- Ratings of events
- ''The meeting with this person was very important.''
- Correction of false categorizations
- Criticizing automated event categorizations
- Interactive learning
- Asking for explanation of automated event categorizations
- ''I think you usually drink a cup of coffee as soon as you
enter the office, because you have done so in 34 out of 39
If the user takes the initiative, we have navigation
and retrieval in the personal journal and the user model. The user
can choose different views, filters and services. She can also manually
rate events, like the meeting this morning was very important and
she can correct false automated categorizations from the system,
which gives SPECTER the chance to improve interactive its user model.
Automated categorizations will be indicated by a special icon which
can be inspect by the user. Nevertheless the user is not forced
to do so and can decide by her own how much time she wants to spend
in training the system. SPECTER also provides an explanation for
automated categorizations of events which the user can request.
3.2 SPECTER Has the Initiative
- Asking about events SPECTER could not categorize
- Learning about membership of events
- Learning new categories, e.g., names of emotions
- Summary and explanation of learning progress
- Important events of the day
- Starting a dialog with the user to improve the user model
- Metaphor: Two people getting to know each other
- Happens during idle time
- ''How did you like that salesperson who just helped you?''
If SPECTER take the initiative he can ask the
user about events he couldnít categorize. By that the system can
learn about membership of events and also new categories, like names
of emotions. SPECTER can give a summary and explanation of its learning
progress, such as presenting important events of the day and interpretations
of them. During idle time, SPECTER can also start a dialogue with
the user to improve the user model. For all this services it is
important that the frequency of such question are limited or can
even turn off. The system should also figure out which are the important
questions to achieve the best improvement for the user model.
4. Binding Services to the Personal Journal
SPECTER provides several services to the user,
like using external devices, sending emails or checking bank balances.
The user can start this services manually. But it would be desirable
if SPECTER could recognize which services the user start recurrently
and trigger them by itself. But then the question arises what is
the appropriate situation for such a trigger? For that SPECTER can
have a look on the personal journal and user model and use domain
knowledge like ontolgies. He proposes such a trigger to the user
who can criticize it and help SPECTER to find a better trigger or
she can modify the trigger according to her ideas.
The figure below shows in detail how a trigger
can be determined.
4.1 Decision Trees
To determine an automated trigger we have to make
predictions in which situation the service will occur. The algorithm
we use for that is building decision trees. In the Figure below
you can see two decision trees that the user might deal with in
connection with the EC-card purchase service. Each node of a tree
is labeled with an attribute, and each edge specifies a possible
value (or range of values) for the attribute. Each leaf of the tree
is labeled as positive or negative, indicating the decision that
results if a path is traversed through the tree that leads to this
leaf. In the case of service triggering, a positive result means
that SPECTER should establish the goal of invoking the service.
(Whether or not the service is actually invoked can depend on other
factors, such as the existence of competing goals.)
The decision tree on the left means that an EC-card
purchase will be likely occur, if the store is edeka and the price
is greater than 117,5 Euros. A decision tree depends on the attributes
which are involved with it. If the user says that the specific store
shouldn't be important for this choice, we get another decision
tree which depends on price and timeOfDay.
4.2 Critiquing of Decision Trees
If the user has determined which attributes are
relevant for triggering the service, she can modify it according
to her own ideas. This includes eliminating irrelevant attributes,
selecting paths from the tree, modifying split decisions, and adding
In the example above she has chosen the first
decision tree and has added the attribute timeOfDay. Such
a decision tree can be stored and also be checked how accurate its
predicted results are. Furthermore, the system can point out bad
rules which the user should modify.
5. Trigger Editor
The trigger editor is an application for building
such triggers. Itís implemented as a viewer instance which runs
in the SPECTER browser. SPECTER presents the user a list of attributes
which are related to the service. The task of the user is to uncheck
the irrelevant attributes. She can also add related attributes according
to the ontology. After that the user presses the Compute-button
and SPECTER computes a trigger and indicates its attributes by red
color. Now a trigger is already determined and can be accepted if
the user donít care about details. For a closer look, the user presses
the Advanced-button and gets an explicit semi natural-language
representation of the computed trigger. She can modify the result
by changing numeric values and adding new conditions. The user can
also inspect directly the underlying decision tree.
Components and processes related to reflection and introspection are embedded in SPECTER's introspection environment, which enables the user to perform the introspection as described before. Its main components are a virtual character, a presentation screen and the journal browser (see the screenshot, from the left to the right). The virtual character guides the user through the introspection. It offers services for the exploitation of recorded memories such as the retrieval of additional information about mentioned objects and the adjustment of service triggers. Here the character serves as the systemís "voice" if the system has to acquire proactively information from the user.