Project Lines

Home Home
Project Members Project Members
Project Lines Project Lines
  Multitasking mit Xybernaut Personal Journal
  Multitasking mit Xybernaut Reflection and Introspection
Publications Publications
Flyer Flyer
provided by

Architecture and Infrastructure


1. Architectural Design

The high-level architecture shown in Figure 1 presents the basic functional blocks of the SPECTER system and their fundamental interrelations. This conceptual base architecture and its modularization into main components directly reflects the underlying organization into major research topics.

Conceptual Architecture

Figure 1: Conceptual Base Architecture

The necessary interconnection of the various components requires an ontological infrastructure for knowledge exchange inside SPECTER. The two central aspects of our basic approach are the use of shared ontologies and to utilize emerging standards for the notation of data and metadata, like XML, XML Schema, RDF/S, OWL, XQuery, etc. Shared ontologies provide common vocabulary to enable declarative representations for data exchange between system components. The knowledge engineering task can draw on general results in the following areas:

The architecture sketch in Figure 2 provides a more detailed view on the different software components of the overall SPECTER system. This diagram as well highlights the important role of user model and personal journal as central and persistent knowledge sources. The modular structure shown here indicates that instrumentation within a mobile context does not only relate to the local environment, but includes also mobile equipment carried around by the user, like a handheld computer and specific sensors to monitor his or her affective state.

Component Architecture

Figure 2: Refined Component Architecture

SPECTER needs to be able to interoperate with different kinds of instrumented environments. A dedicated interface component encapsulates the technical details of gaining access to the specific functionality provided by the current environment.

2. Instrumented Environment

A major challenge for linking into an instrumented environment is the heterogenity of available infrastructure frameworks. Current approaches use different technologies like Jini, UPnP, Bluetooth, Web Services, or custum research prototypes and none of them directly supports semantic interoperability. The approach followed within SPECTER can be characterized as follows:

  • The specific instrumentation is conceptualized as a set of services which can be regarded as virtual sensors and effectors.
  • A separate interface component provides a generic bridge to the local infrastructure and maintains a semantic model of the environment.
  • The representation of services relies on OWL-S and related specifications.
  • Individual Jini services as well as standardized UPnP and Bluetooth profiles can be recast as Semantic Web Services to provide the necessary grounding.

The initial test environment for SPECTER mimics an instrumented shop with RFID equipped products and readers within the shelf and basket (see Figure 3). In a mobile context, such as a shopping scenario, all direct interactions between user and SPECTER itself are performed via a PDA.The laptop on the right-hand side provides the software infrastructure of the instrumented shop, enables communication between shop and mobile assistant, and provides SPECTER with input. Using radio-frequency identification tags, the system can detect what product the user has taken from the shelf or put into the basket.

Instrumented Shop

Figure 3: Experimental Setup for an Instrumented Shop

The initial version of our experimental environment for the shopping scenario relies on the SmartShop software provided by our cooperation partners at Universität des Saarlandes. It provides a Jini-based software infrastructure for instrumented environments and includes basic service implementations forRFID-sensors, object identification and product information. Information access is realized through virtual shared memory using the tuple space paradigm. Future versions of the infrastructure software are planned to incorporate a growing list of more advanced shopping services (e.g. for cross-selling). Figure 3 presents an overview of the different runtime components on the infrastructure level, including standard Jini services as well as environment-specific additions.

Shop Software

Figure 4: Runtime Components of the Infrastructure Software

3. Sample Interaction

This section presents a typical Interaction with the initial prototype of the fully integrated system. This early demonstrator has also been documented in a short video, which is available for download in AVI format (approx. 150 MBytes).

Pick Item

Figure 5: User picks item from shelf

Using RFID tags the system is able to detect what product the user has taken from the shelf. SPECTER stores all observed interactions in an extended episodic memory, the personal journal.

Alternative Recommendation

Figure 6: SPECTER retrieves related offers and recommends alternative product

Depending on what the user chose, SPECTER can provide assistance. An acoustic signal is used to draw the user's attention to new information on the handheld display. In the current context, SPECTER automatically queried the shopping service for product information. In this case, SPECTER recommends an alternative product of the same category that provides more features and can also be found in this store. All interactions between user and SPECTER itself will also be added to the journal. For example, an immediate confirmation provides valuable information as SPECTER is also able to learn from user feedback to further improve its decision making.

Take other Item

Figure 7: User puts item back and takes other product

The user follows the recommendation and puts the item back. This action can also be recognized.

Put Item into Basket

Figure 8: User puts item into basket

The record of the user's actions and affective states in the personal journal can be mined to learn a model of the user's preferences, habits, and typical affective reactions. SPECTER exploits the user model and the personal journal to provide appropriate recommendations and assistance in the user's current situation. Even while walking around, the user can have immediate access to his updated personal journal for further reflection and introspection.

Personal Journal

Figure 9: Interactions can be Viewed within the Personal Journal

Also on the PC at home, the user can explore and modify the personal journal as well as the user model in order to assist Specter in its learning process. Journal entries can be annotated to provide explicit ratings for the corresponding low-level events or high-level interpretations. Such subjective evaluations offer an added value for affect-aware personal assistance. All modifications provided by the user during the reflection and introspection phase will be exploited for the automatic update of the persistent user model.