Intelligent User Interfaces
Frank Shipman
Department of Computer Science
Texas A&M University
What this is about
Designing, building, and evaluating
intelligent user interfaces
Particular technologies used to create
intelligent user interfaces
Issues concerning applicability of
intelligent user interfaces
Intelligent user interfaces (IUI)?
Systems that provide interactive support
based on embedded AI mechanisms
Interfaces to AI functionality and
knowledge representations
Adaptive systems
Cooperative problem-solving systems
Technologies I
User models,
Situation models, and
Programming by Demonstration
Adaptive Interfaces
– interface that can be adapted
– user or situation model
– adaptation strategy
Frequently used for providing
assistance or training to user
User Model
“any information which a program has
which is specific to a particular user.
The information itself could range from
a simple count of errors, to some
complicated data structure which
purports to represent relevant parts of
the user’s knowledge of the problem
Stereotyping vs. Individual
Stereotyping (canonical user modeling)
– provide interfaces for classes of users
– classes might be based on skill (novices,
midrange users, experts) or role
Individual approach
– dynamically adapt to suit each user
– can be based on observed use of system
or self assessment
Representations for User Models
Descriptive method
– modeling the user’s observed behavior
– describes what system has seen user do
Skill-based or cognitive method
– attempt to model the internal cognitive
models and processes of the user
– represents background knowledge, goals,
plans, preferences, misconceptions
Acquisition of User Model
Model based on a combination of:
– Observations of system use
• statistical history
• chronological history
– Self-assessment
– Testing
How can model evolve over time?
– Any of the above
Berkeley UNIX Consultant
Goal: Provide help to new UNIX users
Generates user model based on
“successful” use of UNIX commands
– Explanations of difficult commands can
make use of student’s knowledge.
Intelligent Tutoring Systems
Task is generally well-known
– assignments given to student by system
– systems track partial completion
Systems keep record of student’s
success and failure.
– used to determine future assignments
– used to determine how to help when
student has difficulties
Situation Models
Components of situation:
users, system, environment
– multiple user models
– hardware constraints and load
– device / resource availability
Representing the Environment
Identifying environmental influences
– anticipating use situations
– classes of use vs. detailed model of
Monitoring environment
– direct input devices
– user description
Mars Medical Assistant
Goal: Provide medical support for
astronauts on three year trip to Mars
Consider educational, consultation, and
emergency situations
Models of user and patient
– limited highly-trained user community
– no new users joining during mission
Other Adaptive Systems
Typing completion
– suggests completions for partial terms
based on prior use
Emacs suggestions
– notifies user when more efficient method
available to complete task
Computer Chess Game
– determines quality of own play based on
perceived level of opponent
Programming by Demonstration
Generalizing from demonstrated action
and situation sequences to programs
– knowing what must stay the same
– knowing what are variables and their types
– connecting to programmed application
Programming by Demonstration
Peridot -- demonstration of simple
Marquise -- demonstration of graphical
editors including palettes and modes
DEMO -- demonstrating dynamically
created objects
DEMO2 -- refinement by system based
on multiple demonstrations
Focus on programming animation
– graphical objects
– models of motion and time
Stimulus-response demonstration
– modes for creating objects and behaviors
– mode for demonstrating interaction
Technologies II
Presentation generation,
Design Environments, and
Interface agents
Presentation Generation
Generating dynamic links to information
– enabling user-controlled flow
Generating presentations based on
current situation and/or user
– Use of user or situation model
Generating rhetorical structure/transition
– Scripting events
– Media-based decisions
Presentations and Explanations
– Explainer (Redmiles)
– Explainable Expert System (Moore)
– Story Presentation System (Sgouros, …)
Domain: Graphical program explanation
for software reuse
Creates links between perspectives on
software including source code,
documentation, execution information,
application domain view
Provides user multiple points of access
to better inform about software
Explainable Expert System (EES)
Explains different outcomes in an expert
system / planner
Generates natural language to answer
user’s questions
Keeps dialog history to provide
differential descriptions
EES Architecture for Explanation
Knowledge Base
Query analyzer
User goals
Plan Operators
Text planner
Dialog history
Sentence generator
Focus Information
EES Example
User: “Describe Inderal”
System: “Inderal a drug that …”
User: “Describe Elavil”
System: “Like Inderal, Elavil is used …”
User: “Describe Cafergot”
System: “Cafergot is very different from
the drugs we have been talking about.
Story Presentation System
A dynamic dramatization method for
narrative presentations
Symbolic Plot
Dramatic Effects
Original Story
Story Presentation Analysis
Plot analysis models:
– physical and emotional state changes
– positive and negative interference among
Dramatization uses plot analysis to
determine dramatic events in story
– Lifeline, Rising complication, Reversal of
fortune, Dramatic irony, Happy end
Story Presentation Results
Presentation manager adds dramatic
effects to original story material to
emphasize dramatic events in story
Effects include
– audio: selection of noises or music
– images and video: flashbacks,
flashforwards, images of other characters
Design Environments
Provide a software environment
supporting the activities part of design.
– specification, construction
– argumentation, documentation,
– Framer (Lemke, Fischer)
– JANUS (Morch, McCall, Fischer , ...)
Knowledge-based support for interface
direct manipulation interface builder
Framer 1 -- construction kit approach
Framer 2 -- design environment
Design Environment
Components (1)
– system provides decomposition of task,
– user identifies current focus
Palette & Workspace
– system provides primitive components
– user identifies components used and
organization of components in design
Design Environment
Components (2)
Specification sheets
– system brings design issues to user’s
attention, presents potential answers, and
explains significance and consequences of
design choices
– user symbolically specifies answers to
design issues
Design Environment
Components (3)
– system points out sub-optimal design
decisions, explains why this is believed,
and provides heuristics for making
– users may accept or reject the system’s
Design Environment
Components (4)
– system provides examples
– user selects designs to reuse and modify
Code generator
– system generates an executable
representation of designed interface
Other Design Environments
JANUS -- kitchen design
– designed for non-technical users
XNetwork -- computer network design
– identified need for simulation component
VDDE -- voice dialog design
– another type of interface design with
interesting constraints
Software Agents
One view:
Software processes that have non-trivial
tasks delegated to them which require
independent action and a report on the
Issues for Software Agents (1)
– Should agents be represented as a living
or animated character?
– Does it improve adoption of software?
– Does it create inflated expectations?
– Is it just too annoying?
Issues for Software Agents (2)
Trust and Competence
– How does user develop an informed level
of trust?
– Can agent give self-assessment on likely
outcome of task?
– How can user delegate tasks?
– How can user check on status of delegated
Issues for Software Agents (3)
– How does user set limits on the agent’s
– When does the agent get to interrupt the
user (mixed-initiative dialog)?
Dealing with multiple agents
– How can the user manage many agents?
– How can interactions between agents be
Information Retrieval Agents
Watch user patterns to infer interests or
goals which are used to classify, rank,
or suggest new information
– INFOSCOPE: patterns in Netnews use
– BASAR: patterns in Web access
Issue: the “cold start” problem
– must watch a while before suggesting
Social Filtering
Finding elements liked by others (with
similar preferences)
– requires some notion of preferences
– improves with more users
– Tapestry -- rating of documents
– GroupLens -- collaboration & user profiles
– and CD-NOW
Technologies III
Knowledge manipulation and
Using recognized structure
Interacting with Knowledge
User tasks
– Adding knowledge
– Editing rule bases and object hierarchies
– HITS Knowledge Editor (Terveen)
– Modifier (Girgensohn)
– Hyper-Object Substrate (Shipman)
Knowledge Representations
– text, graphics, audio, video
– hypertext, argumentation
– frames, semantic nets, scripts, rules,
inheritance hierarchies,
HITS Knowledge Editor
Knowledge editor for CYC project
Difficulties of knowledge representation
formalization - articulation in precise detail
comprehension - complex vocabulary, size
modification - location and consistency
Terveen’s Design Principle #1
Provide a workspace in which users
and systems can jointly construct
and manipulate a context for
problem-solving, and in which the
state of the problem-solving is
represented visibly.
Terveen’s Design Principle #2
Deliver intelligent assistance through
Terveen’s Design Principle #3
Exploit the interactive potential of
computational media to manage
the user-system interaction
according to conventions that are
appropriate to the role of each
party in the interaction.
Support Provided by HKE
Inferences -- information inferred from
workspace and existing KB
Troubles -- inconsistencies between
workspace and KB
Suggestions -- relevant representational
issues for users to consider
Support for End-User Modifiability
– Users are not knowledge engineers
– Adding new object class to existing system
Support: suggestions based on
similarity of features and efficiency of
Hyper-Object Substrate
Formalization as difficulty
– cognitive load
– tacit knowledge
– premature structure
Supporting incremental formalization
– Flexible knowledge representation
– Suggestions based on informal content
– Automating knowledge acquisition?
Using Recognized Structure
When can it be used without user
acknowledging correctness?
TileBars (Hearst)
• graphically presents results of text analysis
inferring topic changes within text corpus
Data Mining
• search for patterns within database
Non-Verbal Representations
Non-verbal communication allows
expression of emergent thoughts
Systems need to recognize implicit
– VIKI (Marshall, Shipman, …)
– PerSketch (Saund, Moran)
– Tivoli (Moran, van Melle, Chui, …)
VIKI: An Analyst’s Workspace
Task: collect, organize, and interpret
– evolving understanding
– ephemeral structures
– emerging visual languages
Goal: Recognize visual structures and
PerSketch: Adaptive Drawing Tool
Recognize multiple potential
connections between strokes in drawing
Enables cleaning up drawings, editing
drawn objects
TIVOLI: Meeting Room Support
Manipulating structures on an electronic
Recognize gestures, characters, etc.
Recognize lists, groupings
Map changes in visual structure to
underlying database
Design Considerations
Issues of applicability,
Evaluating interfaces, and
IUI: Appearing Too Smart?
A central danger of creating IUI is, like
ELIZA, appearing too smart
– Creating unrealistic expectations
– Communication difficulties
• false alarm
• garden path
Design guideline: do not hide system’s
knowledge or lack thereof
Evaluation Methods
Cognitive Walkthroughs
Videotapes of use
Usage data from software monitoring
Comparative evaluations
Pre- and post- use testing
Evaluating Interfaces
Deciding between observational and
empirical methods
– Is there a specific question being asked?
– How mature is the technology?
– Is having real tasks in a natural setting
Expectation Agents
Support for greater user participation in
software design
– Software designers encode expectations of
use which are matched against actual use.
– When expectations are not met, a dialog
can be started between designers and

Intelligent User Interfaces