Intelligent User Interfaces Frank Shipman Department of Computer Science Texas A&M University E-mail: email@example.com What this is about Designing, building, and evaluating intelligent user interfaces Particular technologies used to create intelligent user interfaces Issues concerning applicability of intelligent user interfaces Intelligent user interfaces (IUI)? Systems that provide interactive support based on embedded AI mechanisms Interfaces to AI functionality and knowledge representations Adaptive systems Cooperative problem-solving systems Technologies I User models, Situation models, and Programming by Demonstration Adaptive Interfaces Requirements: – interface that can be adapted – user or situation model – adaptation strategy Frequently used for providing assistance or training to user User Model “any information which a program has which is specific to a particular user. The information itself could range from a simple count of errors, to some complicated data structure which purports to represent relevant parts of the user’s knowledge of the problem domain.” Stereotyping vs. Individual Stereotyping (canonical user modeling) – provide interfaces for classes of users – classes might be based on skill (novices, midrange users, experts) or role Individual approach – dynamically adapt to suit each user – can be based on observed use of system or self assessment Representations for User Models Descriptive method – modeling the user’s observed behavior – describes what system has seen user do Skill-based or cognitive method – attempt to model the internal cognitive models and processes of the user – represents background knowledge, goals, plans, preferences, misconceptions Acquisition of User Model Model based on a combination of: – Observations of system use • statistical history • chronological history – Self-assessment – Testing How can model evolve over time? – Any of the above Berkeley UNIX Consultant Goal: Provide help to new UNIX users Generates user model based on “successful” use of UNIX commands – Explanations of difficult commands can make use of student’s knowledge. Intelligent Tutoring Systems Task is generally well-known – assignments given to student by system – systems track partial completion Systems keep record of student’s success and failure. – used to determine future assignments – used to determine how to help when student has difficulties Situation Models Components of situation: users, system, environment Users – multiple user models System – hardware constraints and load – device / resource availability Representing the Environment Identifying environmental influences – anticipating use situations – classes of use vs. detailed model of environment Monitoring environment – direct input devices – user description Mars Medical Assistant Goal: Provide medical support for astronauts on three year trip to Mars Consider educational, consultation, and emergency situations Models of user and patient – limited highly-trained user community – no new users joining during mission Other Adaptive Systems Typing completion – suggests completions for partial terms based on prior use Emacs suggestions – notifies user when more efficient method available to complete task Computer Chess Game – determines quality of own play based on perceived level of opponent Programming by Demonstration Generalizing from demonstrated action and situation sequences to programs Difficulties: – knowing what must stay the same – knowing what are variables and their types – connecting to programmed application code Programming by Demonstration Systems Peridot -- demonstration of simple interface Marquise -- demonstration of graphical editors including palettes and modes DEMO -- demonstrating dynamically created objects DEMO2 -- refinement by system based on multiple demonstrations Pavlov Focus on programming animation Includes: – graphical objects – models of motion and time Stimulus-response demonstration – modes for creating objects and behaviors – mode for demonstrating interaction Technologies II Presentation generation, Design Environments, and Interface agents Presentation Generation Generating dynamic links to information – enabling user-controlled flow Generating presentations based on current situation and/or user – Use of user or situation model Generating rhetorical structure/transition – Scripting events – Media-based decisions Presentations and Explanations Examples: – Explainer (Redmiles) – Explainable Expert System (Moore) – Story Presentation System (Sgouros, …) Explainer Domain: Graphical program explanation for software reuse Creates links between perspectives on software including source code, documentation, execution information, application domain view Provides user multiple points of access to better inform about software Explainable Expert System (EES) Explains different outcomes in an expert system / planner Generates natural language to answer user’s questions Keeps dialog history to provide differential descriptions EES Architecture for Explanation User Question Knowledge Base Query analyzer User goals Plan Operators Text planner Dialog history Response Sentence generator Focus Information EES Example User: “Describe Inderal” System: “Inderal a drug that …” User: “Describe Elavil” System: “Like Inderal, Elavil is used …” User: “Describe Cafergot” System: “Cafergot is very different from the drugs we have been talking about. …” Story Presentation System A dynamic dramatization method for narrative presentations Architecture: Symbolic Plot Description Plot Analysis Dramatic Effects Library Dramatization Original Story Material Presentation Manager Story Presentation Story Presentation Analysis Plot analysis models: – physical and emotional state changes – positive and negative interference among characters Dramatization uses plot analysis to determine dramatic events in story – Lifeline, Rising complication, Reversal of fortune, Dramatic irony, Happy end Story Presentation Results Presentation manager adds dramatic effects to original story material to emphasize dramatic events in story Effects include – audio: selection of noises or music – images and video: flashbacks, flashforwards, images of other characters Design Environments Provide a software environment supporting the activities part of design. – specification, construction – argumentation, documentation, communication Examples: – Framer (Lemke, Fischer) – JANUS (Morch, McCall, Fischer , ...) Framer Knowledge-based support for interface design Approach: direct manipulation interface builder Framer 1 -- construction kit approach Framer 2 -- design environment Design Environment Components (1) Checklist – system provides decomposition of task, – user identifies current focus Palette & Workspace – system provides primitive components – user identifies components used and organization of components in design Design Environment Components (2) Specification sheets – system brings design issues to user’s attention, presents potential answers, and explains significance and consequences of design choices – user symbolically specifies answers to design issues Design Environment Components (3) Critics – system points out sub-optimal design decisions, explains why this is believed, and provides heuristics for making decisions – users may accept or reject the system’s critique Design Environment Components (4) Catalog – system provides examples – user selects designs to reuse and modify Code generator – system generates an executable representation of designed interface Other Design Environments JANUS -- kitchen design – designed for non-technical users XNetwork -- computer network design – identified need for simulation component VDDE -- voice dialog design – another type of interface design with interesting constraints Software Agents One view: Software processes that have non-trivial tasks delegated to them which require independent action and a report on the results. Issues for Software Agents (1) Personification – Should agents be represented as a living or animated character? – Does it improve adoption of software? – Does it create inflated expectations? – Is it just too annoying? Issues for Software Agents (2) Trust and Competence – How does user develop an informed level of trust? – Can agent give self-assessment on likely outcome of task? Delegation – How can user delegate tasks? – How can user check on status of delegated tasks? Issues for Software Agents (3) Control – How does user set limits on the agent’s activity? – When does the agent get to interrupt the user (mixed-initiative dialog)? Dealing with multiple agents – How can the user manage many agents? – How can interactions between agents be predicted? Information Retrieval Agents Watch user patterns to infer interests or goals which are used to classify, rank, or suggest new information Examples: – INFOSCOPE: patterns in Netnews use – BASAR: patterns in Web access Issue: the “cold start” problem – must watch a while before suggesting Social Filtering Finding elements liked by others (with similar preferences) – requires some notion of preferences – improves with more users Examples – Tapestry -- rating of documents – GroupLens -- collaboration & user profiles – Amazon.com and CD-NOW Technologies III Knowledge manipulation and Using recognized structure Interacting with Knowledge User tasks – Adding knowledge – Editing rule bases and object hierarchies Examples – HITS Knowledge Editor (Terveen) – Modifier (Girgensohn) – Hyper-Object Substrate (Shipman) Knowledge Representations Informal – text, graphics, audio, video Semi-formal – hypertext, argumentation Formal – frames, semantic nets, scripts, rules, inheritance hierarchies, HITS Knowledge Editor Knowledge editor for CYC project Difficulties of knowledge representation formalization - articulation in precise detail comprehension - complex vocabulary, size modification - location and consistency Terveen’s Design Principle #1 Provide a workspace in which users and systems can jointly construct and manipulate a context for problem-solving, and in which the state of the problem-solving is represented visibly. Terveen’s Design Principle #2 Deliver intelligent assistance through critics. Terveen’s Design Principle #3 Exploit the interactive potential of computational media to manage the user-system interaction according to conventions that are appropriate to the role of each party in the interaction. Support Provided by HKE Inferences -- information inferred from workspace and existing KB Troubles -- inconsistencies between workspace and KB Suggestions -- relevant representational issues for users to consider Modifier Support for End-User Modifiability – Users are not knowledge engineers Example: – Adding new object class to existing system Support: suggestions based on similarity of features and efficiency of representation Hyper-Object Substrate Formalization as difficulty – cognitive load – tacit knowledge – premature structure Supporting incremental formalization – Flexible knowledge representation – Suggestions based on informal content – Automating knowledge acquisition? Using Recognized Structure When can it be used without user acknowledging correctness? Examples: TileBars (Hearst) • graphically presents results of text analysis inferring topic changes within text corpus Data Mining • search for patterns within database Non-Verbal Representations Non-verbal communication allows expression of emergent thoughts Systems need to recognize implicit structure Examples: – VIKI (Marshall, Shipman, …) – PerSketch (Saund, Moran) – Tivoli (Moran, van Melle, Chui, …) VIKI: An Analyst’s Workspace Task: collect, organize, and interpret information Characteristics: – evolving understanding – ephemeral structures – emerging visual languages Goal: Recognize visual structures and languages PerSketch: Adaptive Drawing Tool Recognize multiple potential connections between strokes in drawing Enables cleaning up drawings, editing drawn objects Example: TIVOLI: Meeting Room Support Manipulating structures on an electronic whiteboard Recognize gestures, characters, etc. Recognize lists, groupings Map changes in visual structure to underlying database Design Considerations Issues of applicability, Evaluating interfaces, and Discussion IUI: Appearing Too Smart? A central danger of creating IUI is, like ELIZA, appearing too smart – Creating unrealistic expectations – Communication difficulties • false alarm • garden path Design guideline: do not hide system’s knowledge or lack thereof Evaluation Methods Interviews Cognitive Walkthroughs Questionnaires Videotapes of use Usage data from software monitoring Comparative evaluations Pre- and post- use testing Evaluating Interfaces Deciding between observational and empirical methods – Is there a specific question being asked? – How mature is the technology? – Is having real tasks in a natural setting important? Expectation Agents Support for greater user participation in software design – Software designers encode expectations of use which are matched against actual use. – When expectations are not met, a dialog can be started between designers and users.