Direct Manipulation Interfaces
Positive acceptance of an application
 Mastery of the interface
 Competence in performing tasks
 Ease in learning originally and in assimilating advanced features
 Confidence in the capacity to retain mastery over time
 Enjoyment in using the interface
 Eagerness to show off interfaces to novices
 Desire to explore more powerful aspects
Direct Manipulation Interfaces
Attributes of direct-manipulation interfaces
 Visibility of the objects and actions of interest
• Example: Driving an automobile
 Rapid, reversible, incremental actions
 Replacement of types commands by a pointing action on the object of
 Example: Dragging a file to a trash can versus “rm file.doc”
• What about: “rm file*.doc”?
• What about “rm *.*” with no undo?
Direct Manipulation Interfaces
Extensions of direct manipulation
 Virtual reality – users are in an immersed environment
• Reality is blocked out via a head-mounted display
• Hand gestures (via a data glove) allow users to point, select,
grasp and navigate
 Augmented reality – user remains in normal
surroundings, but adds a transparent overlay.
• Examples – labeled buildings, hidden plumbing
• Google: Project Glass
• http://www.youtube.com/watch?v=W2E2zcFt9Xo
• http://www.youtube.com/watch?v=tnRJaHZH9lo&feature=related
 Tangible user interfaces – users manipulate physical
• Example – putting several plastic blocks near each other to
create an office floor plan
Direct Manipulation Interfaces
Command-line versus display editors versus word processors
 The Tubeless Interface
Direct Manipulation Interfaces
Command-line versus display editors versus word processors
 Single-line and Multi-line Editors (e.g., IBM MVS, VM, TSO, JCL)
Direct Manipulation Interfaces
Command-line versus display editors versus word processors
Single-line and Multi-line Editors versus WYSIWYG (what you see is what you get) editors (e.g.,
Microsoft Word – early 1990s)
Example: Three basic modes of vi
Command mode (Telling the computer what to do: Low level commands, e.g., move the cursor
to the right one character)
Default when you enter vi.
Most letters, or short sequences of letters, that you type will be interpreted as commands
Pressing Esc when you're in command mode, your terminal will beep at you. This is a very
good way to tell when you're in command mode
Insert mode (Entering the content)
Whatever you type is inserted in the file at the cursor position
Press Esc to end insert mode, and return to command mode.
Line mode (Telling the computer what to: High level commands, e.g. Save)
To enter line mode from command mode, type a colon ( : )
Your cursor moves to the bottom of the screen, by a colon prompt.
Type a line mode command, then press Enter.
Direct Manipulation Interfaces
Command-line versus display editors versus word processors
 Single-line and Multi-line Editors (e.g., vi)
Starting vi and Saving Files
Starting vi: vi filename (start editing filename, create it if necessary)
Saving the file you're working on and/or leaving vi: :wq (write the file to disk and quit)
Quit without saving any changes: :q!
:w! newfile (write all lines from the entire current file into the file 'newfile', overwriting any existing
:n,m w! newfile (write the lines from n to m, inclusive, into the file newfile, overwriting any existing
Direct Manipulation Interfaces
Command-line versus display editors versus word processors
 Single-line and Multi-line Editors
To Move To
one space to the left (also try left arrow)
one line down (also try down arrow)
one line up (also try up arrow)
one space to the right (also try right arrow)
end of current line
beginning of current line
beginning first word on the next line
end of file
line n; use :0 to move the beginning of the file
beginning of next word; 5w moves to the beginning of the 5th word to the right
end of next word
beginning of previous word
one page up
one page down
the matching (, ), [, ], {, or }
(Press % with your cursor on one of these characters to move your cursor its
Direct Manipulation Interfaces
Command-line versus display editors versus word processors
 Single-line and Multi-line Editors
• Searching for text
search down for string
search up for string
repeat last search from present position
• Inserting text
append starting right of cursor
append at the end of the current line
insert starting left of cursor
insert at beginning of the current line
open line below cursor, then enter insert mode
open line above cursor, then enter insert mode
:r newfile
add the contents of the file newfile starting below the current line
Direct Manipulation Interfaces
Command-line versus display editors versus word processors
 Single-line and Multi-line Editors
• Deleting text
delete single character; 5x deletes 5 characters
delete word; 5dw deletes 5 words
delete line; 5dd deletes 5 rows
delete word, leaves you in insert mode (i.e. change word)
change line -- delete line and start insert mode
change character -- delete character and start insert mode
delete from cursor to end of line
change from cursor to end of line -- delete and start insert mode
undo last change
undo all changes to current line
join current line with line that follows (press Enter in insert mode to split line)
Direct Manipulation Interfaces
Command-line versus display editors versus word processors
 Single-line and Multi-line Editors
• Cutting and Pasting
transpose two characters (two commands, x followed by p)
yank (i.e. copy) one line into a general buffer (5yy to yank 5 lines)
yank into the buffer named a
put the general buffer back before the current line
put from buffer a before current line
put the general buffer back after the current line
put from buffer a after the current line
• Miscellaneous Commands
show line number of current line
redraw the entire display
Direct Manipulation Interfaces
Command-line versus display editors versus word processors
 Early 1980s – Text editing was done with line-oriented command languages
 Nroff/troff: Unix based word processor
.nf = no-fill, you use for graphs or
text that you don't want spaces to
be ignored
.ce # = centers by the # of sentences you enter
.ti # = .ti indents but only for one line, so if you have one
sentence that needs to be indented 8 spaces but the rest of
the page is indented 3 you can use .ti 8 for that single
.fi = fill-in, extra space will be ignored and text that is entered
like this, the fill-in command will continue until you enter a .nf
command and vice-versa
.in # = # is the number of spaces you want the text indented,
this command will be in place until you put in a new .in
command...if you want something to not be indented you put in
.in 0
Direct Manipulation Interfaces
Command-line editors
 Who would use these interfaces?
 Why?
Direct Manipulation Interfaces
Command-line editors
 Who would use these interfaces?
 Why?
• Mastery of the interface
• Competence in performing tasks
• Ease in learning originally and in assimilating advanced features
• Confidence in the capacity to retain mastery over time
• Enjoyment in using the interface
• Eagerness to show off interfaces to novices
• Desire to explore more powerful aspects
Direct Manipulation Interfaces
Command-line editors
 Who would use these interfaces?
 Why?
Direct Manipulation Interfaces
Advantages of WYSIWYG Editors
Users see a full page of text
• 20 to 60 lines provides a context for each sentence
The document is seen as it will appear when printed
• Eliminating the clutter of formatting commands
Cursor action is visible
• Indicates where to focus attention and apply action
Cursor motion is natural
• Arrow keys or mouse provide natural physical mechanisms for moving the cursor
Labeled icons make frequent actions rapid
• Toolbar for frequent actions
Immediate display of the results of an action
• Example: Clicking a button to center text provides immediate result
Rapid response and display
• Full page of text in a fraction of a second
Easily reversible actions
• Example: Undo, backspace
Direct Manipulation Interfaces
Technology advancements evolving from word processing
 Integration of graphics, spreadsheets, animations, photographs, etc.
 Desktop publishing software
 Presentation software
 Hypermedia environments and the World Wide Web (hyperlinks to
 Improved macro facilities (e.g., construct, save and edit sequences of
frequently used actions)
 Spell checkers and thesauri
 Grammar checkers
• Use of passive voice
• Excessive use of certain words
• Lack of parallel construction
 Document assemblers
• Contracts
• Wills
Direct Manipulation Interfaces
 1979 – VisiCalc from a Harvard Business School
 254 rows and 63 columns
 Functions within a cell as it relates to other cells
 Simulation of an accountants spreadsheet
 Lotus 1-2-3 dominated the market in the 1980s
 Today Excel dominates
• Graphics displays
• Multiple windows
• Statistical routines
• Database access (e.g., Price List to Service Catalog)
Direct Manipulation Interfaces
Spatial Data Management
 Geographic applications – Nicholas Negroponte at MIT
 Display of the world and able to zoom in on the Pacific Ocean to see markers
for convoys of military ship
Direct Manipulation Interfaces
Spatial Data Management
 ArcView – ESRI
• Global Information Systems (Demo)
• Select type of information to display (roads, population, rainfall, topography,
political boundaries
Direct Manipulation Interfaces
Video Games
 The most exciting, well-engineered, commercially successful application of
direct-manipulation concepts?
 Pong
 Pacman (http://www.activitypad.com/online-games/pacman/)
 Field of action is visual and compelling
 Button presses, joystick motions and knob rotations produce rapid response
on the screen
 No syntax to remember
 Error messages are rare – the results of the action are obvious and easily
 Often there is continuous display of the score (competition between others
and the player themselves)
• Positive reinforcement that encourages mastery
Direct Manipulation Interfaces
Video Games
 Educational Video Games – direct manipulation
• SimCity – education on urban planning
• The Sims – stronger attraction to women then men
Direct Manipulation Interfaces
 Video Games
 Computer Role Playing Games (CRPGs)
• Players assume the role of a fictional character
• Activity takes place in a fictional world
• User’s control many of their character's actions
• Myst
 Massively Multiplayer On-Line Role Playing Games (MMORPGs)
• A large number of players interact with one another in a virtual
• Interaction is in a persistent world
– hosted by the game's publisher
– continues to evolve while the player is away from the game
• Worldwide MMORPGs revenues exceeded half a billion dollars in 2005
• World of Warcraft
Direct Manipulation Interfaces
Video Games versus Business Applications
 Game players
• Engaged in competition with a system or other players
• Seek entertainment and focus on challenge
• May prefer random events
 Application users
• Prefer a strong internal locus of control
• Focus on their tasks and may resent too many playful distractions
• Do not prefer random events
Direct Manipulation Interfaces
Computer-aided Design
 Automobiles, electronic circuitry, aircraft, mechanical engineering
 Structural engineering, floorplans, interiors, landscaping, plumbing, electrical
installation, etc.
 When the design is complete, the program can provide information regarding:
• Current
• Voltage drops
• Fabrication costs
• Manufacturing problems
Direct Manipulation Interfaces
Computer – aided Manufacturing and
Process Control
 Honeywell’s Experion Process Knowledge
• Provides the manager of a oil refinery or
power utility plant with a colorized schematic
of the plant
• Can indicate with a red line a sensor value that
is out of range
• With a single click the operator can get a more
detailed view of the troubling component
• A second click can provide more detailed
information the sensor, or reset a value or
• Basic strategy: eliminate the need for complex
commands that the operator might only need
to recall during a once-a-year emergency
Direct Manipulation Interfaces
 Direct Manipulation in Office Automation
 Xerox Star
• Sophisticated Text Formatting
• Graphics
• Multiple Fonts
• High Resolution
• Cursor Based Interface
 Apple Lisa
• Precursor to the Macintosh
• Hardware and software designs supported
– Pull-down menus
– Multi-window manipulation
– Editing of graphics and text
– Dragging of icons
Direct Manipulation Interfaces
 Direct Manipulation in Office Automation
 MS-DOS Commands vs. Macintosh Direct Manipulation
• Tasks: Creating, copying, renaming, erasing files
• After training and practice, average task times:
– MS-DOS is 5.8 minutes
– Macintosh is 4.8 minutes
After training and practice, average errors:
– MS-DOS is 2.0
– Macintosh is 0.8
Direct Manipulation Interfaces
 Continuing evolution of direct manipulation
 Quicken
 Home Automation
• Direct manipulation on a floor plan of:
– Burglar alarms
– Heat sensors
– Smoke detectors
– Opening/closing curtains or screens
– Air conditioning and heating
– Audio/video speakers or screens
» E.g., users can route sound from a MP3 player located in the living
room to the kitchen by dragging the MP3 icon into the kitchen
 Virtual Worlds
• Travel through the human body
• Ride an electron cloud as it spins around a nucleus
Direct Manipulation Interfaces
 Continuing evolution of direct manipulation
 Problems with direct manipulation
• Spatial or visual representation are not necessarily an improvement over text
– Especially for blind or visually impaired users
• Direct manipulation designs may consume considerable screen space
– May result in scrolling or multiple actions
• Users must learn the meanings of visual representations
– Icon interpretation
• For experienced typists, taking a hand off the keyboard to move a mouse may
take more time then typing the relevant command
• Users may not share the same understanding of the metaphor, analogy, or
conceptual model with the designer (testing is required)
• Browser based applications limit direct manipulation (e.g., drag & drop)
– Require Dynamic HTML, Java or Flash
Direct Manipulation Interfaces
 Advantages of direct manipulation
 Continuous representation of the objects and actions of interest with
meaningful visual metaphors
 Physical actions or presses of labeled buttons, instead of complex syntax
 Rapid, incremental, reversible actions whose effects on the objects of
interest are visible immediately
Direct Manipulation Interfaces
 Advantages of direct manipulation
 Design systems with the following benefits
• Novices can learn basic functionality quickly
• Experts can work rapidly to carry out a wide range of tasks, even defining new
functions and features
• Knowledgeable intermittent users can retain operational concepts
• Error messages are rarely needed
• Users can immediately see whether their actions are furthering their goals, and
if the actions are counterproductive, they can change the direction of their
• Users experience less anxiety because the interface is comprehensible and
because actions are easily reversed
• Users gain confidence and mastery because they are the initiators of action,
they feel in control, and they predict the interface’s responses
Direct Manipulation Interfaces
 The OAI Model and Direct Manipulation
 The object of interest is displayed so that interface actions are close to the
high-level task domain
 Little need for mental decomposition of tasks into multiple interface
commands with complex syntactic forms (e.g., vi: go to line, go to word,
go to character)
 Each action produces a comprehensible result in the task domain that is
visible in the interface immediately
 The closeness of the task domain to the interface domain reduces
operator problem-solving load and stress
 Compared to textual descriptors, visual representations of objects may be
more natural and closer to human innate capabilities
• Action and visual skills emerged well before language in human evolution
Direct Manipulation Interfaces
 Visual Thinking and Icons
 Semiotics – the study of signs and symbols
 Icon – an image, picture or symbol representing a concept
• In computer systems usually less than one inch square (64x64 pixels)
• Smaller icons are often integrated with a window border or toolbar
 Task dependency
• When working on a visual task (e.g., painting program), icons may be useful
• When working on a text-based task, it may be better to stay text based
 Icons with words (or mouse-overs) are useful
Direct Manipulation Interfaces
 Icon related guidelines
 Represent the object or action in a familiar and recognizable manner
 Limit the number of different icons
 Make the icon standout from its background
 3-d icons can be visually distracting
 Ensure that a single selected icon is clearly visible when surrounded by
unselected icons
 Make each icon distinctive from every other icon
 Ensure harmoniousness of each icon as a member of a family of icons
 Design the movement animation (e.g., grayed-out ghost image on a drag)
 Detail information
• Larger shadowing for a larger file
• Color to show the age of a document
• Animation to show how much of a file has printed (document icon absorbed
progressively into the printer icon)
 Explore use of combination of icons
Direct Manipulation Interfaces
 Direct Manipulation Programming
 Example: programming a radio to a set of stations by pressing/holding a
channel selection button
 Phone Services
Direct Manipulation Interfaces
 Direct Manipulation Programming
 Programming in the User Interface
• Sufficient computational generality
– Conditionals (if, then, else)
– Iteration (repeat/while)
• Access to appropriate data structures and operators
– File structures for directories
– Addition, subtraction, etc.
• Ease in programming
– By specification or by demonstration (Flash animation)
– Argument passing
• Simplicity in invocation and assignment of arguments
• Low risk
– High probability of bug free programs
– Halt and resume
Direct Manipulation Interfaces
 Direct Manipulation Programming
 Viscosity – the difficulty of making changes to a program
 Progressive evaluation – the capacity for execution of partial programs
Direct Manipulation Interfaces
 3-Dimensional Interfaces
 Some applications are designed as 2-D to be simpler than real-world
systems http://www.youtube.com/watch?v=Hw-O4zX8qRY
• Constrain movement
• Limit interface actions
• Ensure visibility of interface objects
 Enhanced 3D may be better than 3D
• Flying through objects
• Multiple simultaneous views of objects
• X-ray vision
• Shrink/expand objects
• Group/ungroup components
• Going back in time
 Less than successful 3D interfaces
• Air-traffic control
• Showing altitude by perspective drawing only adds clutter when compared to
an overview from directly above
Direct Manipulation Interfaces
 4-Dimensional Interfaces
 4-D Anyone?
Direct Manipulation Interfaces
 Second Life
 Multi-user environment where users interact
 Users can choose avatars (fantasy images, desirable characteristics)
Direct Manipulation Interfaces
 Relationship to:
 Personality Theory
 Social Psychology
 Spatial Cognition
 Applicable to business meetings, community discussion groups,
political forums?
 Blaxxun envision
Direct Manipulation Interfaces
 3D Desktops and Workplaces
 Microsoft’s Task Gallery
 Intel’s Grand Canyon
 Xerox PARC’s Information Visualizer
 No successful products yet
Direct Manipulation Interfaces
 Tips for effective 3D interfaces
 Use occlusion, shadows, perspective and other 3D techniques carefully
 Minimize the number of navigation steps for users to accomplish their
 Keep text readable (better rendering, good contrast with background, an
no more than 30-degree tilt)
 Avoid unnecessary visual clutter, distraction, contrast shifts and reflections
 Simplify user movement (keep movements planar, avoid surprises like
going through walls)
 Organize groups of items in aligned structures to allow rapid visual search
 Enable users to construct visual groups to support spatial recall (e.g.,
placing items in corners)
Direct Manipulation Interfaces
 Guidelines for inclusion of 3D features
 Provide users overviews so they can see the big picture
 Allow teleportation (rapid context shifts by selecting destination in an
 Offer x-ray vision
 Provide history keeping (recording, undoing, replaying, editing)
 Permit rich user actions on objects (save, copy, annotate, share, send)
 Give users control over explanatory text (pop-up, floating, screen tips)
 Offer tools to select, mark and measure
 Implement dynamic queries to rapidly filter out unneeded items
 Enable landmarks to show themselves even at a distance
 Allow multiple coordinated views (users can be in more than one place at
a time)
Direct Manipulation Interfaces
 Teleoperation
Mahru Humanoid Robot Real-Time Teleoperation
 Derived from direct manipulation and process control
 Physical processes taking place in a remote location
• Clean-up in a nuclear reactor
 Need adequate feedback in sufficient time to permit effective decision
• Manufacturing
• Medicine (consultation, radiology)
• Military operations (drones)
 Home automation
• Answering machines
• Security systems
• Energy control
• Appliances
Direct Manipulation Interfaces
 Design to accommodate teleoperation issues
 Slow response times and time delays
• Transmission delay (time for command to reach the microscope)
• Operation delay (time until the microscope responds)
 Incomplete feedback
• The microscope can transmit its current position, but operates so slowly it
cannot indicate the exact current position
 Unanticipated interferences
• The slide is accidentally moved by a person at the local site
 May be better for the user to specify a destination (rather than a motion)
and wait until the action is completed
Direct Manipulation Interfaces
 Telemedicine
 Remote examination
 Remote surgery
 Telepathology
• Magnification
• Focus
• Illumination
• Position
Direct Manipulation Interfaces
 Virtual and Augmented Reality
 “Being in” as opposed to “Looking at”
 Architectural applications
• Wall-sized image to give perspective
• Animation to simulate movement (left to right)
• Treadmill to simulate walking toward, walk through doors, stairs
• Replace projector with a head-mounted display
 Some applications are better when “looked at”
• Air-traffic control
• To surgeons want to “Be in” the patients body?
 Training using virtual reality
• Fifth Dimension Technologies www.5DT.com
Direct Manipulation Interfaces
 Virtual and Augmented Reality
 The CAVE (National Center for Supercomputing Applications)
 An immersive virtual reality facility designed for the exploration of and
interaction with spatially engaging environments.
 The stereoscopic capabilities, coupled with its uniquely immersive design,
enable scientists and researchers to interact with their data
• An atmospheric scientist can actually "climb inside" of a hurricane and visualize
its complex and chaotic elements from any angle or visual perspective
• A biological researcher, examining a tightly coiled strand of DNA, can virtually
"unravel" this strand and manipulate it in an environment that preserves the
critical depth information of the data.
• Teach a child to cross a street
Direct Manipulation Interfaces
 Artificial Reality
 VideoPlace – Myron Krueger
• Surround the user with an artificial reality which responded to their movements and
• The users were able to visually see the results of their actions on screen, through the use
of colored silhouettes.
• The users had a sense of presence while interacting with onscreen objects and other
• The sense of presence was enough that users pulled away when their silhouettes
intersected with those of other users.
eHarmony 2.0?
Direct Manipulation Interfaces
 Applications of virtual environments
 Phobia treatment
• Acrophobia
 Pain Control
• Immersive environments provide distractions for patients
 Interior Design
• OAI model
– Click, drag, enlarge objects
– Room painting tool
Direct Manipulation Interfaces
 Augmented Reality
 See the real world with an overlay of additional information
• See wires or plumbing behind walls
• Tourist glasses – label buildings in a historic town
• Molecular biology
Direct Manipulation Interfaces
 Virtual environments dependent on integration of multiple technologies
 Visual Display
• Normal Display
– 12 to 17 inches diagonally at a normal viewing distance of 70 cm subtends
a visual angle of about 5-degrees
• Large Screen
– 17 to 30 inches can cover 20 to 30-degrees
• Head Mounted Displays
– 100 degrees horizontally and 60-degrees vertically
– Head motion produces new images so users perceive 360-degrees
– Displays must approach 100-millisecond delay in presenting images to
approach real time
Direct Manipulation Interfaces
 Virtual environments dependent on integration of multiple technologies
 Head-position sensing
• Head-mounted displays can provide differing views depending on head position
 Hand-position sensing
• DataGlove
Direct Manipulation Interfaces
 Virtual environments dependent on integration of multiple technologies
 Force feedback and haptics
• Hand-operated remote-control devices for performing chemistry experiments or
for handling nuclear materials
• Gives users a sense of grasp
 Sound input and output
• Training of Army tank crews while using realistic sounds of battle resulted in:
– Elevated heart rates, more rapid breathing, and increased perspiration
• Speech recognition for initiating actions and making menu selection
– Keyboard and mouse use is restricted
 Other sensations
• Tilting and vibration of flight simulators
 Collaborative and competitive virtual environments
• Two people at remote sites working together while seeing each others actions
and the object of interest
Direct Manipulation Interfaces
 Definition, benefits, and drawbacks of direct manipulation
 Definition
• Visual representation (metaphor) of the “world of action”
• Objects are always shown
• Rapid, incremental, and reversible actions
• Replacement of typing with pointing/selecting
• Immediate visibility of results of actions
 Benefits over commands
• Less syntax reduces error rates
• Errors are more preventable
• Faster learning and higher retention
• Encourages exploration
Direct Manipulation Interfaces
 Definition, benefits, and drawbacks of direct manipulation
 Concerns
• Increased system resources (possibly)
• Some actions may be cumbersome (e.g., form fillin forcing use of mouse)
• History and other tracing may be difficult
• Visually impaired users may have more difficulty
Direct Manipulation Interfaces
 Piaget’s Four Stages of Development
 Sensorimotor (birth to 2 years)
 Preoperational (2 to 7 years)
 Concrete operational (7 to 11 years)
• Physical actions on an object are comprehensible
• Children acquire the concept of conservation or invariance
• Direct manipulation brings activity to this stage
 Formal operational (begins at 11 years)
• Symbol manipulation to represent actions on objects

Soarian™ User Interface