University of Milano-Bicocca, Doctoral School of Science
Ph.D. Program in Informatics, Cycle XXIV
Department of Informatics, Systems and Communication
Diego Bernini
Ph.D. research status and thesis hypothesis
Software Architectures for
Responsive Environments:
the case of Interactive Art & Design
Advisor: Prof. Francesco Tisato
Tutor: Prof. Carla Simone
September 23th 2009
Reference scenario
Research topic, status and thesis hypothesis
State of the art
Plan of the activities
September 23th 2009
D. Bernini, Ph.D. research status
Software Architecture
• Having a sound software architecture is a key aspect of
ICT systems
• In different classes of ICT systems relevant high-level
concepts are not turned into proper elements in their
software architecture
– need of proper architectural abstractions
– examples: time and space in surveillance systems and Geographic
Information Systems (Micucci et al., 2006) (Micucci et al., 2009)
• This leads to several issues
– intermix of conceptual, design aspects and implementation
– strong dependency on specific technologies
September 23th 2009
D. Bernini, Ph.D. research status
Responsive environments
• Real physical spaces equipped with ICT technologies
capable of sensing and responding accordingly to entities
that inhabit them (Silva and Vasconselos 2007) (Pinhanez and
Bobick 2003)
• Different aims
– From more concrete …
• Automation of offices, shops, houses (Elrod et al., 1993)
– …to more aesthetics
• Interactive Art & Design (Bullivant 2007) (Bullivant 2006)
• Various related research areas, different characterizations
– From Software Engineering, Ubiquitous (Pervasive) Computing
– to Human-Computer Interaction, Ambient Intelligence
September 23th 2009
D. Bernini, Ph.D. research status
4 Interactive Art
• ICT-supported interactive art installations
– Art Installations: artworks where space is central
• “art that appropriates space to its own artistic ends” (Bestor 1996)
– Interactive Art installations: the viewer become an active player
– ICT-supported: some interactions are achieved by ICT
• For itself: they are integrant part of the installation
• For the exhibition: valorization of traditional artworks/installations
– Interactive museums and galleries
(Studio Azzurro,
September 23th 2009
D. Bernini, Ph.D. research status
(Rafael Lozano Hemmer,
.. in Interactive Design
• Various classes of interactive small-scale architectural &
design interventions (Castle 2007) (Bullivant 2007) (Bullivant
• Interactivity: the environment enhance the user experience
• Interactive Architecture & Design, urban-open & closed spaces
(Interactive tunnel,
Vito Acconci Studio and CSAI,
(Bandini et al 2008))
September 23th 2009
(Digital Pavillion,
ONL [Oosterhuis_Lénárd] design office
D. Bernini, Ph.D. research status
High-level concepts: examples
• Time
– real-time responsiveness: the interactions may require strong
time constraints
– temporal input analysis and time-driven output
• Space
– locating objects in multiple spaces (e.g. sensor network space vs.
environment topology)
• Multi-modality
– Multiple heterogeneous sensing and responding
• Sensing:
– Strict physical environmental properties
» Pressure, temperature, humidity, lighting condition, noise levels
– Entity properties
» Presence, identification, motion, contact, physiological
• Responding: audio and visual effects, commanding of specific
September 23th 2009
D. Bernini, Ph.D. research status
Current practice
• Various development tools, but
– no explicit treatment of the previous high level concepts!
– low level of abstraction
• Hardware computing devices: “low level” programming
– E.g. Arduino, input, output & computing device, C programming
– oriented to specific technologies
• Different visual programming language & tools, but too specific
– E.g. Cycling’74 Max, oriented to audio and image processing
• … consequently, they hardly address issues like:
– Management of complexity: do not add complexity to the
inherent complexity of the desiderata interactions
– Integration: how to integrate new and heterogeneous
– Scalability: from few homogenous sensors to a lot of
heterogeneous sensors
September 23th 2009
D. Bernini, Ph.D. research status
Research topic
Filling the gap between low-level, technology oriented
aspects and high-level concepts of the previous
response environments
• What are suitable architectural abstractions for the
development of responsive environments in Interactive
Art & Design? (Trifonova 2008) (Edmonds et al., 2005) (Machin
September 23th 2009
D. Bernini, Ph.D. research status
What I Have Done
• Characterization of a thesis hypothesis
• Investigation of the state of the art
• Identification of case studies and collaborations (in
• GAS (“Grandi Attrezzature Scientifiche”) project (UniMiBDISCo): location aware, multimodal applications
– my overall collaboration: architectural abstractions for the
integration of heterogeneous components
• they allow the establishment of communication flows based on
space and time
– my current specific task: design and implementation of a
supporting platform (in progress)
September 23th 2009
D. Bernini, Ph.D. research status
Thesis hypothesis (what)
• Devising effective architectural abstractions for
responsive environments in Interactive Art and Design
– Goal 1: abstract from the specific technologies (hardwareinput/output and computing devices- and software- specific
programming languages & components-)
– Goal 2: go beyond ad-hoc approaches, fulfillment of real-time
and multimodal requirements in an effective, robust and scalable
– Goal 3: are usable by artists & designers, not only technologists
– Goal 4: are applicable to other purposes, e.g. Video-surveillance,
• Design and development of experimental applications
based on the proposed abstractions
– One for Interactive Art & Design, another for a different context
September 23th 2009
D. Bernini, Ph.D. research status
Thesis hypothesis (how and who)
• Core approach: Software Architectures (Prof. Tisato)
enhanced with other disciplinary perspectives
Ambient Intelligence
Interaction Design/Human-Computer Interaction
Collaborative work and Knowledge Management
Prof. Tisato, Prof. Archetti, Prof. Bandini, Prof. De Michelis, Prof.
Schettini, Prof. Simone...
• Identified case studies, collaborations and abroad period
– A specific workgroup among CSAI center, DISCo-ITIS and DISCoSAL labs
– Studio Azzurro (ref. Prof. De Michelis)
– GAS project
– Abroad period: acquiring knowledge & skills in interactivity vs.
Art & Design vs. ICT
• e.g. The Mixed Reality Lab, University of Nottingham (ref. Prof. De
September 23th 2009
D. Bernini, Ph.D. research status
State of Art: a reference schema
Different conceptual
ICT System organization:
models are adopted:
– It does NOT imply a centralized design organization from
Data flow models
stateful/reflexive, to emergent behaviors to Agents
ICT System
Computing & Communication
September 23th 2009
Output flow
Physical environment
D. Bernini, Ph.D. research status
Responsive environment
Input flow
State of the Art: enabling technologies
• Input: sensors, vision & tracking systems, tangible &
wearable devices
– Values of strict physical environmental properties and entity
properties, a survey in (Cook and Das 2007)
– Common technologies:
• Wireless sensor networks, a survey in (Yick et al., 2008)
• Cameras, image detection and tracking systems
– Object tracking, human motion capture ad analysis, surveys in (Yilmaz et
al., 2006) (Moeslund et al., 2006)
– Face recognition (Colombo et al., 2007) and other systems: DISCoImaging and Vision Lab
• Tangible interfaces and wearable devices (Harrison and Hudson 2008)
• Output: lights, projectors, displays, speakers, sound
synthesizers and other specific devices
– Some common protocols: DMX (lights), MIDI (sound), Power Line
Communication (PLC) and X10 (general electronic devices)
September 23th 2009
D. Bernini, Ph.D. research status
State of the Art: tools & programming languages
• Arduino
– “an open-source electronics prototyping platform
based on flexible, easy-to-use hardware and software”
• Cycling ’74 Max
– Visual programming language,
– System as computational graph of objects (patches) representing
input/output devices, processing and data containers
• Objects: self-contained program, they communicate by messages
Data flow oriented model, high extensible
MSP package: audio synthesis
Jitter package: video processing
Support for execution in local networks
• Others: Pure Data (PD), vvvv, Quartz Composer
– Core-model similar to MAX, but different focus
September 23th 2009
D. Bernini, Ph.D. research status
State of the Art: architectural abstractions
• Software Components
– (Heineman and Council 2001), (Shaw and Garlan 1996),
component styles (Garlan 1996) (Maier and Rechtin 2000)
– Specific component typologies: components with ports and
connectors (Caflisch et al., 2005), (Senthil et al., 2009), targeted
to responsive environments: context components (Jacquet et al.,
2005), e-Gadgets (Markopoulos et al., 2004)
• Time and space awareness
– time (De Paoli and Tisato 1995), (Micucci et al., 2006), space
(Micucci et al., 2009), GAS project
• Emergent conceptual and architectural models
– Multi-Agent models, Cellular Automata (CA) models, suitable for
emergent and lifelike behaviors
– (Bandini et al., 2008), (Silva and Vasconcelos 2007), (Sparacino et
al., 2001)
September 23th 2009
D. Bernini, Ph.D. research status
State of the art: conceptual frameworks
• Time and space
– time, space and audience vs. interactivity (Benford et al., 2009)
• Multi-modality
– multi-modality categories (Bongers and Veer 2007)
– typologies of sensing (Benford et al., 2005)
– human movement representation and responsive environments
(Loke and Robertson 2009)
September 23th 2009
D. Bernini, Ph.D. research status
Plan of Activities
• Phase 1 (investigation) - 12 months
– Investigation of the state of the art (completed)
– Identification of case studies & collaborations (currently in
• Phase 2 (research) – 12 months
– Devising and development of the architectural abstractions, by
generalizing the analysis of the identified case studies
• Phase 3 (experiment) – 8 months
– Design and development of experimental applications based on
the proposed abstractions
• Phase 4 (research) - 4 months
– Refinement of the architectural abstractions on the basis of the
previous experimental applications
September 23th 2009
D. Bernini, Ph.D. research status
Wishful thinking
• How artists/designers would like to conceive the
interactivity, the audience/user participation,
independently from available technologies?
– From collaborations with artists/designers!
– This is NOT A GOAL of the thesis
September 23th 2009
D. Bernini, Ph.D. research status
Target Conferences and Journals
• Conferences (and related Symposia)
– IEEE/IFIP Conference on Software Architecture (WICSA) and European
Conference on Software Architecture (ECSA)
– ACM Special Interest Group on Graphics and Interactive Techniques
(SIGGRAPH) conferences
• ACM’s Creativity & Cognition conferences
• ACM’s International conference on Digital Interactive Media in
Entertainment and Arts
– ACM Special Interest Group for Computer–Human Interaction (SIGCHI)
– ACM International Conference on Multimedia
– International Conference on Tangible and Embedded Interaction (TEI)
• Journals
ACM Transactions on Computer Systems (TOCS)
Pervasive and Mobile Computing, Elsevier (PMC)
ACM Transactions on Computer-Human Interaction (TOCHI)
ACM Transactions on Multimedia, Applications, and Computing
September 23th 2009
D. Bernini, Ph.D. research status
• Attended courses:
– “Knowledge Management: conoscenza, condivisione,
tecnologia”, Prof. Simone
– “Interaction Design”, Prof. De Michelis
– Others
“Laboratorio di Comunicazione” (from QUASI_SI Ph.D. Program)
“Comunicazione della Scienza” (Doctoral School of Science)
“Gestione dei progetti di ricerca” (Doctoral School of Science)
English course (DISCo), ongoing
• Next courses:
– 2/3 internal courses
• “Aspetti epistemologici dell’informatica”, DISCo, Prof. Bandini
• Enabling technologies: courses of Prof. Bisiani & Schettini?
– 2/1 external courses/Ph.D. schools
September 23th 2009
D. Bernini, Ph.D. research status
Other collaborations
• F.I.R.B. - Integrated Systems for Emergency project
– two papers related to my Master’s degree thesis
• Bernini, D., Toscani, D., and Frigerio, M. A software architecture for
the deployment of executable transformation models. Proceedings
of the 2009 International Conference on Wireless Communications
and Mobile Computing: Connecting the World Wirelessly, ACM
(2009), 47-51.
• Toscani, D., Frigerio, M., and Bernini, D. Dynamic update of data
analysis models in emergency systems. Proceedings of the 2009
International Conference on Wireless Communications and Mobile
Computing: Connecting the World Wirelessly, ACM (2009), 37-41.
– workshop participations and talks
• Leipzig, 21-24 June 2009, International Workshop on Advanced
Topics in Mobile Computing for Emergency Management:
Communication and Computing Platforms (MCEM 2009),
presentation of (Bernini et al., 2009)
• Rome, 3-4 April 2009, InSyEme internal workshop, talk “From
Application Requirements to Application Architecture”
September 23th 2009
D. Bernini, Ph.D. research status
References (1)
Overview of Responsive Environments, Interactive Art & Design
– Bullivant, L. Responsive Environments: architecture, art and design. Victoria and
Albert Museum, 2006.
– Bullivant, L. 4dSocial: Interactive Design Environments. John Wiley & Sons, 2007.
– Castle, H. Editorial. Architectural Design 77, 4 (2007), 4-5.
– Elrod, S., Hall, G., Costanza, R., Dixon, M., and Rivières, J.D. Responsive office
environments. Commun. ACM 36, 7 (1993), 84-85.
– Edmonds, E.A., Weakley, A., Candy, L., Fell, M., Knott, R., and Pauletto, S. The
studio as laboratory: combining creative practice and digital technology research.
Int. J. Hum.-Comput. Stud. 63, 4-5 (2005), 452-481.
– Machin, C.H.C. Digital Artworks: Bridging the Technology Gap. Proceedings of the
20th UK conference on Eurographics, IEEE Computer Society (2002), 16.
– Trifonova, A., Jaccheri, L., and Bergaust, K. Software engineering issues in
interactive installation art. International Journal of Arts and Technology 1, 1
(2008), 43-65.
September 23th 2009
D. Bernini, Ph.D. research status
References (2)
Enabling technologies, tools & programming languages
– Bestor, C. MAX as an overall control mechanism for multidiscipline installation art.
Computers & Mathematics with Applications 32, 1 (1996), 11-16.
– Cook, D.J. and Das, S.K. How smart are our environments? An updated look at the
state of the art. Pervasive Mob. Comput. 3, 2 (2007), 53-73.
– Colombo, A., Cusano, C., and Schettini, R. Face^3 a 2D+3D Robust Face
Recognition System. Proceedings of the 14th International Conference on Image
Analysis and Processing, IEEE Computer Society (2007), 393-398.
– Harrison, C. and Hudson, S.E. Scratch input: creating large, inexpensive,
unpowered and mobile finger input surfaces. Proceedings of the 21st annual ACM
symposium on User interface software and technology, ACM (2008), 205-208.
– Pinhanez, C.S. and Bobick, A.F. Interval scripts: a programming paradigm for
interactive environments and agents. Personal Ubiquitous Comput. 7, 1 (2003), 121.
– Moeslund, T.B., Hilton, A., and Krüger, V. A survey of advances in vision-based
human motion capture and analysis. Comput. Vis. Image Underst. 104, 2 (2006),
– Yick, J., Mukherjee, B., and Ghosal, D. Wireless sensor network survey. Comput.
Netw. 52, 12 (2008), 2292-2330.
– Yilmaz, A., Javed, O., and Shah, M. Object tracking: A survey. ACM Comput. Surv.
38, 4 (2006), 13.
September 23th 2009
D. Bernini, Ph.D. research status
References (3)
Software architectures and architectural abstractions
Caflisch, L., Savigni, A., Schettini, R., and Tisato, F. A software architecture for real-time,
embedded monitoring systems. Advanced Video and Signal Based Surveillance, 2005. AVSS
2005. IEEE Conference on, (2005), 540-545.
De Paoli, F. and Tisato, F. Architectural Abstractions for Real-Time Software. Proceedings of
the Second Asia Pacific Software Engineering Conference, IEEE Computer Society (1995), 199.
Heineman, G.T. e Councill, W.T., cur. Component-based software engineering: putting the
pieces together. Addison-Wesley Longman Publishing Co., Inc., Boston, MA, USA, 2001.
Garlan, D. and Shaw, M. An Introduction to Software Architecture. Carnegie Mellon
University, 1994.
Jacquet, C., Bourda, Y., and Bellik, Y. An architecture for ambient computing. Intelligent
Environments, 2005. The IEE International Workshop on (Ref. No. 2005/11059), (2005), 47-54.
Markopoulos, P., Mavrommati, I., and Kameas, A. End-User Configuration of Ambient
Intelligence Environments: Feasibility from a User Perspective. In Ambient Intelligence. 2004,
Maier, M.W. and Rechtin, E. The art of systems architecting (2nd ed.). CRC Press, Inc., 2000.
Micucci, D., Oldani, M. and Tisato, F. Time-Aware Multi Agent Systems. In Proocedings of
Multiagent Systems and Software Architecture (MASSA). D. Weyns and T. Holvoet (Eds.),
2006, 71-78.
Micucci, D., Tisato, F., and Adorni, M. Engineering spatial concepts. Knowl. Eng. Rev. 24, 1
(2009), 77-93.
Senthil, R., Kushwaha, D.S., and Misra, A.K. An improved component model for component
based software engineering. SIGSOFT Softw. Eng. Notes 32, 4 (2007), 9.
Shaw, M. e Garlan, D. Software architecture: perspectives on an emerging discipline. PrenticeHall, Inc., Upper Saddle River, NJ, USA, 1996.
September 23th 2009
D. Bernini, Ph.D. research status
References (4)
Emergent conceptual and architectural models
– Bandini, S., Bonomi, A., Vizzari, G., et al. A CA-Based Approach to Self-Organized
Adaptive Environments: The Case of an Illumination Facility. Proceedings of the
2008 Second IEEE International Conference on Self-Adaptive and Self-Organizing
Systems Workshops - Volume 00, IEEE Computer Society (2008), 1-6.
WITH SOFTWARE AGENTS. Appl. Artif. Intell. 21, 4-5 (2007), 469-488.
– Sparacino, F., Davenport, G., and Pentland, A. Media in performance: interactive
spaces for dance, theater, circus, and museum exhibits. IBM Syst. J. 39, 3-4
(2000), 510, 479.
Conceptual frameworks
– Benford, S., Giannachi, G., Koleva, B., and Rodden, T. From interaction to
trajectories: designing coherent journeys through user experiences. Proceedings
of the 27th international conference on Human factors in computing systems,
ACM (2009), 709-718.
– Benford, S., Schnädelbach, H., Koleva, B., et al. Expected, sensed, and desired: A
framework for designing sensing-based interaction. ACM Trans. Comput.-Hum.
Interact. 12, 1 (2005), 3-30.
– Bongers, B. and Veer, G.C. Towards a Multimodal Interaction Space:
categorisation and applications. Personal Ubiquitous Comput. 11, 8 (2007), 609619.
– Loke, L. and Robertson, T. Design representations of moving bodies for
interactive, motion-sensing spaces. Int. J. Hum.-Comput. Stud. 67, 4 (2009), 394410.
September 23th 2009
D. Bernini, Ph.D. research status

Presentazione corso di formazione