IMPLEMENTATION OF AN AFFECTIVE
CONVERSATIONAL AGENT FOR
CONTROLLING A HI-FI SYSTEM
Justo Saavedra
CONTENTS
Introduction
System Architecture
Emotional System
• What´s the project about? Hi-Fi
• Objectives
• Overview
• SOAP
•
•
•
•
Needs
Appraisals
Emotions
Dialogue Features
Object Oriented
Design
• Classes Implemented
Hi-Fi System
• Response Generation
• Synthesizer
Supervisor
Demonstration Video
Conclusion
Future Work
• Overview of the Supervisor
Máster en Sistemas Electrónicos para Entornos Inteligentes
THE HI-FI SYSTEM
Front
End
Speech
Recognition
Language
Understanding
Module
Dialogue
Manager
Synthesizer
Generation
Response
Module
Máster en Sistemas Electrónicos para Entornos Inteligentes
Execution
Module
OBJECTIVES
Improve user satisfaction when interacting with dialogue
module for controlling the Hi-Fi system…
The Adaptation of a psychological model of emotions to the
servant agent.
Design and development of the architectural communication
system to be used.
Improvement of the existing dialogue module.
Include the new Emotional Speech Synthesizer
Máster en Sistemas Electrónicos para Entornos Inteligentes
PURPOSE
In the future
emotional systems
will play an
important role in
the development of
intelligent/affective
systems.
Both recognition of
the user emotions
(smile detection) and
emotional response
from the system are
highly desirable.
Why?
Máster en Sistemas Electrónicos para Entornos Inteligentes
There is a wide
field of
applications
which can be
benefited from
integrating an
emotional system
(education,
entertainment
systems, social
robots, etc)
SYSTEM ARCHITECTURE
Reuse, as much as possible, the previous
dialogue system.
The new servers in our dialogue system,
similarly to the Galaxy-II, can be in the
same machine or in different machines
giving broad flexibility to operate and
distribute computational processing
Must provide easiness to add/delete new
modules and functionalities.
Máster en Sistemas Electrónicos para Entornos Inteligentes
SYSTEM ARCHITECTURE
Speech
recognizer
Language
Understand
ing Module
Dialogue
Manager
Execution
Module (IR,
Roomba)
Response
Generator
Main
Program
Loop (Hub)
Opencv
Webcam
server
Synthesiser
Client
Opencv
Client
Emotional
system
server &
Supervisor
Synthesiser
server
Máster en Sistemas Electrónicos para Entornos Inteligentes
SIMPLE OBJECT ACCESS
PROTOCOL
SOAP is a lightweight protocol for exchange of
information
Decentralized, distributed environment.
It is an XML-based portable protocol
Service Requester
EmotionClient
Service Provider
EmotionServer
Gets Emotions
Máster en Sistemas Electrónicos para Entornos Inteligentes
EMOTIONAL SYSTEM
Adaptation of the
Maslow´s Hierarchy of
Needs to a robotic system
Ethics
Success
Social
Safety
Survival
Máster en Sistemas Electrónicos para Entornos Inteligentes
EMOTIONAL SYSTEM
Task1
Ethics
Success
Task2
Appraisals
Social
Safety
Task3
Survival
Surprise
Fear
Happiness
Sadness
Anger
Shame
Neutral
Máster en Sistemas Electrónicos para Entornos Inteligentes
EMOTIONAL SYSTEM
Task1
Ethics
Relevance
Success
Task2
• Importance of the
situation: Going
critical/Overcoming
Social
Safety
Task3
Desirability
Survival
Surprise
Fear
Happiness
Sadness
Anger
Shame
• degree of valence of
the change
provoked by the
event
Neutral
Máster en Sistemas Electrónicos para Entornos Inteligentes
EMOTIONAL SYSTEM
Task1
Task2
Task3
Ethics
Relevance
Success
Desirability
Social
Urgency
Safety
• Estimates the time
to get to the critical
level
Survival
Unexpectedness
Surprise
Sadness
Fear
Anger
Happiness • Based on the
Shame
acceleration
assuming a linear
prediction model.
Neutral
Máster en Sistemas Electrónicos para Entornos Inteligentes
EMOTIONAL SYSTEM
Task1
Relevance
Ethics
Desirability
Success
Task2
Social
Urgency
Safety
Task3
Unexpectedness
Survival
Unfamiliarity
Surprise
Fear
Happiness
Sadness
Anger
Shame
Neutral
• Is the state a
probable situation
or not
Changeability
• range of the need
variation
Máster en Sistemas Electrónicos para Entornos Inteligentes
EMOTIONAL SYSTEM
Task1
Relevance
Ethics
Urgency
Success
Task2
Social
Unexpectedness
Safety
Task3
Unfamiliarity
Survival
Changeability
Surprise
Sadness
Fear
Anger
Neutral
Happiness Controllability
Shame
• Whether or not an
event is controllable
depends on the
capabilities
Priority (Weight)
Máster en Sistemas Electrónicos para Entornos Inteligentes
EMOTIONAL SYSTEM
Task1
Relevance
Desirability
Urgency
Unexpectedness
Unfamiliarity
Changeability
Controllability
Priority (Weight)
Ethics
Success
Task2
Social
Safety
Task3
Survival
Surprise
Fear
Happiness
Sadness
Anger
Shame
Neutral
Máster en Sistemas Electrónicos para Entornos Inteligentes
EMOTIONAL SYSTEM
Task1
Task2
Task3
Kur  urgency  Ku  unexpectedness
 Kf  unfamiliarity

 Emotion  
Kc  changeability  Kcont  controlability

amiliarity
ability
 Kagent
  sign  weight  relevance  Kd  desirability

100

Máster en Sistemas Electrónicos para Entornos Inteligentes
 Kagent



SUCCESS RELATED DIALOGUE
FEATURES
For each phrase recognized and processed by the HiFi, it calculates
several features associated with the phrase, words, concepts and
actions understood.
These features are passed to the emotional system in order to
calculate the success events to be generated
Speech Recognition Failures
Confidence Average
Max Confidence Recognized Words Percentage
High Confidence Understood Concepts Percentage
Máster en Sistemas Electrónicos para Entornos Inteligentes
OBJECT ORIENTED DESIGN
Several classes were implemented
in order to provide code
reusability and flexibility.
Emotions and needs share
attributes, and methods in order
to calculate, and update its status
LevelTimeHistory
CIOProccess
CNeed
CEmotion
Máster en Sistemas Electrónicos para Entornos Inteligentes
OBJECT ORIENTED DESIGN
• Statistics Methods (Mean,
Standard Deviation)
• Storage of History Levels
LevelTimeHistory
CIOProccess
CNeed
CEmotion
Máster en Sistemas Electrónicos para Entornos Inteligentes
OBJECT ORIENTED DESIGN
LevelTimeHistory
•Process Running (Threads)
CIOProccess
CNeed
CEmotion
Máster en Sistemas Electrónicos para Entornos Inteligentes
OBJECT ORIENTED DESIGN
• Weight (Appraisals)
• Critical Values
• Need Formulas
CNeeds
Csurvival
Csafety
Csocial
Csuccess
Cethics
• Time Constants
• EmotionsUpdate(K)
Formula
CAngry
CShame
CEmotions
Chappy
Cfear
Máster en Sistemas Electrónicos para Entornos Inteligentes
Csad
CSurprise
HI-FI SYSTEM->RESPONSE
GENERATION
Allows the user to emulate the responses by providing input
parameters (template, slot, status, value, relation, emotion). Its
functionality was expanded to adapt emotional behaviour
[V_CANCI][REPRO_CD]
¿Qué canción del cedé [V_DISCO] quiere?.
¿Qué canción del cedé [V_DISCO] quiere escuchar?.
[V_CANCI][REPRO_CD][FRIEND]
Tío, ¿Qué canción del cedé [V_DISCO] quieres?.
[V_CANCI][REPRO_CD][FRIEND][SADNESS]
Perdona tío, ¿Qué canción del cedé [V_DISCO] quieres?.
Máster en Sistemas Electrónicos para Entornos Inteligentes
HI-FI SYSTEM->RESPONSE
GENERATION
Example:
[V_CANCI][REPRO_CD][FRIEND][HAPPINESS]
1. Look for this template
2. If not found -> Exclude the last optional parameter and look for
the new template [V_CANCI][REPRO_CD][FRIEND]
3. If not found -> Exclude the last optional parameter and look for
the new template [V_CANCI][REPRO_CD]
4. If not found -> Failure
Templates:
[V_CANCI][REPRO_CD]
¿Qué canción del cedé [V_DISCO] quiere?.
¿Qué canción del cedé [V_DISCO] quiere escuchar?.
[V_CANCI][REPRO_CD][FRIEND]
Tío, ¿Qué canción del cedé [V_DISCO] quieres?.
Máster en Sistemas Electrónicos para Entornos Inteligentes
SYNTHESIZER
A new synthesizer developed by the GTH was used. It takes into
account the emotions and the intensity value.
Surprise
Fear
Who wins?
Happiness
Sadness
PositiveEmotions
Neutral
Anger
NegativeEmotions
Shame
Máster en Sistemas Electrónicos para Entornos Inteligentes
Emotional
Phrase
SYNTHESIZER
A new synthesizer developed by the GTH was used. It takes into
account the emotions and the intensity value.
Surprise
23.9
Fear
15.0
Happiness 25.0
Who wins?
Sadness
16.1
PositiveEmotions
Neutral
10.0
NegativeEmotions
Anger
10.0
Shame
0.0
Máster en Sistemas Electrónicos para Entornos Inteligentes
Emotional
Phrase
SYNTHESIZER
A new synthesizer developed by the GTH was used. It takes into
account the emotions and the intensity value.
Surprise
23.9
Fear
15.0
Happiness
25.0
Sadness
16.1
Neutral
10.0
Anger
10.0
Shame
0.0
Negative Emotions = 23.9+15+16.1+10 = 65
Positive Emotions = 23.9+25 = 48.9
Wins: Negative Emotions
Dominant: Sadness
Intensity: 16.1/(23.9+15.0+25.0+16.1+10+10)
Máster en Sistemas Electrónicos para Entornos Inteligentes
SUPERVISOR
Monitors/Simulate/Debug/Fine-Tune
Its users
interface
permits
real time
monitoring
• Monitoring of the
emotions
• Appraisal calculations due
to certain interaction
• Decay due to time of the
different needs\emotions
values
Máster en Sistemas Electrónicos para Entornos Inteligentes
SUPERVISOR->HOW IT WORKS?
• Using SOAP, the supervisor system interacts with emotional
system (both send and receive data).
• The supervisor as well as the emotional system acts as a
client/server depending if they are receiving or sending data.
Hi-Fi
Emotional
System
Máster en Sistemas Electrónicos para Entornos Inteligentes
Supervisor
SUPERVISOR
Máster en Sistemas Electrónicos para Entornos Inteligentes
CONCLUSIONS
The affective system implemented and the
improvement of the dialogue response generation
permits the future evaluation of the system (Compare
with previous non-emotional model)
The architecture and features of the affective system
(independent of task) make it possible to adapt it to
other applications domains (games, tutoring systems)
The communication architecture implemented give
the flexibility to change/include/delete modules with
easiness
Máster en Sistemas Electrónicos para Entornos Inteligentes
FUTURE WORK
Implementing more dialog features
Fill in the necessary templates to provide
better response generations
Implement more events (ethic/social)
Evaluate the system
Máster en Sistemas Electrónicos para Entornos Inteligentes
DEMONSTRATION
Máster en Sistemas Electrónicos para Entornos Inteligentes
QUESTIONS?
Máster en Sistemas Electrónicos para Entornos Inteligentes
Descargar

Slide 1