```Automatic and Data Driven
Pitch Contour Manipulation
with Functional Data Analysis
Michele Gubian, Lou Boves
Nijmegen, The Netherlands
Francesco Cangemi
Laboratoire Parole et Langage
University of Provence, Aix-en-Provence, France
Outline



Pitch Contour Manipulation

Context and problem

Sketch of proposed approach
Use of Functional Data Analysis (FDA)

Case study

Data preparation

Functional PCA

Functional synthesis and listening
Conclusions
2
Context

Languages can express oppositions using intonation

Question/Statement opposition in Neapolitan Italian
QUESTION
STATEMENT
“Milena lo vuole amaro (?)” = Milena drinks it (her coffee) bitter (?)

What are the intonation cues that listeners use?

Perceptual experiments where listeners judge stimuli
whose pitch (F0) contour has been manipulated

STEP 1: extract pitch contours from speech data

STEP 2: modify pitch contours

STEP 3: re-synthesize speech
3
Pitch Contour Manipulation
F0

Use of an intonation model


Stylization
Manual changes
time
POSSIBLE IMPROVEMENTS


Handle dynamic detail

Locally (e.g. concavity/convexity)

Long range correlation
Derive useful variation modes directly and
automatically from data
4
A data driven approach
Functional
Data
Analysis
x
5
Question/Statement opposition
in Neapolitan Italian
DATA

2 male speakers


“Milena lo vuole amaro (?)” = Milena drinks it (her coffee) bitter (?)

“Valeria viene alle nove (?)” = Valeria arrives at 9 (?)

“Amelia dorme da nonna (?)” = Amelia sleeps at grandma’s (?)

2 modalities = Q / S

5 repetitions

2 x 3 x 2 x 5 - 3 discarded = 57 utterances
6
Data Preparation

Sampled F0 curves have to be turned into functions

A basis of functions (B-splines) expresses each original curve

Decide how much detail to retain (smoothing)
7
Data Preparation (2)

Landmark registration

Align points in time that are deemed as having the same
meaning across the dataset
8
Classic
Principal Component Analysis (PCA)
PC1
x
x
x x x
x
x xx
xxx xx x x
x x
xx
x
x xx x
x
x x
x
x
x
x
x
x
x
PC2
salary
25
65
age
9
Functional PCA
10
PC-based signal reconstruction
+ 1.65 x
mean(t)
- 0.46 x
PC1(t)
PC2(t)
11
Manipulated stimuli
12
Conclusions

A data driven approach is possible in the exploration of
intonation phenomena

FDA provides automatic tools to describe variation in a set of
pitch contours extracted from real utterances


provided that the relevant landmarks are annotated
The same tools allow to construct artificial contours with
desired perceptual characteristics

Smooth and global variation are applied

Variations come from a statistical analysis of data

The process is automatic
13
14
```