How do you feel today?

Transkript

1 How do you feel today? Detecting emotions in speech November 15, 27 Sona Patel, Rahul Shrivastav, James Harnsberger, Mini Shrivastav Department of Communication Sciences and Disorders Motivation Clinical Applications Aprosodia 1 Autism spectrum disorders Technical Applications Speech recognition and speech synthesis Alternative and Augmentative Communication 1 Rosenbek et al. (26); Stringer (1996) Methods of Emotion Perception Two methods of perceiving emotions in speech: 1. Semantic information: I won the state lottery! Nlottery 2. Suprasegmental information: The yellow house has a tall fence Hyh

2 Background Current research - acoustic analysis on emotions to determine emotions that can be perceived in speech Criteria for selecting emotions Ekman s big six 2 : anger, joy, sadness, fear, surprise and disgust Biologically relevant 3 : Irreducibility 3 : Are the emotions perceivable in speech? 2 Ekman (1993) 3 Ortony and Turner (199) Goals Long-term goal: Predict the emotion within a speech signal Short-term objectives: I. Select emotion terms II. Discriminate emotions in speech a) To determine the perceptual mechanisms for emotion classification b) To determine the emotion categories that can be cued by suprasegmental information alone Exp 1: Selecting Emotion Terms Begin with 7 emotion terms Methods 18 participants Emotion terms grouped according to similarity Analysis 1. Summarize data 2. Calculate frequencies of emotion pairs 3. Hierarchical clustering scheme (HCS) 4 4 Johnson (1967)

3 Exp 1: Clustering Exp 1: Clustering Exp 1: Clustering Results

4 Recall Project Goal Short-term objectives: I. Select emotion terms:,,,,,,, Embarrassed, Exhausted, Funny, Happy, Interested, Jealous, Lonely, Love,, Sad, Surprised, Suspicious II. DISCRIMINATE EMOTIONS IN SPEECH a) To determine the perceptual mechanisms for emotion classification Exp 2 Methods: Emotional Stimuli Recorded 1 actor and 1 actress Sentences 2 sentences Nonsense sentences using pseudo-words Speakers allowed to repeat productions until satisfied Order of emotions spoken randomized Exp 2: Perceptual Task Same-different discrimination task 1 pair of utterances presented at a time All stimulus pairs presented at least twice Testing done in six 2-hour sessions Are the speakers expressing the same emotion in both sentences? Examples apair coficofu

5 Exp 2: Data Analysis. 48% 4% 14% 1% 21% 1% 47% 11% 1 4% 21% 15% 42% 5% 1% 1% 18% 15% 3 1% 1 11% 5% 2 8% 8% 2 4% 11% 18% 1 19% 4% 6% 9% 1 22%... Similarity matrix for Same judgments Exp 2: Data Analysis. 2% 29% 49% 5 38% 48% 5 48% 39% 5 39% % 4 8% 5 45% 48% 31% 5 46% % 46% 46% 36% 29% 45% 49% 3 39% 44% 49% 35% 49% 48% 4 32% 41% 49% 3 35% 48% %... Similarity matrix for Different judgments Exp 2: Signal Detection Theory with with ; with Different Hit (H) False Alarm (F) Same Miss (M) Correct Rejection (C) Goal: Determine distances between emotion categories using H and F

6 Exp 2: Signal Detection Theory Different Same with H = 63 M = 17 with ; with Frequencies F = 16 C = 154 Exp 2: Signal Detection Theory Different Same with H H + M =.79 M H + M =.21 with ; with F F + C =.41 C F + C =.59 Proportions Calculate d * [6,7] H =.79; F =.41; d = 2.27 MacMillan and Creelman (1991); Kaplan, MacMillan, & Creelman (1978) Exp 2: Signal Detection Analysis d = 1 (highly similar phenomena) d = 3 (discriminable phenomena) d relatively unbiased measure of sensitivity

7 Exp 2: Signal Detection Results d values for each emotion pair How to interpret? Exp 2: Multidimensional Scaling (MDS) Visual display of perceptual distances in a multidimensional space Determine number of dimensions needed to represent the data 472 mi 151 mi 345 mi 522 mi 582 mi 965 mi Exp 2: Multidimensional Scaling Try 2 dimensions (2D) 472 mi 345 mi 472 mi mi 345 mi 522 mi 582 mi mi 582 A 2D solution is best

8 Exp 2: MDS Results Used MDS to find number of dimensions to explain 19 emotions R R 2 and stress per dimension R 2 Stre ss Stre ss Exp 2: MDS Results Emotions separated in Dimension 1 Embarrassed Love Suspicious Funny Surprised Lonely Sad Exhausted Jealous Interested Happy Exp 2: MDS Results 1D 2D 3D 4D LOW Lonely LOW LOW Funny Arousal HIGH Embarrassed Exhausted Love Sad Funny Happy Valence HIGH Jealous Funny Sad Love Question vs. Statement Sad Interested Suspicious Tension HIGH Surprised Happy

9 Conclusions and Future Work 4 dimensions Future work: Determine the acoustic correlates of the 4 dimensions Determine the emotion categories that can be perceived in speech using only suprasegmental information Derived Stimulus Configuration Euclidean distance model happ 2 funn surp anxi sadd cote inte love 1 lone cofi cofu resp exha emba suspbore Dimension 2 anno jeal -1-2 angr Dimension 1 Dimension 3 Contact Info Sona Patel - sona9@ufl.edu Department of Communication Sciences & Disorders University of Florida, FL References Ekman, P. (1993). Facial Expression and Emotion. American Psychologist, 48 (4), Johnson, S., Hierarchical Clustering Schemes. Psychometrika, 32(3), Macmillan, N. A., & Creelman, C. D. (1991). Detection Theory: A User's Guide. New York: Cambridge University Press. Kaplan, Macmillan and Creelman, (1978). Tables of d for variable-standard discrimination paradigms. Behavior Research Methods & Instrumentation, 1, Ortony, A., Turner, J.T., 199. "What's basic about basic emotions?" Psychological Review, 97(3): pp Rosenbek, J.C. Rodriguez, A, Hieber, B., Leon, S., Crucian, G., Ketterson, T., Ciampitti, M. Singletary, F., Heilman, K., Gonzalez-Rothi, L., 26. Effects of two treatments for aprosodia secondary to acquired brain injury. Journal of Rehabilitation Research & Development, 43(3),