Menu of the last lecture More on details of acoustic phonetics (continued) Characteristics of human hearing Fundamental frequency and pitch again Four

Size: px

Start display at page:

Download "Menu of the last lecture More on details of acoustic phonetics (continued) Characteristics of human hearing Fundamental frequency and pitch again Four"

ゆたかかなり
5 years ago
Views:

1 Cognitive Media Processing Nobuaki Minematsu

2 Menu of the last lecture More on details of acoustic phonetics (continued) Characteristics of human hearing Fundamental frequency and pitch again Fourier analysis of speech signals Simple hearing tests Technology for acoustic analysis of speech Source-filter model of speech production Cepstrum method to separate source and filter Advanced analysis tool of STRAIGHT Some morphing examples Spectrums/waveforms of various language sounds Vowels, semivowels, liquids, nasals, voiced fricatives, unvoiced fricatives, glottals, voiced plosives, unvoiced plosives, voiced affricatives, and unvoiced affricatives Speech recognition as spectrum reading Summary

Waveform to spectrum From waveforms to spectrums Windowing + FFT + log-amplitude Insensitivity of human ears on phase characteristics of speech Human ears are basically deaf to phase differences in

3 Waveform to spectrum From waveforms to spectrums Windowing + FFT + log-amplitude Insensitivity of human ears on phase characteristics of speech Human ears are basically deaf to phase differences in speech. It is not impossible for us to discriminate acoustically two sounds with different phase characteristics but we don t discriminate them linguistically. No language treats those two sounds as two different phonemes. Insensitivity to phase differences speech waveforms phase characteristics amplitude characteristics source characteristics lter characteristics

4 1 octave = doubling F0 Mathematical mechanism of music (scale) = y = log 10 (x)

5 Harmonic structure Speech waveforms and their log power spectrum Guitar sound waveforms and their linear power spectrum Fundamental tone + 2nd harmonic + 3rd harmonic +... Fourier series of periodical signals

6 Acoustic phonetics Spectrum of a vowel sound Resonance = concentration of the energy on specific bands that are determined only by the shape of a tube used for sound generation. Timbre = energy distribution pattern over the frequency axis

7 Fourier series and speech production Periodical signals are decomposed into Periodical signals have line shaped spectrums. Fourier series of a train of impulses A train of impulses g(t) = k= δ(t kt 0) Fourier series of a train of impulses g(t) = n= α ne jnω 0t sinusoidal waveforms α n = 1 T 0 2 T 0 g(t)e jnω0t dt = 1 T0 2 T 0 T 0 δ(t)e jnω0t dt = 1 2 T 0 T 0 2 Fourier transform of a train of impulses G(ω) = 1 T 0 n= 1 e jnω 0 t e jωt dt = 2π T 0 n= δ(ω nω 0) Vowel production as convolution of an impulse response Vocal tract (tube) functions as a filter : impulse response of Glottal source waveform : g(t), output waveform : s(t) s(t) =h(t) g(t) h(t)

8 How sounds are processed in the ears

9 Modeling of speech production Mathematical modeling of speech production -- source & filter model -- Linear independence between source and filter R(ω) G(ω) pulses H(ω) S(ω) source white noise filter

spectrums of source and filter fine structure of the spectrum

10 Modeling of vowel production Mathematical modeling of speech production -- source & filter model -- Separation between the spectrums of source and filter fine structure of the spectrum log-amplitude envelope of the spectrum + the spectrum of speech log-amplitude

Extraction of spectrum envelopes Cepstrum

spectrum with pitch harmonics Smoothing (LPF)

waveform = train of sampled data windowing

11 Extraction of spectrum envelopes Cepstrum method Windowing + FFT + log-amplitude --> a spectrum with pitch harmonics Smoothing (LPF) of the fine spectrum into its smoothed version waveform = train of sampled data windowing time log-power spectrum frequency cepstrum liftering time low quefrency band high quefrency band peak picking spectrum envelope fundamental frequency

12 Advanced technology for analysis STRAIGHT [Kawahara 06] High-quality analysis-resynthesis tool Decomposition of speech into Fundamental frequency, spectrographic representations of power, and that of periodicity High-quality speech morphing tool input speech F0 periodicity map spectrogram morph F0 periodicity map spectrogram resynthesized speech T-F coordinate T-F coordinate Spectrographic representation of power F0 adaptive complementary set of windows and spline based optimal smoothing Instantaneous frequency based F0 extraction With correlation-based F0 extraction integrated Spectrographic representation of periodicity Harmonic analysis based method

13 Advanced technology for analysis Spline-based optimum smoothing reconstructs the underlying smooth time-frequency representation.

14 Various sounds in languages

Cognitive Media Processing @ 2011-2012 Vowels Characteristics of vowels

vowels of /a/: energy distribution over a wide frequency range Back vowels

15 Cognitive Media Vowels Characteristics of vowels Front vowels of /i/ and /e/: resonance at higher frequency bands Middle vowels of /a/: energy distribution over a wide frequency range Back vowels of of /u/ and /o/: lower bands are dominant in energy distribution Unvoiced vowels

Cognitive Media Processing @ 2011-2012 Voiced

voiced plosives and unvoiced plosives /b/, /d/,

tract at a time and abrupt release of air flow

vibration --> radiation from the skin Transitional

16 Cognitive Media Voiced plosives and unvoiced plosives Characteristics of voiced plosives and unvoiced plosives /b/, /d/, /g/ /p/, /t/, /k/ Complete closure in the vocal tract at a time and abrupt release of air flow Buzz-bar: closed vocal tract + vocal fold vibration --> radiation from the skin Transitional parts from/to neighboring vowels are useful in identifying nasal sounds.

17 What are these? Hint : they are numbers. Spectrum reading This is the task that is done by a speech recognizer.

Title of each lecture Theme-1 Multimedia information and humans Multimedia information and interaction between

synesthesia - Theme-2 Speech communication technology - articulatory & acoustic phonetics - Speech communication

technology - speech synthesis - Theme-3 A new framework for human-like speech machines #1 A new framework for

18 Title of each lecture Theme-1 Multimedia information and humans Multimedia information and interaction between humans and machines Multimedia information used in expressive and emotional processing A wonder of sensation - synesthesia - Theme-2 Speech communication technology - articulatory & acoustic phonetics - Speech communication technology - speech analysis - Speech communication technology - speech recognition - Speech communication technology - speech synthesis - Theme-3 A new framework for human-like speech machines #1 A new framework for human-like speech machines #2 A new framework for human-like speech machines #3 A new framework for human-like speech machines #4 c 1 c D c 4 c 2 c 3

19 Speech Communication Tech. - Speech recognition Nobuaki Minematsu

20 Today s menu Fundamentals of speech recognition Acoustic analysis and acoustic matching Acoustic models for speech recognition From word models to subword models Speech recognition using grammars A small demo of automatic broadcast captioning

21 Speech waveforms

22 Spectrums of speech Harmonic structure --> pitch

23 Spectrums of speech F F M Spectrum - fine structures of pitch = spectrum envelope

24 Extraction of spectrum envelope spectrum vocal tract filter glottal source Analytical approximation of vocal tract filter Homomorphic analysis Linear predictive analysis

25 Simulation of vowel production using an acoustic tube

26 Resonance characteristics of an acoustic tube In the case of A1 >> A2, acoustic characteristics of the tube are calculated by treating them as two independent tubes. Simulation of vowel /i/ Helmholtz resonance frequency

27 Resonance characteristics of an acoustic tube observed predicted observed predicted

28 Spectrum

29 Waveforms --> spectrums --> sequence of feature vectors

30 Distance measure between two spectrums distance cepstrum distance

31 Dynamic Time Warping (DTW) Temporal alignment between two feature sequences

32 Dynamic Time Warping (DTW) The locally optimal paths are searched for and the globally optimal path is obtained using the local results.

33 Today s menu Fundamentals of speech recognition Acoustic analysis and acoustic matching Acoustic models for speech recognition From word models to subword models Speech recognition using grammars A small demo of automatic broadcast captioning

34 Markov Process If a signal at t = n-1 is known, the signals at t < n-1 has no effect on the signal at t = n. The signal at t = n depends only on the signal at t=n-1.

35 Hidden Markov Process transition state previous observations Observation sequence (Hidden) state sequence current state Previous observations cannot determine the current state uniquely. Signals (features) are observed but states are hidden.

36 HMM as generative model CLOSURE BURST RELEASE VOWEL Probabilistic generative model State transition is modeled as transition probability. Output features are modeled as output probability.

37 Parameters of HMM transition state Transition prob. : Output prob. : Forward prob. Backward prob.

38 Output probability of observation sequence (Trellis) S1 (a,b) = (0.7,0.3) 0.4 S2 0.8 (a,b) = (0.6,0.4) 1.0 x1.0 a b b x0.6 x0.6 x x x x0.4 x x0.2 x x0.2 x x

39 Output probability of observation sequence (Viterbi) S1 S2 1.0 x1.0 a b b x0.6 x0.6 x x x x0.4 x x0.2 x x0.2 x x The maximum likelihood path is only adopted.

40 Estimation of HMM parameters Forward prob. Backward prob. Represents association of ot with state j

41 Estimation of HMM parameters µ j =! Forward prob. # j =! j (t)" j (t) o t # t! j (t)" j (t) #! j (t)" j (t) (o t µ j )(o t µ j ) t t! j (t)" j (t) # t # t " Backward prob. state time

42 Estimation of HMM parameters When the number of training data is 1, When the number of training data is R (>1), #speakers = several thousands

43 Estimation of HMM parameters (sharing) The same value can be shared among different states. { } = S

44 Estimation of HMM parameters (embedded training) Training strategy where temporal labels are not available. A sentence HMM is built by concatenating word HMMs. In a sentence, words of the same kind are shared. { } = S

45 Recognition of isolated words path

46 Recognition of isolated words state input frame

47 Today s menu Fundamentals of speech recognition Acoustic analysis and acoustic matching Acoustic models for speech recognition From word models to subword models Speech recognition using grammars A small demo of automatic broadcast captioning

48 Phonemes The minimum units of spoken language Vowels Consonants short vowels long vowels plosives fricatives affricates semi-vowels nasals

49 Word lexicon (word dictionary) Examples required for automated call centers

50 Tree lexicon (compact representation of the words) The following words are stored as a tree.

51 Tree-based lexicon using phoneme HMMs Generation of state-based network containing all the candidate words

52 Coarticulation and context-dependent phone models Acoustic features of a specific kind of phone depends on its phonemic context. model of /k/ = *-k+* = a-k+a model of /k/ preceded by /a/ and succeeded by /i/ monophone = a-k+i trihphone a-k+i e-k+o a-k+u a-k+e a-k+o i-k+o... A phoneme is defined by referring to the left and the right context (phoneme)

53 Clustering of phonemic contexts Number of logically defined trihphones = N x N x N (N 40) Clustering of the contexts to reduce #triphones. *-a+* Context clustering is done based on phonetic attributes of the left and the right phonemes.

54 Unit of acoustic modeling word model phoneme model merit: demerit: use: merit: demerit: use: Within-word coarticulation is easy to be modeled. For new words, actual utterances are needed. #models will be easily increased. Small vocabulary speech recognition systems Easy to add new words to the system. Long coarticulation effect is ignored. Every word has to be represented as phonemic string. Large vocabulary speech recognition systems

55 Today s menu Fundamentals of speech recognition Acoustic analysis and acoustic matching Acoustic models for speech recognition From word models to subword models Speech recognition using grammars A small demo of automatic broadcast captioning

56 Continuous speech (connected word) recognition Repetitive matching between an input utterance and word sequences that are allowed by a specific language Constraints on words and their sequences (ordering) Vocabulary: a set of candidate words Syntax: how words can be concatenated to each other. Semantics: can be represented by word order?? Examples of unaccepted sentences (lexical error) (syntax error) (semantic error)

57 Representation of syntax (grammar) specific expression specific expression variable

58 Network grammar with a finite set of states A sentence is accepted if it ends at one of the final states.

59 Speech recognition using a network grammar grammatical state word HMM grammatical state word HMM When a grammatical state has more than one preceding words, the word of the maximum probability (or words with higher probabilities) is adopted and it will be connected to the following candidate words.

60 Viterbi search algorithm

61 Probabilistic decision bag A bag B Observation: You pick a ball three times. The colors are. Probabilities of P( A) and P( B) Decision: The bag used is supposed to be B.

62 Statistical framework of speech recognition A = Acoustic, W = Word P(bag ) --> P(bag=A ) or P(bag=B ) P( bag=a) : prob. of bag A s generating. P(bag) --> P(bag=A) or P(bag=B) Which bag is easier to be selected? If we have three bags of type-a and one bag of type-b, then The bag used is supposed to be A.

63 N-gram language model The most widely-used implementation of P(w) Only the previous N-1 words are used to predict the following word. (N-1)-order Markov process N-1 = 1 --> bi-gram N-1 = 2 --> tri-gram I m giving a lecture on speech recognition technology to university students. P(a I m, giving), P(lecture giving, a), P(on a, lecture), P(speech lecture, on), P(recognition on, speech),...

64 Development of a speech recognition system acoustic model input speech hypothesis generation word matching probability calculation phoneme HMM grammar language model results of recognition efficient pruning lexicon decoder

65 Today s menu Fundamentals of speech recognition Acoustic analysis and acoustic matching Acoustic models for speech recognition From word models to subword models Speech recognition using grammars A small demo of automatic broadcast captioning

66 ASR under various conditions!!!!!

67 Automatic broadcast captioning

Fig. 3 Flow diagram of image processing. Black rectangle in the photo indicates the processing area (128 x 32 pixels).

Fig. 1 The scheme of glottal area as a function of time Fig. 3 Flow diagram of image processing. Black rectangle in the photo indicates the processing area (128 x 32 pixels). Fig, 4 Parametric representation