MIDI 1 2 3 2 1 Modeling Performance Indeterminacies for Polyphonic Midi Score Following and Its Application to Automatic Accompaniment Nakamura Eita 1 Yamamoto Ryuichi 2 Saito Yasuyuki 3 Sako Shinji 2 Sagayama Shigeki 1 Abstract: Score following plays an important role in automatic accompaniment, which is an automated performance of accompaniment in synchrony with human performances. This paper describes the score following capable of following performances with ornaments and improvised phrases. We construct a probabilistic model of ornaments based on hidden markov model and discuss a method of describing the structure of more indeterminate improvisational phrases. A score following algorithm based on the model is proposed and its effectiveness is evaluated using human-played performances. An automatic accompaniment system using the algorithm is built and its operation is tested. Keywords: score following, automatic accompaniment, performance indeterminacies, hidden markov model, tempo estimation. 1. 1 The University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo 113 0033, Japan 2 Nagoya Institute of Technology, Gokiso-cho, Showa-ku, Nagoya 466 8555, Japan 3 KisarazuNCT, 2-11-1 Kiyomidai-Higashi, Kisarazu, Chiba 292 0041, Japan Dannenberg [1] Vercoe [2] [3] c 2012 Information Processing Society of Japan 1
Fig. 1 1 Various realizations of a trill. [4], [5] MIDI Raphael [6] Cont [4] MIDI [5] MIDI MIDI 2. 2.1 [8] a) b) c) d) e) f) a) d) [3], [5], [7], [8] e) f) 1 f) 2.2 τ t (onset time) X = {(τ i, c i )} I i=1 I i τ i i c i c i c i c i c i S = {(t m, s m )} M m=1 m t m c 2012 Information Processing Society of Japan 2
s m MIDI s m MIDI s m [8] M X S X S i M i M M τ im 3. HMM 3.1 3.2 HMM 2 p(t m, s m ) p(t m, s m i m, i m 1, t m 1 ) (δt m, s m ) = p(t m, s m i m, i m 1, t m 1 ) (1) δt m = t m t m 1 (inter-onset interval, ioi) p(i m, t m i m 1, t m 1 ) δt m S = {(t m, s m )} M m=1 p(s) = Q = p(s Q)p(Q) (2) i 1,,i m m=1 M a im 1,i m (δt m, s m ) (3) Q = {i m } M m=1 a im 1,i m = p(i m, t m i m 1, t m 1 ) a im 1,i m (δt m, s m ) Hidden Markov Model, HMM HMM ioi (δt m, s m ) = b (ioi) i m 1,i m (δt m )b (evt) i m (s m ) (4) 3.3 a im 1,i m a im 1,i m = δ im 1+1,i m 0 a im 1,i m i m = i m 1 + 2 i m = i m 1 i m < i m 1 i m = i m 1 + d d 1 a im 1,i m HMM 2 ioi c 2012 Information Processing Society of Japan 3
2 Fig. 2 HMM i i Topology of state transition probability for the performance HMM. The i th state corresponds to the i th musical action (chord, ornament etc.) of the performace score. 4. HMM 4.1 i HMM ioi ioi 35 msec [9] b (ioi) i,i (δt) δt 35 msec b (evt) i (s) s 4.2 2 2 HMM 1 *1 shake 3 ioi 30 < δt < 200 msec b (ioi) i,i (δt) b (evt) i (s) s *1 0 4.3 ioi 1 HMM 1 ( ) 1 HMM ioi 30 < δt < 100 msec b (ioi) i,i (δt) 4.4 1 HMM 5. 5.1 HMM HMM HMM ioi HMM HMM 5.2 c 2012 Information Processing Society of Japan 4
HMM HMM ( ) HMM HMM 5.3 HMM ioi 6. 6.1 3 HMM X S Q = {i m } M m=1 Bayes argmax Q = argmax i 1,,i M p(q S) = argmax [p(s Q)p(Q)] (5) Q [ M ] a im 1,i m (δt m, s m ) m=1 (6) HMM Viterbi t M 1 i M 1 ˆp im 1 = max i 1,,i M 2 ˆp im [ M 1 m=1 a im 1,i m (δt m, s m ) ] (7) = max i M 1 [ˆpiM 1 a im 1,i M b im 1,i M (δt M, s M ) ] (8) Viterbi [8] HMM 6.2 HMM r im = (τ im+1 τ im )/(t im+1 t im ) HMM 7. 7.1 [8] MIDI 3 Beethoven 3 4 Mozart 2 c 2012 Information Processing Society of Japan 5
8. 3 Fig. 3 Estimation of score location for a performance with trills. In the piano roll, the vertical lines show onsets, and the blue bold lines show onsets where the score location estimation is updated. HMM HMM MusicXML midi HMM 4 Fig. 4 Estimation of score location for a performance with arpeggi. Beethoven 7.2 Eurydice [8], [10] [1] R. Dannenberg, An on-line algorithm for real-time accompaniment, Proc. ICMC, pp. 193 198, 1984. [2] B. Vercoe, The synthetic performer in the context of live performance, Proc. ICMC, pp. 199 200, 1984. [3] N. Orio et al., Score Following: State of the Art and New Developments, in New Interfaces for Musical Expression, 2003. [4] A. Cont, ANTESCOFO: Anticipatory synchronization and control of interactive parameters in computer music, Proc. ICMC, 2008. [5] D. Schwarz, Nicola Orio and N. Schnell, Robust Polyphonic Midi Score Following with Hidden Markov Models, Proc. ICMC, 2004. [6] C. Raphael, Music Plus One: A system for expressive and flexible musical accompaniment, Proc. ICMC, 2001. [7], HMM MIDI,, MUS, pp. 109 116, 2006. [8] Eurydice:, [9] MIDI,, 48(1), pp. 237 247, 2007. [10] Eurydice:, 96, 2012. c 2012 Information Processing Society of Japan 6