Vol. 48 No. 1 GTTM GTTM GTTM 3 GTTM 1 GTTM GTTM GTTM 5),6) 7) 9) 10),11) GTTM 1 GTTM 2 (1) (2) (1) GTTM (2) GTTM exgttm exgttm 4),12) 14) GTTM 1

Vol. 48 No. 1 Jan. 2007 GTTM Generative Theory of Tonal Music GTTM GTTM GTTM GTTM exgttm exgttm exgttm Grouping Structure Generator Based on Music Theory GTTM Masatoshi Hamanaka, Keiji Hirata and Satoshi Tojo This paper describes a grouping system which segments a music piece into units such as phrases or motives, based on the Generative Theory of Tonal Music (GTTM). Previous melody segmentation methods have only focused on detecting local boundaries of melodies, while the grouping analysis of GTTM aims at building a hierarchical structure including melodic repetition as well as such local boundaries. However, as the theory consists of a number of structuring rules among which the priority is not given, groups are acquired only by the ad hoc order of rule application. To solve this problem, we propose a novel computational model exgttm in which those rules are reformalized for computer implementation. The main advantage of our approach is that we attach a weight on each rule as an adjustable parameter, which enables us to assign priority to the application of rules. In this paper, we show the process of grouping analysis by exgttm, and show the experimental results. 1. Generative Theory of Tonal Music GTTM 1) PRESTO Japan Science and Technology Agency NTT NTT Communication Science Laboratories Japan Advanced Institute of Science and Technology 1 GTTM 2) 4) GTTM 3 2 284

Vol. 48 No. 1 GTTM 285 2 GTTM GTTM 3 GTTM 1 GTTM GTTM GTTM 5),6) 7) 9) 10),11) GTTM 1 GTTM 2 (1) (2) (1) GTTM (2) GTTM exgttm exgttm 4),12) 14) GTTM 15) GTTM 12) 2 20) 2 GTTM GTTM GTTM exgttm 3 exgttm 4 5 2. GTTM GTTM 4

286 Jan. 2007 (b) 2 GPR3 (a) (b) (c) (d) 4 2.1 GPR Fig. 1 1 Simple example of time-span tree. 2 Fig. 2 Grouping structure, metrical structure and timespan tree. 4 /2 /1 /2 /4 2 1 (a) <---> 1(b) 2 1 C4 GTTM 2 4 3 16) 19) 2 Grouping Preference Rule; GPR GPR GPR1 GPR2 GPR3 GPR4 GPR2 3 GPR5 GPR6 GPR7 7 GPR2 (a)/ GTTM 1 GTTM 2 2.1.1 GTTM (i) (ii) (ii) (i) (i) (ii) GTTM GPR6 GTTM (i) (ii) 2.1.2 GTTM

Vol. 48 No. 1 GTTM 287 Fig. 3 3 Simple example of conflict between rules. 2.1.3 3 GPR3a a GPR6 b 4 8 1 GPR1 GPR3a GPR6 2.2 exgttm exgttm 2.2.1 exgttm 2 exgttm 1 exgttm 3 1 GTTM GPR2a D 2a 1 0 GPR5 D 5 0 1 2 GTTM GTTM 2.1.3 GTTM exgttm extended-gttm GTTM executable-gttm GTTM 3 GTTM GPR6 GTTM exgttm 3 exgttm 2 21) 2 2.2.2 (1) 1 (2) (3) (4) (5) (6) 2 (7) (4) (5) (6) 3. exgttm 4 MusicXML 22) GroupingXML GTTM homophony

288 Jan. 2007 4 Fig. 4 Processing flow of the system. monophony 3.1 GPR R R {1, 2a, 2b, 3a, 3b, 3c, 3d, 4, 5, 6} 10 D R (i) 0 D R (i) 1 R {1, 2a, 2b, 3a, 3b, 3c, 3d, 4, 5, 6} 1 15 5 10 GPR7 time-span and prolongational stability GPR7 3.1.1 MusicXML 6 6 ρ i ι i η i δ i α i β i 6 τ i i ε i i 1 Table 1 15 Fifteen adjustable parameters. S R R {2a, 2b, 3a, 3b, 3c, 3d, 4, 5, 6} (0 S R 1) (19) (23) ˆσ GPR5 (0 ˆσ 0.1) (14) W m GPR6 (0 W m 1) (15) W l GPR6 (0 W l 1) (15) W s GPR6 i (0 W s 1) (16) T 4 GPR4 GPR 2 3 GPR4 (0 T 4 1) (13) T low (0 T low 1) (20) ˆε i i f i i υ i i 4 MIDI { τ i+1 ˆε i if τ i+1 ˆε i 0 ρ i = (1) ι i = τ i+1 τ i (2) η i = f i+1 f i (3) δ i = υ i+1 υ i (4) α i = ε i+1 τ i+1 ε i τ i ˆε i+1 τ i+1 ˆε i τ i (5) β i = ι i+1 ι i (6) 3.1.2 GPR2 3 4 GPR2 3 4 4 n 1 n 2 n 3 n 4 i i i +1 D R (i) =1D R (i) =0 GPR2a/ n 2 n 3

Vol. 48 No. 1 GTTM 289 Fig. 5 5 GPR Relationship between paremeters and GPRs. GPR3an 2 n 3 n 1 n 2 n 3 n 4 GPR3a 1 if η i 1 < η i and D 3a (i) = η i > η i+1 ) (9) Fig. 6 6 Calculation of basic parameters. n 1 n 2 n 3 n 4 GPR2a { 1 if ρ i 1 <ρ i and ρ i >ρ i+1 D 2a (i) = (7) GPR2b n 2 n 3 n 1 n 2 n 3 n 4 GPR2b { 1 if ι i 1 <ι i and ι i >ι i+1 D 2b (i) = (8) GPR3b n 2 n 3 n 1 n 2 n 3 n 4 GPR3b 1 if δ i 1 =0, δ i 0, and D 3b (i) = δ i+1 =0) (10) GPR3cn 2 n 3 n 1 n 2 n 3 n 4 GTTM 1) GPR3c 1 if α i 1 =0, α i 0, and D 3c (i) = α i+1 =0 (11) GPR3d n 2 n 3 n 1 n 2 n 3 n 4

290 Jan. 2007 GPR3d 1 if β i 1 =0, β i 0, and D 3d (i) = β i+1 =0 (12) GPR4 GPR2 3 GPR2 3 GPR4 P ρ (i) P ι (i) P η (i) P δ (i) P α (i) P β (i) GPR2a 2b 3a 3b 3c 3d T 4 0 T 4 1 GPR2a 2b 3a 3b 3c 3d 1 if max(p ρ (i),p ι (i),p η (i), D 4 (i) = P δ (i),p α (i),p β (i))>t 4 (13) ρ i /(ρ i 1 + ρ i + ρ i+1 ) P ρ (i) = if ρ i 1 + ρ i + ρ i+1 > 0 P ι (i) =ι i /(ι i 1 + ι i + ι i+1 ) η i /( η i 1 + η i + η i+1 ) P η (i) = if η i 1 + η i + η i+1 > 0 δ i /(δ i 1 + δ i + δ i+1 ) P δ (i) = if δ i 1 + δ i + δ i+1 > 0 α i /(α i 1 + α i + α i+1 ) P α (i) = if α i 1 + α i + α i+1 > 0 β i /(β i 1 + β i + β i+1 ) P β (i) = if β i 1 + β i + β i+1 > 0 3.1.3 GPR5 GPR5 symmetry GTTM 2 2 7 D 5 (i) Fig. 7 Degree of symmetry D 5 (i). D 5 (i) σ σ 4 1 7 (a) a D 5 (i) b D 5 (i) 7(b) σ 0 D 5 (i) σ { } D 5 (i) = 1 (τi τ mid ) 2 exp (14) 2πσ 2σ 2 τ mid = ε end τ start 2 start end start end D 5 (i) σ

Vol. 48 No. 1 GTTM 291 8 Fig. 8 Similarity of parallel phrases. ˆσ ˆσ (ε end τ start )=σ 3.1.4 GPR6 GPR6 i D 6 (i) 0 D 6 (i) 1 D 6 (i) i r j i j r GTTM 2 D 6 (i) W m W l W s i W m W l W s 0 1 21) 8 2 7 6 6/7 4/6 exgttm r 1 exgttm 4 1 m 1 m r [m, m + r) m + r N O P N(m) = [m, m +1) O(m, n) = [m, m +1) [n, n +1) r r 1 N(m, r) = N(m + j) j=0 r 1 O(m, n, r) = O(m + j, n + j) j=0 [m, m + r) [n, n + r) P (m, n, r) [m, m + r) [n, n + r) { O(m, n, r) G(m, n, r) = N(m, r)+n(n, r) (1 W m) } P (m, n, r) + O(m, n, r) W m r W l (15) L G(m, n, r) 1 m n L r +1 1 r L G(m, n, r) =0 r W l W l 0 1 W l > 0 r W l 0 1

292 Jan. 2007 i i b e t head(m) [m, m +1) i i tail(m) [m, m +1) beat(i) i [m, m +1) m b(i) (i = head(beat(i)) i tail(beat(i))) e(i) (i head(beat(i)) i = tail(beat(i))) t(i) (i = head(beat(i)) i = tail(beat(i))) i A(i) = G(beat(i),n,r) (1 W s ) if b(i) holds and N(n) 1 G(beat(i) r, n r, r) W s L/2 L if e(i) holds and N(n) 1 G(beat(i),n,r) (1 W s ) n=1 r=1 + G(beat(i) r, n r, r) W s if t(i) holds and N(n) 1 (16) 2 L 1/2 A(i) N(1,L) A max = max(a(1),a(2),,a(n(1,l)) D 6 (i) =A(i)/A max (17) D 6 (i) i D 6 (i) 9 3.1.5 GPR1 GPR1 GPR1 D 1 (i) =1 9 D 6 (i) Fig. 9 Degree of parallelism D 6 (i). 10 B low (i) Fig. 10 Low-level strength of boundary B low (i). D 1 (i) =0 10 B low (i) 0 1 B low (i) D 1 (i) i 1 1 if B low (i 1) B low (i), B low (i) B low (i +1), D 1 (i) = (18) and D 1 (i 1) = 0

Vol. 48 No. 1 GTTM 293 D high (i) =D low (i) B high (i) (22) D R (i) S R B high (i) = max i R ( R D R (i ) S R ) (23) R {2a, 2b, 3a, 3b, 3c, 3d, 4, 5, 6} i 5 4. Fig. 11 11 Construction of hierarchical grouping structure. B low (i) = max i D R (i) S R R ( R D R (i ) S R ) (19) R =(2a, 2b, 3a, 3b, 3c, 3d, 6) 3.2 D 1 (i) D 2a (i) D 2b (i) D 3a (i) D 3b (i) D 3c (i) D 3d (i) D 6 (i) T low i D low (i) =1 D low (i) =0 B low (i) (18) 1 if B low (i) >T low and D low (i) = D i (1) = 1 (20) 3.3 D low (i) D 1 (i) D 2a (i) D 2b (i) D 3a (i) D 3b (i) D 3c (i) D 3d (i) D 4 (i) D 5 (i) D 6 (i) B high (i) 0 1 B high (i) B low (i) D 4 (i) D 5 (i) î 11 B high (i) î = argmax i D high (i) (21) P 0 P 1 R 0 R 1 P R P = (24) R = (25) 4.1 GTTM GTTM 1 8 100 GTTM 4.1.1 100 Finale 23) MusicXML Dolet 3.1.1 MusicXML τ i ε i f i MusicXML ˆε i υ i 8 8

294 Jan. 2007 13 GroupingXML Fig. 13 GroupingXML viewer. 12 Fig. 12 GroupingXML GroupingXML. 1 0.8 ˆε i υ i 1 0.8 4.1.2 1 GroupingXML GroupingXML Xpointer 24) Xlink 25) MusicXML 12 3 GTTM 4.2 1 100 1 10 13 GroupingXML GroupingXML 10 14 GUI Fig. 14 GUI for configuring parameters. 14 GroupingXML S R (R {2a, 2b, 3a, 3b, 3c, 3d, 4, 5, 6}) W m W s W l T 4 T low 0 1 0.1 ˆσ 0.01 0.10 0.01 P 0.77 R 0.79 15 100 0.9 51 0.9 55 4.3

Vol. 48 No. 1 GTTM 295 (b) 4 5 6 7 GPR2b 4 5 D 6 (i) op. 23 17 17 (a) 15 P R Fig. 15 Histgram of precision and recall. 16 K. 331 Fig. 16 Analysis of Mozart Sonata K. 331. 4.3.1 2 2.2.1 1 GTTM K. 331 16 4 5 5 6 2 S 2a S 2b S 3a S 2a S 2b S 3a D 6 (i) (a) 17 (b) 1 18 GPR1 2 4.3.2 19 20 100 1 GPR5 2 21 3 1 3 4 6 7 9 1 1 3 4 6

296 Jan. 2007 17 op. 23 Fig. 17 Analysis of Chopin, Ballade, op. 23. 18 Fig. 18 Analysis of Bizet, L Arlesienne, Farandole. 19 Fig. 19 Numbers of groups in correct data and system outputs. 1 1 9 1 20 Fig. 20 Numbers of grouping hierarchies in correct data and system outputs. 1 6 7 9 3 1

Vol. 48 No. 1 GTTM 297 21 Fig. 21 Analysis of Tchaikovsky, Album pour enfants, Waltz. 22 2 (a) (b) Fig. 22 Analysis of two songs which has same parameter sets. (a) Tchaikovsky, The Nutcracker, March. (b) Elgar, Pomp and Circumstance Marches, op. 39. No.1 58 58 0.76 0.75 58 42 0.79 0.83 3 4.3.3 22 2 S 5 S 6 5. exgttm 3 exgttm GTTM exgttm GTTM 26)

298 Jan. 2007 Web CGI 100 GTTM GPR7 1) Lerdahl, F. and Jackendoff, R.: A Generative Theory of Tonal Music, The MIT Press, Cambridge (1983). 2) Cooper, G. and Meyer, L.B.: The Rhythmic Structure of Music, The University of Chicago Press, Chicago (1960). 3) Narmour, E.: The Analysis and Cognition of Basic Melodic Structure, The University of Chicago Press, Chicago (1990). 4) Temperley, D.: The Cognition of Basic Musical Structures, The MIT Press, Cambridge (2001). 5) GTTM Vol.43, No.2, pp.1512 1526 (1992). 6) Hirata, K. and Aoyagi, T.: Computational Music Representation on the Generative Theory of Tonal Music and the Deductive Object- Oriented Database, Computer Music Journal, Vol.27, No.3, pp.73 89 (2003). 7) Todd, N.: A Model of Expressive Timing in Tonal Music, Musical Perception, Vol.3, No.1, pp.33 58 (1985). http://staff.aist.go.jp/m.hamanaka/atta/grouping.html 8) Widmer, G.: Understanding and Learning Musical Expression, Proc. ICMC1993, pp.268 275 (1993). 9) Hirata, K. and Hiraga, R.: Ha-Hi-Hun plays Chopin s Etude, Working Notes of IJCAI-03 Workshop on Methods for Automatic Music Performance and their Applications in a Public Rendering Contest, pp.72 73 (2003). 10) GTTM 2002-MUS-46, pp.29 36 (2002). 11) Hirata, K. and Matsuda, S.: Interactive Music Summarization based on Generative Theory of Tonal Music, Journal of New Music Research, Vol.32, No.2, pp.165 177 (2003). 12) Stammen, D.R. and Pennycook, B.: Real-time Segmentation of Music using an Adaptation of Lerdahl and Jackendoff s Grouping Principles, Proc. ICMPC1994, pp.269 270 (1994). 13) Cambouropoulos, E.: The Local Boundary Detection Model (LBDM) and its application in the study of expressive timing, Proc. ICMC2001, pp.290 293 (2001). 14) Ferrand, M., Nelson, P. and Wiggins, G.: Memory and Melodic Density: A Model for Melody Segmentation, Proc. XIV CIM 2003, pp.95 98 (2003). 15) Hamanaka, M. and Hirata, K.: Applying Voronoi Diagrams in the Automatic Grouping of Polyphony, Information Technology Letters, Vol.1, No.1, pp.101 102 (2002). 16) Hamanaka, M., Hirata, K. and Tojo, S.: Automatic Generation of Grouping Structure based on the GTTM, Proc. ICMC2004, pp.141 144 (2004). 17) ATTA exgttm 2005-MUS-61, pp.19 26 (2005). 18) Hamanaka, M., Hirata, K. and Tojo, S.: Automatic Generation of Metrical Structure based on the GTTM, Proc. ICMC2005, pp.53 56 (2005). 19) Hamanaka, M., Hirata, K. and Tojo, S.: ATTA: Automatic Time-span Tree Analyzer based on Extended GTTM, Proc. ISMIR2005, pp.358 365 (2005). 20) Nord, T.: Toward Theoretical Verification: Developing a Computer Model of Lerdahl and Jackendoff s Generative Theory of Tonal Music, Ph.D. Thesis, The University of Wisconsin, Madison (1992). 21) Hewlett, W.B. (Ed.): Melodic Similarity: Concepts, Procedures, and Application, Computing

Vol. 48 No. 1 GTTM 299 in Musicology, Vol.11, The MIT press, Cambridge (1998). 22) Recordare L.L.C.: MusicXML 1.1 Tutorial (2006). http://www.recordare.com/xml/ musicxml-tutorial.pdf 23) Finale: PG Music Inc. (2006). http://www.pgmusic.com/finale.htm 24) W3C, XML Pointer Language (XPointer) (2002). http://www.w3.org/tr/xptr/ 25) W3C, XML Linking Language (XLink) Version 1.0 (2001). http://www.w3.org/tr/xlink/ 26) Heikki, V.: Lerdahl and Jackendoff Revisited (2000). http://www.cc.jyu.fi/heivalko/ articles/lehr jack.htm ( 18 5 7 ) ( 18 10 3 ) 2003 2003 2004 PD 2004 2004 2005 NICI 2001 2001 SCI 5th World Multiconference on Systemics Cybernetics and Informatics in Art 2003 2005 ICMC2005 Best Paper Award Journal of New Music Research Distinguished Paper Award 1987 NTT 1990 1993 ICOT 13 15 t-room 1981 1983 1986 1988 1995 2000 1997 1998 ACL Folli