情報処理学会インタラクション 2015 IPSJ Interaction INT /3/7 1,a) 1,b) 1,c) CD Robust PCA Subharmonic Summation MIREX2014 GUI GUI A Vocal Expression Ed

Transcription

1 情報処理学会インタラクション 215 IPSJ Interaction INT15 215/3/7 1,a) 1,b) 1,c) CD Robust PCA Subharmonic Summation MIREX214 GUI GUI A Vocal Expression Editing System based on Singing Voice Separation and F Estimation for Music Recordings Ikemiya Yukara 1,a) Itoyama Katsutoshi 1,b) Yoshii Kazuyoshi 1,c) Abstract: This paper presents a novel system that enables users to edit vocal expressions of singing voices (e.g., vibrato, glissando, and kobushi) included in real-world music recordings while preserving the original accompanying sounds. Active music listening has recently gained a lot of attention for providing users with a way of modifing existing music signals as they like. In particular, editing accompanied singing voices is one of the most challenging problems. Although a promising method was proposed for directly converting the timbres of existing singing voices into those of another singer s voice, it had been infeasible to edit the characteristic patterns of singing F contours. In this paper we propose a method that significantly improves singing voice separation based on robust PCA (RPCA) and vocal F estimation based on subharmonic summation (SHS) by using the mutual dependency between these tasks (the proposed method took first place in the singing voice separation track of an international music recognition contest called MIREX 214). We developed a GUI for adding arbitrary kinds of vocal expressions to users specified regions of existing singing voices included in commercial CD recordings. 1. [1] CD MP3 1 Graduate School of Informatics, Kyoto University a) ikemiya@kuis.kyoto-u.ac.jp b) itoyama@kuis.kyoto-u.ac.jp c) yoshii@kuis.kyoto-u.ac.jp MIDI [2] [3,4] [5] 215 Information Processing Society of Japan 128

2 [6] TANDEM-STRAIGHT [7] F [8] F F [9] [1] CD GUI 1 F GUI F [11] F F F [12] [13] F Robust Principal Component Analysis (RPCA) [14] F F F MIREX214 [15] 1 *1 2. GUI (F) F 5Hz 8Hz F 3 F 2 [11] *1 ikemiya/demo/interaction215/ 215 Information Processing Society of Japan 129

3 ビブラート こぶし á ê ˆ グリスアップ グリスダウン 楽譜に指定された音高 Ä ˆ Ý ˆô ¾ â ê u ˆô ¾ 3 F 2 F F c (t) l c (t) h F F F á w 2 GUI F (small middle large) 2.3 Q Robust PCA (RPCA) [16] F GUI 3. ( 1) 2.2 F 3.2 F F F 3.1 Q Q [17] Q x(n) 215 Information Processing Society of Japan 13

4 4 X(n, k) = 1 n+ N k /2 N k { F j=n N k /2 x(j)a k(j n + N k /2) (1) a k (n) = w(n/n k ) exp( i2πnf k /f s ) N k = Q fs f k, Q = (2 1/fratio 1) 1 qrate k f k k [Hz] f s w(t) [, 1] fratio 1 qrate n 1 [msec] Q t f X(t, f) 3.2 F F F F 4 Robust PCA F Robust PCA Robust PCA (RPCA) [14] 2 minimize L + λ S 1 (subject to L + S = X) (2) X L S 1 L1 λ X RPCA L S [16] { 1 S(t, f) > L(t, f) M r (t, f) = otherwise (3) X(t, f) Subharmonic Summation F RPCA Xs rpca (t, f) Subharmonic Summation (SHS) [18] F SHS F F H(t, s) = N h n P (t, s + 12 log 2 n), (4) n=1 t s [cents] P (t, s) t s [cents] N h n n 1 SHS A *2 SHS H(t, s) F (t) F (t) = arg max H(t, s) (5) c (t) l s c (t) h c (t) l c (t) h t ([cents]) *2 replaygain.hydrogenaud.io/proposal/equal loudness.html 215 Information Processing Society of Japan 131

5 Original spectrum Original spectrum (vocal) 1 1 X(t, f) M b(t, f) M h (t, f) X s(t, f) RPCA mask Harmonic mask Masked spectrum X s(t, f) E(t, f) X shift(t, f) Estimated spectrum envelope Simply-shifted spectrum Corrected spectrum [cents] : RPCA F [cents] 6 : F RPCA F 5 F (F) [ Ht h w 2 1 < C(f) < Hh t + w 2 M h (t, f) = Ht h = F t + 12 log 2 h, 1 h H otherwise F t t F [cents] C(f) f [cents] H w [cents] RPCA X s (t, f) X m (t, f) X s (t, f) = M r (t, f)m h (t, f)x(t, f), X m (t, f) = X(t, f) X s (t, f) (6) 3.3 X s (t, f) X m (t, f) [Hz] a [cents] 12 log 2 a [cents] b b [7] ( 6) F (DAP) [19] E(t, f) m E(t, f) X shift (t, f) = A t X s (t, f m) (7) E(t, f m) A t X new (t, f) = X m (t, f m) + b X shift (t, f) (8) 3.4 Q Q [17] Q [2] 4. F 215 Information Processing Society of Japan 132

6 db RPCA RPCA-F RPCA-F-GT Ideal 1 MIREX214 NSDR [db] [22] [23] [24] [25] db 5 SDR SIR SAR NSDR 3 RPCA 25 RPCA-F RPCA-F-GT 2 Ideal SDR SIR SAR NSDR 7 MIR-1K kHz 16bits Q fratio.5 (2 bins per octave) qrate.2 1 [msec] RPCA k [16] w 12 [cents] F RPCA: RPCA [16] RPCA-F: RPCA + RPCA-F-GT: RPCA + F Ideal: Q (STFT) MIR-1K * khz BSS Eval Toolbox [21] source-to-interference ratio (SIR) sources-to-artifacts ratio (SAR) source-to-distortion ratio (SDR) Normalized SDR (NSDR) SIR SAR SDR NSDR SDR (db) *3 sites.google.com/site/unvoicedsoundseparation/mir-1k [%] GW1 RNA1 [26] c: [cents] 8 7 RPCA (RPCA) (RPCA-F) 1 MIREX 214 [15] *4 [22 26] 4.3 F 2.2 F RWC Music Database: Popular Music (RWC-MDB-P-21) [27] 94 ±c [cents] F c F 8 (a) 5 [cents] c = 1 [cents] 1 % c c = 4 [cents] 9% F F 4 F *4 music-ir.org/mirex/wiki/214:singing Voice Separation Results 215 Information Processing Society of Japan 133

7 6 5 TANDEM-STRAIGHT [cents] 9 : TANDEM-STRAIGHT [cents] 1 í d ˆô ¾ íâ d d d / JUDY AND MARY : [Hz] [cents] 7 TANDEM-STRAIGHT [7] TANDEM-STRAIGHT 9 2 [cents] TANDEM-STRAIGHT 2 [cents] TANDEM-STRAIGHT (DAP) / JUDY AND MARY Q Web [28] 5. F GUI F 4.5 F F 215 Information Processing Society of Japan 134

8 Songle [13] 6. F GUI F F [11] JSPS , JST CREST OngaCREST [1] Goto, M.: Active Music Listening Interfaces Based on Signal Processing, ICASSP (27). [2] Yoshii, K., Goto, M., Komatani, K., Ogata, T. and Okuno, H. G.: Drumix: An Audio Player with Real-time Drum-part Rearrangement Functions for Active Music Listening, IPSJ Journal (27). [3] Itoyama, K., Goto, M., Komatani, K., Ogata, T. and Okuno, H. G.: Instrument Equalizer for Query-by- Example Retrieval: Improving Sound Source Separation based on Integrated Harmonic and Inharmonic Models, ISMIR (28). [4] Fritsch, J. and Plumbley, M. D.: Score Informed Audio Source Separation using Constrained Nonnegative Matrix Factorization and Score Synthesis, ICASSP (213). [5] Rafii, Z., Germain, F. G., Sun, D. L. and Mysore, G. J.: Combining Modeling of Singing Voice and Background Music for Automatic Separation of Musical Mixtures, IS- MIR (213). [6] Saito, T. and Goto, M.: Acoustic and Perceptual Effects of Vocal Training in Amateur Male Singing, INTER- SPEECH (29). [7] Kawahara, H., Morise, M., Takahashi, T., Nisimura, R., Irino, T. and Banno, H.: Tandem-STRAIGHT: A Temporally Stable Power Spectral Representation for Periodic Signals and Applications to Interference-free Spectrum, F, and Aperiodicity Estimation, ICASSP (28). [8] Ohishi, Y., Mochihashi, D., Kameoka, H. and Kashino, K.: Mixture of Gaussian Process Experts for Predicting Sung Melodic Contour with Expressive Dynamic Fluctuations, ICASSP (214). [9] (213). [1] Fujihara, H. and Goto, M.: Concurrent Estimation of Singing Voice F and Phonemes by Using Spectral Envelopes Estimated from Polyphonic Music, ICASSP, pp (211). [11] Ikemiya, Y., Itoyama, K. and Okuno, H. G.: Transcribing Vocal Expression from Polyphonic Music, ICASSP (214). [12] Bryan, N. J. and Mysore, G. J.: An Efficient Posterior Regularized Latent Variable Model for Interactive Source Separation, ICML (213). [13] Mauch, M. Songle: (213). [14] Candes, E. J., Li, X., Ma, Y. and Wright, J.: Robust Principal Component Analysis?, J. ACM (211). [15] Downie, J. S.: The Music Information Retrieval Evaluation Exchange (25 27): A Window into Music Information Retrieval Research Acoustical Science and Technology, Vol. 29, pp (28). [16] Huang, P.-S., Chen, S. D., Smaragdis, P. and Hasegawa- Johnson, M.: Singing-Voice Separation from Monaural Recordings Using Robust Principal Component Analysis, ICASSP (212). [17] Schorkhuber, C. and Klapuri, A.: Constant-Q Transform Toolbox for Music Processing, SMC Conference (21). [18] Hermes, D. J.: Measurement of pitch by subharmonic summation, J. Acoust. Soc. Am., Vol. 83, No. 1, pp (online), DOI: / (1988). [19] El-Jaroudi, A. and Makhoul, J.: Discrete All-Pole Modeling, IEEE Trans. on Signal Proc. (1991). [2] Irino, T. and Kawahara, H.: Signal Reconstruction from Modified Auditory Wavelet Transform, IEEE Trans. on Signal Proc. (1993). [21] Vincent, E., Gribonval, R. and Févotte, C.: Performance Measurement in Blind Audio Source Separation, IEEE Trans. on Audio, Speech and Language Processing, Vol. 14, pp (26). [22] Huang, P.-S., Kim, M., Hasegawa-Johnson, M. and Smaragdis, P.: Singing-Voice Separation from Monaural Recordings using Deep Recurrent Neural Networks, ISMIR (214). [23] Yen, F., Luo, Y.-J. and Chi, T.-S.: Singing Voice Separation using Spectro-Temporal Modulation Features, IS- MIR (214). [24] Liutkus, A., Fitzgerald, D., Rafii, Z., Pardo, B. and Daudet, L.: Kernel Additive Models for Source Separation, IEEE TSP (214). [25] Rafii, Z. and Pardo, B.: Music/Voice Separation using the Similarity Matrix, ISMIR, pp (212). [26] Jeong, I.-Y. and Lee, K.: Vocal Separation from Monaural Music Using Temporal/Spectral Continuity and Sparsity Constraints, Signal Processing Letters, Vol. 21, pp (214). [27] Goto, M., Hashiguchi, H., Nishimura, T. and Oka, R.: RWC Music Database: Popular, Classical, and Jazz Music Databases, ISMIR, pp (22). [28] ac.jp/members/ikemiya/demo/interaction215/. 215 Information Processing Society of Japan 135