動画コンテンツ動画 1 動画 2 動画 3 生成中の映像入力音楽選択された素片テンポによる伸縮音楽的構造 A B B B B B A C C : 4) 6) Web Web 2 2 c 2009 Information Processing S

Size: px

Start display at page:

Download "動画コンテンツ動画 1 動画 2 動画 3 生成中の映像入力音楽選択された素片テンポによる伸縮音楽的構造 A B B B B B A C C : 4) 6) Web Web 2 2 c 2009 Information Processing S"

くにひとかいじ
6 years ago
Views:

1 Web An Automatic Music Video Creation System by Reusing Dance Video Content Sora Murofushi, 1 Tomoyasu Nakano, 2 Masataka Goto 2 and Shigeo Morishima 1 This paper presents a system that automatically generates a dance video clip appropriate to music by segmenting and concatenating existing dance video clips. Although there were previous works on automatic music video creation, they did not support various associations between music and video. To model such various associations, our system uses a large amount of fan-fiction content on the web, and selects video segments appropriate to music by using linear regression models for multiple clusters. By introducing costs representing temporal continuity and music structure of the generated video clip as well as associations between music and video, this video creation problem is solved by minimizing the costs by Viterbi search. 1. 1) CG 2),3) 4) 6) 7) 8) 1 1 Waseda University 2 National Institute of Advanced Industrial Science and Technology (AIST) 1 c 2009 Information Processing Society of Japan

2 動画コンテンツ動画 1 動画 2 動画 3 生成中の映像入力音楽選択された素片テンポによる伸縮音楽的構造 A B B B B B A C C : 4) 6) Web Web 2 2 c 2009 Information Processing Society of Japan

3 楽曲音楽のフレーム特徴量抽出小節特徴量抽出 (DCT) 素片番号データベース構築フェーズ動画生成フェーズ 1 d d+1 動画像の伸縮連結動画コンテンツの出力テンポ推定... pt 音楽的構造 A A A B B ビタビ探索による素片選択 ( 時間的連続性音楽的構造の考慮 ) 2 pt pt 収集動画コンテンツ fps 画面サイズ変更リサンプリングテンポ推定再生数 C E 1 フレーム B A テンポによる選別写像音楽特徴空間各素片との距離計算動画コンテンツ群映像のフレーム特徴量抽出音楽のフレーム特徴量抽出小節特徴量抽出 (DCT) 素片分割データベース構築再生数 D 映像特徴空間 x 1 fps 2 30 fps 2 A x Web 上 d i (t) 2 B fps C 2 D fps frame per second c 2009 Information Processing Society of Japan

4 3 THE Live for You 9) Web 10) 10, kHz 60 1 khz E(t) 2 T E(t) R a (τ) R a(τ) = 1 T T (E(t) E(t + τ)). (1) t=1 R a(τ) E(t) bpm 0.5 τ 1.0 P (t + τ) E(t) R c (τ) P (t + τ) E(t) R c (τ) 1 16 Spectral Flux 2 16 khz 1 khz Spectral Flux 2 R c (τ) = 1 T 6 Zero-crossing rate MFCC 5 6 ( ) 7 8 T (E(t) P (t + τ)). (2) t=1 R c(τ) 4/ fps 2.1 n 30 fps 33 ms ),8) 11) khz khz 44.1 khz 4 c 2009 Information Processing Society of Japan

5 fps Spectral Flux 5 4 Spectral Flux S t STFT: Short-term Fourier Transform t f S(t, f) A s(t) = N 0 (S(t, f) S(t 1, f)) 2 df, (3) N Zero-crossing rate 6 MFCC Mel-Frequency Cepstral Coefficients ),8) 1 30 fps A o (t) A b (t) A o(t) = 1 P (O x(t, b k ) O x(t 1, b k )) 2 + (O y(t, b k ) O y(t 1, b k )) 2, (4) P A b (t) = 1 Q b k =0 Q B(t, b n ) B(t 1, b n ), (5) b n=0 B(t, b n) t b n bin Q bin Q = 128 O x (t, b k ) t b k O y (t, b k ) P HSV (HSB) Hue Saturation Value Brightness ) DCT: Discrete Cosign Transform 16 DCT DCT % c 2009 Information Processing Society of Japan

6 3.4.1 k-means V c w w = α log 10 (V c) β. (6) 0.5 α = 2 β = 7 10,000 w = 1 10,0000 w = RefraiD 12) RefraiD N n i n (n = 1, 2,..., N) D M t d (t,m) (t = 1, 2,..., T m, m M) d(i n, d (t,m) ) c(i n, d (t,m) ) ( d ( ) v(d(t,m), f) v f n(i n, f) ) 2 if p ch (v(d (t,m), f)) = 1 i n, d (t,m) = ch(n) = 1 ( p c v(d(t,m), f) v f n (i n, f) ) (7), 2 otherwise c ( ) d ( ) i n, d (t,m) + c(in 1, d (τ,µ) ) if µ = m, τ = (t 1) i n, d (t,m) = min s(n) s(n 1) τ,µ p t d ( (8) ) i n, d (t,m) + c(in 1, d (τ,µ) ) otherwise v(d (t,m), f) v n (i n, f) n i n p c p ch (v(d (t,m), f)) = 0 n ch(n) = 0 ch(n) n 1 p t s(n) s(n 1) p c p t N d min d min = argmin c ( ) i N, d (t,m) t,m p t = 5.0, p c = 1.0 k-means k = (9) 6 c 2009 Information Processing Society of Japan

7 元のコンテンツのシーン推定された音楽的構造 [1] Vol. 50, No. 3, pp (2009). 生成された動画のシーン t [s] [2] Goto, M.: An Audio-based Real-time Beat Tracking System for Music With or Without Drum-sounds, Journal of New Music Research, Vol. 30, No. 2, pp (2001). 3 RefraiD 12) 4. [3] D Vol. 90-D, No. 8, pp (2007). [4] Foote, J., Cooperand, M. and Girgensohn, A.: Creating music videos using automatic media analysis, Proceedings of the tenth ACM international conference on Multimedia, pp (2002). [5] Hua, X.-S., Lu, L. and Zhang, H.-J.: AVE: automated home video editing, Proceedings of the eleventh ACM international conference on Multimedia, pp (2003). [6] Hua, X.-S., Lu, L. and Zhang, H.-J.: Automatic music video generation based on temporal pattern analysis, Proceedings of the 12th annual ACM international conference on Multimedia, pp (2004). [7] 2007-MUS-069, pp (2007). [8] Gillet, O. and Richard, G.: Comparing Audio and Video Segmentations for Music Videos Indexing, Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2006), pp. V 21 V 24 (2006). [9] THE IDOLM@STER OFFICIAL WEB, [10] [11] Tzanetakis, G. and Cook, P.: Musical Genre Classification of Audio Signals, IEEE Transactions on Speech and Audio Processing, Vol. 10, No. 5, pp (2002). [12] Goto, M.: A Chorus-Section Detection Method for Musical Audio Signals and Its Application to a Music Listening Station, IEEE Transactions on Audio, Speech, and Language Processing, Vol. 14, No. 5, pp (2006). 7 c 2009 Information Processing Society of Japan

IPSJ SIG Technical Report Vol.2015-MUS-106 No.10 Vol.2015-EC-35 No /3/2 BGM 1,4,a) ,4 BGM. BGM. BGM BGM. BGM. BGM. BGM. 1.,. YouTube 201

IPSJ SIG Technical Report Vol.2015-MUS-106 No.10 Vol.2015-EC-35 No /3/2 BGM 1,4,a) ,4 BGM. BGM. BGM BGM. BGM. BGM. BGM. 1.,. YouTube 201 BGM 1,4,a) 1 2 2 3,4 BGM. BGM. BGM BGM. BGM. BGM. BGM. 1.,. YouTube 2015 1 100.. Web.. BGM.BGM [1]. BGM BGM 1 Waseda University, Shinjuku, Tokyo 169-8555, Japan 2 3 4 JST CREST a) ha-ru-ki@asagi.waseda.jp.

動画コンテンツ 動画 1 動画 2 動画 3 生成中の映像 入力音楽 選択された素片 テンポによる伸縮 音楽的構造 A B B B B B A C C : 4) 6) Web Web 2 2 c 2009 Information Processing S

動画コンテンツ動画 1 動画 2 動画 3 生成中の映像入力音楽選択された素片テンポによる伸縮音楽的構造 A B B B B B A C C : 4) 6) Web Web 2 2 c 2009 Information Processing S