Technical Papers Web-based Video Annotation and its Applications Daisuke Yamamoto Katashi Nagao Graduate School of Information Science, Nagoya University yamamoto@nagao.nuie.nagoya-u.ac.jp, http://www.nagao.nuie.nagoya-u.ac.jp/members/yamamoto.xml EcoTopia Science Institute, Nagoya University nagao@nuie.nagoya-u.ac.jp, http://www.nagao.nuie.nagoya-u.ac.jp/members/nagao.xml keywords: video annotation, Web, video retrieval, video simplification, community support system. 67 Summary In this paper, we developed a Web-based video annotation system, named ivas (intelligent Video Annotation Server). Audiences can associate any video content on the Internet with annotations. The system analyzes video content in order to acquire cut/shot information and color histograms. And it also automatically generates a Web page for editing annotations. Then, audiences can create annotation data by two methods. The first one helps the users to create text data such as person/object names, scene descriptions, and comments interactively. The second method facilitates the users associating any video fragments with their subjective impression by just clicking a mouse button. The generated annotation data are accumulated and managed by an XML database connected with ivas. We also developed some application systems based on annotations such as video retrieval, video simplification, and video-contentbased community support. One of the major advantages of our approach is easy integration of hand-coded and automatically-generated (such as color histograms and cut/shot information) annotations. Additionally, since our annotation system is open for public, we must consider some reliability or correctness of annotation data. We also developed an automatic evaluation method of annotation reliability using the users feedback. In the future, these fundamental technologies will contribute to the formation of new communities centered around video content. 1. MPEG-7 [MPEG 02] [Davis 93, Ricoh 02, Smith 00, Nagao 01, Nagao 02, Nagao 03] Web
68 20 1 G 2005 2. 3 Informedia [Wactlar 96] [ 96] [Davis 93, Ricoh 02, Smith 00, Nagao 01, Nagao 02, Nagao 03] [ 02] OVID [Oomoto 93] [ 96, 95] SceneNavi [ 01] 2 (http://www.2ch.net/) TV 1 Pradhan [Pradhan 97]
69 1 3 2 3. ivas(intelligent Video Annotation Server) 1 3 1 2 ivas ivas XML 6 Proxy Model [Hirotsu 99] 3 2 PC Web HDD TV DVD ( 3) 3 3 3 4 Web [ 92] XML
70 20 1 G 2005 1 a 56 8 8 b 31 0 2 c 72 1 3 d 29 0 3 7 3 89.3 91.7 4 4. ivas 3 [Weiss 94] stratified model 5 4 1 4 20 5 XML 2 4 2 4 3
71 2 ID Cookie 6 2 I 1 I 2...I n N(, 2 ) I k I k (t)= N(t i,m) (1) I k t i m ( 6) 6 5. A k e k (good) g k (bad) b k c k c k 1 <c k < 1 c k c k e k e k = s gk a b k + t c k (2) g k + a b k s t s + t =1 t a g all,b all a = g all b all (3) e k 1 <e k < 1 (2) p (good) G (bad) B p p = d(g + B) G a B (4) G + a B d(x) d(x) =1 exp( τ x) (5) τ τ>0 r k r k =(1 d(g k + b k )) p + d(g k + b k ) e k (6) r k r k 1 <r k < 1
72 20 1 G 2005 p = 1 6. 7 6 1 Web ( 7) [ 02] [ 03] > > > 8 6 2 [Nagao 01, Nagao 02, Nagao 03] 8 Web 9 6 3
73 3 a 19 5 33 10 56 56 b 0 30 5 30 31 c 0 00 6 12 72 d 0 20 3 50 32 9 C P j P k D C (P j,p k ) Dc(P j,p k )= (I pj I nj ) (I pk I nk )dt (7) I pj (1) P j I nj I pk I nk P k 7. ivas [ 02] 3 5 a,b,c,d 4 30 4 A 30 174 2153 239 68 B 30 160 1225 234 70 C 30 222 2132 334 61 D 30 97 1224 159 44 1 1 10 4 A B C D E 5 NHK ONLINE MEMBERS(https://members.nhk.or.jp/) 92 3 7 1 (A) (B) (C) (D) (4) 7 1 (2) s =1,t=0 4 (3) a =4
74 20 1 G 2005 10 4 (5) 0.5 τ =0.2 7 1 A >B >C D 6 153 (6) (2) 23 130 ( r k > 0.4 r k < 0.4) 17 3 153 289 7 2 26 (7) 11 11 1 11 11 (7) 7 3 5 ivas ivas
75 5 5 6 1 2 3 4 5 0 3 7 12 2 3.50 0 3 8 11 8 3.80 0 2 11 16 1 3.53 0 2 7 10 11 4.00 ivas 1 1 11 11 6 3.67 8. Web http://www.nagao.nuie.nagoya-u.ac.jp/ivas/ [Davis 93] Davis, M.: An Iconic Visual Language for Video Annotation., in Proceedings of IEEE Symposium on Visual Language, pp. 196 202 (1993) [Hirotsu 99] Hirotsu, T., Takada, T., Aoyagi, S., Sato, K., andsugawara, T.: Cmew/U-a multimedia Web annotation sharing system, in TENCON 99. Proceedings of the IEEE Region 10 Conference, Vol. 1, pp. 356 359 (1999) [MPEG 02] MPEG, : MPEG-7, MPEG-7 Consortium, http://www.mp7c.org/ (2002) [Nagao 01] Nagao, K., Shirai, Y., andsquire, K.: Semantic Annotation andtranscoding: Making Web Content More Accessible, IEEE MultiMedia, Vol. 8, No. 2, pp. 69 81 (2001) [Nagao 02] Nagao, K., Ohira, S., andyoneoka, M.: Annotation-BasedMultimedia Summarization andtranslation, in Proceedings of the Nineteenth International Conference on Computational Linguistics (COLING-02), pp. 702 708 (2002) [Nagao 03] Nagao, K.: Digital Content Annotation and Transcoding, Artech House Publishers, London (2003) [Oomoto 93] Oomoto, E. andtanaka, K.: OVID: Design and Implementation of a Video-Object Database System, IEEE Transactions on Knowledge and Data Engineering, Vol. 5, pp. 629 643 (1993) [Pradhan 97] Pradhan, S., Tajima, K., and Tanaka, K.: Querying Video Databases based on Description Substantiality andapproximations, in Proceeding of the IPSJ International Symposium on Information Systems and Technologies for Network Society, World Scientific, pp. 183 190 (1997) [Ricoh 02] Ricoh, : Movie Tool, http://www.ricoh.co.jp/src/multimedia/movietool/ (2002) [Smith 00] Smith, J. R. andlugeon, B.: A Visual Annotation Tool for Multimedia Content Description, in Proceedings of SPIE Photonics East, Internet Multimedia Management Systems (2000) [Wactlar 96] Wactlar, D., Kanade, T., Smith, A., and Stevens, M.: Intelligent Access to Digital Video: Informedia Project, IEEE Computer, Vol. 29, No. 5, pp. 140 151 (1996) [Weiss 94] Weiss, R., Duda, A., and Gifford, D. K.: Content- BasedAccess to Algebraic Video, in Proceedings of IEEE First International Conference on Multimedia Computing and Systems, Boston, MA (1994) [ 95],, 51 7, 3, pp. 301 302 (1995) [ 96],, D-II, J79-D-II, pp. 559 567 (1996) [ 01],,,,,, 43 (2001) [ 02],, 65, No. 3, pp. 95 96 (2002) [ 96],, TV, D-279 (1996) [ 92],, Vol. 33, No. 4, pp. 543 550 (1992) [ 03], http://chasen.aist-nara.ac.jp/ (2003) [ 02],,,,,,,,,,,, PRMU2002-30 June (2002) 2004 4 2 2003, 2003 1987, 1987-1991, 1991-1999, 1994, 1996-1997, 1999-2001, 2001-2002, 2002-2004, 2004