22_15.dvi

Size: px

Start display at page:

Download "22_15.dvi"

ありさあくや
5 years ago
Views:

1 Vol. 2 No (Feb. 2009) Generating Diverse Katakana Variants via Backward- Forward Transliteration for Information Retrieval Hiroyuki Hattori, 1 Kazuhiro Seki 2 and Kuniaki Uehara 3 In Japanese, it is quite common for the same word to be written in multiple ways. This is especially true for katakana words which are typically used for transliterating foreign languages. For example, Los Angeles can be written in katakana as (rosanjerusu), (rosanzerusu), (rosuanjerusu), or (rosuanzerusu), all considered legitimate. This ambiguity becomes a critical problem for automatic processing such as information retrieval. To tackle this problem, we propose asimplebuteffectiveapproachforgeneratingkatakanavariantsforagiven katakana word based on phonemic representation of the original language for a given word. The proposed approach is first evaluated through a manual assessment of the variants it generates. It is also shown that the approach is beneficial for information retrieval when applied for query replacement, retrieving a large number of potentially relevant documents. 1. 1) Y Los Angeles 1 Google, Inc. 2 Organization of Advanced Science and Technology, Kobe University 3 Graduate School of Engineering, Kobe University 145 c 2009 Information Processing Society of Japan

2 146 2) Yahoo! Japan 1 Alisha Keys Alicia Keys ),4) 5),6) OR ) % Masuyama 8) Masuyama % 86.3% 9)

3 ) /æ/ /a/ /e/ Chandler/tSændl@/ 4 (1) (2) (3) (4) 3.2 Knight 9) Gregory 11) Knight diteeru Table 2 Table 1 1 Katakana characters and their phonetic representations. a ta ma gi bi i chi mi gu bu u tsu mu ge be e te me go bo o to mo za pa ka na ya ji pi ki ni yu zu pu ku nu yo ze pe ke ne ra zo po ko no ri da n sa ha ru ji v shi hi re zu su hu ro de se he wa do so ho ga ba 2 Compound katakana characters and their phonetic representations. di tsi chyo pyu du tse nya pyo ti tso nyu gya tu she nyo gyu si je hya gyo wi che hyu jya we kya hyo jyu wo kyu mya jyo va kyo myu dya vi shya myo dyu ve shyu rya dyo vo shyo ryu bya vyu chya ryo byu tsa chyu pya byo Knight 3 diteeru ru 1 L

4 148 Table Knight 9) A fragment of English-Japanese phonemic mappings. D d do ER aa a ar er EY ee a e IH i L r ru T t to tto UH u uu diteeru Fig. 1 Possible partitions for diteeru. r u 1 L UH Knight φ φ diteeru Fig. 2 Possible English phoneme sequences for diteeru. noisy channel model 3.2 a ER EY 1 d-i-t-ee-ru e ee 2 AH EH EY IY J = j 1...j n j i E = e 1...e n e i P (E J) Ê Ê = arg max P (E J) E (1) = arg max P (J E)P (E) E

5 149 P (J E)P (E) = P (j i e i)p (e i e i 1) (2) i P (e 1 e 0)=P (e 1) Knight 9) (2) e i 12) 1 2 P (j i e i) 8,000 Knight 9) P (e i e i 1) CMU 1 127,000 1,571 = CMU 39 2 J (2) Ê D-IH-T-EY-L 3.4 Ê J J K J K K K 2 1 Ê J K P (K ) i P (j i ê i) 1 P (K ) n 2 (2) 1 Knight n 3 EDICT 2 13,124 K 4 Table 4 Examples of katakana variants generated for 10 6 P (K ) i P (j i êi) K K Yahoo! API K K K K Infoseek 3 Yahoo! jwb/jedict.html 3

6 ) % 13.98% % 7) 17.6% Table 5 5 Individual results for quality judgment of generated katakana variants % % Google (

7 NTCIR-3 Web 13) NTCIR-3 DM2&RL1 DM2&RL1 H A ) tfidf 15) Base Base Phone EDICT Rule Rule Yahoo! API Base Phone Rule 1,000 4 R P R P Precision Recall Precision = TP TP+FP Recall = TP TP+FN TP true positive FP false positive FN false negative 4 Base Phone Rule Rule (3) N HO 3 26 Fig. 3 Twenty-six katakana queries from NTCIR-3. 4 NTCIR R P Fig. 4 R P curves for NTCIR dataset.

8 152 Rule Base P hone Rule 6 Rule Phone 1, ,000 Table 6 Precision at top 1,000 retrieved documents by query replacement. Rule Phone N HO Phone Rule Rule Phone 10 NTCIR 1 NTCIR 2 NTCIR 6 NTCIR 8 NTCIR capsaicin 1 10 NTCIR 7 C 7 6

9 Table 7 Top 10 documents retrieved by query replacement for polyphenol. Y/N 1 A Y 2 C N 3 N 4 N 5 N 6 C Y 7 C Y 8 N 9 N 10 N NTCIR 16) n n Rule Phone Phone NTCIR-3 Web NTCIR Yahoo! API ,420 2,040,000 1, ,000 12,400, Phone Rule Phone Rule 25 Rule Phone Phone Rule 100 R P 5 Phone Rule 0.2

10 154 5 Phone Rule R P Fig. 5 Comparison of R P curves for our proposed and existing approaches Rule Phone % 66.6% % 60.1 R P 1) Vol.2, pp (1983). 2) Brill, E. and Moore, R. C.: An improved error model for noisy channel spelling correction, Proc. 38th Annual Meeting of the Association for Computational Linguistics, pp (2000). 3) 44 pp (1992). 4) FleCS

155 Vol.87, No.11, pp.83 90 (1992). 5) Vol.J77-D-II, No.2, pp.380 387 (1994). 6) Vol.35, No.12, pp.2745 2750 (1994). 7) Vol.J86-D-II, No.3, pp.418 428 (2003). 8) Masuyama, T. and Nakagawa, H.

and Graehl, J.: Machine Transliteration, Computational Linguistics, Vol.24, No.4, pp.599 612 (1998). 10) (1978). 11) Gregory, G., Yan, Q. and David, A.E.

12) Frederick, J.: Statistical Methods for Speech Recognition, MITPress(1998). 13) Eguchi, K., Oyama, K., Ishida, E., Kando, N. and Kuriyama, K.

11 155 Vol.87, No.11, pp (1992). 5) Vol.J77-D-II, No.2, pp (1994). 6) Vol.35, No.12, pp (1994). 7) Vol.J86-D-II, No.3, pp (2003). 8) Masuyama, T. and Nakagawa, H.: Web-based acquisition of Japanese katakana variants, Proc. 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp (2005). 9) Knight, K. and Graehl, J.: Machine Transliteration, Computational Linguistics, Vol.24, No.4, pp (1998). 10) (1978). 11) Gregory, G., Yan, Q. and David, A.E.: Mining the Web to create a language model for mapping between English names and phrases and Japanese, Proc. IEEE/WIC/ACM International Conference on Web Intelligence, pp (2004). 12) Frederick, J.: Statistical Methods for Speech Recognition, MITPress(1998). 13) Eguchi, K., Oyama, K., Ishida, E., Kando, N. and Kuriyama, K.: Overview of the Web retrieval task at the third NTCIR workshop, Technical Report NII E, National Institute of Informatics (2003). 14) Salton, G. and McGill, M.J.: Introduction to Modern Information Retrieval, McGraw-Hill, Inc. (1983). 15) Jones, K.S.: Statistical interpretation of term specificity and its application in retrieval, Journal of Documentation, Vol.28, No.1, pp (1972). 16) Voorhees, E.M. and Harman, D.K. (Eds.): TREC: Experiment and Evaluation in Information Retrieval, The MIT Press (2005) Ph.D. ACM SIGIR AAAI ( ) ( ) ( )

ル札幌市公式ホームページガイドライン

ル札幌市公式ホームページガイドライン平成 16 年 1 月 8 日総 ) 広報部長決裁企 ) 情報化推進部長決裁最近改正平成 23 年 3 月 10 日 ...3...3...4...5...5...5...5...6...6...7...8...9...9...10...11...11...12...12...13...13...14...15...15...16...17...18...19...20