Vol. 51 No (Mar. 2010) Maximal Marginal Relevance MMR Support Vector Machine SVM feature-based feature-based feature-based Featur

Size: px

Start display at page:

Download "Vol. 51 No (Mar. 2010) Maximal Marginal Relevance MMR Support Vector Machine SVM feature-based feature-based feature-based Featur"

つかさうすい
5 years ago
Views:

1 Vol. 51 No (Mar. 2010) Maximal Marginal Relevance MMR Support Vector Machine SVM feature-based feature-based feature-based Feature-based 3 1 Cue Phrase for important sentences; CP CP Conditional Random Fields CRF feature-based Class Lecture Summarization Based on Important Sentence Extraction Yasuhisa Fujii, 1 Kazumasa Yamamoto, 1 Norihide Kitaoka 2 and Seiichi Nakagawa 1 This paper describes summarization methods based on important sentence extraction for the summarization of class-room lecture. First, we compare two summarization techniques; a Maximal Marginal Relevance and a feature-based method which uses a Support Vector Machine (SVM) as a classifier. We show that the latter is superior to the former. Second, we improve the feature-based summarizer by three different types of approaches. In the first approach, we propose a technique that extracts cue phrases for important sentences (CPs) that often appear in important sentences and thus can be used as a feature to the summarizer. We formulate CP extraction as a labeling problem of word sequences and use Conditional Random Fields (CRF) for labeling. The second approach presents a novel sentence extraction framework that takes into account the consecutiveness of important sentences based on the observation that important sentences tend to be extracted consecutively by human. We deal with this consecutiveness by applying this new features to a feature-based summarizer. The third approach provides a way to reduce redundancy in the summary. Experimental result shows that our method outperforms traditional sentence extraction methods using these aproaches. 1. 1),2) 3) 7) 8) 4) 6),9),10) CSJ 8),11) 13) CJLC 14) 1 Department of Information and Computer Sciences, Toyohashi University of Technology 2 Department of Media Science, Graduate School of Information Science, Nagoya University 1094 c 2010 Information Processing Society of Japan

2 1095 CSJ 8) 12) 1 MMR 6),9),10),15) feature-based 8),16) MMR feature-based SVM feature-based MMR feature-based 3 1 Cue Phrases for important sentences; CP CP 2 3 MMR 2 3 MMR feature-based Feature-based CJLC 14) , ms CJLC SPOJUS 17) HMM 2 2 CSJ 18) Accuracy 49.1% Correct 55.8% 19) 2.2 CJLC 6 CJLC 25% 3 man3/6 man3/6 16) man3/ % 25%

3 1096 ID Acc. [%] Corr. [%] 1 Table 1 Details of speech materials. man3/5 man3/5 man3/6 κ F ROUGE-4 L11M L11M L11M L11M L11M L11M L11M L11M Average man3/6 man3/ man3/5 1 κ F Rouge-4 κ =0.387 man3/5 κ =0.469 man3/5 man3/ % man3/6 2.3 Rouge 20) 21) κ κ 22) F Rouge-N 20) κ κ 2 κ = P (A) P (E) 1 P (E) (1) P (A) =A B (2) P (E) =A B (3) F F Precision Recall F -measure = Precision = M H M 2 Precision Recall, (4) Precision + Recall, Recall = M H. H H M Rouge-N Rouge-N N-gram 1 Rouge-N Rouge-N = S {Ref-Summaries} S {Ref-Summaries} Count gram N S match(gram N ) Count(gram gram N S N ) Ref-Summaries man3/6 gram N S S

4 1097 N Count match (gram N ) gram N Count(gram N ) gram N N=4 Rouge-4 3. MMR 6),9),10),15) feature-based 8),16) MMR feature-based 3.1 Maximal Marginal Relevance Carbonell Maximal Marginal Relevance MMR 23) 15) 6) MMR MMR 15) MMR 1 (1) S rk S rk S nrk tf i i tf i =(tf i,1, tf i,2,...,tf i,w ), (5) ( ) fŵ tf i,w = f w log, (6) f w f w w ŵ tf i,w Term Frequency TF D tf i,w w f w i (1) (2) (3) S max =argmax R = N = S rk = 0, S rk = φ, S nrk = {tf 1, tf 2,...,tf N }, D = S S nrk S N S S nrk {λ(sim(s, D)) (1 λ)(sim(s, S rk )) } (7) S rk = S rk S rk + S max, S rk +1 S rk = S rk {S max }, S nrk = S nrk {S max } (4) S rk RS rk (2) 1 MMR Fig. 1 Algorithm of MMR. i tf i,w =0 (2)S nrk (7) S max (7) Sim λ (3) S max (4) (2) MMR TF 1 (2) λ κ λ =0.6 κ 3.2 Feature-based Feature-based

5 ) ),16) 50% 15) 1 Repeated words 8),16) Words in slide texts Term Frequency (TF) (6) TF 16) Duration Power and F0 F0 F0 ESPS 25) Rate of Speech Pause CSJ 10 16) feature-based ChaSen 24) ) div 2 div 2 2 div = div = i Score(S i)=wx + b, (8) x w b w b SVM 27) SVM svm perf 28) (8) w b 3 4-fold MMR feature-based 25% LM11M SVM

6 1099 Table 2 2 MMR feature-based Summarization result of MMR and feature-based summarizer. 4. Feature-based Manual ASR MMR Feature-based κ F Rouge-4 κ F Rouge-4 L11M L11M L11M L11M L11M L11M L11M L11M Average L11M L11M L11M L11M L11M L11M L11M L11M Average LM11M0041 MMR feature-based MMR feature-based GMM 15) 6) MMR feature-based 29) feature-based MMR feature-based MMR 1 feature-based 1 feature-based feature-based MMR 3 feature-based feature-based Cue Phrases for important sentences; CP CP CP CP CP CP CP CP CP CP CP CP CRF CP CRF CP CP CP Conditional Random Fields: CRF Conditional Random Fields CRF 30) x y P (y x) CRF P (y x) P (y x) = exp Θ, Φ(x, y) y Y exp Θ, Φ(x, y) (9)

7 Table 3 Labeling rule. 0 1 CP 2 CP 3 CP 1 CP Fig. 2 2 Procedure of cue phrases for important sentences extraction. Θ Φ(x, y) x y A, B A B CP CP 2 CP CRF CP CP CP CP CP CP / *.* CP CP CP e 1 C I(e) Th N 2 P I(e) =C I(e)/(C I(e)+C N (e)) Th R C N (e) e Th N =10 Th R = o CP - CP Fig. 3 Labeling CPs in training data. o describes a word in a CP candidate, and - a word not in the CP candidate. 4 CRF Fig. 4 Graphical representation of CRF. CP 3 CP.* 0 1 CP CP CRF CRF 4 CRF 1 CP CP

8 1101 CRF CRF++ 1 CRF SVM 3 4-fold CP CP CRF CP CP CP CP CRF CRF CRF CRF 4.2 Feature-based feature-based man3/ % % / taku/software/crf++/ 2 CSJ 33% / i { 10 if Si 1 is extracted. dynamic(i) = (10) 01 otherwise i j diff i,j diff i,j = f j(s i) f j(s i 1). (11) repeated words (8) / S imp S imp =argmax Score(S) (12) S D S S

9 1102 subject to S imp = R D R Score(S) (8) / (12) { g0(i 1,j) g 0(i, j) =max (13) g 1(i 1,j) { g0(i 1,j 1) + score(i 0) g 1(i, j) =max (14) g 1(i 1,j 1) + score(i 1), i j g 1(i, j) g 0(i, j) i i j score(i 1) score(i 0) i (8) I R (= J/I) max(g 0(I,J),g 1(I,J)) 4.3 Feature-based MMR feature-based MMR Feature-based (5) imp i rdun(i) rdun(i) =Sim(tf i, Imp), (15) S S imp tf i if tf imp Imp = i imp otherwise, S imp S imp Sim (16) imp imp (15) (12) (12) (13) g 0(i, j) g 1(i, j) W W CP CP CP 4 CP 4 0 CP 19.9% 10.6% CP CP CP 4 κ Precision Precision Precision ) (16)

10 1103 noun * noun * noun * * noun noun noun noun * noun * noun noun noun * noun * * noun * noun * noun * noun 5 CP * CP Fig. 5 Examples of extracted CPs ( * means arbitrary words) which are extracted from the words in parenthesis. Table 4 4 CP Precision κ Important sentence extraction results based on CP extraction (Precision, κ). Trn. Precision κ CP L11M L11M L11M L11M Manual L11M L11M L11M L11M Average L11M L11M L11M L11M ASR L11M L11M L11M L11M Average Precision TF Repeated words CP CP 5 CP Table 5 Summarization result with CP feature and features which take into account consecutiveness of important sentences and redundancy. Trn. Condition κ F Rouge CP Manual CP ASR Human CP % %κ CP CP CP CP CP / CP CP *

11 1104 CP CP κ F Rouge κ F Rouge Δκ =0.013 ΔF =0.010 ΔRouge-4 = Accuracy 49.1% Correct 55.8% feature-based 2 MMR feature-based WER 50% WER 50% 4) κ F κ =0.404 F =0.560 κ =0.391 F =0.550 κ =0.469 F =0.597 Rouge Rouge 5. MMR feature-based feature-based feature-based 3 1 CP CP 2 3

12 1105 1) Glass, J., Hazen, T.J., Hetherington, L. and Wang, C.: Analysis and processing of lecture audio data; Preliminary investigations, Proc. HLT-NAACL 2004, pp.9 12 (2004). 2) Lamel, L., Adda, G., Bilinski, E. and Gauvain, J.L.: Transcribing Lectures and Seminars, Proc. Interspeech, pp.4 8 (2005). 3) SLP (2006). 4) Zhu, X. and Penn, G.: Summarization of Spontaneous Conversations, Interspeech, pp (Sep. 2006). 5) Chen, Y., Chiu, H., Wang, H. and Chen, B.: A Unified Probabilistic Generative Framework for Extractive Spoken Document Summarization, Interspeech, pp (2007). 6) Daniel, R. and Martins, D.: Extractive Summarization of Broadcast News: Comparing Strategies for European Portuguese, TSD, Vol.4629, pp , Springer (2007). 7) Vol.91-D, No.2, pp (2008). 8) Vol.12, No.6, pp.3 24 (2005). 9) Xie, S. and Liu, Y.: Using Corpus and Knowledge-based Similarity Measure in Maximum Marginal Relevance for Meeting Summarization, ICASSP, pp (2008). 10) Liu, Y. and Xie, S.: Impact of Automatic Sentence Segmentation on Meeting Summarization, ICASSP, pp (2008). 11) CSJ NLC Vol.103, No.517, pp (2003). 12) SP Vol.105, No.132, pp.1 6 (2005). 13) Kikuchi, T., Furui, S. and Hori, C.: Automatic Speech Summarization Based on Sentence Extraction and Compaction, ICASSP, pp (2003). 14) Vol.50, No.2, pp (2009). 15) Murray, G., Renal, S. and Carletta, J.: Extractive Summarization of Meeting Recording, Interspeech, pp (2005). 16) Togashi, S., Yamaguchi, M. and Nakagawa, S.: Summarization of Spoken Lectures Based on Linguistic Surface and Prosodic Information, IEEE/ACL Workshop on Spoken Language Technology, pp (2006). 17) N-best 1-best Vol.87-DII, No.3 (2004). 18) Furui, S., Maekawa, K. and Isahara, H.: A Japanese National Project on Spontaneous Speech Corpus and Processing Technology, Proc. ASR2000, pp (2000). 19) Proc. 2nd Spoken Document Processing Workshop, pp (2008). 20) Lin, C. and Hovy, E.: Automatic Evaluation of Summaries Using N-gram Co- Occurence Statistics, The Human Language Technology Conference, pp (2003). 21) Nenkova, A.: Summarization Evaluation for Text and Speech: Issues and Approaches, Interspeech, pp (2006). 22) Fleiss, J.L.: Measuring Nominal Scale Agreement Among Many Rater, Psychological Bulletin, Vol.76, pp (1971). 23) Carbonell, J. and Goldstein, J.: The Use of MMR, Diversity-Based Reranking for Reordering Documents and producing summaries, Proc. ACM SIGIR, pp (1998). 24) version (2000). 25) Entropicspeech Technology: ESPS Manual Pages (1998). au/ roberto/research/speech/local/entropic/espsdoc/manpages/indexes/ 26) Hirao, T., Isozaki, H., Maeda, E. and Matsumoto, Y.: Extracting Important Sentences with Support Vector Machines, Proc. COLING, pp (2002). 27) Vapnik, V.N.: The Nature of Statistical Learning Theory, Springer, New York, NY, USA (1995). 28) Joachims, T.: Training Linear SVMs in Linear Time, Proc. ACM Conference on Knowledge Discovery and Data Mining (KDD), pp (2006). 29) Lin, S.-H., Chen, Y.-T., Wang, H.-M. and Chen, B.: A Comparative Study of Probabilistic Ranking Models for Spoken Document Summarization, ICASSP, pp (2008). 30) Lafferty, J. and McCallum, F.P.A.: Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data, Proc. 18th International Conference on Machine Learning (2001). ( ) ( )

13 IEEE Nanyang Technologycal University Singapore Visiting Assosiate Professor ISCA IETE 13

IPSJ-TOD

IPSJ-TOD Vol. 3 No. 2 91 101 (June 2010) 1 1 1 2 1 TSC2 Automatic Evaluation of Text Summaries by Using Paraphrase Kazuho Hirahara, 1 Hidetsugu Nanba, 1 Toshiyuki Takezawa 1 and Manabu Okumura 2 The evaluation