Morphological Analysis System JUMAN Copyright 2016 Kyoto University All rights reserved. Licensed under the Apache License, Version 2.0 (the Li

Size: px

Start display at page:

ゆゆこかりこめ
6 years ago
Views:

1 JUMAN++ version

2 Morphological Analysis System JUMAN Copyright 2016 Kyoto University All rights reserved. Licensed under the Apache License, Version 2.0 (the License ); you may not use this file except in compliance with the License. You may obtain a copy of the License at Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an AS IS BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Version 1.01 September 2016 Version 1.00 September 2016

3 1 1 2 JUMAN Python JUMAN JUMAN i

4 A 25 A.1 (JUMAN.grammar) A A A.2 (JUMAN.kankei) A A A.3 (JUMAN.katuyou) A A B 28 B B B B B ii

5 1 JUMAN Chasen MeCab JUMAN++ RNN(recurrent neural network) RNN [6, 7] JUMAN++ JUMAN 3 Wikipedia JUMAN++ JUMAN++ JUMAN++ CREST ( : ) CREST JUMAN++ JUMAN JUMAN nl-resource@nlp.ist.i.kyoto-u.ac.jp 1

6 2 JUMAN JUMAN++ OS: Linux ( CentOS 6.7 ) : 4GB : 2GB gcc (4.9 ) Boost C++ Libraries (1.57 ) 1 gperftool 2 libunwind 3 (gperftool 64bit ) 2.2 JUMAN++ JUMAN++ JUMAN++ % wget ( 600MB) % tar xjvf jumanpp-1.01.tar.xz % cd jumanpp-1.01 %./configure % make % sudo make install /usr/local/./configure --prefix=/path/to/somewhere/

7 2.3 JUMAN++ jumanpp UTF-8 4 # ##JUMAN % cat cake.txt # S-ID: % cat cake.txt jumanpp # S-ID: JUMAN++: * 0 * 0 " : / : - : " 9 1 * 0 * 0 NIL 2 * " : / : " -s, --specifics N N-Best 2.4 -B, --beam width Beam 4.1 (default: width = 5) --partial force-single-path 4.1 -v, --version --debug -h, --help 2.4 JUMAN (default) ID ID ID ID 4 3

8 : 6 1 * 0 * 0 " : / : : ; - : " 9 1 * 0 * 0 NIL 9 2 * 0 * 0 NIL 2 * " : / 2 * " : / 2 * " : / " * ID 0 \ \ \ 1 6 * 0 * 0 " : / " (-s) N-best JUMAN \t N-best ID ID ID ID ID ID # - : # MA-SCORE rank1: rank2: rank3: rank4: rank5: / 7 1 * 0 * 0 : : : :1;2;3;4; / 9 3 * 0 * 0 : : : : / 9 1 * 0 * 0 : : : :1;2;3; ; / 2 * : : / : : / : : : : ; / 9 2 * 0 * 0 : : : :1;2;4;5 4

9 / : : : : / 2 * : : : :1; / 2 * : : : :1; / 2 * : : : :1; / 2 * : : : : / 2 * : : : : :2 JUMAN JUMAN N-best RNN JUMAN JUMAN++ JUMAN++ script script/server.rb, script/client.rb JUMAN++ server.rb --cmd JUMAN++ TCP port 1234 $ ruby script/server.rb --cmd "jumanpp -B 5" --host host.name --port 1234 JUMAN++ client.rb --host <hostname> port 1234 $ echo " " ruby script/client.rb --host host.name --port

10 6 1 * 0 * 0 " : / : - : " 9 1 * 0 * 0 NIL 2 * " : / : " 2.6 Python python pyknp python JUMAN++ % wget % tar xvf pyknp-0.3.tar.gz % cd pyknp-0.3 % sudo python setup.py install [--prefix=path] pyknp python 2.7 pyknp python 2, python 3 sample/python juman.py #-*- encoding: utf-8 -*- from pyknp import Jumanpp import sys import codecs sys.stdin = codecs.getreader( utf_8 )(sys.stdin) sys.stdout = codecs.getwriter( utf_8 )(sys.stdout) # Use Juman++ in subprocess mode jumanpp = Jumanpp() result = jumanpp.analysis(u" ") for mrph in result.mrph_list(): print u" :%s" % (mrph.midasi) pyknp Readme KNP sample/python knp.py 6

11 2.7 JUMAN++ ##JUMAN++ ##JUMAN++ set-lattice N N-best N ##JUMAN++ set-beam width width ##JUMAN++ set-force-single-path ##JUMAN++ unset-force-single-path 7

12 3 3.1 JUMAN++ JUMAN [1] 1. (cf. JUMAN.grammar) ( ) (cf. JUMAN.kankei) 3. (cf. JUMAN.katuyou) JUMAN++ dic.bin, dic.da JUMAN 8

13 3.2.1 dic ; BNF ::= ( # ) ( # ( # )) ::= ( ) ::= ( ) ::= # # ::= ( # ) ::= ( # ) NIL ::= ( # ) NIL # # # # # # # (") \" : 2 ; : : ; : : - ; Wikipedia Wiktionary Web 9

14 JUMAN ContentW.dic 3 Noun.koyuu.dic 8 Postp.dic Suffix.dic Rendaku.dic 3 ( : ) % echo " " jumanpp 6 4 * 0 * 0 " : / : : : " 6 1 * 0 * 0 " : / : ; - : " Onomatopeia.dic ( ) Wikipedia Wikipedia.dic Wikipedia 2016/06/01 83 JUMAN Wikipedia Wikipedia :Wikipedia Wikipedia Wikipedia 10

15 Wiktionary Wiktionary.dic Wiktionary 2016/06/01 Wikipedia 2,000 Wiktionary Wiktionary :Wiktionary Web Web.dic Web 1 1 : B.1 22 B.2 12 ( ) : : : : B.3 B.4 11

16 3 : / : / B.5 ) : : : : dict-build dict-build/userdic/ dict-build install.sh --prefix /path/to/somewhere/ % make % sudo./install.sh CD-ROM Readme EUC-jp UTF

17 2. ( ) Web Web 1,000 JUMAN

18 1: : 4 JUMAN++ Recurrent Neural Network (RNN) JUMAN MeCab [6] : 1-3 gram : RNN RNN RNN B N-best N N-best --force-single-path 14

19 * 0 * 0 " : / : " 2 * " : / : : / " " : / " 2 * " : / : : : / " " : / " 12 * 0 * 0 * 0 " : / " " : / " 15

20 12 * 0 * 0 * 0 " : / " " : / " ( ) 6 7 * 0 * 0 " : " 6 7 * 0 * 0 " : " 6 7 * 0 * 0 " : " 1 2 * 0 * 0 NIL 6 7 * 0 * 0 " : " 1 2 * 0 * 0 NIL 6 7 * 0 * 0 " : " * 0 * 0 * 0 " " 2 * " : / : " 16

21 4.2.4 ( ) : * 0 * 0 " : " 9 1 * 0 * 0 NIL 2 * " : / : : : / " 4.3 JUMAN JUMAN JUMAN++ JUMAN JUMAN++ JUMAN++ 9 v ( : / v ) : JUMAN KNP : JUMAN % echo " " juman 6 1 * 0 * 0 " : / : : 6 1 * 0 * 0 " : / : : " 9 1 * 0 * 0 NIL 2 * " : / " 9 1 * 0 * 0 NIL

22 3 * " : / : : / " " : / " : JUMAN++ % echo " " jumanpp 6 1 * 0 * 0 " : / : : 6 1 * 0 * 0 " : / : : " 9 1 * 0 * 0 NIL 6 1 * 0 * 0 " : / v : " 9 1 * 0 * 0 NIL 3 * " : / : : / " " : / " JUMAN JUMAN JUMAN++ JUMAN : JUMAN % echo " " juman 2 * " : / " 5 * " " 9 2 * 0 * 0 " " " " : JUMAN++ % echo " " jumanpp 2 * " : / " 5 * NIL 9 2 * 0 * 0 NIL " : / " 18

23 JUMAN JUMAN++ : JUMAN % echo " " juman 6 7 * 0 * 0 " : " : JUMAN++ % echo " " jumanpp 6 7 * 0 * 0 " : " 19

24 解析済み Web テキスト学習学習基本モデル RNNLM タグ付きコーパス再学習本システム RNN 言語モデル ( 再学習 ) 2: 5 JUMAN++ JUMAN++ Web Exact Soft Confidence-Weighted Learning [5] [5, 6, 7] 5.1 knp. script $ cat xxxx.knp... yyyy.knp ruby script/corpus2train.rb > train.fmrp train.fmrp 5.2 JUMAN++ jumanpp --train -t, --train : -i : (default: 10) -o, --outputmodel : (default: output.mdl) -C : Exact Soft Confidence-Weighted C (default: 1.0) -P : Exact Soft Confidence-Weighted ϕ (default: 1.65) -B, --beam : (default: 5) --output-intermediate-model : 20

25 weight.mdl % jumanpp --train train.fmrp --outputmodel trained.mdl % sudo rm /usr/local/share/jumanpp/weight.mdl.map % sudo cp trained.mdl /usr/local/share/jumanpp/weight.mdl ( /usr/local/ JUMAN++ ) ITERATION: /50476 avg: loss:0 /, --output-intermediate-model partial JUMAN++ \t JUMAN++ 21

26 echo " " jumanpp 7 2 * 0 * 0 NIL 8 * 0 * 0 * 0 " : / " 6 2 * 0 * 0 " : / : " 2 * " : / : : / " " : / " $ echo " \t " jumanpp --partial 7 2 * 0 * 0 NIL 6 1 * 0 * 0 " : / : : - " 9 2 * 0 * 0 NIL 6 2 * 0 * 0 " : / : " 2 * " : / : : / " " : / " sample/part-sample.txt 5.2 % cat sample/part-sample.txt jumanpp --partial ruby script/corpus2train.rb > partial.fmrp % cat train.fmrp partial.fmrp > part_train.fmrp % jumanpp --train part_train.fmrp --outputmodel part_trained.mdl 5.4 Faster RNNLM (HS/NCE) toolkit

27 Faster RNNLM (HS/NCE) toolkit RNN Faster RNNLM toolkit _ _ _ _ _ _ _ _ _ _ _ _ _ JUMAN script % cat corpus.txt jumanpp ruby script/corpus2train.rb ruby script/fullmrp2basep.rb > data_for_lm.txt Faster RNNLM toolkit --nce --direct data for LM.txt data for LM.train Validation data for LM.valid lang.mdl, lang.mdl.nnet % faster-rnnlm --rnnlm lang.mdl --train data_for_lm.train --valid data_for_lm.valid --nce 22 --hidden direct direct-order 3 -bptt 4 --use-cuda 1 -independent Faster RNNLM toolkit toolkit readme 6 Windows(Cygwin) ( ) 23

28 [1], [2] 42, [3] NL-101, [4] JUMAN 2, [5] Jialei Wang, Peilin Zhao and Steven C.H. Hoi Exact Soft Confidence-Weighted Learning Proceedings of 29th International Conference on Machine Learning, [6] Hajime Morita, Daisuke Kawahara and Sadao Kurohashi Morphological Analysis for Unsegmented Languages using Recurrent Neural Network Language Model Proceedings of EMNLP 2015: Conference on Empirical Methods in Natural Language Processing, [7] RNN 78,

29 A BNF # # NIL A.1 (JUMAN.grammar) A.1.1 ::= ( ) ( ( ) ) ::= ( # ) ( # %) ::= ::= ( # ) ( # %) A.1.2 (( %)) ; (( ) (( ) ( ) ( ) ( ) ( ) ( ))) ; ;... ; (( ) (( ) ( ) ( ) ( %) ( %) ( %))) ; 25

30 A.2 (JUMAN.kankei) A.2.1 ::= ( ( ) ) ::= ( # ) ( # # ) ::= # # A.2.2 (( ) ( )) ; (( ) ( )) (( ) ( )) ; ; ; 26

31 A.3 (JUMAN.katuyou) A.3.1 ::= ( # ( ) ) ::= ::= ( # ) ::= # * A.3.2 ( (( * ) ( ) ( * ) ( ) ( ) ( ) ( ) ( * ) ( ) ( ) ( ) ( )) ) ( (( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( )) ) 27

32 B B.1 / / ( ) ID ID 28

33 B

35 - - ( ) ( ) ( ) ( ) ( ) ( ) - ( ) - 31

36 B.3 32

37 33

38 34

39 B.4 ( 4,000 ) Web 1,500 2,000 : : :7: : : :45: Web 150 : ( 50,000 ) : : : : : : : : : : : : : : : : : : : : : 35

40 ( 700 ) : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : B.5 : : : : : : : : 36

41 : : : : : : : : : : ( ) : 37

JUMAN++ version

JUMAN++ version 1.0 28 9 Morphological Analysis System JUMAN++ 1.0 Copyright 2016 Kyoto University All rights reserved. Licensed under the Apache License, Version 2.0 (the License ); you may not use this