JUMAN++ version

Size: px

Start display at page:

Download "JUMAN++ version"

ともみあんさい
6 years ago
Views:

1 JUMAN++ version

2 Morphological Analysis System JUMAN Copyright 2016 Kyoto University All rights reserved. Licensed under the Apache License, Version 2.0 (the License ); you may not use this file except in compliance with the License. You may obtain a copy of the License at Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an AS IS BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Version 1.0 September 2016

3 1 1 2 JUMAN Python JUMAN JUMAN i

4 A 26 A.1 (JUMAN.grammar) A A A.2 (JUMAN.kankei) A A A.3 (JUMAN.katuyou) A A B 29 B B B B B ii

5 1 JUMAN Chasen MeCab JUMAN++ RNN(recurrent neural network) RNN [6, 7] JUMAN++ JUMAN 3 Wikipedia JUMAN++ JUMAN++ JUMAN++ CREST ( : ) CREST JUMAN++ JUMAN JUMAN [email protected] 1

6 2 JUMAN JUMAN++ OS: Linux ( CentOS 6.7 ) : 4GB : 2GB gcc (4.9 ) Boost C++ Libraries (1.57 ) 1 gperftool 2 libunwind 3 (gperftool 64bit ) 2.2 JUMAN++ % wget ( 1.3GB) % tar xjf jumanpp-1.0.tar.xz % cd jumanpp-1.0 % ls jumanpp-src jumanpp-resource jumanpp-manual-1.0.pdf README.md README_ja.md

7 JUMAN++ $JPPRC % export JPPRC=/usr/local/share/jumanpp-resource/ ( /usr/local/share/jumanpp-resource/ ) % sudo mv jumanpp-resource $JPPRC % cd jumanpp-src %./configure --enable-default-resource-path=$jpprc % make % sudo make install /usr/local/./configure --prefix=/path/to/somewhere/ 2.3 JUMAN++ jumanpp UTF-8 4 # ##JUMAN % cat cake.txt # S-ID: % cat cake.txt jumanpp # S-ID: JUMAN++: * 0 * 0 " : / : - : " 9 1 * 0 * 0 NIL 2 * " : / : " 4 3

8 -D, --dir path -s, --specifics N N-Best 2.4 -B, --beam width Beam 4.1 (default: width = 5) --partial force-single-path 4.1 -v, --version --debug -h, --help ( -D, --dir ) $HOME/.jumanpprc.jumanpprc % echo [ ] > $HOME/.jumanpprc 1. ( -D, --dir ) 2. $HOME/.jumanpprc 3. --enable-default-resource-path JUMAN (default) ID ID ID ID : 6 1 * 0 * 0 " : / : : ; - : " 9 1 * 0 * 0 NIL 9 2 * 0 * 0 NIL 2 * " : / 2 * " : / 2 * " : / " 4

9 * ID 0 \ \ \ 1 6 * 0 * 0 " : / " (-s) N-best JUMAN \t N-best ID ID ID ID ID ID # - : # MA-SCORE rank1: rank2: rank3: rank4: rank5: / 7 1 * 0 * 0 : : : :1;2;3;4; / 9 3 * 0 * 0 : : : : / 9 1 * 0 * 0 : : : :1;2;3; ; / 2 * : : / : : / : : : : ; / 9 2 * 0 * 0 : : : :1;2;4; / : : : : / 2 * : : : :1; / 2 * : : : :1; / 2 * 0 1 5

10 2 : : : :1; / 2 * : : : : / 2 * : : : : :2 JUMAN JUMAN N-best RNN JUMAN JUMAN++ JUMAN++ jumanpp-src/script script/server.rb, script/client.rb JUMAN++ server.rb --cmd JUMAN++ TCP port 1234 $ ruby script/server.rb --cmd "jumanpp -B 5" --host host.name --port 1234 JUMAN++ client.rb --host <hostname> port 1234 $ echo " " ruby script/client.rb --host host.name --port * 0 * 0 " : / : - : " 9 1 * 0 * 0 NIL 2 * " : / : " 2.6 Python python pyknp python JUMAN++ 6

11 % wget % unzip pyknp-0.3.zip % cd pyknp-0.3 % sudo python setup.py install [--prefix=path] pyknp python 2.7 pyknp python 2, python 3 jumanpp-src/sample/python juman.py #-*- encoding: utf-8 -*- from pyknp import Jumanpp import sys import codecs sys.stdin = codecs.getreader( utf_8 )(sys.stdin) sys.stdout = codecs.getwriter( utf_8 )(sys.stdout) # Use Juman++ in subprocess mode jumanpp = Jumanpp() result = jumanpp.analysis(u" ") for mrph in result.mrph_list(): print u" :%s" % (mrph.midasi) pyknp Readme KNP jumanpp-src/sample/python knp.py 2.7 JUMAN++ ##JUMAN++ ##JUMAN++ set-lattice N N-best N ##JUMAN++ set-beam width width 7

12 ##JUMAN++ set-force-single-path ##JUMAN++ unset-force-single-path 8

13 3 3.1 JUMAN++ JUMAN [1] 1. (cf. JUMAN.grammar) ( ) (cf. JUMAN.kankei) 3. (cf. JUMAN.katuyou) JUMAN++ dic.bin, dic.da JUMAN 9

14 3.2.1 dic ; BNF ::= ( # ) ( # ( # )) ::= ( ) ::= ( ) ::= # # ::= ( # ) ::= ( # ) NIL ::= ( # ) NIL # # # # # # # (") \" : 2 ; : : ; : : - ; Wikipedia Wiktionary Web 10

15 JUMAN ContentW.dic 3 Noun.koyuu.dic 8 Postp.dic Suffix.dic Rendaku.dic 3 ( : ) % echo " " jumanpp 6 4 * 0 * 0 " : / : : : " 6 1 * 0 * 0 " : / : ; - : " Onomatopeia.dic ( ) Wikipedia Wikipedia.dic Wikipedia 2016/06/01 83 JUMAN Wikipedia Wikipedia :Wikipedia Wikipedia Wikipedia 11

16 Wiktionary Wiktionary.dic Wiktionary 2016/06/01 Wikipedia 2,000 Wiktionary Wiktionary :Wiktionary Web Web.dic Web 1 1 : B.1 22 B.2 12 ( ) : : : : B.3 B.4 12

17 3 : / : / B.5 ) : : : : $JPPRC $JPPRC/source $JPPRC/source/userdic/ $JPPRC/source % make % sudo cp jumanpp_dic/* $JPPRC/ CD-ROM Readme EUC-jp UTF

18 2. ( ) Web Web 1,000 JUMAN

19 1: : 4 JUMAN++ Recurrent Neural Network (RNN) JUMAN MeCab [6] : 1-3 gram : RNN RNN RNN B N-best N N-best --force-single-path 15

20 * 0 * 0 " : / : " 2 * " : / : : / " " : / " 2 * " : / : : : / " " : / " 12 * 0 * 0 * 0 " : / " " : / " 16

21 12 * 0 * 0 * 0 " : / " " : / " ( ) 6 7 * 0 * 0 " : " 6 7 * 0 * 0 " : " 6 7 * 0 * 0 " : " 1 2 * 0 * 0 NIL 6 7 * 0 * 0 " : " 1 2 * 0 * 0 NIL 6 7 * 0 * 0 " : " * 0 * 0 * 0 " " 2 * " : / : " 17

22 4.2.4 ( ) : * 0 * 0 " : " 9 1 * 0 * 0 NIL 2 * " : / : : : / " 4.3 JUMAN JUMAN JUMAN++ JUMAN JUMAN++ JUMAN++ 9 v ( : / v ) : JUMAN KNP : JUMAN % echo " " juman 6 1 * 0 * 0 " : / : : 6 1 * 0 * 0 " : / : : " 9 1 * 0 * 0 NIL 2 * " : / " 9 1 * 0 * 0 NIL

23 3 * " : / : : / " " : / " : JUMAN++ % echo " " jumanpp 6 1 * 0 * 0 " : / : : 6 1 * 0 * 0 " : / : : " 9 1 * 0 * 0 NIL 6 1 * 0 * 0 " : / v : " 9 1 * 0 * 0 NIL 3 * " : / : : / " " : / " JUMAN JUMAN JUMAN++ JUMAN : JUMAN % echo " " juman 2 * " : / " 5 * " " 9 2 * 0 * 0 " " " " : JUMAN++ % echo " " jumanpp 2 * " : / " 5 * NIL 9 2 * 0 * 0 NIL " : / " 19

24 JUMAN JUMAN++ : JUMAN % echo " " juman 6 7 * 0 * 0 " : " : JUMAN++ % echo " " jumanpp 6 7 * 0 * 0 " : " 20

25 解析済み Web テキスト学習学習基本モデル RNNLM タグ付きコーパス再学習本システム RNN 言語モデル ( 再学習 ) 2: 5 JUMAN++ JUMAN++ Web Exact Soft Confidence-Weighted Learning [5] [5, 6, 7] 5.1 knp. jumanpp-src/script $ cat xxxx.knp... yyyy.knp ruby jumanpp-src/script/corpus2train.rb > train.fmrp train.fmrp 5.2 JUMAN++ jumanpp --train -t, --train : -i : (default: 10) -o, --outputmodel : (default: output.mdl) -C : Exact Soft Confidence-Weighted C (default: 1.0) -P : Exact Soft Confidence-Weighted ϕ (default: 1.65) -B, --beam : (default: 5) --output-intermediate-model : 21

26 weight.mdl % jumanpp --train train.fmrp --outputmodel trained.mdl % sudo cp trained.mdl $JPPRC/weight.mdl ITERATION: /50476 avg: loss:0 /, --output-intermediate-model partial JUMAN++ \t JUMAN++ echo " " jumanpp 7 2 * 0 * 0 NIL 22

27 8 * 0 * 0 * 0 " : / " 6 2 * 0 * 0 " : / : " 2 * " : / : : / " " : / " $ echo " \t " jumanpp --partial 7 2 * 0 * 0 NIL 6 1 * 0 * 0 " : / : : - " 9 2 * 0 * 0 NIL 6 2 * 0 * 0 " : / : " 2 * " : / : : / " " : / " jumanpp-src/sample/part-sample.txt 5.2 % cat sample/part-sample.txt jumanpp --partial ruby script/corpus2train.rb > partial.fmrp % cat train.fmrp partial.fmrp > part_train.fmrp % jumanpp --train part_train.fmrp --outputmodel part_trained.mdl 5.4 Faster RNNLM (HS/NCE) toolkit 10 Faster RNNLM (HS/NCE) toolkit RNN

28 Faster RNNLM toolkit _ _ _ _ _ _ _ _ _ _ _ _ _ JUMAN jumanpp-src/script % cat corpus.txt jumanpp ruby script/corpus2train.rb ruby script/fullmrp2basep.rb > data_for_lm.txt Faster RNNLM toolkit --nce --direct data for LM.txt data for LM.train Validation data for LM.valid lang.mdl, lang.mdl.nnet % faster-rnnlm --rnnlm lang.mdl --train data_for_lm.train --valid data_for_lm.valid --nce 22 --hidden direct direct-order 3 -bptt 4 --use-cuda 1 -independent Faster RNNLM toolkit toolkit readme 6 Windows(Cygwin) ( ) 24

29 [1], [2] 42, [3] NL-101, [4] JUMAN 2, [5] Jialei Wang, Peilin Zhao and Steven C.H. Hoi Exact Soft Confidence-Weighted Learning Proceedings of 29th International Conference on Machine Learning, [6] Hajime Morita, Daisuke Kawahara and Sadao Kurohashi Morphological Analysis for Unsegmented Languages using Recurrent Neural Network Language Model Proceedings of EMNLP 2015: Conference on Empirical Methods in Natural Language Processing, [7] RNN 78,

30 A BNF # # NIL A.1 (JUMAN.grammar) A.1.1 ::= ( ) ( ( ) ) ::= ( # ) ( # %) ::= ::= ( # ) ( # %) A.1.2 (( %)) ; (( ) (( ) ( ) ( ) ( ) ( ) ( ))) ; ;... ; (( ) (( ) ( ) ( ) ( %) ( %) ( %))) ; 26

31 A.2 (JUMAN.kankei) A.2.1 ::= ( ( ) ) ::= ( # ) ( # # ) ::= # # A.2.2 (( ) ( )) ; (( ) ( )) (( ) ( )) ; ; ; 27

32 A.3 (JUMAN.katuyou) A.3.1 ::= ( # ( ) ) ::= ::= ( # ) ::= # * A.3.2 ( (( * ) ( ) ( * ) ( ) ( ) ( ) ( ) ( * ) ( ) ( ) ( ) ( )) ) ( (( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( )) ) 28

33 B B.1 / / ( ) ID ID 29

34 B

36 - - ( ) ( ) ( ) ( ) ( ) ( ) - ( ) - 32

37 B.3 33

38 34

39 35

40 B.4 ( 4,000 ) Web 1,500 2,000 : : :7: : : :45: Web 150 : ( 50,000 ) : : : : : : : : : : : : : : : : : : : : : 36

41 ( 700 ) : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : B.5 : : : : : : : : 37

42 : : : : : : : : : : ( ) : 38