() (MeCab) 1 Juman ChaSen 2 MeCab ChaSen 1.3 MeCab MeCab OS Windows MeCab [] [Binary package for MS-Windows] [] sourceforge.net [mecab-win32] Mac OS

Size: px

Start display at page:

Download "() (MeCab) *1 Juman ChaSen *2 MeCab ChaSen 1.3 MeCab MeCab OS Windows MeCab [] [Binary package for MS-Windows] [] sourceforge.net [mecab-win32] Mac OS"

ゆきさちとく
7 years ago
Views:

1 RMeCab MeCab RMeCab MeCab RMeCab RMeCab RMeCab MeCab N-gram MeCab RMeCab 1.1 MeCab R MeCab RMeCab 1.2 ishida-m@ias.tokushima-u.ac.jp 1

4........................................... 16 2.5............................................ 19 2.6 N-gram................................................ 21 2.7.

2 () (MeCab) *1 Juman ChaSen *2 MeCab ChaSen 1.3 MeCab MeCab OS Windows MeCab [] [Binary package for MS-Windows] [] sourceforge.net [mecab-win32] Mac OS X Linux [mecab] Mac OS X Linux [mecab-ipadic] Windows Windows MeCab 0.97 mecab-0.97.exe [OK] Shift Jis MeCab C Program Files Mac OS X Unix Downloads Terminal Mac OS X DVD *1 *2 2

net [mecab-win32] Mac OS X Linux [mecab] Mac OS X Linux [mecab-ipadic] Windows Windows MeCab 0.

1 1 MeCab # # ** $ cd Downloads $ tar zxvf mecab-0.**.tar.gz $ cd mecab-0.** $./configure --with-charset=utf-8 $ make $ sudo make install # $ tar zxf mecab-ipadic-2.

3 1 1 MeCab # # ** $ cd Downloads $ tar zxvf mecab-0.**.tar.gz $ cd mecab-0.** $./configure --with-charset=utf-8 $ make $ sudo make install # $ tar zxf mecab-ipadic ****.tar.gz $ cd mecab-ipadic **** $./configure --with-charset=utf-8 $ make $ sudo make install Windows MeCab [Enter] C work test.txt res.txt 3

4 C: Program Files MeCab bin > mecab c: work test.txt > c: work res.txt test.txt res.txt 1 1 EOS,*,*,*,*,*,,,,,*,*,*,*,,,,,,*,*,*,,,,,*,*,*,*,,,,,,*,*,*,,,,,*,*,,,,,,*,*,*,,,,,,,*,*,*,*,,, 1 1 MeCab (), 1, 2, 3,,,,, EOS (end of sentence) (token) (type) MeCab 1 2 EOS,,*,*,*,*,,,,,*,*,*,*,,,,,*,*,*,*,,,,*,*,*,,,,,,*,*,*,,,,,,,*,*,*,*,,,,,*,*,*,*,,,,,*,*,*,*,,,,,*,*,*,*,,,,,*,*,,,,,,,*,*,*,*,,,

5 8 MeCab CSV () () MeCab R MeCab R R MeCab RMeCab 1.4 RMeCab RMeCab R MeCab R RMeCab RMeCab R MeCab R MeCab 2 RMeCab *1 OS RMeCab RMeCab 0.50 RMeCab 0.50.zip RMeCab 0.50.tgz, RMeCab 0.50.tar.gz.zip Windows Mac OS X.tgz Unix.tar.gz Windows RMeCabInstall.txt Windows R RMeCab *2 [1] Windows R *3 R R [] - [ zip ] RMeCab ***.zip ( 1 2)*** RMeCabInstall.txt RMeCabInstall.bat *1 *2 *3 RMeCabInstall.txt R MeCab RMeCabInstall.bat MeCab bin libmecab.dll R library RMeCab libs libmecab.dll 5

txt Windows R RMeCab *2 [1] Windows R *3 R R [] - [ zip ] RMeCab ***.zip ( 1 2)*** RMeCabInstall.txt RMeCabInstall.

RMeCabInstall.bat ( 1 3) 1 1 2 RMeCab 1 3 Mac OS X R [] - [] [CRAN] [] [install] RMeCab ***.

6 RMeCabInstall.bat ( 1 3) RMeCab 1 3 Mac OS X R [] - [] [CRAN] [] [install] RMeCab ***.tgz *** Linux R R R getwd() ** > install.packages("rmecab_0.**.tar.gz", destdir=".", repos = NULL) 6

2 RMeCab RMeCab R RMeCab Windows R [] - [ ] RMeCab ( 2 1 ) R library(rmecab) [Enter] R 2 1 RMeCab RMeCab 2 1 *1 2 1 2 1 RMeCab *2

7 2 RMeCab RMeCab R RMeCab Windows R [] - [ ] RMeCab ( 2 1 ) R library(rmecab) [Enter] R 2 1 RMeCab RMeCab 2 1 * RMeCab *2 Windows data.zip Mac OS X Unix data.tar.gz Windows data zip [] [] [] zip data data *1 *2 7

8 RMeCabC RMeCabText RMeCabDF RMeCabFreq docmatrix collocate collscores Ngram N, N-gram NgramDF N, N-gram docngram N N-gram rmsign RMeCab 2 1 RMeCab C (C:) R R getwd() 2.1 RMeCab RMeCab RMeCabText() RMeCabFreq() MeCab RMeCabC() RMeCabC() MeCab > res <- RMeCabC("") > res [[1]] 8

9 "" [[2]] "" [[3]] "" [[4]] "" #... > res[[1]] # "" > unlist(res)... "" "" "" ""... > x <- "" # > res <- RMeCabC(x) > unlist(res) R [[]] res[[1]] R unlist() ( x) RMeCabC() RMeCabC() 2 1 () > res <- RMeCabC("", 1) > unlist(res) # 9

10 "" "" "" "" > res <- RMeCabC("", 0) > unlist(res) # "" "" "" "" ( 2 ) () > res <- RMeCabC("") > res2 <- unlist(res) > res2 "" "" "" "" "" "" "" > res2[names(res2) == ""] "" "" "" "" > names(res2) == "" # [1] TRUE FALSE TRUE FALSE TRUE FALSE TRUE RMeCabC() res2 res2 names() == (TRUE) (FALSE) [] TRUE FALSE res2 TRUE which() TRUE any() > res3 <- names(res2) == "" 10

11 > res3 [1] TRUE FALSE TRUE FALSE TRUE FALSE TRUE > which(res3) [1] > any(res3) [1] TRUE which() TRUE any() TRUE TRUE RMeCabText() RMeCabText() 1 10 MeCab RMeCab data data yukiguni.txt > res <- RMeCabText("yukiguni.txt") > res [[1]] [1] "" "" "" "*" "*" [6] "*" "*" "" "" "" [[2]] [1] "" "" "" "" "*" "*" "*" "" [9] "" "" [[3]] [1] "" "" "" "*" [5] "*" "" "" "" # RMeCabFreq() RMeCabFreq() Windows Linux Mac OS X Windows > res <- RMeCabFreq("yukiguni.txt") length = 13 > res 11

txt") > res [[1]] [1] "" "" "" "*" "*" [6] "*" "*" "" "" "" [[2]] [1] "" "" "" "" "*" "*" "*" "" [9] "" "" [[3]] [1] "" ""

12 Term Info1 Info2 Freq #... res Term Info1 Info2 Freq data kumo.txt > pt1 <- proc.time() # > res <- RMeCabFreq("kumo.txt") length = 447 > pt2 <- proc.time() > # > pt2 - pt1 # MeCab Windows Mac OS X Linux RMeCabFreq() length = 447 () () 447 Linux Mac OS X 446 Linux Mac OS X MeCab OS MeCab,,*,*,*,*,,,,,*,*,,,,,,,*,*,,,,, Windows,,*,*,*,*,,,,,*,*,,*,*,,,,*,*,,*,*,*,,,,*,*,,,,, 12

13 Windows Mac OS X Linux OS OS MeCab MeCab MeCab *1 Unix OS Windows MeCab 2.2 MeCab Windows MeCab Mac OS X Linux Mecab C: Program Files MeCab bin > mecab,,,,*,*,,,,,,,*,*,,,,,*,*,,,,,,*,*,*,,,,, EOS CSV,-1,-1,1000,,,,,*,*,,, MeCab ID ID motohiro.csv C data ( "C:\data" ) ID ID -1 MeCab Windows []-[]-[ ]-[] cd MeCab bin MeCab *1 13

mecab,,,,*,*,,,,,,,*,*,,,,,*,*,,,,,,*,*,*,,,,, EOS CSV,-1,-1,1000,,,,,*,*,,, MeCab ID ID 1 2

14 C: Program Files MeCab mecab-dict-index.exe MeCab motohiro.csv \ () C: data > cd C: Program Files MeCab bin C: Program Files MeCab bin > mecab-dict-index.exe \ -d c: Program Files MeCab dic ipadic \ -u ishida.dic -f shift-jis -t shift-jis \ c: data motohiro.csv reading c: data mecabdic.csv... 1 emitting double-array: 100% ########################################### done! done mecab-dict-index.exe ishida.dic C: data MeCab C: Program Files MeCab dict dicrc Windows ([]-[]-[]-[]) userdic = C: data ishida.dic MeCab C: Program Files MeCab bin > mecab,,,,*,*,,,,,,,*,*,,,,*,*,*,,,,, EOS *1 *1 14

reading c: data mecabdic.csv... 1 emitting double-array: 100% ########################################### done! done mecab-dict-index.exe ishida.

15 2.3 R RMeCabDF() RMeCabDF() RMeCabDF() data photo.csv ID, Sex, Reply 1, F, 2, M, 3, F, 4, F, 5, M, 2 2 CSV > # > dat <- read.csv("photo.csv") > res <- RMeCabDF(dat, 3) # () > res <- RMeCabDF(dat, 3, 1) # RMeCabDF() res length(res) [[]] res res[[1]] res[[1]] 5 > res[[1]] "" "" "" "" "" 15

16 2.4 (term-document matrix) T erm doc1 doc2 doc doc1, doc2, doc3 doc1: doc2: doc3: RMeCabText() doc1 doc2 doc docmatrix() docmatrix() 1 data doc doc1.txt, doc2.txt, doc3.txt > res <- docmatrix("doc", pos = c("","","")) > res docs terms doc1.txt doc2.txt doc3.txt [[LESS-THAN-1]]

1 docmatrix() docmatrix() 1 data doc doc1.txt, doc2.txt, doc3.

17 [[TOTAL-TOKENS]] docmatrix() 1 2 pos docmatrix() minfreq [[LESS-THAN-2]] 2 docmatrix() () () 2 [[LESS-THAN-1]] 1 1 minfreq minfreq 2 [[TOTAL-TOKENS]] pos ()pos > res <- docmatrix("doc", pos = c("","")) > res docs terms doc1.txt doc2.txt doc3.txt [[LESS-THAN-1]] [[TOTAL-TOKENS]]

minfreq minfreq 2 [[TOTAL-TOKENS]] pos ()pos > res <- docmatrix("doc", pos = c("","")) > res

18 [[TOTAL-TOKENS]] 2 minfreq 1 > res <- docmatrix("doc", pos = c("",""), minfreq = 2) > res docs terms doc1.txt doc2.txt doc3.txt [[LESS-THAN-2]] [[TOTAL-TOKENS]] [[LESS-THAN-2]] 2 1 morikita > res <- docmatrix("morikita", pos = c("",""), minfreq = 2) > nrow(res) [1] 11 > res docs terms morikita1.txt morikita2.txt morikita3.txt [[LESS-THAN-2]] [[TOTAL-TOKENS]] () morikita1.txt 2 0 morikita3.txt 18

c("",""), minfreq = 2) > nrow(res) [1] 11 > res docs terms morikita1.txt morikita2.txt morikita3.

19 1 [[LESS-THAN-2]] 2.5 () CPU (local weight) (global weight) (normalization) 3 (term frequency; TF) IDF (inverse document frequency) (2002) (1999) docmatrix() docmatrix() tf (), tf2 (: logarithimic TF)tf3 (2 : binary weight) idf ()idf2 ( IDF)idf3 ( IDF)idf4 () norm () weight * tf idf > res <- docmatrix("doc", pos = c("","",""), weight = "tf*idf") > res docs 19

(inverse document frequency) (2002) (1999) 2.5.

20 terms doc1.txt doc2.txt doc3.txt doc1.txt 1 tf idf id f = log N n i + 1 N n i w i 2 idf log2(3/3) log2(3/2) + 1) log2(3/1) + 1) tf weight *norm > res <- docmatrix("doc", pos = c("","",""), weight = "tf*idf*norm") > res docs terms doc1.txt doc2.txt doc3.txt () 20

584963 tf weight *norm > res <- docmatrix("doc", pos = c("","",""), weight = "tf*idf*norm") > res docs terms doc1.txt doc2.txt doc3.txt 0.3132022 0.2563399 0.2271069 0.4964137 0.

21 8 docmatrix() (t f id f ) 2 doc1.txt = tf*idf N-gram N-gram N N [ - ] [ - ] [ - ] N 2 bi-gram () 2 3 N bi-gram bi-gram

22 Ngram() Ngram() N bi-gram N-gram Ngram() R bi-gram > res <- Ngram("yukiguni.txt") file = yukiguni.txt Ngram = 2 length = 38 > nrow(res) [1] 38 > res # Ngram Freq 1 [-] 1 2 [-] 1 3 [-] 1 4 [-] 1 5 [-] 1 6 [-] 1 # [-] 1 35 [-] 1 36 [-] 1 37 [-] 1 38 [-] 1 bi-gram > res <- Ngram("yukiguni.txt", type = 1, N = 2) file = yukiguni.txt Ngram = 2 length = 25 22

23 > nrow(res) [1] 25 > res Ngram Freq 1 [-] 1 2 [-] 1 3 [-] 1 4 [-] 1 5 [-] 1 #.. 20 [-] 1 21 [-] 1 22 [-] 1 23 [-] 1 24 [-] 1 25 [-] 1 bi-gram tri-gram tri-gram N 3 3-gram > # bi-gram > res <- Ngram("yukiguni.txt", type = 2, N = 2) file = yukiguni.txt Ngram = 2 length = 13 > nrow(res) [1] 13 > res Ngram Freq 1 [-] 2 2 [-] 3 3 [-] 2 4 [-] 3 5 [-] 2 6 [-] 2 7 [-] 1 8 [-] 1 9 [-] 6 10 [-] 1 11 [-] 1 12 [-] 1 13 [-] 2 23

24 > > # tri-bram > res <- Ngram("yukiguni.txt", type = 2, N = 3) file = yukiguni.txt Ngram = 3 length = 20 > nrow(res) [1] 20 > res Ngram Freq 1 [--] 1 2 [--] 1 3 [--] 2 4 [--] 1 5 [--] 1 # [--] 1 17 [--] 1 18 [--] 1 19 [--] 1 20 [--] 1 Ngram() type 1 N-gram > res <- Ngram("yukiguni.txt", type = 1, N = 2, pos = "") file = Ngram = 2 length = 7 > res Ngram Freq 1 [-] 1 2 [-] 1 3 [-] 1 4 [-] 1 5 [-] 1 6 [-] 1 7 [-] 1 pos = "" N-gram 24

25 N-gram N-gram RMeCab NgramDF() NgramDF() Ngram() N-gram > kekkadf <- NgramDF("yukiguni.txt", type = 1, N = 2, pos = "") file = yukiguni.txt Ngram = 2 > kekkadf Ngram1 Ngram2 Freq bi-gram (Freq) 1 Ngram() [- ] docngram() docngram() Ngram() 1 type N Ngram() data doc > res <- docngram("doc") > nrow(res) [1] 16 > res Text Ngram doc1.txt doc2.txt doc3.txt [-] [-]

26 [-] # 9 Ngram() docngram() N-gram %in% (collocation) () (node) () collocate() RMeCab collocate() 1 node () span span 3 > res <- collocate("kumo.txt", node = "", span = 3) > nrow(res) [1] 33 > res[25:33,] Term Span Total [[MORPHEMS]] [[TOKENS]]

27 Span Total 2 [[MORPHEMS]] [[TOKENS]] Span collocate() T MI T T (Barnbrook, 1996, p.97) ( - ) Church et al. (1991) ( 4 ) 4/ (4/ ) T T T (Church et al., 1991) MI MI 2 ( ) R 27

28 > log2( 4 / ((4/1808) * 10 * 3 * 2)) MI MI 1.58 (Barnbrook, 1996) MI T T MI RMeCab T MI collocate() ( res) 1 collocate() node span collscores() > res2 <- collscores(res, node = "", span = 3) > res2[25:33,] Term Span Total T MI NA NA [[MORPHEMS]] NA NA 33 [[TOKENS]] NA NA NA [[MORPHEMS]] [[TOKENS]] NA T MI

29 bi-gram, 21 collocate(), 8, 26 colscores(), 8 collscores(), 28 docmatrix(), 8, 16 docngram(), 8, 25 FALSE, 10 IDF, 19 MeCab, 2, 13 MI, 27 Ngram(), 8, 22 NgramDF(), 8 NgramDF(), 25 proc.time(), 12 RMeCab, 1, 5 RMeCabC(), 8 RMeCabDF(), 8, 15 RMeCabFreq(), 8, 11 RMeCabText(), 8, 11 rmsign(), 8 TF, 19 tri-gram, 23 TRUE, 10 T, 27 unlist(), 9, 12 MeCab, 2 RMeCab, 5, 26, 19, 19, 19, 19, 17, 1, 4, 4, 16, 4, 26, 4, 17, 4, 21 29

30 Barnbrook, Geoff (1996) Language and Computers: Edinburgh. Church, K. W., W. Gale, P. Hanks, and D. Hindle (1991) Using statistics in lexical analysis, in Using On-line Resources to Build a Lexicon: Lawrence Erlbaum, pp (2007) R S-PLUS 1 (1999) 5 - (2002) (2006) R 3 30

() (MeCab) 1 Juman ChaSen 2 MeCab ChaSen 1.3 MeCab MeCab OS Windows MeCab [] [Binary package for MS-Windows] [] sourceforge.net [mecab-win32] Mac OS

() (MeCab) *1 Juman ChaSen *2 MeCab ChaSen 1.3 MeCab MeCab OS Windows MeCab [] [Binary package for MS-Windows] [] sourceforge.net [mecab-win32] Mac OS RMeCab 2008 11 8 1 MeCab RMeCab 1 1.1.............................................. 1 1.2............................................ 1 1.3 MeCab......................................... 2 1.4 RMeCab..........................................