78-2 (2002) p.172-193 1
1 4 2 4 3 5 4? 7 5 9 6 10 7 11 8 13 9 16 10 17 11 19 12 20 13 21 2
( ) ( )? 3
1 N i p i log p i i p i log p i i N i q i N i p i log q i N i p i { ( log q i ) ( log p i ) } = N i p i log p i q i > 0 KL-divergence 0 < p i < 1, 0 < q i < 1, i p i = i q i = 1 i p i log p i 1 2 0100100010100111001110101001101110011 autreasjkloiuxdecollkfdryuhhgreuiolpjhgfreasjoijnbvcxzswert N i p i log p i p i? p i n i i N = i n i p i = n i /N i? 1 () 4
1 0 1 0 0 1 0 0 0 1 0 1 0 0 1 1 1 0 0 1 1 1 0 1 0 1 0 0 1 1 0 1 1 1 0 0 1 1 2 3 4 5 i 01 00 10 00 10 10 01 11 00 11 10 10 10 01 10 11 10 01 (1) 010 010 001 010 011 100 111 010 100 110 111 001 (1) 0100 1000 1010 0111 0011 1010 1001 1011 1001 (1) 01001 00010 10011 10011 10101 00110 11100 (11) 1 2 3 2 3 0100100010100111001110101001101110011 i n i = 1 i n i = 0 0 log 0 = 0 0 log 0 + 0 log 0 +... + 1 log 1 = 0 autreasjkloiuxdecollkfdryuhhgreuiolpjhgfreasjoijnbvcxzswert 1 1 (0?) 3 2 0010 1 3 N N 1 5
( ) MDL (Minimum Description Length principle) MDL ( ) i p i log p i autreasjkloiuxdecollkfdryuhhgreuiolpjhgfreasjoijnbvcxzswert 1 autreasjkloiuxdecollkfdryuhhgreuiolpjhgfreasjoijnbvcxzswert 2 1 1 O(1) O(N) N i p i q i q i i q i i p i q i y y i < y y i+1 i i q i ( ) q i exp ( (y i θ) 2 ) 2σ 2 i, i = y i+1 y i MDL θ P θ q i = P (i θ) 6
i n i log P (i θ) + (θ ) P, θ 4? θ θ θ = θ i π i θ i log π i 4? 399 MDL 10 2 n log n π(n) 1/n π(n) =? n n log n. log n log n + log log n + log log log n... log 5 π(n) n 4 θ π(θ) (prior distribution) 5 2 2 7
MDL π(n) 1000 1000 2 10 π(n) π(n) = 1 n 2 10 π(n) π(n) 1/n Zipf 1/f n 6 2 π(n) MDL n L log L 3 2 2 1 3 1/L improper prior Lindley paradox 7 MDL 6 7 8
5 (b) (a) (b) (c) (d) 20 19 1000 999 (c) (b) (c) 8 ( )(c) overfit (d) 8? 9
6? 3 2 2 ( ) 1. 2. 3. 9 1. AIC ( ) 1. 10 AIC 11 12 9 i p i q i 10 11 AIC 12 http://tswww.ism.ac.jp/kitagawa/homepage/index.html 10
log P ( θ) - (θ ) P θ θ θ σ 2 y = f θ (x) P ({y i } θ) exp ( ) 1 2σ 2 (y i f θ (x i )) 2 1 2σ 2 (y i f θ (x i )) 2 + (θ ) i i 13 2 θ 33 4 i (y i f(x i )) 2 3 1 3 2 ( ) 2. (cross validation) 3. (MDL,BIC, ) 1. 2. 3. AIC 7? m m 2 m 13 σ 2 11
14 m m 2 m 15 254 0.99 1 m 256 m = 256 2 256 2 256 1 χ 2 254 m m 250 AIC MDL χ 2? n = 1, 2,... n 2 1 m m 1 m m 1 1 11111111... 14 m 15 12
( ) 16 ( ) 17. 18 Complexity is complicated. Complexity is a complicated matter.? 19 III 8 16 Knuth ( ) Knuth 17 18 n, m 19 13
2 DNA DNA 3 TTG, CTT, CTC, 11 20 21 20 Vol.4 No.6 pp.695-703(1989). 21 14
( Jounal of Royal Statistical Society A ) MRI 22 1 0 22 15
H.S. ( ) ( ) 9 = + = + = + AIC MDL = / = / 16
= / AIC MDL 23 10? ( )... 23 17
i p i log p i discommunication ( ) 18
11 ( ) 24? [ ] 25 26. 24 25 AIC MDL/BIC 26 19
12 27 28 (intention) 29 27 28 29 20
i p i log p i i p i i p i log p i 30 13 representation re- 31 representation presentation representation representive presentation representation presentation representation 30 31 recognition cognition 21
representation) (presentation) 32 33 34 35 36 void solid void void solid CG 37 32 33 34 35 36 (1997) 37 22
23