Duality in Bayesian prediction and its implication

Size: px

Start display at page:

Download "Duality in Bayesian prediction and its implication"

ぜんぺいかやぬま
5 years ago
Views:

1 $\theta$ Duality in Bayesian prediction and its implication Toshio Ohnishi and Takemi Yanagimotob) a) Faculty of Economics, Kyushu University b) Department of Industrial and Systems Engineering, Chuo University \S 1. Bayes (1) Bayes $p(x;\theta)\pi(\theta;c, \delta)$ $p(x;\theta, \tau)$ $\tau$ $\lambda(c, \delta)$ (2) $\pi(\theta \tau)$ $\lambda(\tau)$ $\xi\in\xi$ Bayes $p_{\xi}(x;\theta)\pi_{\xi}(\theta)$ $\lambda(\xi)$ Bayesian model averaging (Hoeting et al., 1999). $\lambda(\xi)$ prior averaging density $m_{\xi}(x)=e[p_{\xi}(x;\theta) \pi_{\xi}(\theta)]$ $m(x)=e[m_{\xi}(x) \lambda(\xi)]$ $E[f(y) p(y)]$ $p(y)$ $f(y)$

2 105 $x$ Bayes $\lambda(\xi x)=\frac{\lambda(\xi)m_{\xi}(x)}{m(x)}$ (1.1) posterior averaging density $p(x;\theta)\pi(\theta)$ Bayes Bayes $y$ Bayes $p(y;\theta)$ $q(y x)$ $q(y x)=p(y;\hat{\theta})$ 2 $D(q(y x),$ $p(y;\theta))$ $D(p(y;\theta),$ $q(y x))$ Bayes $D(p(y),$ $q(y));=e$ $[$log{p(y)/q(y)} $ p(y)]$ Kullback-Leibler divergence e-divergence $m$-divergence (Amari &Nagaoka, 2000). 2 1 Pythagrean difference Pythagras $PD(p_{1},p_{2},p_{3}):=D(p_{1},p_{3})-D(p_{1},p_{2})-D(p_{2},p_{3})$ $E$ $[PD(p_{1},p_{2},p_{3})]$ $E[D(p_{1},p_{3})]$ $=0$ $E[D(p_{1},p_{2})]$ $p(x;\theta)$ 1 $\hat{\theta}_{m}$ (MLE) $\leq$ $\log\frac{p(x;\hat{\theta}_{m})}{p(x;\theta)}=d(p(y;\hat{\theta}_{m}), p(y;\theta))$ (1.2) (Kullback, 1959). ) $p(x;\theta)$ (3 $\hat{\theta}_{s}$ Stein (Stein, 1981) $E[\log\frac{p(x;\hat{\theta}_{S})}{p(x;\theta)}-D(p(y;\hat{\theta}_{S}), p(y;\theta)) p(x;\theta)]=0$ (1.3)

3 106 MLE MLE Stein MLE (1.2) (1.3) $e$-divergence \S 2. $e$-divergence $e$-divergence Bayes $\min_{q(y x)}e[d(q(y x), p_{\xi}(y;\theta)) \pi_{\xi}(\theta x)\lambda(\xi x)]$ (2.1) (1.1) $\pi_{\xi}(\theta x)$ $\lambda(\xi x)$ $p_{\xi}(x;\theta)\pi_{\xi}(\theta)\lambda(\xi)=\pi_{\xi}(\theta x)\lambda(\xi x)m(x)$ posterior averaging density $\pi_{\xi}(\theta x)\lambda(\xi x)$ $(\theta, \xi)$ $x$ Corcuera &Giummole (1999) Bayes $\min_{q(y x)}e[d(q(y x), p_{\xi}(y;\theta)) \pi_{\xi}(\theta x)]$ $q_{\xi}^{e}(y x)\propto\exp\{e[\log p_{\xi}(y;\theta) \pi_{\xi}(\theta x)]\}$ (2.2) Pythagras E PD $(q(y x), q_{\xi}^{e}(y x), p_{\xi}(y;\theta)) \pi_{\xi}(\theta x)]=0$ (2.3) (Yanagimoto &Ohnishi, 2009). Bayes (2.1) $\min_{q(yx)}e[d(q(y x), q_{\xi}^{e}(y x)) h(\xi)]$ (2.4)

4 107 $h(\xi)$ canonical weight (2.4) Bayes (2.1) (2.3) $\min_{q(y x)}e[d(q(y x), q_{\xi}^{e}(y x)) \lambda(\xi x)]$ posterior averaging density canonical $\lambda(\xi x)$ weight $h(\xi)$ Definition 2.1. (i) (2.2) $q_{\xi}^{e}(y x)$ $f^{e}(y x;h) :=\exp\{e[\log q_{\xi}^{e}(y x) h(\xi)]-\psi_{x}(h)\}$ (2.5) $\exp\{\psi_{x}($ $)\}$ $f^{e}(y x;h)$ canonical weight $h$ $q_{\xi}^{e}(y x)$ $e$-mixture (ii) canonical weight $h$ mean weight $t_{x}(\xi;h) :=E[\log q_{\xi}^{e}(y x) f^{e}(y x;h)]$. (2.6) $f^{e}(y x;h),$ $\psi_{x}(h)$ $t_{x}(\xi;h)$ $h$ $\log q(x x)$ Bayesian $\log$ -likelihood Bayesian $\log$ -likelihood ratio $e$-divergence Theorem 2.1. (2.5) $f^{e}(y x;h)$ Pythagras $E [PD(q(y x), f^{e}(y x;h), q_{\xi}^{e}(y x)) h(\xi)]=0$ (2.4) $E[\log\frac{f^{e}(x x;h)}{q_{\xi}^{e}(x x)}-d(f^{e}(y x;h), q_{\xi}^{e}(y x)) h(\xi)]=0$ (2.7)

5 108 (2.7) (2.4) $(h)$ $\psi$x $F$ Gateaux ( ) canonical weight $h$ $F$ ( ) $h_{1}$ $h_{2}-h_{1}$ Gateaux $\delta_{g}f(h_{1};h_{2}-$ $h_{1})$ $\delta_{g}f(h_{1};h_{2}-h_{1}) :=\lim_{\betaarrow 0}\frac{F(h_{1}+\beta(h_{2}-h_{1}))-F(h_{1})}{\beta}$ Canonical weight $h$ Gateaux Ohnishi &Yanagimoto (2013) Mean weight (2.6) $\psi_{x}(h)$ Gateaux $\delta_{g}\psi_{x}(h_{1};h_{2}-h_{1})=e[t_{x}(\xi;h_{1}) h_{2}(\xi)-h_{1}(\xi)].$ Shannon entropy $H[p(y)]:=E[-\log p(y) p(y)]$ $p(y)$ Shannon entropy Theorem 2.2. $s(\xi)=t_{x}(\xi;h)$ Shannon entropy $\max H[q(y x)]$ subject to $E[\log q_{\xi}^{e}(y x) q(y x)]=s(\xi)$ (2.4) $f^{e}(y x;h)$ $p_{1}(y),p_{2}(y)$ $p(y; \eta)=\exp\{\eta\log\frac{p_{1}(y)}{p_{2}(y)}-\psi(\eta)\}p_{2}(y)$ 2 $\mu=\psi (\eta)$

6 109 $)$ 1 Shannon entropy $\max E[-\frac{q(y)}{p_{2}(y)}\log\frac{q(y)}{p_{2}(y)} p_{2}(y)],$ subject to $E[\log\frac{p_{1}(y)}{p_{2}(y)} q(y)]=\mu.$ 2 $)$ $e$-divergence $\min_{q(y1x)}\{\eta D(q(y), p_{1}(y))+(1-\eta)d(q(y), p_{2}(y))\}.$ Bayesian $\log$ -likeliho $od$ Theorem 2.3. Canonical weight $h_{x}^{\uparrow}(\xi)$ $\delta_{g}\log f^{e}(x x;h_{x}^{1};h-h_{x}^{\dagger})=0$ for any $h$. (2.8) $f^{e}(y x;h_{x}\dagger)$ $\log\frac{f^{\epsilon}(x x;h_{x}\dagger)}{q_{\xi}^{e}(x x)}=d(f^{e}(y x;h_{x}^{\uparrow}), q_{\xi}^{e}(y x))$ for any $\xi.$ $h_{x}^{*}(\xi)$ (1.1) posterior $h_{x}^{*}(\xi)$ averaging density $:=\lambda(\xi x)$ Theorem 2. 1 $f^{e}(y x;h_{x}^{*})$ Bayes (2. 1) Theorem 2.3 $\mathcal{q}^{e}$ Theorem 2.4. $E[\log\frac{f^{e}(x x;h)}{q_{\xi}^{e}(x x)}-d(f^{e}(y x;h), q_{\xi}^{e}(y x)) \lambda(\xi x)m(x)]=0$. (2.9) (2.8) Bayesian -likelihood $h_{x}^{\uparrow}(\xi)$ $\log$ $\log f^{e}(x x;h)$ $\mathcal{q}^{e}$ $f^{e}(y x;h_{x}^{*})$ $f^{e}(y x;h_{x}^{\uparrow})$ Bayesian -likelihood $\log$ Theorem 2.3 MLE

7 110 Yanagimoto & Ohnishi (2011) (2.9) (2.4) $-\psi_{x}$ ( ) $h$ Theorem 2.5. Canonical weight $h_{x}^{c}(\xi)$ $\delta_{g}\psi_{x}(h_{x}^{c};h-h_{x}^{c})=0$ for any $h.$ $f^{e}(y x;h_{x}^{c})$ $D(f^{e}(y x;h_{x}^{c}), q_{\xi}^{e}(y x))=-\psi_{x}(h_{x}^{c})$. $f^{e}(y x;h_{x}^{c})$ $q_{\xi}^{e}(y x)$ prior averaging density $-\psi_{x}(h_{x}^{c})$ \S 3. $m$-divergence $e$-divergence $m$-divergence Bayes $\min_{q(y x)}e[d(p_{\xi}(y;\theta), q(y x)) \pi_{\xi}(\theta x)\lambda(\xi x)]$ (3.1) Shannon entropy $m$-divergence \S 2 Shannon entropy Aitchson (1975) Bayes $\min_{q(y x)}e[d(p_{\xi}(y;\theta), q(y x)) \pi_{\xi}(\theta x)]$ $q_{\xi}^{m}(y x) :=E[p_{\xi}(y;\theta) \pi_{\xi}(\theta x)]$ (3.2)

8 111 Pythagras $E[PD(p_{\xi}(y;\theta), q_{\xi}^{m}(y x), q(y x)) \pi_{\xi}(\theta x)]=0$ (3.3) (Yanagimoto &Ohnishi, 2009). \S 2 $\min_{q(y x)}e[d(q_{\xi}^{m}(y x), q(y x)) h(\xi)]$ (3.4) $h(\xi)$ canonical weight (3.4) (3.3) (3.1) $\min_{q(y1x)}e[d(q_{\xi}^{m}(y x), q(y x)) \lambda(\xi x)]$ $\lambda(\xi x)$ $h(\xi)$ Definition 3.1. (i) (3.2) $q_{\xi}^{m}(y x)$ $f^{m}(y x;h):=e[q_{\xi}^{m}(y x) h(\xi)]$ (3.5) $f^{m}(y x;h)$ canonical weight $h$ $q_{\xi}^{m}(y x)$ (ii) canonical weight $h$ $m$ -mixture entropy weight $t_{x}(\xi;h)=-\log f^{m}(x x;h)-d(q_{\xi}^{m}(y x), f^{m}(y x;h))$ (3.6) (3.4) Shannon entropy $m$-divergence Theorem 3.1. (i) (3.5) $f^{m}(y x;h)$ Pythagras $E [PD(q_{\xi}^{m}(y x), f^{m}(y x;h), q(y x)) h(\xi)]=0$ (3.7)

9 112 (3.4) $f^{m}(y x;h)$ $E[H[f^{m}(y x;h)]-h[q_{\xi}^{m}(y x)]-d(q_{\xi}^{m}(y x), f^{m}(y x;h)) h(\xi)]=0$. (3.8) $\psi_{x}$ ( ) $-\psi_{x}(h) :=H[f^{m}(y x;h)]-e[h[q_{\xi}^{m}(x x)] h(\xi)]$ - $\psi$x( ) (3.8) (3.4) $\psi$x( ) Gateaux entropy weight $\delta_{g}\psi_{x}(h_{1};h_{2}-h_{1})=e[t_{x}(\xi;h_{1}) h_{2}(\xi)-h_{1}(\xi)].$ (3.4) Theorem 3.2. $s(\xi)=t_{x}(\xi;h)$ Bayesian $\log$-likelihood $\max\log q(x x)$ $q(y x)$ subject to $-\log q(x x)-d(q_{\xi}^{m}(y x), q(y x))=s(\xi)$ (3.4) $f^{m}(y x;h)$ Shannon entropy (3.8) Theorem 3.3. Canonical weight $h_{x}\dagger(\xi)$ $\delta_{g}h[f^{m}(y x;h_{x}^{\dagger};h-h_{x}^{\uparrow})]=0$ for any $h$. (3.9) $f^{m}(y x;h_{x}\dagger)$ $H[f^{m}(y x;h_{x}^{\dagger})]-h[q_{\xi}^{m}(y x)]=d(q_{\xi}^{m}(y x), f^{m}(y x;h_{x}^{\dagger}))$ for any $\xi.$

10 113 $h_{x}^{*}(\xi)=\lambda(\xi x)$ \S 2 $\mathcal{q}^{m}$ Theorem 3.4. $E[H[f^{m}(y x;h)]-h[q_{\xi}^{m}(y x)]-d(q_{\xi}^{m}(y x),$ $f^{m}(y x;h)) \lambda(\xi x)m(x)]=0.$ (3.9) Shannon entropy $H[f^{m}(y x;h)]$ $h_{x}\dagger(\xi)$ $\mathcal{q}^{m}$ $f^{m}(y x;h_{x}^{*})$ $f^{m}(y x;h_{x}\dagger)$ (3.4) $-\psi_{x}$ ( ) $h$ Theorem 3.5. Canonical weight $h_{x}^{c}(\xi)$ $\delta_{g}\psi_{x}(h_{x}^{c};h-h_{x}^{c})=0$ for any $h.$ $f^{m}(y x;h_{x}^{c})$ $D(q_{\xi}^{m}(y x), f^{m}(y x;h_{x}^{c}))=-\psi_{x}(h_{x}^{c})$ for any $\xi.$ \S 4. \S 4.1. Mean weight entropy weight Definition 2.1 (ii) (2.6) mean weight $t_{x}(\xi;h)$ $t_{x}(\xi;h)=-h[f^{e}(y x;h)]-d(f^{e}(y x;h), q_{\xi}^{e}(y x))$

11 114 $h_{x}^{\uparrow}$ (2.8) mean weight Theorem 2.3 Corollary 4.1. Canonical weight $h_{x}^{\uparrow}$ mean weight $t_{x}(\xi;h_{x}^{\uparrow})=\log q_{\xi}^{e}(x x)-a_{x}[f^{e}(y x;h_{x}^{\dagger})]$ $A_{x}[p(y)]$ $:=\log p(x)+h[p(y)]$ $p_{\xi}(x;\theta)$ $\hat{\theta}_{m\xi}$ MLE $\log q_{\xi}^{e}(x x)=\log p_{\xi}(x;\hat{\theta}_{m\xi})-d(p_{\xi}(y;\hat{\theta}_{m\xi}), q_{\xi}^{e}(y x))$ $t_{x}(\xi;h_{x}^{\uparrow})=\log p_{\xi}(x;\hat{\theta}_{m\xi})-\{d(p_{\xi}(y;\hat{\theta}_{m\xi}), q_{\xi}^{e}(y x))+a_{x}[f^{e}(y x;h_{x}^{\uparrow})]\}$ AIC (Akaike, 1973) ( ) ( ) Definition 3.1 (ii) (3.6) entropy weight Entropy weight (3.6) Theorem 3.3 $h_{x}\dagger$ Corollary 4.2. (3.9) canonical weight entropy weight $t_{x}(\xi;h_{x}^{1})=h[q_{\xi}^{m}(y x)]-a_{x}[f^{m}(y x;h_{x}^{\uparrow})]$ $p_{\xi}(y;\theta)$ Corollary 4.2 $\theta_{m\xi}=$ $H[p_{\xi}(y;\theta)]$ $H[q_{\xi}^{m}(y x)]=h[p_{\xi}(y;\theta_{m\xi})]-d(p_{\xi}(y;\theta_{m\xi}), q_{\xi}^{m}(y x))$

12 115 $t_{x}(\xi;h_{x}^{\uparrow})=h[p_{\xi}(y;\theta_{m\xi})]-\{d(p_{\xi}(y;\theta_{m\xi}), q_{\xi}^{m}(y x))+a_{x}[f^{m}(y x;h_{x}^{\uparrow})]\}$ ( Shannon entropy) ( ) \S $\alpha-d$ ivergence $\alpha$-divergence $D_{\alpha}(p(y;\theta), q(y x)):=e[u_{\alpha}(\frac{q(y x)}{p(y;\theta)}) p(y;\theta)],$ $u_{\alpha}(r):= \frac{4}{1-\alpha^{2}}(1-r$ $)$. $-1<\alpha<1$ $\alpha$-divergence $e$-divergence $m$-divergence $u_{1}(r):=r\log r$ $u_{-1}(r):=-\log r$ $e$-divergence $\alpha=+1,$ $m$-divergence $\alpha=-1$ $\alpha$-divergence $B$ayes $\min_{q(y x)}e[d_{\alpha}(p_{\xi}(y;\theta), q(y x)) \pi_{\xi}(\theta x)\lambda(\xi x)]$ (4.1) \S 2 \S 3 Yanagimoto &Ohnishi (2009) $\min_{q(y x)}e[d_{\alpha}(q_{\xi}^{\alpha}(y x), q(y x)) h(\xi)]$ (4.2) $q_{\xi}^{\alpha}(y x)$ Bayes $\min_{q(y x)}e[d_{\alpha}(p_{\xi}(y;\theta), q(y x)) \pi_{\xi}(\theta x)]$ $q_{\xi}^{\alpha}(y x)\propto(e[\{p(y;\theta)\}^{\frac{1-\alpha}{2}} \pi_{\xi}(\theta x)])^{\frac{2}{1-\alpha}}$

13 116 (Corcuera&Giummole, 1999). (4.2) $h(\xi)$ canonical weight Definition 4.1. (i) canonical weight $h$ $q_{\xi}^{\alpha}(y x)$ $\alpha$-mixture $f^{\alpha}(y x;h):= \frac{1}{c_{x}(h)}(e[\{q_{\xi}^{\alpha}(y x)\}^{\frac{1-\alpha}{2}} h(\xi)])^{\frac{2}{1-\alpha}}$ (4.3) $c_{x}(h)$ (ii) canonical weight $h$ divergence weight $t_{x}(\xi;h)=u_{\alpha}(f^{\alpha}(x x;h))-d_{\alpha}(q_{\xi}^{\alpha}(y x), f^{\alpha}(y x;h))$ (4.4) \S 2 \S 3 Theorem 4.1. (4.3) $f^{\alpha}(y x;h)$ (4.2) $E[u_{-\alpha}(\frac{q_{\xi}^{\alpha}(x x)}{f^{\alpha}(x x;h)})-d_{\alpha}(q_{\xi}^{\alpha}(y x), f^{\alpha}(y x;h)) h(\xi)]=0$. (4.5) (4.5) (4.2) $u_{-\alpha}(c_{x}$ $)$ ( ) $\psi_{x}(h)$ $\psi_{x}(h):=-u_{-\alpha}(c_{x}(h))$ (4.6) Theorem 4.2. $s(\xi)=t_{x}(\xi;h)$ $\min u_{\alpha}(q(x x))$ $q(y x)$ subject to $-D_{\alpha}(q_{\xi}^{\alpha}(y x),$ $q(y x))+u_{\alpha}(q(x x))=s(\xi)$ (4.2) $f^{\alpha}(y x;h)$ $u_{\alpha}(r)$ Bayesian -likelihood $\log$ $\log q(x x)$

14 $ $ $\mathcal{q}^{\alpha}$ 117 $\alpha=-1$ $\alpha=+1$ (4.5) $u_{-\alpha}( \frac{q_{\xi}^{\alpha}(x x)}{f^{\alpha}(x x;h)})$ canonical weight $h_{x}\dagger(\xi)$ $u_{-\alpha}(1/r)$ $r$ $\delta_{g}\log f^{\alpha}(x x;h_{x}\dagger;h-h_{x}^{\dagger})=0$ for any. (4.7) $h$ Theorem 4.2 $\alpha=+1$ $\alpha=-1$ $h_{x}^{\uparrow}$ Theorem 4.3. (4.7) $f^{\alpha}(y x;h_{x}\dagger)$ $u_{-\alpha}( \frac{q_{\xi}^{\alpha}(x x)}{f^{\alpha}(x x;h_{x}^{\dagger})})=d_{\alpha}(q_{\xi}^{\alpha}(y x),$ $f^{\alpha}(y x;h_{x}^{\uparrow}))$ for any. (4.8) Theorem4.3 $h_{x}^{*}(\xi)=\lambda(\xi x)$ $\mathcal{q}^{\alpha}$ Theorem 4.4. $E[u_{-\alpha}(\frac{q_{\xi}^{\alpha}(x x)}{f^{\alpha}(x x;h)})-d(q_{\xi}^{\alpha}(y x), f^{\alpha}(y x;h)) \lambda(\xi x)m(x)]=0.$ $h_{x}\dagger$ (4.7) Bayesian $\log$ -hkelihood $f^{\alpha}(y x;h_{x}^{*})$ $\log f^{\alpha}(x x;h)$ $f^{\alpha}(y x;h_{x}^{\uparrow})$ (4.6) $\psi_{x}(h)$ $h$ prior

15 118 averaging density Theorem 4.5. Canonical weight $h_{x}^{c}(\xi)$ $\delta_{g}\psi_{x}(h_{x}^{c};h-h_{x}^{c})=0$ for any $h.$ $f^{\alpha}(y x;h_{x}^{c})$ $D(q_{\xi}^{\alpha}(y x), f^{\alpha}(y x;h_{x}^{c}))=-\psi_{x}(h_{x}^{c})$ for any $\xi.$ REFERENCES Akaike, H. (1973). Information theory as an extension of the maximum hkelihood principle. Pages in B.N. Petrov and F. Csaki (editors) Second Intemational Symposium on Information Theory. Akademiai Kiado, Budapest. Aitchison, J. (1975). Goodness of prediction fit. Biometrika, 62, Amari, S-$I$. and Nagaoka, H. (2000). Methods of Information Geometry. American Mathematical Society, Load Island. Corcuera, J. $M$. and Giummole, F. (1999). $A$ generalized Bayes rule for prediction. Scandinavian Joumal of Statistics, 26, Hoeting, J. $A$., Madigan, D., Raftery, A. $E$. and Volinsky, C. $T$. (1999). Bayesian model averaging: a tutorial. Statistical Science, 14, Kullback, S. (1959). Information Theory and Statistics. Wiley, New York. Ohnishi, T. and Yanagimoto, T. (2013). Twofold structure of duality in Bayesian model averaging. Joumal of the Japan Statistical Society, to appear. Stein, C. $M$. (1981). Estimation of the mean of a multivariate normal distribution. Annals of Statistics, 9, Yanagimoto, T. and Ohnishi, T., (2009). Bayesian prediction of a density function

16 119 in terms of $e$ -mixture. Journal of Statistical Planning and Inferen ce, 139, Yanagimoto, T. and Ohnishi, T., (2011). Saddlepoint condition on a predictor to reconfirm the need for the assumption of a prior distribution. Journal of Statistical Planning and Inference, 141,

橡表紙参照.PDF

橡表紙参照.PDF CIRJE-J-58 X-12-ARIMA 2000 : 2001 6 How to use X-12-ARIMA2000 when you must: A Case Study of Hojinkigyo-Tokei Naoto Kunitomo Faculty of Economics, The University of Tokyo Abstract: We illustrate how to