COE SITAIE- ICE IEICE IEICE IEICE IEICE (PRMU) () IEEE Committee Members of IT Society Japan ChapterIEEE Computational Intelligence Society Japan Chap

Transcription

1 12 Collection of Technical Reports of the 12th Workshop on Information-Based Induction Sciences (IBIS 2009) IBIS

2 COE SITAIE- ICE IEICE IEICE IEICE IEICE (PRMU) () IEEE Committee Members of IT Society Japan ChapterIEEE Computational Intelligence Society Japan Chapter DMSM ii

3 IBIS; Information-Based Induction Sciences IBIS IBIS IBIS IBIS IBIS IBIS iii

4 pre-image IBIS IBIS 2009 iv

5 12 IBM NEC NTT ATR NTT v

6

7 Preface The IBIS Workshop is well established as a top-ranked research conference in machine learning in Japan. Since 1998 the IBIS Workshop has been a leading interdisciplinary forum where researchers and practitioners in various machine-learning-related disciplines can collaborate, including information theory, statistical science, statistical physics, computer science, data mining, and services sciences. In the last decade of IBIS Workshops, we have seen drastic changes in many aspects of society: Massive information exchange through the Internet has become an essential part of the social infrastructure, and machine learning is receiving increasing attention from everyone as a basic and essential tool to extract useful knowledge from an enormous amount of data of the Internet age, just as quantum physics is the basis of semiconductor engineering. The IBIS Workshop this year aims at being: International in promoting world-class research presentations from the Japanese community Open in encouraging new research attempts from new contributors and research domains Sound in supporting innovative research that matters in society while balancing theoretical and practical research work Besides the technical program that covers all aspects of machine learning and related research, the IBIS Workshop this year features seven organized sessions, where top-notch researchers talk about the latest trends and activities in various research domains such as financial risk management, speech and audio processing, bioinformatics, network theories, machine learning of ranking, pattern recognition, and real-world applications in computer vision and plant monitoring. To encourage discussions about ongoing studies also under submission to other conferences and journals, this year s IBIS Workshop accepted highquality technical presentations on two tracks. The Technical Track accepted vii

8 4-8 page technical papers through a review process by the Program Committee, while submissions to the Discussion Track needed only an abstract in the form of presentation slides up to 2 pages. The accepted papers in the Technical Track are electronically published on the IBIS Web site as IBIS 2009 Technical Reports. We hope the IBIS Workshop this year inspires your research by providing a high-quality technical program and well organized sessions with top-notch researchers, while also serving as an opportunity for networking between researchers from different areas. Oct. 19, 2009 IBIS 2009 Organizing Committee viii

9 IBIS 2009 Committee Organizing Committee Chair: Jun ichi Takeuchi (Kyushu University) Program Committee Chair: Tsuyoshi Idé (IBM) Program Committee Vice-Chair: Shinichi Nakajima (Nikon) Program Committee Members: Akihiro Inokuchi (Osaka University) Shigeyuki Oba (Kyoto University) Satoshi Oyama (Hokkaido University) Hisashi Kashima (the University of Tokyo) Tsuyoshi Kato (Ochanomizu University) Masanori Kawakita (Kyushu University) Masashi Sugiyama (Tokyo Institute of Technology) Jun Sese (Ochanomizu University) Takayuki Nakata (NEC) Kazushi Mimura (Hiroshima City University) Daichi Mochihashi (NTT) Jun Morimoto (ATR) Takehisa Yairi (the University of Tokyo) Shinji Watanabe (NTT) ix

10 目 次 IBIS2009のプログラムについて 井 手 剛, 中 島 伸 一 企 画 セッション 金 融 リスクと 統 計 的 学 習 1 金 融 リスク~ 企 業 倒 産 の 判 別 問 題 の 現 状 と 課 題 5 山 下 智 志 ( 統 計 数 理 研 究 所 ) 金 融 リスクとコピュラ~ 依 存 構 造 がリスクに 及 ぼす 影 響 6 吉 羽 要 直 ( 日 本 銀 行 金 融 研 究 所 統 計 数 理 研 究 所 ) 極 値 統 計 学 7 高 橋 倫 也 ( 神 戸 大 学 海 事 科 学 研 究 科 ) 企 画 セッション 音 声 音 響 処 理 と 機 械 学 習 スパース 表 現 による 音 響 信 号 処 理 14 亀 岡 弘 和 ( 日 本 電 信 電 話 株 式 会 社 NTTコミュニケーション 科 学 基 礎 研 究 所 ) 機 械 学 習 に 基 づく 音 楽 情 報 処 理 15 吉 井 和 佳 ( 産 業 技 術 総 合 研 究 所 情 報 技 術 研 究 部 門 ) 音 声 系 列 パターン 認 識 のための 識 別 学 習 16 中 村 篤 ( 日 本 電 信 電 話 株 式 会 社 NTTコミュニケーション 科 学 基 礎 研 究 所 ) LCore: 言 葉 と 動 作 によるコミュニケーションを 学 習 するロボットの 知 能 化 技 術 17 岩 橋 直 人 (( 独 ) 情 報 通 信 研 究 機 構 知 識 創 成 コミュニケーション 研 究 センター) 企 画 セッション 化 学 構 造 とその 数 理 木 構 造 および 化 学 構 造 に 対 する 特 徴 ベクトル: 埋 め 込 み 検 索 構 造 推 定 18 阿 久 津 達 也 ( 京 都 大 学 化 学 研 究 所 バイオインフォマティクスセンター) 企 画 セッション 疎 グラフ 上 のダイナミクス 疎 なグラフ 上 のダイナミクスの 解 析 とその 応 用 20 三 村 和 史 ( 広 島 市 立 大 学 大 学 院 情 報 科 学 研 究 科 知 能 工 学 専 攻 ) 経 路 積 分 を 用 いたランダムネットワーク 上 の 同 期 現 象 の 解 析 21 一 宮 尚 志 ( 京 都 大 学 大 学 院 理 学 研 究 科 数 学 教 室 ) 疎 結 合 位 相 振 動 子 ネットワークのcavity 法 による 解 析 22 上 江 洌 達 也 ( 奈 良 女 子 大 学 大 学 院 人 間 文 化 研 究 科 ) 企 画 セッション ランキング 学 習 の 最 前 線 Learning to Rank Methods 23 Hang Li (Microsoft Research Asia) 企 画 セッション パターン 認 識 の 新 潮 流 パターンとは 何 か 非 記 号 計 算 と 一 般 対 象 の 情 報 計 量 24 石 川 博 ( 名 古 屋 市 立 大 学 大 学 院 システム 自 然 科 学 研 究 科 ) 企 画 セッション 広 がる 機 械 学 習 応 用 のフロンティア

11 顔 と 人 体 画 像 認 識 に 生 きる 機 械 学 習 46 勞 世 竑 (オムロン( 株 ) 技 術 本 部 ) 品 質 問 題 を 解 くプロセスデータ 解 析 技 術 : 産 業 応 用 の 現 状 と 課 題 53 加 納 学 ( 京 都 大 学 大 学 院 工 学 研 究 科 ) テクニカルセッションA 代 理 ベイズ 学 習 と 隠 れマルコフモデルへの 応 用 54 山 崎 啓 介 ( 東 京 工 業 大 学 ) カーネルマルコフ 連 鎖 モンテカルロ 法 による 測 定 誤 差 モデル 推 定 62 赤 穂 昭 太 郎 ( 産 業 技 術 総 合 研 究 所 ), 伊 庭 幸 人 ( 統 計 数 理 研 究 所 ) 一 次 元 正 規 分 布 のなす 空 間 への 曲 線 あてはめ 68 藤 木 淳 ( 産 業 技 術 総 合 研 究 所 ), 赤 穂 昭 太 郎 ( 産 業 技 術 総 合 研 究 所 ) 行 と 列 の 生 成 による 線 形 計 画 ブースティング 74 畑 埜 晃 平 ( 九 州 大 学 ), 瀧 本 英 二 ( 九 州 大 学 ) Multiple Kernel Learning for Object Classification 81 中 島 伸 一 (( 株 )ニコン),Binder Alexander(Fraunhofer Institute FIRST),Müller Christina(Technische Universität Berlin),Wojcikiewicz Wojciech(Technische Universität Berlin),Marius Kloft(Technische Universität Berlin),Brefeld Ulf(Technische Universität Berlin),Müller Klaus-Robert(Technische Universität Berlin),Kawanabe Motoaki(Fraunhofer Institute FIRST) 変 分 ベイズ 法 を 用 いた 混 合 ベルヌーイ 分 布 学 習 の 相 図 について 89 梶 大 介 ( 東 京 工 業 大 学 大 ), 渡 辺 澄 夫 ( 東 京 工 業 大 学 ) 領 域 ベースの 隠 れ 変 数 を 用 いた 決 定 論 的 画 像 領 域 分 割 95 三 好 誠 司 ( 関 西 大 学 ) 共 起 成 分 の 含 意 関 係 に 基 づくデータマイニングの 実 験 と 考 察 99 二 木 克 也 ( 北 海 道 大 学 ), 湊 真 一 ( 北 海 道 大 学 ) High-Precision Speaker Verification by Adaptive Weighting of Local MFCC Features 105 坂 井 俊 亮 ( 筑 波 大 学 ) オンライン 学 習 可 能 な 多 重 スケールでの 時 間 発 展 を 考 慮 したトピックモデル 113 岩 田 具 治 ( 日 本 電 信 電 話 株 式 会 社 ), 山 田 武 士 ( 日 本 電 信 電 話 株 式 会 社 ), 櫻 井 保 志 ( 日 本 電 信 電 話 株 式 会 社 ), 上 田 修 功 ( 日 本 電 信 電 話 株 式 会 社 ) Observational Reinforcement Learning 120 Simm Jaak( 東 京 工 業 大 学 ), 杉 山 将 ( 東 京 工 業 大 学 ), 八 谷 大 岳 ( 東 京 工 業 大 学 ) Matching between Piecewise Similar Curve Images 128 岩 田 一 貴 ( 広 島 市 立 大 学 ), 林 朗 ( 広 島 市 立 大 学 ) 時 系 列 パターンの 多 数 決 型 識 別 器 の 設 計 136 福 冨 正 弘 ( 九 州 大 学 ), 小 川 原 光 一 ( 九 州 大 学 ), 馮 尭 楷 ( 九 州 大 学 ), 内 田 誠 一 ( 九 州 大 学 ) 変 分 ベイズ 法 における 確 定 的 アニーリングとハイパーパラメータの 部 分 最 適 化 について 144 永 田 賢 二 ( 東 京 大 学 ), 片 平 健 太 郎 ( 科 学 技 術 振 興 機 構 ), 岡 ノ 谷 一 夫 ( 科 学 技 術 振 興 機 構, 独 立 行 政 法 人 理 化 学 研 究 所 ), 岡 田 真 人 ( 東 京 大 学 大 学 院 )

12 Radon 変 換 を 介 した 医 用 画 像 再 構 成 における 画 像 修 復 152 庄 野 逸 ( 電 気 通 信 大 学 ), 岡 田 真 人 ( 東 京 大 学 大 学 院 ) クラウドコンピューティングを 用 いた 粒 子 フィルタのためのMapReduceアルゴリズム 159 石 垣 司 ( 産 業 技 術 総 合 研 究 所 ), 中 村 和 幸 ( 明 治 大 学 ), 本 村 陽 一 ( 産 業 技 術 総 合 研 究 所 ) Virtual Concept Drift 環 境 におけるRBFNNのモデル 選 択 167 山 内 康 一 郎 ( 中 部 大 学 ) 機 械 学 習 を 用 いたスプログ 検 出 におけるHTML 構 造 の 類 似 性 の 利 用 片 山 太 一 ( 筑 波 大 学 ), 芳 中 隆 幸 ( 東 京 電 機 大 学 ), 宇 津 呂 武 仁 ( 筑 波 大 学 ), 河 田 容 英 (( 株 )ナビッ クス), 福 原 知 宏 ( 東 京 大 学 ) ベーテ 自 由 エネルギーとLoopy belief Propagation に 現 れるグラフ のゼータ 関 数 につい て 渡 辺 有 祐 ( 総 合 研 究 大 学 院 大 学 ), 福 水 健 次 ( 統 計 数 理 研 究 所, 総 合 研 究 大 学 院 大 学 ) 劣 モジュラカットとその 応 用 190 河 原 吉 伸 ( 大 阪 大 学 ), 永 野 清 仁 ( 東 京 工 業 大 学 ), 津 田 宏 治 ( 産 業 技 術 総 合 研 究 所 ),Bilmes Jeff(Washington University) 重 み 付 きカーネルマシンの 多 次 元 パス 追 跡 法 に 関 する 一 考 察 198 烏 山 昌 幸 ( 名 古 屋 工 業 大 学 大 学 院 ), 原 田 尚 幸 ( 名 古 屋 工 業 大 学 大 学 院 ), 竹 内 一 郎 ( 名 古 屋 工 業 大 学 大 学 院 ) ベイズ 確 率 文 脈 自 由 文 法 のための 高 速 構 文 木 サンプリング 法 206 武 井 俊 祐 ( 東 京 大 学 大 学 院 ), 牧 野 貴 樹 ( 東 京 大 学 ), 高 木 利 久 ( 東 京 大 学 ) VC Theory and a Concentration Inequality for Sums of Eigenvalues of Wishart Matrix 214 上 野 康 隆 ( 東 北 大 学 大 学 院 ), 赤 間 陽 二 ( 東 北 大 学 大 学 院 ) テクニカルセッションB 1 次 元 線 形 動 的 システムの 特 異 性 とベイズ 汎 化 誤 差 への 影 響 220 内 藤 卓 人 ( 東 京 工 業 大 学 ), 山 崎 啓 介 ( 東 京 工 業 大 学 ) 品 質 の 異 なる 二 つのデータ 集 合 間 の 転 移 学 習 の 解 析 225 赤 穂 昭 太 郎 ( 産 業 技 術 総 合 研 究 所 ), 神 嶌 敏 弘 ( 産 業 技 術 総 合 研 究 所 ) 条 件 付 きエントロピー 最 小 化 に 基 づく 教 師 付 き 次 元 削 減 手 法 231 日 野 英 逸 ( 早 稲 田 大 学 ), 村 田 昇 ( 早 稲 田 大 学 ) ネットワーク 科 学 の 方 法 を 用 いたWebページネットワークの 構 造 による 分 類 239 中 川 帝 人 ( 名 古 屋 大 学 ), 鈴 木 泰 博 ( 名 古 屋 大 学 ) 化 合 物 -タンパク 質 活 性 空 間 における 特 徴 選 択 243 新 島 聡 ( 京 都 大 学 大 学 院 ), 奥 野 恭 史 ( 京 都 大 学 大 学 院 ) 大 幾 何 マージン 最 小 分 類 誤 り 学 習 法 250 渡 辺 秀 行 ( 独 立 行 政 法 人 情 報 通 信 研 究 機 構 ), 片 桐 滋 ( 同 志 社 大 学 ), 山 田 幸 太 ( 同 志 社 大 学 ), マクダーモット エリック( 日 本 電 信 電 話 株 式 会 社 ), 中 村 篤 ( 日 本 電 信 電 話 株 式 会 社 ), 渡 部 晋 治 (NTT), 大 崎 美 穂 ( 同 志 社 大 学 ) 点 過 程 を 特 徴 付 ける 統 計 量 の 時 間 変 化 を 推 定 する 258 下 川 丈 明 ( 京 都 大 学 ), 篠 本 滋 ( 京 都 大 学 )

13 生 物 学 情 報 への 機 械 学 習 解 析 の 応 用 (Toxicogenomicsへの 展 開 ) 263 武 藤 裕 紀 ( 中 外 製 薬 株 式 会 社 ), 松 下 智 哉 ( 中 外 製 薬 株 式 会 社 ), 芦 原 基 起 ( 中 外 製 薬 株 式 会 社 ) Ellipsoidal Support Vector Machines 268 門 馬 道 也 (NEC) 確 率 伝 搬 法 による 確 率 的 画 像 処 理 における 統 計 的 性 能 評 価 276 片 岡 駿 ( 東 北 大 学 ), 安 田 宗 樹 ( 東 北 大 学 ), 田 中 和 之 ( 東 北 大 学 ) Chow-Liuアルゴリズムの 一 般 化 と 木 の 複 雑 さを 考 慮 した 修 正 版 について 石 田 悠 ( 大 阪 大 学 大 学 院 ), 鈴 木 譲 ( 大 阪 大 学 大 学 院 ) Kullback-Leibler Importance Estimation Procedureを 用 いたRestricted Boltzmann Machineの 学 習 アルゴリズム 桜 井 哲 治 ( 東 北 大 学 ), 安 田 宗 樹 ( 東 北 大 学 ), 田 中 和 之 ( 東 北 大 学 ) 非 定 常 2 値 時 系 列 データから 隠 れた 構 造 を 読 み 取 る - 神 経 科 学 データへの 応 用 - 瀧 山 健 ( 東 京 大 学 ), 岡 田 真 人 ( 東 京 大 学 大 学 院 ) 部 分 空 間 SVMのための 交 差 部 分 空 間 学 習 井 之 上 直 矢 ( 東 京 工 業 大 学 ) 線 形 時 間 異 種 混 合 モデル 選 択 のための 期 待 情 報 量 基 準 最 小 化 法 312 藤 巻 遼 平 (NEC), 森 永 聡 (NEC), 門 馬 道 也 (NEC), 青 木 健 児 (NEC), 中 田 貴 之 (NEC) Extending the Use of Instrumental Variables for the Identification of Direct Causal Effects in SEMs Chan Hei(Center for Service Research,AIST), 黒 木 学 ( 大 阪 大 学 ) 独 立 性 の 一 般 化 に 基 づく 統 計 モデルの 拡 張 327 藤 本 悠 ( 青 山 学 院 大 学 ), 村 田 昇 ( 早 稲 田 大 学 ) 近 傍 ハッシュを 用 いた 高 速 なグラフカーネル 335 比 戸 将 平 (IBM 東 京 基 礎 研 究 所 ), 鹿 島 久 嗣 ( 東 京 大 学 ) Hannan-Quinn の 命 題 は 線 形 回 帰 でも ガウス 型 Baysian ネットワークの 構 造 推 定 でも 正 しい 鈴 木 譲 ( 大 阪 大 学 大 学 院 ) 非 線 形 プリコーディングの 統 計 力 学 的 解 析 349 林 愛 空 ( 東 京 工 業 大 学 ), 樺 島 祥 介 ( 東 京 工 業 大 学 ) Latent Dirichlet Allocationの 量 子 アニーリング 変 分 ベイズ 学 習 佐 藤 一 誠 ( 東 京 大 学 ), 栗 原 賢 一 (Google 東 京 ), 田 中 宗 ( 東 京 大 学 ), 宮 下 精 二 ( 東 京 大 学 ), 中 川 裕 志 ( 東 京 大 学 ) 個 性 を 考 慮 した 周 期 的 全 身 運 動 のオンライン 予 測 365 松 原 崇 充 ( 奈 良 先 端 科 学 技 術 大 学 院 大 学 ), 玄 相 昊 (ATR), 森 本 淳 (ATR)

14 2009 Technical Report on Information-Based Induction Sciences 2009 (IBIS2009) IBIS 2009 Overview of the IBIS 2009 technical program Tsuyoshi Idé Shinichi Nakajima Abstract: This report reviews the features of the IBIS 2009 technical program, including statistics about IBIS 2009 submissions. Keywords: IBIS 2009, Poster Session, Organized Session 1 2 Information-Based Induction Sciences; IBIS, 1 1 IBIS Google Amazon 2 ICML 1 NIPS 2 IBIS 2009, IBM, goodidea@jp.ibm.com, IBIS 2009 Program Committee Chair, IBM Research Tokyo, IBIS 2009,, nakajima.s@nikon.co.jp, IBIS 2009 Program Committee Vice-Chair, Nikon 1 International Conference on Machine Learning 2 Neural Information Processing Systems 1: IBISInformation-Based Induction Sciences IBIS ISBN IBIS IBIS IBIS 2008 IBIS 2008 IBIS 1

15 2 2 IBIS 2008 IBIS IBIS 2009 IBIS 2 IBIS IBIS IBIS IBIS 2 IBIS IBIS IBIS 2009 IBIS IBIS /4 IBIS Web IBIS 2008 IBIS

16 Discussion (27) 38% Other (12) Technical (45), 62% 2: 27% For award (33) 73% 3 7 IBIS 2009 COE GCOE GCOE 7 IBIS 2008 IBIS Microsoft Research Asia Hang Li 3

17 4 IBIS IT ibis-workshop.org URL / IBIS 3 IBIS 3: IBIS [1],,, IBIS 98, IT IBIS IBIS IBIS [1] 4

18 2009 Technical Report on Information-Based Induction Sciences 2009 (IBIS2009) Abstract:, 5

19 2009 Technical Report on Information-Based Induction Sciences 2009 (IBIS2009) Abstract: /, 6

20 2009 Technical Report on Information-Based Induction Sciences 2009 (IBIS2009) Statistics of Extremes Rinya Takahashi Abstract: Statistics of extremes deals with disastrous rare events like heavy rains, strong winds, earthquakes, etc. For planning embankments and other constructions, or to sell the insurance to cover the loss, the assessment of the possible severeness is essential. Frequently, the possibility of the events, which people have never experienced, must be estimated, that is we have to extrapolate statistically available data. A standard procedure is to make use of the two types of basic distributions, the generalized extreme value distributions (GEV), and the generalized Pareto distributions (GP). According to the available data one of two distributions is adopted, the GEV for block maxima and the GP for threshold exceedances. Keywords: F X F X 1, X 2,..., X n Z n = max{x 1, X 2,..., X n } block maxima) F (upper boundary) x F = sup{x : F (x) < 1} Z n x F (threshold exceedances or (n ) Z n Peaks Over Threshold) 2 F a n > 0b n R (n = 1, 2,...) G ( ) Zn b n P x = P (Z n a n x + b n ), , = F n (a n x + b n ) L G(x) tel , r-taka@maritime.kobe-u.ac.jp, Faculty of Maritime Sciences, Kobe University, 5-1-1, Fukae- Minami-Machi, Higashi-Nada-ku, Kobe, G (exreme value distribution, EVD) F D(G) F 7 a n

21 G a n b n ΛΦ α Ψ α GumbelFréchet Weibull F f Exp(1) F (x) = 1 e x, f(x) = e x, x 0. F n (x + log n) = = { 1 e (x+log n)} n {1 + e x n } n e e x, n. ( < x < ). a n = 1b n = log n Pareto ( α > 0) F (x) = 1 1/x α, f(x) = αx α 1, x 1. F n (n 1/α x) = { } n 1 1 = {1 + x α (n 1/α x) α n e x α, n. (x > 0). a n = n 1/α b n = 0 ( α > 0) } n F (x) = 1 (1 x) α, f(x) = α(1 x) α 1, 0 x 1. F n (n 1/α x + 1) = {1 ( n 1/α x) α} n [Gnedenko (1943)de Haan (1970)] F D(Φ α ) x F = and 1 F (tx) lim x 1 F (x) = t α, t > 0. F D(Ψ α ) x F < and 1 F (x F (x F x)t) lim = t α, t > 0. x x F 1 F (x) F D(Λ) s( ) > 0 s.t. ( ) = ( ) xf x = s(x) = 1 F (x + ts(x)) lim = e t. x xf 1 F (x) (1 F (y))dy <, x < x F. xf x 1 F (y) dy satisfies ( ). 1 F (x) 3 1 (ξ R) (generalized extreme value, GEV) G ξ (x) = exp { (1 + ξx) 1/ξ}, 1 + ξx > 0, ξ = 0 G 0 (x) = lim ξ 0 G ξ (x) = exp{ exp( x)} = Λ(x) = {1 + ( x)α n } n e ( x)α, n. (x 0). a n = n 1/α b n = 1 = x F 3 G n ξ (A n x + B n ) = G ξ (x) [Fréchet (1927)Fisher and Tippett (1928) Gnedenko (1943);Trinity Theorem ] G(x) Λ(x) = exp( exp( x)), x R, Φ α (x) = exp( x α ), x 0, α > 0, Ψ α (x) = exp( ( x) α ), x 0, α > 0. 8 Φ α (x) = G 1/α (α(x 1))Ψ α (x) = G 1/α (α(x + 1)) (max stable) n A n > 0 B n R 3 F n (a n x + b n ) G ξ (x). n a n x + b n = z ( ) z P (Z n z) = F n bn (z) G ξ. a n

22 G ξ n a( ) a n t Pareto F n (a n x + b n ) = [ 1 { 1 F (a n x + b n ) }] Fréchet (ξ > 0) n = [ 1 n { 1 F (a n x + b n ) }/ n ] n exp [ lim n n{ 1 F (a n x + b n ) }] lim n{1 F (a nx+b n )} = log G ξ (x) = (1+ξx) 1/ξ. n x = 0 lim n{1 F (b n)} = 1. n 2 n{1 F (a n x + b n )} lim = (1 + ξx) 1/ξ. n n{1 F (b n )} n P (X > a n x + b n X > b n ) = 1 F (a nx + b n ) 1 F (b n ) (1 + ξx) 1/ξ. P (X b n a n x X > b n ) 1 (1 + ξx) 1/ξ. H ξ (x) = 1 + log G ξ (x) = 1 (1 + ξx) 1/ξ ( ) y P (X b n y X > b n ) H ξ. a n H ξ b n G ξ (z) ξ 0 g ξ (z) = (1 + ξ z) 1/ξ 1 exp { (1 + ξ z) 1/ξ}, F F u 1 + ξz > 0, F u (y) = P (X u y X > u), 0 y x F u Gumbel (ξ = 0) Weibull (ξ < 0) H ξ (ξ > 0ξ = 0ξ < 0) ξ > 0 ξ = 0 ξ < 0 min{x 1, X 2,..., X n } = max{ X 1, X 2,..., X n } 3 GEV(µ, σ, ξ) { [ ( )] } 1/ξ z µ G(z) = exp 1 + ξ σ 1 + ξ(z µ)/σ > 0. g 0 (z) = exp { z exp( z) }, ( ) z µ = G ξ, σ G ξ GEV(0, 1, ξ) µ R σ > 0 ξ R (Generalized Pareto, GP) b n G(z) ξ < 0 Weibull z < µ σ/ξ ξ = 0 Gumbel < z < ξ > 0 Fréchet z > µ σ/ξ ξ = 0 [Pickands (1975)] F D(G ξ ) lim u x F F u (a(u)y) = H ξ (y), y 0, F u (a(u)y) < 1. < z <, GEV( 2.5, 1, 0.4) 0 GEV(0, 1, 0) GEV(2.5, 1, 0.4) 0 9

23 GP(σ, ξ) ( H(y) = ξ y ) 1/ξ ( y ) = Hξ, 1 + ξy/σ > 0. σ σ H ξ GP(1, ξ) σ > 0 ξ R H(y) ξ < 0 0 < y < σ/ξ ξ = 0 H 0 (y/σ) = limh ξ (y/σ) = 1 e y/σ ξ 0 GEV(µ, σ, ξ) ξ 0 0 < y < ξ > 0 Pareto 1: GEV( 2.5, 1, 0.4) 0 e 11 = ξ 2 p, e 12 = ξ { Γ(2 + ξ) p }, GEV(0, 1, 0)GEV(2.5, 1, 0.4) 0 e 13 = σξ(p/ξ q), e 22 = 1 2 Γ(2 + ξ) + p, [ Γ(2 + ξ) 1 e 23 = σ + q p ] ξ ξ 1 + γ, [ e 33 = σ 2 π 2 ( γ + 1 ) ] 2 2q ξ ξ + p ξ 2. 2: GP(1, ξ)ξ = 0.4, 0, GEV) 0 < y < nx h zi µ i H ξ (y) l(µ, σ, ξ) = n log σ (1 + 1/ξ) log 1 + ξ σ i=1 (1 + ξ y) 1/ξ 1, 1 + ξy > 0, ξ 0, nx h zi µ i 1/ξ h ξ (y) = 1 + ξ, σ exp( y), 0 < y <, ξ = 0, i=1 1 + ξ(z i µ)/σ > 0, i = 1,..., n, GP(1, ξ)ξ = 0.4, 0, 0.4 ξ = 0 nx zi µ nx n zi µ o l(µ, σ) = n log σ exp, σ σ {z 1, z 2,..., z n } GEV(µ, σ, ξ) 10 i=1 i=1 ( µ, σ, ξ) ( µ, σ) (Prescott and Walden, 1980) n σ 2 ξ 2 e 11 e 12 e 13 e 22 e 23 e 33, (µ, σ, ξ) Γ( ) Ψ(r) = d log Γ(r)/dr p = (1 + ξ) 2 Γ(1 + 2ξ)q = Γ(2 + ξ){ψ(1 + ξ) + (1 + ξ)/ξ}γ = Euler GEV(µ, σ, ξ)µ Rσ > 0ξ R ξ > 0.5 Smith (1985) ξ < ξ < 0.5

24 (µ, σ, ξ) l(µ, σ, ξ 0 ) µ σ { { ξ : 2 l( µ, σ, ξ) max l(µ, σ, ξ)} χ 2 1(0.05) } µ, σ = { ξ : max l(µ, σ, ξ) l( µ, σ, ξ) } µ, σ GEV(µ, σ, ξ) 1 1/T R T G(R T ) = G ξ ( RT µ σ ) = 1 1/T (µ, σ, ξ) (R T, σ, ξ) µ + σ [{ log(1 1/T ) } l(r T, σ, ξ) ξ ]/ 1 ξ, ξ 0, nx» «R T = µ + σ [ log { log(1 1/T ) }] n log σ (1 + 1/ξ) log y ξ T + ξ zi R T σ, ξ = 0. i=1 nx» «1/ξ R T (return period) T y ξ T + ξ zi R T σ i=1 (return level) T = 200 R 200 R T 95% n n n T R T : 2 l( R b T, bσ, b o o ξ) max l(r T, σ, ξ) χ 2 1(0.05) σ, ξ n = R T : max σ, ξ R T (µ, σ, ξ) l(rt, σ, ξ) l( R b T, bσ, ξ) b o µ + σ [ y b ξ T ẑ T = 1]/ ξ, ξ 0, µ + σ [ ] log y T, ξ = 0, GP(σ, ξ) y T = log(1 1/T ) V ( R T ) R T T V R T ξ ξ = ξ 0 ξ 95% R T µ = R T σ [ y ξ T 1] /ξ 3.2 (GP) {y 1, y 2,..., y n } GP(σ, ξ) ξ 0 V ( µ, σ, ξ)» RT T = RT µ, R T σ, R T n ξ l(σ, ξ) = n log σ (1 + 1/ξ) log(1 + ξ y i /σ), h = 1, (y ξ T 1)/ξ, σy ξ T ( log y T )/ξ σ(y ξ T 1)/ξ2i i=1. ( µ, σ, ξ) 1 + ξ y i /σ > 0, i = 1, 2,..., n, ξ < 0 ξ = 0 R = µ σ/ ξ l(σ) = n log σ 1 n y i σ R T = [ 1, 1/ξ, σ/ξ 2] ( σ, ξ) σ GP(σ, ξ) Fisher ξ R T [ ] n (1 + ξ)/σ 2 1/σ (1 + ξ)(1 + 2ξ) 1/σ 2 11 i=1

25 ξ > 1/2 σ = σ u ξ u u ξ u u [ ] V n = 1 2σ 2 (1 + ξ) σ(1 + ξ) n σ(1 + ξ) (1 + ξ) 2 u u Smith, 1985 F (VaR p ) = 1 p (threshold) (Value at Risk) Y GP(σ, ξ) F (x) ξ < 1 E(Y ) = 0 (1 H(y))dy = σ 1 ξ H ξ v > 0 Y v Y > v σ = (σ + ξv) ξv : Y (mean excess) e(v) e(v) = E(Y v Y > v) Y v Y > v GP(σ + ξv, ξ) ( ζ u, σ, ξ) e(v) = σ + ξv 1 ξ = σ 1 ξ + ξ 1 ξ v, [ ζu V (1 = ζ ] u )/n 0 T 0 V Nu v ξ = 0 e(v) F F (x) = (1 F (u))f u (y) + F (u), y = x u. u F u GP ( ) x u F (x) (1 F (u))h ξ + F (u). σ 1 H(y + v) P (Y v > y Y > v) = 1 H(v) = ζ u = 1 F (u) ( ) 1/ξ 1 + ξ(y + v)/σ { = ( ) 1/ξ 1 + ξv/σ VaR p = u + σ (ζu ) ξ 1}, ξ p ( ) 1/ξ y = 1 + ξ σ + ξv u GP(σ + ξv, ξ) ( σ, ξ) ζ u N u /n ξ n N u u ( )b VaR p = u + σ ξ ξ ζu 1 p, VaR p 0 T = (0, 0) VaR p u u ( σ u, ξ V ( VaR p ) VaR T p V VaR p, u ) 12

26 [5] Fisher, R. A. and Tippett, L. H. C. (1928). Limiting [ VaR T VaRp p =, VaR p ζ u σ, VaR ] forms of the frequency distribution of the largest p ξ or smallest member of a sample. Proc. Cambridge ( ζ u, σ, ξ) Philos. Soc. 24, [6] Fréchet, M. (1927). Sur la loi de probabilité de l écart maximum. Ann. Soc. Math. Polon. 6, GEV ξ 95% [7] Gnedenko, B. (1943). Sur la distibution limite du terme maximum d une serie aleatoire. Ann, Math. {ξ : max l(σ, ξ) l( σ, ξ) 44, Translated and reprinted in: Breakthroughs in Statistics, Vol.I, 1992, eds. S. Kotz and 1.921} σ N. L. Johnson, Springer-verlag, pp VaR p σ = (VaR p u)ξ (ζ u /p) ξ 1 l(var p, ξ) VaR p likelihood estimation of the parameters of the generalized extreme value distribution. Biometrika 67, max ξ l(var p, ξ) ζ u [10] Smith, R. L. (1985). Maximum likelihood estimation in a class of nonregular cases. Biometrika 72, [11] (2004). r r (2004) 52 1 de Haan (1970) Embrechts et al. (2001) Coles (2001) Beirlant et al. (2004) [1] Beirlant, J., Goegebeur, Y., Segers, J. and Teugels, J. (2004). Statistics of Extremes, Theory and Applications. Wiley. [2] Coles, S. G. (2001). An Introduction to Statistical Modeling of Extreme Values. Springer. [3] Embrechts, P., Klüppelberg, C. and Mikosch, T. (2001). Modelling Extremal Events for Insurance and Finance, 3rd ed. Springer. [4] de Haan, L. (1970). On Regular Variation and Its Application to the Weak Convergence of Sample Extremes. Mathematical Centre Tracts 32, Mathematisch Centrum. Amsterdam. 13 [8] Pickands, J. (1975). Statistical inference using extreme order statistics. Ann. Statist. 3, [9] Prescott, P. and Walden, A. T. (1980). Maximum

27 2009 Technical Report on Information-Based Induction Sciences 2009 (IBIS2009) Abstract: NTT, 14

28 2009 Technical Report on Information-Based Induction Sciences 2009 (IBIS2009) Abstract: (1) (2) (3) (4), 15

29 2009 Technical Report on Information-Based Induction Sciences 2009 (IBIS2009) Abstract: NTT, 16

30 2009 Technical Report on Information-Based Induction Sciences 2009 (IBIS2009) LCore: Abstract: LCore LCore (), 17

31 2009 Technical Report on Information-Based Induction Sciences 2009 (IBIS2009) Feature Vectors for Trees and Chemical Structures: Embedding, Search and Pre-Image Tatsuya Akutsu Abstract: In this short article, we briefly review our recent results on feature vectors on tree structures. For edit distance between trees, it is shown that the edit distance between two ordered trees can be approximated within a factor of O(h) by using the edit distance between the corresponding Euler strings, and the edit distance between two unordered trees can be approximated within a factor of O(h) by using feature vectors consisting of the numbers of occurrences of subtrees (each induced by a node and its descendants), where h is the minimum height of input trees. For the pre-image problem on trees (i.e., inferring a tree from a feature vector consisting of the numbers of occurrences of vertex-labeled paths), it is shown that the problem can be solved in polynomial time in the size of an output graph if the graphs are trees whose maximum degree is bounded by a constant and the lengths of given paths and alphabet size are bounded by constants, but is NP-hard even for trees of bounded degree if the maximum length of paths is not bounded. A practical branch-and-bound algorithm for the pre-image problem is also reviewed. Keywords: feature vector, kernel methods, embedding, tree edit, graph pre-image 1 L 1 [4] L 1, , tel , takutsu@kuicr.kyoto-u.ac.jp, Bioinformatics Center, Institute for Chemical Research, Kyoto University, Gokasho, Uji, Kyoto , Japan. 2 T 1 T 2 O(n 3 ) NP [6] T 1 T 2 D T (T 1, T 2 ) T s(t ) T 1, T 2 h 1 2 D S(s(T 1 ), s(t 2 )) D T (T 1, T 2 ) (2h+1)D S (s(t 1 ), s(t 2 )) [2]D S (s 1, s 2 ) s 1, s 2 18

32 [3] φ(t ) 1 2h + 2 φ(t 1) φ(t 2 ) 1 D T (T 1, T 2 ) φ(t 1 ) φ(t 2 ) 1 [7] L 1 3 (SVM) SVM v φ 1 (v) pre-image [5] [1] v φ 1 (v) NP NP 1 pre-image [8] 2030 φ 1 (v) 4 pre-image [1] T. Akutsu and D. Fukagawa, Inferring a graph from path frequency, Lecture Notes in Computer Science, 3537: , [2] T. Akutsu, A relation between edit distance for ordered trees and edit distance for Euler strings, Information Processing Letters, 100: , [3] T. Akutsu, D. Fukagawa, and A. Takasu, Approximating tree edit distance through string edit distance, Algorithmica, to appear. [4] A. Andoni and K. Onak, Approximating edit distance in near-linear time, Proc. ACM Annual Symposium on Theory of Computing, , [5] G.H. Bakir, A. Zien, and K. Tsuda, Learning to find graph pre-images, Lecture Notes in Computer Science, 3175: , [6] P. Bille, A survey on tree edit distance and related problem, Theoretical Computer Science, 337: , [7] D. Fukagawa, T. Akutsu, and A. Takasu, Constant factor approximation of edit distance of bounded height unordered trees, Lecture Notes in Computer Science, 5721:7 17, [8] Y. Ishida, L. Zhao, H. Nagamochi, and T. Akutsu, Improved algorithms for enumerating tree-like chemical graphs with given path frequency, Genome Informatics, 21:53 64,

33 2009 Technical Report on Information-Based Induction Sciences 2009 (IBIS2009) Abstract:, 20

34 2009 Technical Report on Information-Based Induction Sciences 2009 (IBIS2009) Abstract: COE, 21

35 2009 Technical Report on Information-Based Induction Sciences 2009 (IBIS2009) cavity Abstract: 1 (LDPCC) (cavity ) (belief propagation), uezu@kirin.phys.nara-wu.ac.jp, uezu@cc.nara-wu.ac.jp 22

36 2009 Technical Report on Information-Based Induction Sciences 2009 (IBIS2009) Learning to Rank Methods Hang Li Abstract: As an interdisciplinary field between machine learning and information retrieval, learning to rank is concerned with automatically constructing a ranking model using training data. Learning to rank technologies have been successfully applied to many tasks in information retrieval, and have been attracting more and more attention recently in the machine learning and information retrieval communities. In this talk I will introduce first explain the problem formulation of learning to rank, and relations between learning to rank and the other learning tasks. I will then describe in details about learning to rank methods developed in recent years, including pointwise, pairwise, and listwise approaches. Microsoft Research Asia, hangli@microsoft.com 23

37 2009 Technical Report on Information-Based Induction Sciences 2009 (IBIS2009) Abstract: Keywords:,,, , [4, 5, 7, 8, 12, 13] MDL (Minimum Description Length) [9, 10] MDL 2 MDL MDL 24

38 π x (3, 0), (14, 0), (159, 0), (2653, 0), R MDL MDL MDL

39 1:. [14] MDL 2:

40 2 3 Postscript Bézier [3] [1] N R {0, 1} X R 2 C I X C π 1 : X C X X s = s 0 s 1 s n N {0, 1} {(i, s i ) i = 0,, n} 3 E 3 M 27

41 E 3 X x X m(x) M {(x, m(x)) E 3 M x X} MDL R R x, y x y R R 2 X Y prec X : X X 2 prec Y : Y Y 2 f : X Y a, b X prec X (a, b) = prec Y ( f (a), f (B)) X Y X φ κ δ η Y W s X, Y, Z, W s(x) X, s(y) Y, s(z) Z, s(w) W (1)(2) s(z) s(x), s(y), s(w) s(x) t S t s S s(s ) S s(s ) t S t(z) t X s(z) = t(z) s t s s(x) = A X A i). ii). ψ Z 28

42 iii). iv). v). T s T s T T t S T t {s Γ(S ) s T = t} Γ(S t) 2. S = (S i ) i I S φ j : 2 S 2 T, (S, T S ) M = (φ j ) j J (S, S, M ) I, J (S, S, M ) φ M φ : 2 S 2 T, (S, T S ) φ S, T dm : M S cdm : M S φ : 2 dm(φ) 2 cdm(φ) = {0}, 2 = {0, 1},, n = {0, 1,, n 1} X 2 X 1 X 0 1 x X x : 1 X 1. I S = (S i ) i I S s I (s i ) i I i I s i S i S S S = (S i ) i I S = S i S S s = (s i ) i I s(s ) s i s S S s(s ) S Γ(S ) T S T S T S s S T S T s(s ) out = dm 1 : S 2 M in = cdm 1 : S 2 M S S out(s ) = {φ M dm(φ) = S } in(s ) = {φ M cdm(φ) = S } 3. (S, S, M ) S s in(s ) S S S S \ S s(s ) = φ(s(dm(φ))) (1) S S s(s ) = φ in(s ) φ in(s ) φ(s(dm(φ))). (2) (S, S, M ) S S (1) (2) (S, S, M ) Γ(S, S, M ) T S t Γ(T ) Γ(S, S, M t) = Γ(S, S, M ) Γ(S t) Γ(S, S, M t) (S, S, M ) S T t 29

43 4.2.2 S = {1, X, Y, Z, W} S = {W} W w M = {w, φ, ψ, η, δ, κ} w : W, φ : 2 Y 2 X, ψ : 2 Z 2 Y, η : 2 W 2 Y, δ : 2 W 2 X, κ : 2 X 2 W, W w (S, S, M ) φ ψ X (1) Y (2) Z (3) κ δ η W (4) φ Y X 1 1 w 1 {w} S i (i) (3) S 1 X S 2 Y S S 4 = W s in(x) = {φ, δ}, dm(φ) = Y, s(x) = φ(s(y)) δ(s(w)), s(w) = κ(s(x)) {w} w (3) 4. (S, S, M ) X S T S, t T X A (S, S, M t) s s(x) = A s (S, S, M, T, t, X) XT t (S, S, M, T, t, X) (S, S, M ) S = S = T = {X}, M =, t(x) = A 5. (S, S, M ) X, Y S T S t T X A s(x) = A s Γ(S, S, M t) φ : 2 X 2 Y Γ(S, S, M t) s s(y) = φ(s(x)) (S, S, M, T, t, X, Y) X, Y, T, t (S, S, M, T, t, X, Y) φ : 2 X 2 Y φ (S, S, M ) i) X id : X X X ω : X 1 X 1 cmpl : 2 X 2 X A X cmpl(a) = c A = X \ A. ii) f i : X Y i (i = 1,, n) f 1 f n : X Y 1 Y n ( f 1 f n )(x) = ( f 1 (x),, f n (x)) 30

44 f : X Y z : 1 Z f z ω : X Z f (z ω) : X Y Z f z : X Y Z X Y 1 Z iii) X 1 X 2 X n π i : X 1 X 2 X n X i i π i π j : X 1 X n X i X j π i j π i π j π k π i jk iv) f : X Y f : 2 X 2 Y A X f (A) = { f (x) x A} Y v) f : X Y f 1 : 2 Y 2 X A Y f 1 (A) = {x X f (x) A} X y Y f 1 (y) f 1 ({y}) vi) f : X X f 0 id n f n f n iv) f n : 2 X 2 X n f n f 1 : 2 X 2 X n 5.2 X X V (x, y) X X y x x y sub : X X V len : V R len R 1 (1) V (2) 1 π X 2 (3) sub 1 (4) π X X 1 (4) X (5) (S, S, M ) S = (S 1,, S 5 ), S =, S 1 = R, S 2 = V, S 3 = S 5 = X, S 4 = X X M = (len 1 : 2 S 1 2 S 2, sub 1 : 2 S 2 2 S 4, π 2 1 : 2 S 3 2 S 4, π 1 : 2 S 4 2 S 5 ). 1 (4) π 1 π 1 : S 4 S 5 π 1 : 2 S 4 2 S 5 T = {S 1, S 3 } t t(s 1 ) = {r}, t(s 3 ) = {p} r p X s Γ(S, S, M t) (1) s(s 2 ) = len 1 (s(s 1 )) = len 1 (t(s 1 )) = len 1 ({r}) = {v V len(v) = r}, s(s 4 ) = sub 1 (s(s 2 )) π 2 1 (s(s 3 )) = {(x, y) X X x y s(s 2 ), y s(s 3 )}, s(s 5 ) = {π 1 ((x, y)) X (x, y) s(s 4 )} = {x X x y s(s 2 ), y s(s 3 )} = {x X len(x p) = r} s s(s 5 ) p r (S, S, M, T, t, S 5 ) t(s 3 ) = {p, q} r p q 2 S 3 t(s 1 ) = {r, t} r t R X (1) ((len π 1 ) π 2 ) 1 V X (2) (sub π 2 ) 1 T = {S 1 } t Γ(T ) X X (3) π 1 X (4) t(s 1 ) = {(r 1, p 1 ), (r 2, p 2 ), } (5) s(s 2 ) = {(v, x) V X (len(v), x) t(s 1 )}, s(s 3 ) = {(x, y) X X (x y, y) s(s 2 )}, s(s 4 ) = {x X y X, (len(x y), y) t(s 1 )}, s(s 4 ) (5) 31

45 X (S, S, M ) V (1) 1 π 1 X (3) 1 π 2 V R (2) sub 1 mult (6) π X X 1 (4) X (5) mult : V R V T = {S 1, S 3 } t t(s 1 ) = {v}, t(s 3 ) = {p}, p X v V s Γ(S, S, M t) (1) s(s 2 ) = π 1 1 ({v}) = {(v, c) V R c R}, s(s 4 ) = sub 1 (mult(s(s 2 ))) π 1 2 ({p}) = sub 1 ({cv V c R}) π 1 2 ({p}) = {(x, p) X X c R, x p = cv}, s(s 5 ) = {x X c R, x p = cv} = {p + cv c R}. s s(s 5 ) p v π V 2 (1) X V (2) π 1 1 id X (3) X (4) add : X V X X (x, w) x + w T = {S 1, S 3 } S t t(s 1 ) = {v}, t(s 3 ) = {p}, (8) p X v V s Γ(S, S, M t) (1), (2) s(s 2 ) = {(x, w) x s(s 4 ), w s(s 1 )}, s(s 4 ) = {p} {x + w (x, w) s(s 2 )} add (7) = {p} {x + w x s(s 4 ), w s(s 1 )} (9) (9) D = {p, p + v, p + 2v, p + 3v, } s(s 4 ) s(s 4 ) p v D v s(s 4 ) = X (9) 1. S g : S N S n = g 1 (n) i N η : 2 S 2 S η(s i ) S i+1, η(s ) = η(s n ). S S = S 0 η(s ) S = η n (S 0 ). n=0. n N, x S n+1 n m S n S m = i) x S 0 ii) m n x η(s m ) S m+1 S n+1 S = S 0 η(s ) = S 0 η(s n ). x η(s n ) S n+1 η(s n ) η(s n ) S n+1 η(s n ) = S n+1 S n = η(s n 1 ) = η(η(s n 2 )) = = η n (S 0 ) S = S n. n=0 1 (7) 1 π V 2 (1) n=0 X V N (2) π 13 1 id X N (3) X N (4) φ n=0 π 1 X (5) (10) 32

46 φ : X V N X N φ((x, w, k)) = (x + w, k + 1) (8) t u p v p u v r u p v t(s 1 ) = {v}, t(s 3 ) = {(p, 0)}. (9) s(s 4 ) =s(s 3 ) φ(s(s 2 )) ={(p, 0)} {(x + w, k + 1) (x, k) s(s 4 ), w s(s 1 )} (11) g : s(s 4 ) N g((x, k)) = k η : 2 s(s 4) 2 s(s 4) η(a) = {(x + w, k + 1) (x, k) A, w s(s 1 )}. g η 1 (11) s(s 4 ) = η({(p, 0)}) n=0 = {(p, 0), (p + v, 1), (p + 2v, 2), (p + 3v, 3), } Γ(S, S, M t) s(s 5 ) = D s D (S, S, M, T, t, S 5 ) t(s 1 ) = {v, v} s(s 4 ) ={(p, 0)} {(x + v, k + 1) (x, k) s(s 4 )} {(x v, k + 1) (x, k) s(s 4 )} s(s 5 ) = {, p 3v, p 2v, p v, p, p+v, p+2v, p+3v, }. t(s 1 ) = {v, v, u, u} 3(a) 5.4 (4) (10) (S, S, M ) V (1) π 2 1 X V N (2) π 13 1 id X N (5) X N (6) φ R (3) len 1 V (4) sub 1 1 π 1 π X 2 π (7) X X 1 (8) X (9) (10) S 7 (4) T = {S 1, S 3, S 5 } t(s 1 ) = {v, v, u, u}, t(s 3 ) = {r}, t(s 5 ) = {(p, 0)}, (a) (b) (c) 3: (a) (b) (d) (S, S, M, T, t, S 9 ) 3(b) (S, S, M ) V (1) π 2 1 X V N (2) π 13 1 id X N (5) X N (6) T = {S 1, S 3, S 5 } φ π 1 1 π V 1 (3) V R (4) sub 1 mult 1 π X 2 π (7) X X 1 (8) X (9) t(s 1 ) = {v, v}, t(s 3 ) = {u}, t(s 5 ) = {(p, 0)}. (S, S, M, T, t, S 9 ) u {, p 3v, p 2v, p v, p, p + v, p + 2v, p + 3v, } 3(c) 5.5 V (1) X (3) π 1 1 π 2 1 X V (2) X (4) A V B X t(s 1 ) = A, t(s 3 ) = B t A (10) add 33

47 (a) (b) (c) (d) 4: (a) B(b) B (c) A (d) B A B 4(a) 4(b) B A 4(c) 4(d) B MDL succ : N N mult : N N N (S, S, M ) N N (1) id N N (2) φ = (π 1 mult) ((succ π 1 ) π 2 ) (n, m) (n + 1, m(n + 1)) T = {S 1 } t t(s 1 ) = {(0, 1)} s Γ(S, S, M t) s(s 2 ) = s(s 1 ) φ(s(s 2 )) = {(0, 1)} φ(s(s 2 )) (12) g : N N N g((n, m)) = n g φ 1 (12) s(s 2 ) = φ({(0, 1)}) n=0 = {(0, 1), (1, 1), (2, 2), (3, 6),, (n, n!), } (13) φ Γ(S, S, M t) s(s 1 ) = {(0, 1)} (13) s (S, S, M, T, t, S 2 ) N N (1) id N N (2) N (3) π 1 1 id N N (4) π 2 N (5) (S, S, M, T, t, S 3, S 5 ) (S, S, M ) N + N + (1) id N + N + (2) φ φ π 1 N + (3) N + φ = π 2 addadd : N + N + N + φ((n, m)) = (m, n+m) T = {S 1 } t t(s 1 ) = {(1, 1)} s Γ(S, S, M t) s(s 2 ) = {(1, 1)} φ(s(s 2 )) (n, m) s(s 2 ) (1, 1) (n, m) φ(s(s 2 )) (m n, n) s(s 2 ) (1, 1) (2n m, m n) s(s 2 ) 2 N + (1, 1) s(s 2 ) = φ({(1, 1)}) = {(1, 1), (1, 2), (2, 3), (3, 5), } n=0 s(s 3 ) = {1, 1, 2, 3, 5, 8, } (S, S, M ) π 12 cmpl π φ C C N (1) 1 C C (2) C (3) id C C N (4) π 2 1 C (5) C φ φ((c, z, k)) = (c, c + z 2, k + 1) 34

48 T = {S 4, S 5 } t t(s 4 ) = {(c, 0, 0) C C N c C}, t(s 5 ) = {z C z > 2} s Γ(S, S, M t) s(s 1 ) = φ(s(s 1 )) s(s 4 ) = φ(s(s 1 )) {(c, 0, 0) C C N c C}. g : s(s 1 ) N g((c, z, k)) = k 1 s(s 1 ) = φ(s(s 4 )) n=0 = {(c, z c n, n) C C N c C, n N} z c n zc 0 = 0zc n+1 = (zc n) 2 + c Γ(S, S, M t) s s(s 1 ) = {(c, z c n, n) C C N c C, n N}, s(s 2 ) = {(c, z c n) C C c C, n N, z c n > 2}, s(s 3 ) = {c C n N, z c n > 2}, s(s 4 ) = {(c, 0, 0) c C}, s(s 5 ) = {z C z > 2}. z c n n c z c n n N z c n > 2 s(s 3 ) M M M a) M idω π i M b) f : X Y, g : Y Z M g f : X Z M c) f i : X Y i (i = 1,, n) M f 1 f n : X Y 1 Y n M X x x : 1 X M M X x x : 1 X X M 7. M M M f M f M a) f M idω π i f M = 1 b) f M f M f M 1 f M i) f M = 1 + min ii) g,h M f =g h ( g M + h M ), f M = 1 + min f 1,, f n M f = f 1 f n ( f 1 M + + f n M ). 8. M S (S, S, M ) M a) M S f : X Y f : 2 X 2 Y b) M S f : X Y f 1 : 2 Y 2 X M S 9. M S (S, S, M ) M S M φ M S φ MS M S φ = f φ = f 1 f M S φ MS = f MS 35

49 (4) M S = {len, sub} M S 1 (6) {sub, mult} 6.2 X A X M S A (S, S, M, T, t, X) (S, S, M ) M S A X M S S = T = {X}, M =, t t(x) = A 0 [11] 10. X A M S P Y P Y A (S, S, M, T, t, X) a) (S, S, M ) M S b) T P c) T T t(t) A M S P I(A M S, P) φ MS log 2 P T (t(t)) φ M T T I(A M S, P) = X X P I(A M S, P) log 2 P X (A). T 10 1 P 1 P 1 ( ) = 0, P 1 (1) = 1 I(A M S, P) P = {1} T = {1} s 1 (1) = 1 T s I(A M S, P) P = {1} I(A M S ) (4) r 1 (1) len R 1 (2) V (3) 1 (4) p X (5) π 2 1 sub 1 (14) X X (6) π 1 X (7) r R p X r : 1 R p : 1 X 1 (6) v V (1) 1 π 1 p X (3) 1 π 2 V R (2) sub 1 mult X X (4) π 1 X (5) (15) M S = {p, v, sub, mult} sub 1 mult M S φ : 2 S 2 T ψ : 2 T 2 U M S S ψ φ φ ψ U S T U 1 3 C I(C {r, p, len, sub}) 6 L I(L {p, v, sub, mult}) 7 (14) (15) len, sub, mult M E = {len, sub, mult} X V R I(C M E ) 6, I(L M E ) 7 36

50 A X A 7 [6] 7.1 [2] f (x) = f x f (x) f x 12. f 1,, f l : N k N g : N l N h : N k N h(n 1,, n k ) = g(m 1,, m l ) m i = f i (n 1,, n k ), 1 i l (16). 13. f : N k N g : N k+2 N h : N k+1 N h(0, n 1,, n k ) = f (n 1,, n k ) (17) h(m + 1, n 1,, n k ) = g(p, m, n 1,, n k ) p = h(m, n 1,, n k ) (18). 14. f : N k+1 N g : N k N g(n 1,, n k ) = f (x 1,, x n, y) = 0 t < y y f (x 1,, x n, t), f (x 1,, x n, t) 0 (19) y. i) π i : N k N, ii) succ : N N, succ(n) = n + 1, iii) zero : N N, zero(n) = M N = {0, succ} 0 : 1 N 0(0) = f : N k N A N k φ f (A) = { f (n 1,, n k ) (n 1,, n k ) A, f (n 1,, n k ) } φ f : 2 Nk 2 N 5 1. M N 18. f : N k N r : N k N l φ f r (A) ={( f (n 1,, n k ), r(n 1,, n k )) (n 1,, n k ) A, f (n 1,, n k ) } N N l φ f r : 2 Nk 2 N Nl. M N π i : N k N N k π i N r : N k N l φ πi r N N l N k π i r succ : N N zero : N N N ω 0 1 N

51 r : N N l φ zero r N (0 ω) r N N l 1 (). f 1,, f l : N k N g : N l N M N (16) h : N k N. i = 1,, l φ fi id : 2 Nk 2 N Nk M N M N N k (1) φ f1 id φ fl id N N k (2 1 ) s. ((π 1 π 1 ) π 2 ) 1 N N k (2 l ) ((πl π 1 ) π 2 ) 1 N l N k (3) π 1 N (5) N l (4) φ g s(s 2i ) = {( f i (x), x) x s(s 1 ); f i (x) }, i = 1,, l s(s 3 ) = {(( f 1 (x),, f l (x)), x) s(s 4 ) = {( f 1 (x),, f l (x)) x s(s 1 ); f 1 (x),, f l (x) } x s(s 1 ); f 1 (x),, f l (x) } s(s 5 ) = {g( f 1 (x),, f l (x)) x s(s 1 ); f 1 (x),, f l (x), g( f 1 (x),, f l (x)) }. h r : N k N m φ h r N k φ f1 (id r) φ fl (id r) N N k N m. ((π 1 π 1 ) π 23) 1 N N k N m ((π l π 1 ) π 23 ) 1 N l N k N m N N m φ g id π 13 N l N m 2 (). f : N k N g : N k+2 N M N (17) (18) h : N k+1 N. N N k (1) π 2 π 23 1 N N N k (2) N k (4) φ f 0 id N N N k (5) s s(s 4 ) = {x (m, x) s(s 1 )} id π 1 N (3) s(s 5 ) ={( f (x), 0, x) (m, x) s(s 1 ), f (x) } φ g (succ π2 ) π 3 {(g(p, m, x), m + 1, x) (p, m, x) s(s 5 ), g(p, m, x) } ={( f (x), 0, x), (g( f (x), 0, x), 1, x), (g(g( f (x), 0, x), 1, x), 2, x), (m, x) s(s 1 )} ={(h(i, x), i, x) i = 0, 1, ; (m, x) s(s 1 )} s(s 2 ) ={(h(m, x), m, x) (m, x) s(s 1 ); h(m, x) } s(s 3 ) ={h(m, x) (m, x) s(s 1 ); h(m, x) }. h r : N k N n φ h r 1 N N k π 23 π 2 π 34 1 (id r) N k φ f 0 id N N N k N n π 14 N N n π N N N k φ g (succ π2 ) π 3 3 (). f : N k+1 N M N (19) h : N k N. N k (1) id 0 N k N (4) φ π1 π 2 f π 1 (succ π 2 ) N k N N (2) N k N N (5) π 23 id π π 3 1 N N (6) π 1 succ N (3) succ 0 N (7) 38

52 N k φ id 0 r N k N N N n id π 1 (succ π 2 ) π 4 N k N N n φ π1 π 2 ( f π 12 ) π 4 N k N N N n π 3 1 N succ 0 π succ π 234 N N N n π 13 N N n 5: s s(s 3 ) ={t N t 0} s(s 2 ) ={(x, m, t) (x, m, t) s(s 5 ), t 0} s(s 4 ) ={(x, 0) x s(s 1 )} {(x, m + 1) (x, m, t) s(s 2 )} ={(x, 0) x s(s 1 )} {(x, m + 1) (x, m, t) s(s 5 ), t 0} s(s 5 ) ={(x, m, f (x, m)) (x, m) s(s 4 ); f (x, m) } ={(x, 0, f (x, 0)) x s(s 1 ); f (x, 0) } {(x, m + 1, f (x, m + 1)) t 0, (x, m, t) s(s 5 ); f (x, m + 1) } ={(x, m, f (x, m)) x s(s 1 ); m N, f (x, m) ; f (x, t), f (x, t) 0, t = 0,, m 1} s(s 6 ) ={(m, f (x, m)) x s(s 1 ), m N, f (x, m), f (x, m) = 0; f (x, t), f (x, t) 0, t = 0,, m 1} s(s 7 ) ={m m N, x s(s 1 ), f (x, m), f (x, m) = 0; f (x, t), f (x, t) 0, t = 0,, m 1} h r : N k N n φ h r N 2 = {0, 1} σ 2 N N σ = {(i, σ[i]) i = 0, 1,, σ 1} {( σ, 0), ( σ, 1)} (20) σ σ σ[i] i 2 σ N σ = σ 1 i=0 2 i σ[i] + 2 σ 1 (21) ϵ 0 ϵ 0, 0 1, 1 2, 00 3, 10 4, 01 5, 11 6, 000 7, 100 8, 010 9, , (20) N N (21) id N N (1) N N (2) ((succ π 1 ) π 2 ) 1 s N N (4) (0 id) 1 N (3) π 1 (succ π 2 ) s(s 4 ) ={(n, m + 1) (n, m) s(s 2 )} s(s 2 ) ={(n, m) (n + 1, m) s(s 4 )} s(s 1 ) ={(n, m) (n + 1, m 1) s(s 2 )} s(s 1 ) (22) (n, m) s(s 1 ) (n, m), (n 1, m+1), (n 2, m+ 2),, (0, m + n) s(s 2 ). s(s 3 ) ={s (0, s) s(s 2 )} ={n + m (n, m) s(s 1 )} (22) add (succ 0) 1 id N N (1) N N (2) (succ 1) 1 s succ 1 N (3) N N (4) id 0 s(s 3 ) ={n (n, 0), (n, 1) s(s 1 )} id ((succ π 1 ) π 2 ) 1 (23) 39

53 s(s 4 ) ={(i, 0) i = 0,, n 1; n s(s 3 )} {(i, 1) i = 0,, n; n s(s 3 )} s(s 2 ) =s(s 1 ) s(s 4 ) (20) σ s(s 1 ) = σ s(s 2 ) = {(i, σ[i]) i = 0, 1,, σ 1} {( σ, 1)} (23) S 1 S 2 φ π 121 (π N N (1) 12 0) N N N 1 (2) π 1 (add π 22 ) π 3 s N N N (4) (π 12 (succ π 3 )) 1 N N (3) s(s 4 ) ={(n, m, k) (n, m, k + 1) s(s 2 )} s(s 2 ) ={(n, 2m, k) (n, m, k + 1) s(s 2 )} {(n, m, n) (n, m) s(s 1 )} s(s 3 ) ={(n, 2 n m) (n, m) s(s 1 )} (24) (24) S 1 S 3 ψ N N (1) ψ φ N N (4) (π 1 0) 1 (π 1 1) 1 π 12 1 N N (2) π 2 N (3) ((succ π 1 ) (succ π 2 )) 1 succ add N N N (5) N N (6) π 13 1 (0,0) (25) s s(s 1 ) = σ (20) σ n 1 k n = 2 i σ[i] i=0 s(s 4 ) ={(n, 2 n σ[n]) n = 0, 1,, σ 1} {( σ, 2 σ )} s(s 5 ) ={(n, m, k) (n, m) s(s 4 ), (n, k) s(s 6 )} s(s 6 ) ={(n + 1, m + k) (n, m, k) s(s 5 )} {(0, 0)} ={(n, k n ) n = 0,, σ } {( σ + 1, k σ + 2 σ )} s(s 2 ) ={(n, m) (n + 1, m + 1) s(s 6 ), (n, 0), (n, 1) s(s 1 )} ={( σ, k σ + 2 σ 1)} s(s 4 ) ={k σ + 2 σ 1} = {N σ }. (25) π succ (succ π 2 ) π 3 N (1) N N N (2) N N N (3) (succ π 1 ) π 3 N N (4) N N (7) π 23 ((succ π 1 ) π 2 ) 1 id (0 id) 1 π 1 0 (succ π 2 ) N N (5) id N N (8) π 2 1 N (10) ((succ π 1 ) π 23 ) 1 π 23 (0 id) 1 id 0 s s(s 1 ) = {N} s(s 2 ) ={(N + 1, 0, 0)} id π 1 1 π 13 succ N (6) 1 N N (9) π 2 0 N N (11) {(n, m + 1, k) (n + 1, m, k) s(s 2 )} {(n, 0, k + 1) (n, k) s(s 5 )} (26) (n, 0, k) s(s 2 ) 3 k (n, 0, k), (n 1, 1, k),, (0, n, k) s(s 2 ) n (n/2, n/2, k) s(s 2 ) (n/2, k) s(s 9 ) ((n 1)/2, (n+1)/2, k) s(s 2 ) ((n + 1)/2, k) s(s 4 ), ((n 1)/2, k) s(s 7 ) ( n/2, k) s(s 8 ) n/2 = 0 ( n/2, k) s(s 5 )( n/2, 0, k + 1) s(s 2 ) n (n/2, k) s(s 9 ) (k, 0) s(s 11 ), ((n 1)/2, k) s(s 7 ) (k, 1) s(s 11 ) x 0 = N + 1, xk 1 x k =, 2 y k = x k 2x k+1 (0, y 0 ), (1, y 1 ),, (M, y M ) s(s 11 ) 40

54 M N + 1 < 2 M+1 n s(s 7 ) s(s 9 ) (0, M) M s(s 10 ) (M, 0) s(s 11 ). (21) N = N σ M = σ x k = σ 1 i=k y k = σ[k] x σ +1 = 0 y σ = 1 2 i k σ[i] + 2 σ k, (k = 0,, σ 1) (0, σ[0]),, ( σ 1, σ[ σ 1]), ( σ, 1), ( σ, 0) s(s 11 ) σ = s(s 11 ) (26) M M N (S, S, M ) S S = N N T = N N M (20) 2 S 2 T (S, S, M, {1}, s 1, S, T). M N M 1 M N 6 M N 19. U σ K U (σ) = min p. p 2,U(p)=σ 3. U c U N σ I( σ M N ) 6K U (σ) + c U.. 2 U M N (S, S, M ) p U(p) = σ p = K U (σ) S S = N N T = N N Γ(S, S, M s 1 ) s s(s ) = p s(t) = σ 0 N (A,0) succ N (A,1) succ succ N (A, p ) i = 0, 1,, p 1 S A,i S N (A,i) id p[i] N N id p[i] : N N N p[i] = 0 id 0 p[i] = 1 id (succ 0) S A, p S id 0 N (A, p ) N N id 1 s s(s ) = p s(t) = σ p 1 succ id p[i] succ 1 id (succ 0) 5 6 p + σ 8 σ M N σ σ X 2 = {0, 1} AND OR 1 0 X X 0 1 f : X 2 41

55 X 2 X Y X X Y 2 X 2 Y X χ χ : 2 X 2 X f 2 X χ( f ) = 1 f χ X C X f C f C 4. M N (S, S, M ) (S, S, M, {1}, s 1, S ) σ σ S = N N σ U d U, e U N σ 2 K U (σ) d U I( σ M N ) + e U. S T t x t S T φ in(t) t T y φ t X x t t y φ t t φ X C Γ(S, S, M s 1 ) C X C i) φ M : a) φ f : U T f : 2 U 2 T C t T \ f (U) y φ t = 0 t f (U) y φ t = x u. u f 1 (t) b) φ f : T U f 1 : 2 U 2 T C t T y φ t = x f (t) ii) in(t) S \ S T t T C x t = y φ t. φ in(t) iii) in(t) S T t T C x t = y φ t. φ in(t) iv) (S, S, M, {1}, s 1, S ) 1 0 x 0 C x 0 = 1 x X C χ x C X X ν : X N X g S s g T S, t T g(x t ) = 1 t s g (T) s g g C Γ(S, S, M s 1 ) g C T S \ S g(x t ) = φ in(t) g(y φ t ) t s g (T) g(x t ) = 1 φ in(t), g(y φ t ) = 1 φ in(t), t φ(s g (dm(φ))) t φ(s g (dm(φ))) φ in(t) (1) T S g(x t ) = φ in(t) g(y φ t ) t s g (T) g(x t ) = 1 φ in(t), g(y φ t ) = 1 φ in(t), t φ(s g (dm(φ))) t φ(s g (dm(φ))) φ in(t) (2) g(x 0 ) = 1 s g Γ(S, S, M s 1 ) s g Γ(S, S, M s 1 ) g C i N X X i X 0 = {x 0 }, 42

56 X i+1 = X i {x X x X i }. x X X Y χ x Y χ x Y X g C g(x 0 ) = 1 X i x g(x) = 1 X τ i x X i \ X i 1 τ(x) = i x X χ x x = y j y j τ(y j ) < τ(y) y j τ(x) = τ(y j ) + 1 χ x x = y j τ(x) = min y j τ(y j ) + 1 τ X h h(x) = 1 τ(x) <. h C χ C h i) χ x = y j y j a) h(x) = 1 y j h(y j ) = 0 b) h(x) = 0 y j h(y j ) = 1 h y j h(y j ) = 0 τ(y j ) = τ(x) = y j τ(y j ) < τ(y j ) k y j X k τ(x) = k + 1 ii) χ x = y j a) h(x) = 1 y j h(y j ) = 0 b) h(x) = 0 y j h(y j ) = 1 y j h(y j ) = 0 τ(y j ) = τ(x) = y j h(y j ) = 1 τ(y j ) < τ(x) τ(y j ) + 1 τ(x) < x X X τ(x) Y x x Y x, y Y x, χ y = y = y j y j Y x, y Y x, χ y = y = y j arg min y j τ(y j )+1=τ(y) ν(y j ) Y x. C Y x y y j Y x y j τ(y j ) < τ(y) χ y y = y j y = y j τ Y x Y x i = Y x X i y Yx i χ y y = y j y j Yx i 1 y = y j y j Yx i 1 i = 1,, τ(x) Yx i Yi 1 x IsOne(x) 1 for each Z X, s. t. x 0, x Z 2 Z 0 {x 0 } 3 for i = 1, 2, 4 Z i Z i 1 5 for each z Z 6 z Z i 1 7 until Z i = Z i 1 z Z i 8 until i x Z i 1 Z X Z Z = {x 0, x} k ν 1 (k) Z i X i IsOne(x) x X i τ(x) <. τ(x) < Y x 2 Y x Z i Y i x Yi 1 x Y i x Z i x Z τ(x) IsOne(x) σ S = N N (S, S, M, {1}, s 1, S ) Γ(S, S, M s 1 ) s s(s ) = σ h C s h (S ) = σ s h h t S h(x t ) = 1 t σ (i, b) S (i, b) σ τ(x (i,b) ) < 43

57 1 (S, S, M, {1}, s 1, S ) 2 ρ 3 for i = 0, 1, 4 IsOne(x t ) 0: t = (i, 0) 1: t = (i, 1) i > 0 2: t = (i 1, 1 ρ[i 1]) 5 until 6 if i > then ρ[i 1] ρ 8 else if 9 then ρ[i] 0 10 else if 1 11 then ρ[i] 1 i = 0, 1,, σ 1 σ[i] i = σ 0 1 i = σ σ (S, S, M ) I( σ M N ) U U σ (S, S, M, {1}, s 1, S ) I( σ M N ) σ d U, e U N K U (σ) d U I( σ M N ) + e U 3 4 I( σ M N ) K U (σ)

58 [1] L. Blum, F. Cucker, M. Shub, and S. Smale, Complexity and Real Computation, Springer Verlag, [2] G. S. Boolos, J. P. Burgess, and R. C. Jeffrey, Computability and Logic, Fifth Edition, Cambridge University Press, [3] P. Bürgisser, M. Clausen, and M. A. Shokrollahi, Algebraic Complexity Theory, Springer Verlag, [4] G. J. Chaitin, On the Length of Programs for Computing Finite Binary Sequences, J. Assoc. Comput. Mach., vol. 13, pp , [5] G. J. Chaitin, On the Length of Programs for Computing Finite Binary Sequences: Statistical Considerations, J. Assoc. Comput. Mach., vol. 16, pp , [6] H. Ishikawa, Representation and Measure of Structural Information, arxiv: November Revised, June [7] A. N. Kolmogorov, Three Approaches to the Quantitative Definition of Information, Problems of Information Transmission, vol. 1, pp. 4 7, [8] M. Li and P. Vitányi, An introduction to Kolmogorov complexity and its applications (2nd ed.), Springer- Verlag, [9] J. Rissanen, Modeling by the shortest data description, Automatica, vol. 14, pp , [10] J. Rissanen, Information and Complexity in Statistical Modeling, Springer, [11] C. E. Shannon. The mathematical theory of communication. Bell System Tech. J., vol. 27, pp , , [12] R. J. Solomonoff, A Preliminary Report on a General Theory of Inductive Inference, Report ZTB-135, Zator Co., Cambridge, MA, [13] R. J. Solomonoff, A Formal Theory of Inductive Inference, Information and Control, vol. 7, pp. 1 22, , [14] A. M. Turing, On Computable Numbers, with an Application to the Entscheidungsproblem, Proceedings of the London Mathematical Society, Ser. 2, Vol. 42, pp ,

59 情 報 論 的 学 習 理 論 テクニカルレポート 2009 Technical Report on Information-Based Induction Sciences 2009 (IBIS2009) 顔 と 人 体 画 像 認 識 に 生 きる 機 械 学 習 * Machine Learning as a Tool for Face and Human Detection 勞 世 竑 Shihong Lao 概 要 : 近 年, 顔 画 像 認 識 技 術 が 急 速 に 実 用 化 してきた.その 代 表 的 な 応 用 と して, 広 く 普 及 されたのは 顔 検 出 機 能 を 搭 載 したデジタルカメラやビデオカ メラである.その 他 にセキュリティ 分 野 における 顔 認 識 技 術 の 応 用 や, 自 動 マーケティングにおける 性 別 年 齢 推 定 技 術 の 応 用 なども 実 用 化 されている. これらの 技 術 の 実 現 には 機 械 学 習 がなくてはならない 重 要 な 役 割 を 果 たして いる. 本 稿 では, 主 に 顔 検 出 の 開 発 を 事 例 として,どのように 機 械 学 習 技 術 を 応 用 し,そしてどのようにこれらの 技 術 が 実 応 用 されているかを 紹 介 する. Keywords: 顔 検 出, 人 体 検 出,SGF 特 徴 量,AdaBoost,Real AdaBoost 1 はじめに デジタルカメラやビデオカメラに 顔 検 出, 顔 認 識 の 機 能 が 搭 載 され, 身 近 に 最 先 端 のパターン 認 識 技 術 の 恩 恵 を 受 けることにできるようになった.これが 可 能 になっ たのは,ハードウェアの 進 歩 による 面 もあるが, 機 械 学 習 の 理 論 と 技 術 の 発 展 によって, 大 量 データによる 学 習 が 可 能 になり, 性 能 的, 速 度 的 に 実 用 的 なアルゴリズム が 実 現 できたお 陰 だと 言 っても 過 言 ではない. 本 稿 では, 顔 画 像 処 理 の 実 応 用 の 現 状 を 簡 単 に 紹 介 し てから, 主 に 顔 検 出 技 術 の 実 用 化 に 際 して, 実 施 したい くつかの 改 良 と 機 械 学 習 を 行 う 際 の 工 夫 について 説 明 す る. 2 顔 画 像 処 理 技 術 の 概 況 近 年, 顔 検 出 や 顔 認 識 などの 技 術 がリアルタイム で 実 現 できるようになったことで, 顔 画 像 処 理 の 応 用 分 野 が 急 速 に 広 がっている. 図 1に, 主 な 要 素 技 術 とそれらを 応 用 したアプリケーションの 例 を 示 す. * 企 画 セッション 広 がる 機 械 学 習 応 用 のフロンティア オムロン 株 式 会 社, コアテクノロジーセンタ 木 津 川 市 木 津 川 台 9-1, tel , lao_shihong@omron.co.jp, OMRON Corp. 顔 検 出 顔 による オートフォーカス 顔 器 官 検 出 顔 サイズ 正 規 化 赤 目 補 正 20 代, 女 性 逆 光 逆 光 顔 認 識 セキュリティ アルバム 検 索 顔 属 性 推 定 自 動 マーケティング 調 査 顔 状 態 推 定 ドライバーモニター 表 情 認 識 笑 顔 シャッター 図 1 顔 画 像 処 理 技 術 の 要 素 技 術 例 顔 画 像 処 理 の 主 な 応 用 分 野 として, 次 のようなも のがあげられる: 1) デジタル 機 器 分 野 写 真 やビデオの 画 質 向 上 に 繋 がる 応 用 を 目 的 とし て,デジタルカメラ,ビデオカメラのオートフォ ーカスやプリンタ, 写 真 現 像 機 などにおける 自 動 補 正 機 能 が 実 用 化 されている.また, PC や 携 帯 電 話 などにおける 個 人 認 証 や 映 像 インデクシング, 写 真 整 理 などに 応 用 されている. 2) エンタテインメント 分 野 顔 写 真 を 楽 しむためのサービスも 多 い. 写 真 シー ル 機 の 写 真 画 質 向 上, 顔 写 真 による 占 いサービス や 写 真 変 装 サービスが 若 年 層 を 中 心 に 使 われてい る.また 携 帯 型 ゲーム 機 器 にも 顔 画 像 処 理 が 導 入 されている. 3) 車 載 分 野 顔 画 像 処 理 によってわき 見 や 居 眠 りを 検 出 するド ライバーモニターが 搭 載 された 事 例 がある. 4) セキュリティ 分 野 46

60 駅, 空 港, 商 業 施 設 などでの 不 審 者 検 出, 出 入 国 管 理, 施 設 へのアクセスコントロールなどの 分 野 で 顔 認 識 技 術 が 実 用 化 されている. の 良 い 顔 検 出 技 術 として 知 られていたが, 計 算 速 度 に 関 しては 実 時 間 処 理 には 及 ばなかった. 3 顔 検 出 技 術 顔 画 像 処 理 の 中 で, 最 も 基 本 的 で, 重 要 な 技 術 と して 顔 検 出 技 術 が 上 げられる.デジタル 写 真 や 映 像 が 急 速 に 普 及 している 中 で, 顔 検 出 技 術 の 役 割 がま すます 重 要 になってきた. 例 えば,コンピュータで 顔 認 識 を 行 う 場 合 は 画 像 の 中 からどこに 顔 があるか をまず 検 出 する 必 要 がある. 特 にデジタルカメラに おける 顔 オートフォーカスや 顔 オートアイリスには カメラに 搭 載 でき,リアルタイム 処 理 できる 顔 検 出 技 術 が 必 要 である. 領 域 切 出 し 大 きい 顔 検 出 顔 非 顔 を 判 定 小 さい 顔 検 出 顔 非 顔 3.1 顔 検 出 の 課 題 顔 検 出 の 難 しさ コンピュータで 顔 をロバストに 検 出 する 難 しさは 顔 の 多 様 性 に 由 来 する. 人 には 個 人 差 があり, 性 別, 年 齢, 人 種 などの 違 いによって, 顔 の 見 た 目 はかな り 違 う.また, 同 じ 人 の 顔 でも, 顔 の 向 き, 照 明 環 境 と 表 情 によっても 見 え 方 が 違 う. 実 アプリケーシ ョンの 場 合, 顔 の 大 きさ, 回 転 角 度 も 変 化 している ことが 多 いので,さらに 複 雑 になる( 図 2). 向 き 眼 鏡 照 明 表 情 図 2 顔 の 多 様 性 従 来 の 顔 検 出 技 術 とその 課 題 人 はまったく 努 力 せずに, 自 然 に 顔 を 見 つけるこ とができる.しかしながら,コンピュータで 顔 を 検 出 するには 画 像 の 中 で 順 番 に 領 域 を 切 り 出 して,そ の 領 域 が 顔 かどうかを 判 定 することによって 顔 の 位 置 を 特 定 する( 図 3).そのため, 膨 大 な 数 の 領 域 の 判 別 が 必 要 となり, 高 速 な 判 別 器 が 重 要 になる. 90 年 後 半 CMU の Rowley らが 開 発 したニューラル ネットワークによる 顔 検 出 技 術 は 学 習 技 術 を 使 った 顔 検 出 のフレームワークを 確 立 し, 当 時 は 最 も 性 能 図 3 領 域 切 り 出 して 顔 かどうかを 逐 一 判 別 する ことで 顔 を 検 出 する 方 法 2001 年 ごろ,ViolaとJones[1]が 高 速 に 顔 検 出 を 行 う 手 法 を 提 案 し, 実 時 間 での 顔 検 出 を 可 能 にした. この 手 法 は 近 年 最 も 脚 光 を 浴 びたパターン 認 識 手 法 の 一 つである. 従 来 の 顔 識 別 器 は 入 力 された 画 像 が どんなものであってもすべて 同 じ 処 理 を 行 って 顔 か どうかを 判 別 する. 彼 らは 画 像 の 中 の 多 くの 部 分 は 顔 ではなく,ほとんどの 領 域 は 明 らかに 顔 に 似 てな いことに 注 目 し,これらの 領 域 においてはより 単 純 な 計 算 で 高 速 に 顔 ではないと 判 断 できることを 巧 妙 に 利 用 して 高 速 化 を 実 現 した. 彼 らの 方 法 には 3 つ 重 要 な 貢 献 がある: 1. 積 分 画 像 を 用 いて, 単 純 で 解 像 度 に 関 係 なく 高 速 演 算 できる Haar-like の 特 徴 量 を 導 入 した. 2. AdaBoost アルゴリズムを 導 入 して 上 述 した 高 速 に 計 算 できる 特 徴 量 を 使 った 弱 判 別 器 を 選 定 し, 線 形 結 合 で 強 判 別 器 を 構 築 する 方 法 を 提 案 した. 弱 判 別 器 は 単 純 な 閾 値 演 算 でバイナリの 出 力 を 決 める. 3. 計 算 量 の 少 ない 判 別 器 と 計 算 量 の 多 い 判 別 器 を 直 列 に 順 番 につなぐ 構 造 の 顔 検 出 器 を 提 案 した. 計 算 量 の 少 ない 識 別 器 で 非 顔 と 判 断 された 場 合 は 途 中 で 計 算 を 打 ち 切 ることで, 画 像 の 中 で 大 部 分 の 領 域 での 計 算 を 減 らし, 高 速 な 顔 検 出 を 実 現 した. ViolaとJonesが 提 案 したHaar-likeの 特 徴 量 は 矩 形 の 中 の 領 域 の 平 均 輝 度 差 である( 図 4 に 白 い 領 域 と 黒 い 領 域 との 平 均 輝 度 差 ).ViolaとJonesは 4 種 類 の Haar-likeの 矩 形 特 徴 量 を 提 案 している.これらの 特 徴 量 の 良 いところは 領 域 の 大 きさによらず, 任 意 領 域 の 平 均 輝 度 値 は 積 分 画 像 を 使 って 3 回 の 足 し 算, 引 き 算 と 1 回 の 割 り 算 で 高 速 に 計 算 できることであ る( 図 5).ここで, 積 分 画 像 の 画 素 S(x, y)と 元 画 像 の 画 素 I(i, j)の 関 係 は 次 のように 定 義 される: 47

61 S( x, y) = i x j y i= 0 j = 0 I( i, j) 図 6 図 7)である.gはその 領 域 の 平 均 輝 度 とし, 元 画 像 をスケール 1/2 s に 縮 小 した 画 像 をあらかじめ 生 成 しておけば, 各 粒 子 gの 値 は 直 接 各 1/2 s に 縮 小 し た 画 像 の 画 素 値 から 得 られるため, 計 算 量 が 少 ない. 図 4 Viola と Jones が 提 案 した 4 種 類 の Haar-like の 特 徴 量, 隣 接 する 白 い 矩 形 領 域 の 平 均 輝 度 と 黒 い 矩 形 領 域 の 平 均 輝 度 の 差 で 表 す reference window g(4,5,2) g(13,3,1) (scale, size, number) (0, 1 1, 576) (1, 2 2, 529) (2, 4 4, 441) A C a c B D b d g(10,14,3) (3, 8 8, 289) 図 6 SGF 特 徴 量 (Sparse Granular Feature)の 概 念 積 分 画 像 図 5 元 画 像 中 のある 領 域 の 輝 度 合 計 を 矩 形 の 大 きさによらず 3 回 の 加 減 算 だけで 求 めることができ る (D=d-c-b+a) Viola らの 手 法 は 非 常 に 高 速 で, 画 期 的 であったが, 正 面 顔 以 外, 横 顔 も 対 応 する 必 要 がある 場 合, 性 能 的, 速 度 的 に 不 十 分 であった. 実 アプリケーション に 応 用 すると, 更 なる 高 速 化 や 検 出 性 能 の 向 上 とハ ードウェア 化 しやすくするためのメモリ 使 用 量 の 削 減 などが 要 求 される. 3.2 実 用 化 に 向 けた 顔 検 出 の 改 良 実 アプリケーションにおいて, 顔 検 出 に 対 して, 以 下 のようなニーズがある: 写 真 印 刷 で 逆 光 や 白 とびの 顔 の 輝 度 を 自 動 補 正 する 場 合,カメラの 持 ち 方 などによって, 顔 が 上 向 きとは 限 らないため, 回 転 した 顔 をも 検 出 する 必 要 がある. 顔 の 向 きは 正 面 だけでなく, 横 顔 の 対 応 も 必 要. タイプのもので, 顔 と 非 顔 の 弱 判 別 器 における 入 力 組 み 込 み 機 器 に 搭 載 するためにはプログラムの データの 分 布 がそれぞれ 分 かれた 場 合 には 有 効 であ ROM と RAM の 使 用 量 を 削 減 する 必 要 がある. るが,そうでない 場 合 は 弱 判 別 器 の 識 別 能 力 が 落 ち これらのニーズに 答 えるために, 特 徴 量, 学 習 ア る 弱 点 がある( 図 8にそのイメージ 図 を 示 す).し ルゴリズム, 検 出 器 の 構 造 に 対 して 改 良 を 加 えてい かしながら, 実 際 の 弱 判 別 器 の 入 力 データの 分 布 は る. 複 雑 なものが 多 いため( 図 9),そこに 改 善 の 余 地 特 徴 量 の 改 良 があった. 筆 者 のグループでは,Haar-likeの 特 徴 量 よりもよ り 高 速 に 計 算 でき,しかも 識 別 能 力 が 高 い SGF 特 徴 量 (Sparse Granular Feature)を 提 案 している[2]. F( π ) = α i gi ( π; x, y, s), α i { 1, + 1} i ここで, π は 入 力 画 像 の 濃 淡 データ, g i ( π; x, y, s) はGranule 粒 子 でx,yは 位 置,sはサイズを 表 すパ ラメータ( 図 7 学 習 した 結 合 粒 子 特 徴 量 の 例 黒 はα=-1, 白 はα=1を 示 す このような SGF 特 徴 量 は 速 度 的 にも 性 能 的 にも 優 れていることが 実 証 できている.また, 計 算 が 単 純 なため,ハードウェアでの 実 現 がしやすいメリット がある 学 習 アルゴリズムの 改 良 ブースティングアルゴリズムは 複 数 の 判 別 能 力 の 低 い 弱 判 別 器 hを 組 合 わせてより 性 能 の 高 い 強 判 別 器 Hを 構 成 する 方 法 である.AdaBoostは 適 応 型 の 学 習 法 で, 弱 判 別 器 を 組 合 わせる 方 法 を 問 題 に 適 応 して 決 める.ViolaとJonesがAdaBoostを 最 初 に 顔 検 出 に 適 用 し, 優 れた 学 習 能 力 があることを 示 した. 但 し, ViolaとJonesの 顔 検 出 アルゴリズムの 場 合, 弱 判 別 器 の 出 力 は 閾 値 処 理 によってバイナリの 値 を 出 力 する 48

62 Pr ob abi lis tic de Positive Negative Pr ob abi lis tic de Positive Negative 1 If f ( x) bin, then h( x ) = ln 2 ここで, j W l = P( f ( x ) bin, y = l), l = ± 1, j = 1,..., n. j W W j + 1 j 1 + ε + ε Feature 閾 値 判 別 に 適 するケース 図 8 弱 判 別 器 の 閾 値 によるバイナリな 判 別 判 別 誤 差 Positive Feature 閾 値 判 別 に 適 しないケース 図 9 弱 判 別 器 の 比 較 例,バイナリの 出 力 の 場 合 は 誤 差 が 大 きい より 判 別 能 力 の 高 い 弱 判 別 器 として, 出 力 値 を 実 数 で 表 せるものを 提 案 した.これは 計 算 上 の 利 便 性 を 考 え,Parewise Function(ロックアップテーブル) によって 実 装 した 図 10).ここで,W +1 とW -1 はそ れぞれ 顔 と 非 顔 の 分 布 で,ヒストグラムで 表 す.こ こで,Real AdaBoostを 使 って 学 習 を 行 うことができ る. j 1 W ln + 1 h = j j h(x) 2 W 1 Negative Output (a) (b) binary (c) real Positive Negative Output 実 験 の 結 果,Real Adaboost 学 習 アルゴリズムは 離 散 Adaboost よりすぐれた 性 能 が 得 られることがわか った. 特 に 最 初 の 数 層 のカスケードにおいて, 少 な い 弱 判 別 器 しか 使 用 してないため,Real Adaboost の ほうがより 正 確 な 確 信 度 を 出 力 できる. SGF 特 徴 量 の 場 合, 可 能 な 形 の 特 徴 量 の 探 索 空 間 非 常 に 広 いため,ヒューリスティックな 探 索 手 法 を 導 入 する 必 要 がある. 特 徴 量 の 探 索 はローカルサー チを 行 い, 以 下 の 3 つの Expansion operator を 使 う: 1. Remove a granule:sgf の 中 の 一 部 を 外 す 2. Add a granule:sgf の 中 に 一 部 を 足 す 3. Refine:SGF の 相 対 位 置 を 調 整 する その 他, 初 期 のSGFの 選 び 方 などにの 内 容 に 関 し ては 参 考 文 献 [2]をご 参 照 ください. F 学 習 データの 整 備 bin j f(x) 高 性 能 な 識 別 器 を 構 築 するために, 最 も 性 能 に 影 Piece-wise function 響 するのは 学 習 データの 質 である. 我 々は 長 年 学 習 図 10 顔 らしさを 実 数 で 出 力 する 弱 判 別 器 データの 充 実 化 を 図 ってきた. 例 えば, 顔 の 向 きに 対 応 するためには 同 時 に 80 種 類 の 異 なる 顔 向 きのデ Real AdaBoost は Schapire と Singer によって 提 案 さ ータを 採 取 するためのマルチカメラ 顔 画 像 データ 収 れたもので,サンプル 空 間 X から 2 値 の 予 測 でなく, 集 装 置 を 作 成 した( 図 12). 確 信 度 に 関 する 実 数 の 空 間 R へ 射 影 する 判 別 器 の 学 習 アルゴリズムである. 以 下,Real Adaboost を 使 って 学 習 を 行 う 時 に, 計 算 上 の 利 便 性 を 考 慮 して, 学 習 した 出 力 は 等 間 隔 に 分 けた 入 力 空 間 における 実 数 型 のテーブルとする. 仮 に 特 徴 量 f が[0, 1]に 正 規 化 したものとし,この 領 域 を n 個 のビンに 分 割 する. bin j = [(j-1)/n, j/n], j = 1,,n そして, 弱 判 別 器 は 次 のように 定 義 できる: F * remove F * * refine F add 図 11 SGF のローカルサーチ 49

63 図 14に 顔 検 出 の 検 出 例 を 示 す. 図 12 マルチアングルの 顔 データ 採 取 装 置 3.3 ハードウェアによる 実 現 デジタルカメラで 顔 検 出 を 行 う 場 合, 速 度 が 非 常 に 重 要 になる.そのため,リアルタイムでの 検 出 を 可 能 にするハードウェアでの 実 装 が 必 要 になる.いか に 少 ないゲート 数 で 高 速 な 検 出 器 を 実 装 できるかは ハードウェア 設 計 者 の 腕 の 見 せ 所 と 言 える. 設 計 に おいて, 重 要 なポイントは2つある: 1. 大 量 演 算 処 理 の 並 列 化 データを 分 割 して 並 列 処 理 するか, 識 別 の 演 算 を 分 解 して 並 列 処 理 するかを 最 適 に 行 う. 2. メモリアクセスの 効 率 化 並 列 処 理 と 同 時 に 考 慮 する 必 要 があるのはメモ リアクセスの 効 率 化. 回 路 規 模 を 減 らすために は 内 部 で 使 うメモリを 最 小 限 にすることが 有 効 で, 共 有 メモリを 設 けるのが 良 いが,そのアク セスのタイミングを 上 手 に 設 計 しないと 他 の 処 理 のメモリアクセスとバッティングしてしまう ことが 起 こり, 計 算 速 度 の 低 下 を 招 くことがあ る. 4 人 体 検 出 の 実 現 に 向 けて また, 収 集 されたデータに 顔 や 顔 器 官 の 正 解 点 位 置 情 報 を 入 力 するためには,コストの 安 い 中 国 での I 顔 検 出 よりも 難 しいのは 人 体 の 検 出 である.それは データ 入 力 を 実 施 した. 顔 よりも 人 体 の 見 え 方 の 変 化 の 幅 がより 広 いからである 顔 の 回 転 角, 向 きの 対 応 現 在 人 体 検 出 の 研 究 開 発 がまだ 模 索 中 で, 顔 検 出 のフ 全 方 向 の 顔 検 出 を 実 現 するために, 顔 の 回 転 角, と 顔 の 向 き( 正 面 顔, 半 横 顔, 横 顔 )への 対 応 はそ レームワークが 人 体 検 出 にも 有 効 であることが 確 認 でき た. 顔 と 違 って, 人 体 の 場 合, 内 部 の 模 様 が 服 装 によっ れぞれの 検 出 器 を 用 意 することで 対 応 している. 顔 て 変 化 するため, 安 定 した 特 徴 抽 出 が 難 しい.そのため, の 向 きが 5 方 向, 顔 の 回 転 角 が 12 方 向 を 用 意 する 必 人 体 の 輪 郭 の 特 徴 を 如 何 に 検 出 できるかがキーポイント 要 があって, 合 計 60 個 の 検 出 器 を 使 っている( 図 13). になる. 図 15に 人 体 の 輪 郭 情 報 を 検 出 するための 特 徴 量 の 例 を 示 す.このような 特 徴 量 を 機 械 学 習 のアルゴリ ズムを 通 して, 人 体 の 検 出 器 を 学 習 することができる 左 90 横 左 90 半 横 正 90 面 右 90 半 横 右 90 横 0 図 16に 人 体 検 出 の 結 果 の 例 を 示 す 図 13 全 方 向 対 応 顔 検 出 器

64 Robust to changes of pose Scale invariant Robust to changes of expression Rotation invariant Robust to bad lighting conditions Robust to occlusion 図 14 顔 検 出 の 結 果 例 図 15 人 体 検 出 のために 開 発 した 勾 配 情 報 を 扱 う SGF 図 16 人 体 検 出 の 結 果 例 5 まとめ 顔 検 出 と 人 体 検 出 の 開 発 において, 機 械 学 習 が 極 めて 重 要 である. 実 用 化 においては 速 度 の 向 上 プロ グラムサイズ 及 び 実 行 時 メモリ 使 用 量 の 削 減 も 重 要 な 課 題 になる. 顔 検 出 において, 実 応 用 多 様 な 顔 の 向 きを 対 応 でき 高 速 に 計 算 できる 特 徴 量 Sparse Granular Feature が 実 用 化 の 鍵 となった.また,Real AdaBoost による 学 習 アルゴリズムは 強 力 な 機 械 学 習 手 法 で, 性 能 向 上 に 有 効 な 手 法 である. 学 習 データ は 量 と 質 も 識 別 器 の 性 能 を 左 右 するため, 地 道 なデ ータ 収 集, 整 理 が 欠 かせない. 人 体 検 出 の 開 発 にも 顔 検 出 の 手 法 が 活 用 されているが, 人 体 の 見 え 方 の 変 化 の 幅 が 広 く,さらに 強 力 な 学 習 アルゴリズムの 出 現 を 期 待 する. 将 来 的 にはよりロバストな 人 体 検 出 や,より 多 くの 種 類 の 物 体 検 出 の 研 究 開 発 につな げて 行 きたい. 51

65 参 考 文 献 [1] P. Viola, M. Jones, Rapid Object Detection using a Boosted Cascade of Simple Features, In Proc. IEEE Conf. on Computer Vision and Pattern Recognition, Kauai, USA, [2] C. Huang, H. Ai, Y. Li, S. Lao, High Performance Rotation Invariant Multi-View Face Detection, IEEE PAMI Vol. 29, No. 4, pp , [3] W. Gao, H. Ai, S. Lao, Adaptive Contour Features in Oriented Granular Space for Human Detection and Segmentation, CVPR2009 pp ,

66 2009 Technical Report on Information-Based Induction Sciences 2009 (IBIS2009) Abstract: Just-In-Time, 53

67 2009 Technical Report on Information-Based Induction Sciences 2009 (IBIS2009) Vicarious Bayes Learning and its Application to HMMs Keisuke Yamazaki Abstract: Hierarchical parametric models, such as Gaussian mixture models, Bayesian networks, and hidden Markov models, are widely used in the information engineering fields. These models are generally expressed as probability functions of the given data space, and there are a number of learning algorithms for each model. However, it is still unknown whether the space is suitable and effective for learning of the function. Therefore, the present paper considers a feature map to a different domain space, and investigates how the map changes the generalization error. Then, we proposed the vicarious learning in the Bayes estimation, which preserves the error value of the original space in a different space. This new learning framework reduces the computational learning and evaluation costs because a simpler space makes the calculation of the likelihood faster. As one of its applications, we can derive a necessary length of training data for HMMs. Keywords: Bayes Learning, Feature Selection, Generalization Error 1 Introduction Hierarchical parametric models, such as Gaussian mixture models, Bayesian networks and hidden Markov models, are used in a number of practical engineering fields. The parameter space of such models can have singularities due to the hierarchical structure or the latent variables. Then, these models are referred to as singular. Models are regular when the parameter space does not include any singularity. The conventional statistical analysis is established on the basis of unique probabilities of the regular model. The analysis is not available for singular models. More statistically, the inverse of the Fisher information matrix is required to describe the convergence of the optimal parameter. The matrices are not positive definite on the singularities, which means that the inverse matrices do not exist. Therefore, the algebraic geometrical method has been developed to reveal the Bayesian generalization error of singular models [6]. Based on the method, the errors of the hierarchical, R2-5, k-yam@pi.titech.ac.jp, Tokyo Institute of Technology, R2-5, 4259 Nagatuta, Midori-ku Yokohama models are revealed (e.g. [8, 7]). The present paper focuses on a relation between the data space and the generalization error. To project the given data to a different space is a common technique in feature extraction and dimensionality reduction. Models dealing with sequential data have large computational cost for learning and evaluation even though they have effective algorithms [4, 3, 2]. A feature map actually seems to reduce the cost because the models can be simplified in the feature space. However, effect of the map on the error has not been studied yet. The feature space has to be designed properly. Let us consider the following simple example: Discrete data x are assumed to be D dimensional binary vectors. Then, the dimension of the data space is 2 D. The most naive modeling is to provide parameters as probability variables of each x. In this case, 2 D 1 variables are required to represent all x. For example, three variables p 1, p 2, p 3 are sufficient for D = 2, i.e. P (00) = p 1, P (01) = p 2, P (10) = p 3. Note that P (11) = 1 p 1 p 2 p 3. A parametric model p(x w), where w is the parameter, generally has to have less number of parameters than 2 D 1. Now, a feature map projects the data to 2 d dimensional space. The feature 54

68 space is assumed to be much smaller than the original one, d D. It is easy to prove that the parameter w cannot be correctly identified when 2 d 1 < dim w. Thus, a theoretical evaluation of a feature map is required to clarify if the feature space is suitable for the parameter learning. Based on the algebraic geometrical method, the present paper proposes an evaluation method the feature map. The main purpose is to design the model p(x w) through the feature space. Therefore, one of the expected applications is the model selection with the cross-validation [5]. The selected model will be finally used in the original space. Our task is to design the optimal model available not for the feature space, but for the original one. Then, the feature space preserving the generalization error is desired because the parameters can be completely estimated. The present paper defines training and test procedures in such feature space as the vicarious Bayes learning, and the feature map as the vicarious feature map. As for the demonstration, the vicarious feature map will be found for hidden Markov models, and a necessary length of data sequences is derived for the complete learning. The remainder of the paper is organized as follows. Section 2 formalizes the Bayes learning and summarizes important results of the algebraic geometrical method. Section 3 proposes the vicarious Bayes learning. Section 4 shows an application to hidden Markov models (HMMs). Sections 5 and 6 present discussions and our conclusion, respectively. 2 The Bayes Learning and the Algebraic Geometrical Method Let us formally define the generalization error. A set of training data X n = {X 1,..., X n } is independently and identically distributed from the true model q(x). The learning model with its parameter w is generative and is represented as p(x w). The generalization error is the average Kullback divergence from q(x) to the predictive distribution p(x X n ), [ ] q(x) G(n) = E X n q(x) ln p(x X n ) dx, (1) where n is the number of the training data and E X n[ ] represents the expectation value over all training samples. The predictive distribution is constructed by p(x w). For example, the maximum likelihood method gives p(x X n ) = p(x ŵ) (2) where ŵ is the maximum likelihood estimator: ŵ = arg max L(w, w Xn ), (3) n L(w, X n ) = p(x i w). (4) i=1 The Bayes estimation yields the predictive distribution p(x X n ) = p(x w)p(w X n )dw, (5) where the posterior p(w X n ) is defined by p(w X n ) = 1 Z(X n ) L(w, Xn )ϕ(w) (6) using a prior ϕ(w) and the normalization factor Z(X n ). The asymptotic form of Eq.(1) is expressed as G(n) = λ n m 1 ( 1 ) n ln n + o n ln n (7) when W t {w : p(x w) = q(x)} =, i.e. p(x w) can attain q(x) [6]. The coefficients are defined as follows: All poles of the zeta function J(z) = H(w) z ϕ(w)dw (8) are real negative and rational, where the Kullback divergence H(w) = q(x) ln q(x) dx (9) p(x w) is analytic. Then, z = λ is the largest pole and m is its order. Eqs (7)-(9) shows that the generalization error is determined by the relation between q(x) and p(x w). Behavior of H(w) in the neighborhood of W t directly affects λ and m. For example, J(z) = w 2jz dw (j = 1, 2,...) when H(w) = w 2j and ϕ(w) is closed to uniform around W t = {0}. It is easily found that J(z) has a factor 1/(2jz + 1) by integrating over w, which implies λ = 1/2j. In this section, we propose the vicarious Bayes learning. 3 Proposed Learning Framework 55

69 3.1 Mapping to a Feature Space As can be noticed in Eqs (7)-(9), the error G(n) depends on the probabilities p(x w) and q(x). Herein, we consider the error value over a different domain space. Let Φ : x y be a feature map. Based on the map, the models on y are defined by q Φ (y) = q(x)δ(y Φ(x))dx, (10) p Φ (y w) = p(x w)δ(y Φ(x))dx, (11) where δ( ) is the Dirac s delta function. The likelihood function Eq (4) is given by L Φ (w, Y n ) = n p Φ (Y i w), (12) i=1 where Y n = {Y 1,..., Y n } = {Φ(X 1 ),..., Φ(X n )}. The Bayes estimation yields the posterior p Φ (w Y n ) and the predictive distribution p Φ (y Y n ) by replacing p(x w) of Eqs (5) and (6) with p Φ (y w). Then, the generalization error defined as [ ] q Φ (y) G Φ (n) = E Y n q Φ (y) ln p Φ (y Y n ) dy, (13) has the asymptotic form, whose coefficients are determined by the zeta function of H Φ (w) = q Φ (y) ln q Φ(y) dy. (14) p Φ (y w) x and/or y should be substituted for dx and/or dy in the discrete space, respectively. 3.2 Vicarious Bayes Learning Comparing Eq (13) with Eq (1), let us determine how the data space affects the parameter learning. It is known that the generalization error implicitly expresses the tuning cost of all essential parameters. For example, the regular model has the coefficients λ = dim w/2 and m = 1. In singular models, λ also depends on the number of parameters to be tuned [6, 8]. The learning process is preserved if a feature map Φ does not change the error. Observing the error change, we can theoretically investigate which factor of the data space is necessary for the parameter learning. We propose the following learning framework: Definition 1 (Vicarious Bayes Learning) The Bayesian parameter learning and its evaluation of the generalization error over a feature space are referred to as the vicarious Bayes learning when G(n) and G Φ (n) have the common asymptotic form. In the present paper, we refer to this preserving map as the vicarious feature map. Herein we are not interested in the case G(n) > G Φ (n) because our purpose is to investigate model properties on the given space x. The smaller error implies that not all parameters have to be estimated in the feature space, which means that the proper form of p(x w) cannot be obtained. 3.3 A Theory of the Feature Map on the Error A novelty of the proposed learning is to restrict the feature map Φ to the vicarious one in the asymptotic manner. Therefore, we study a condition, under which Φ becomes vicarious. The coefficients of the asymptotic error is determined by behavior of the Kullback divergence H(w) or H Φ (w) in the neighborhood of W t. The divergence is expressed as a polynomial form of w because it is analytic. Based on the Noetherian property of polynomial ring, the divergence consists of bases. For example, if polynomials f 1 (w) and f 2 (w) are bases, the nonnegative function H(w) can have the following form, H(w) = f 1 (w) 2 + f 2 (w) 2 + f 3 (w), (15) where f 3 is a sum of squared polynomials with respect to f 1 and f 2, and is higher order than f 2 1 and f 2 2. More precisely, f 3 consists of terms, such as (f 1 f 2 ) 2, (f 1 + f 2 2 ) 2 and (f 2 1 +f 2 ) 2, with coefficients. It naturally holds that f 1 (w) = f 2 (w) = 0 on W t. The error G Φ (n) will have the same asymptotic form if H Φ (w) is given by H Φ (w) = f 1 (w) 2 + f 2 (w) 2 + f 4 (w), (16) where f 4 consists of the same terms as f 3 with different coefficients. condition: Based on this example, we derive a Theorem 1 A feature map Φ becomes vicarious if H(w) and H Φ (w) have the same essential terms for λ and m. This theorem is easily proved according to the relation of Eqs (7)-(9). Note that Theorem 1 shows a sufficient condition: The largest poles in the zeta functions of 56

70 H(w) and H Φ (w) can be at the same position even if the essential parts are different from each other. A general feature map provides insight of the generalization error even when it is not straightforward to find the vicarious one. Theorem 2 For a feature map Φ, it holds that G(n) G Φ (n). (17) The proof is in Appendix. Theorem 2 intuitively shows that the learning in a feature space is more accurate than in the original space because the domain space y is generally simplified based on the definition of p Φ (y w). For example, let us divide elements of vector x into two sets x 1 and x 2, i.e. x = (x 1, x 2 ), and define two feature maps as π i : (x 1, x 2 ) x i for i = 1, 2. The map π i selects the attribute x i. The original model p(x w) is a joint probability of x 1 and x 2. According to the definition, p πi (x i w) is a marginal probability of x i. Then, G(n) measures the error over x 1 and x 2 whereas G πi (n) is about only x i. This derives that G πi (n) should be smaller than G(n). Theorem 2 claims this fact mathematically. A vicarious feature map purifies the domain space when the dimension of the feature space is less than the original one. If π 1 is a vicarious feature map, the attributes in x 1 are essential for the parameter learning and x 2 is nuisance dimension. Considering the map π i, we can regard the vicarious learning as a feature selection in a Bayes scenario. 4 An Application to HMMs In this section, we apply the vicarious Bayes learning to HMMs. We show that a restriction map of a data length can be a vicarious feature map, which derives a necessary length for the parameter learning. 4.1 Model Setting The present paper focuses on the ergodic HMMs, in which the transition connections among the hidden states construct a complete graph. The number of output alphabets and the length of data are M + 1 and L 0, respectively, i.e. x {1,..., M + 1} L 0. The numbers of the hidden states are K + 1 and K in the learning and true models. For simplicity, the initial state is always the first one in both models. The parameter w includes the transition probability a ij, indicating the probability of transition from the ith state to the jth state, and the output probability b im, indicating the probability of generating alphabet m at the ith hidden state. These probabilities satisfy the conditions, a ii = 1 K+1 i j a ij for a ij 0 and b im+1 = 1 M m=1 b im for b im Restriction Maps on the Length We consider the following map of the data space, Φ L : {1,..., M + 1} L0 {1,..., M + 1} L (18) for L L 0, which cuts off the data sequences to change the length into L. For example, Φ 3 ( ) = 484 when L 0 = 10, L = 3 and M = 8. In the similar way to π i, p Φ3 (814 w) is the marginal probability over joint probabilities p(814 w), where means any number from {1,..., 9}. 4.3 Necessary Length for the Vicarious Bayes Learning Using the restriction map, we consider a necessary length for the parameter learning. It can be conjectured that L = 3 is sufficient for the learning according to the following reason: The process to generate data with L = 3 includes two transitions. All transition parameters a ij are used for two transitions in the complete graph. All output parameters b im are used for generating data at all hidden states. Therefore, the information of all parameters w can be extracted from the output sequences when infinitely large number of sequences are given as training data. We hereinafter prove that L = 3 is not enough even when n and derive a necessary length. Lemma 1 The Kullback divergence H ΦL (w) is expressed as a sum of squared terms, which follows the rules: 1. The squared terms monotonically increase with growth of the data length. More precisely, H ΦL2 (w) includes all the terms of H ΦL1 (w) for L1 L2. 2. The significant number of the squared terms NST (L, M) can be expressed as NST (L, M) = (M + 1) L 1 + M 1. (19) 57

71 The proof is in Appendix. Theorem 3 For sufficiently large L 0, vicarious feature maps exist in the series of Φ L. Proof: Based on Lemma 1, NST (L, M) monotonically increases and all squared terms in H ΦL1 (w) are included by those in H ΦL2 (w) for L1 < L2. We consider the series of the squared terms of H(w) = H ΦL0 (w). Let S L be a set of the squared terms f(w) 2 defined by S L = {f(w) 2 : f(w) 2 H ΦL (w), f(w) 2 H ΦL 1 (w)}, where S 0 =. The order of the series is given by (20) S 1, S 2,..., S L,... S L0. (21) The Noetherian property on H(w) shows that there is a constant L e < L 0, which is the number of the essential squared terms for H ΦL (w) = 0, since L 0 is sufficiently large for the parameter learning. More mathematically, L e is the minimum number, such that S Le includes all generating elements for H(w) = 0 in terms of the ideal theory on polynomial ring. Then Φ L for L e L < L 0 is vicarious. (End of Proof) Let dim w be the dimension of the parameter in the learning model. In the present HMMs, dim w = (K + 1)(K + M). (22) Comparing Eq (22) with Eq (19), the following theorem indicates a necessary length: Theorem 4 A necessary length L m for the vicarious Bayes learning of HMMs are represented by L m = arg min L {NST (M, L) dim w}. (23) Proof: The maximum number of the parameters to be tuned is dim w. According to Eqs (7)-(9), we focus on the case p ΦL (y w) = q ΦL (y), i.e., H ΦL (w) = 0. In the case, the squared terms are all zero, where each term is a polynomial of w. To identify all elements of w, the number of the polynomials NST (M, L) should not be less than dim w based on the relation between the number of variables and that of equations. (End of Proof) Theorem 4 shows that L = 3 can not attain the vicarious Bayes learning. For example, let us assume that L = 3, M = 1 and K = 1. NST (1, 3) = (1 + 1) = 4 < dim w = (1 + 1)(2 + 1) = 6, (24) which does not satisfy the condition of L m. The necessary length is derived as L m = arg min L {(1 + 1) L } = 4. (25) Note that this length could not be sufficient, i.e. the vicarious Bayes learning requires longer sequences. Let us assume that H ΦL (w) consists of the square terms f 1 (w) 2, f 2 (w) 2 and f 3 (w) 2, and that dim w = 3. If f 3 (w) is a polynomial with respect to f 1 and f 2, f 1 (w) = f 2 (w) = 0 automatically satisfies f 3 (w) = 0. Then the actual number of the equations to identify w decreases to two, which is less than dim w. In such case, L should be larger to obtain more squared terms in H ΦL (w). 4.4 Experimental Validation of the Minimum Length As seen in the previous part, L m is a necessary length, which implies that longer sequences are required for the vicarious Bayes learning. In this part, we experimentally verify the minimum (necessary and sufficient) length. We suppose that (K, M) = (2, 2) and investigate the generalization error when L = 1,..., 10. The original length is L 0 = 10. The dimension of parameters is dim w = 12 and the number of the squared terms is NST (L, 2) = 3 L According to Theorem 4, L m = 4. The number of training data sequences is n = 500. We used the MCMC method to construct the posterior p ΦL (w Y n ) [1]. The number of parameters to construct the predictive model p ΦL (y Y n ) is N w = 500, i.e. p ΦL (y Y n ) 1 N w 500 p ΦL (y w i ), (26) i=1 where w i is taken from the posterior. In the evaluation, the number of the test data sequences is N = 5000 and the average E Y n[ ] is taken by 100 training sets. Figure 1 shows the results. The horizontal axis indicates the lengths L of training and test data. The vertical one does the generalization error. There are three curves for the true model sizes K 0 = 0, 1, 2. It is 58

72 Generalization error (dim w)/(2n) K_0=0 K_0=1 K_0= Training data length 1: The generalization error w.r.t. the sequence length. derived that when K = K 0 [6]. G(n) = dim w 2n + O ( 1 n 2 ) (27) The horizontal line described as (dim w)/(2n) is the theoretical asymptotic value of the error, dim w/(2n) = 12/(2 500) = The error on K 0 = 2 has to reach the line for G(n) = G ΦL (n). K 0 As can be seen in the graph, the curve of = 2 has a gap between L = 3 and L = 4, and almost reaches the horizontal line at L = 4, which implies that L m = 4 will be the minimum length of the vicarious learning. 5 Discussions The vicarious learning on HMMs reduces the computational cost in both training and test sessions. It is known that the time complexity of p(x w) is O((K + 1) 2 L) in HMMs. Then, the complexity of the likelihood L(w, X n ) is O(n(K + 1) 2 L 0 ) for a given w, which shows the cost O(n(K + 1) 2 L) of L ΦL (w, Y n ) is much less than that of L(w, X n ) when a number of data exist. The MCMC method requires the calculation of L(w, X n ) in each update of the parameter for training. The computation of p(x w) is frequently used for the generalization error in the test session. Therefore, to shorten the length of the data sequences saves a number of computational complexity. The reduction of the computational cost is also effective for cross-validation [5]. In the validation, the given data is divided into training data and validation data. These data sets are used for training and testing respectively. This procedure is then repeated after reversing the roles of the sets, and the generalization error is estimated. This validation method is commonly used for selecting the optimal size of a model, so called the model selection problem. To change the domain space by the vicarious feature map Φ L for L L m is a powerful method for the model selection because both the training and testing sessions are enormously repeated in the cross-validation and G ΦL (n) has the same value as G(n) in any size. 6 Conclusion We proposed the vicarious Bayes learning. It is regarded as a feature selection preserving the Bayes generalization error in an asymptotic manner. We also demonstrated its availability in HMMs. The length restriction is a theoretically guaranteed feature selection on the basis of the vicarious learning framework. Acknowledgement This research was partially supported by the Ministry of Education, Science, Sports and Culture in Japan, Grant-in-Aid for Scientific Research Appendix Proof of Theorem 2 First, let us state the following lemmas without their proofs: Lemma 2 Let λ 1, m 1 and λ 2, m 2 be the largest poles and the orders in the zeta functions of H 1 (w) and H 2 (w), respectively. Then, the following relation holds if H 1 (w) H 2 (w) for all w [8], λ 1 n m 1 1 n ln n λ 2 n m 2 1 n ln n. (28) Lemma 3 (Continuous Log-Sum Inequality) For q(x, y)p(y) ln q(x,y) p(x,y) dxdy <, q(x, y) q(x, y)p(y) ln p(x, y) dxdy ( ) q(x, y)p(y)dy q(x, y)p(y)dy ln dx, (29) p(x, y)p(y)dy where p(y), q(x, y), p(x, y) are all probability density functions. 59

73 Eq (9) is rewritten as H(w) = H y (w)dy, (30) H y (w) = q(x)δ(y Φ(x)) ln q(x) dx. p(x w) (31) Based on Lemma 3, Lemma 2 indicates that H(w) H Φ (w). (32) G(n) G Φ (n), (33) which proves the theorem. (End of Proof) Proof of Lemma 1 First, we focus on L = 3, then generalize the result. We use the following notation: H(w) K(w), (34) where there are positive constants C 1, C 2 such that C 1 K(w) H(w) C 2 (w) (35) in the neighborhood of H(w) = 0. The true model has the true parameter w = {{a ij }, {b im }} for 1 i, j K and 1 m M + 1. It is known that the largest poles in the zeta functions of H(w) and K(w) are the same [8]. It holds that H Φ3 (w) y {p Φ3 (y w) q Φ3 (y)} 2 (36) because y is discrete [7]. To simplify the descriptions, we use the notation, b 1 abab alph path = i,j,k {1,...,M+1} 3 l,m {1,...,K+1} 2 b 1i a 1l b lj a lm b mk, (37) which represents that alph and path is the marginalization over all generation of alphabets and all paths of transitions among the hidden states. For the true parameters {{a ij }, {b im }},. (38) path l,m {1,...,K 0 +1} 2 Then, Eq (36) is rewritten as H Φ3 (w) { b 1 abab b 1a b a b } 2. (39) alph path path Using b im+1 = 1 M m=1 b im, { b 1 abab M+1 b 1a b a b M+1 path path { M = b 1 aba(1 b m ) path m=1 path b 1a b a (1 } 2 M m)} 2. b (40) m=1 Note that the right-hand side of Eq (39) includes terms { path b 1 abab m path b 1a b a b m} 2 (41) for 1 m M. For any constant c, it holds that h 1 (w) 2 + {ch 1 (w) + h 2 (w)} 2 h 1 (w) 2 + h 2 (w) 2. (42) Combining Eq (42) and the presence of the terms in Eq (41), the term in Eq (40) is rewritten as { b 1 abab M+1 path path M { + m=1 { path b 1 abab m path b 1 aba path path M { + m=1 b 1a b a b M+1 b 1a b a } 2 path b 1 abab m path } 2 b 1a b a b m } 2 b 1a b a b m} 2. (43) Based on K+1 i=1 a i = 1, path b 1aba = path b 1ab. Then the term in Eq (43) is rewritten as { b 1a b a } 2 b 1 aba path path { = b 1a b } 2. (44) path b 1 ab path Applying this procedure to b im+1 of the last factor recursively, we can eliminate the b im+1, i.e. M H Φ3 (w) (b 1 b 1) 2 + alph{ M b 1 ab b 1a b } 2 alph path path M + alph{ b 1 aab b 1a a b } 2 path path M + alph{ b 1 abab b 1a b a b } 2, path path (45) 60

74 where M alpha means the marginalization over all generation of alphabets except for M +1. We apply a map b 1m = b 1m b 1m to H Φ3 (w). For simplicity, we use the same symbol b 1m for b 1m. M H Φ3 (w) b alph{ M 1 + b alph path(b 1)ab b 1a b } 2 path M + alph{ 1 + b path(b 1)aab b 1a a b } 2 path M + alph{ 1 + b path(b 1)abab b 1a b a b } 2. path (46) According to Eq (42), M H Φ3 (w) b alph{ M b 1ab b 1a b } 2 alph path path M + alph{ b 1aab b 1a a b } 2 path path M + alph{ b 1abab b 1a b a b } 2. path path (47) Since b 1m is a constant, M H Φ3 (w) b alph{ M ab a b } 2 alph path path M + alph{ aab a a b } 2 path path M + alph{ abab a b a b } 2. (48) path path The number of terms in M alph { }2 is M #b, where #b is the number of b im because only the output probability b im is counted in M alph. Then NST (M, 3) = M +M + M + M 2 = (M + 1) M 1. Hereinafter, let us generalize the proof for any L. Because of the reduction procedure, the right-hand side of Eq (48) includes the terms M M b 2 1, and alph{ ab a b } 2, (49) path path alph which are the term of L = 2. Moreover, the first term appears when L = 1. This shows that the squared terms of L = l includes those of L < l, which proves Lemma 1-1. We count the squared terms. We define that the squared terms in Eqs (39) and Eq (48) are ST 1 L and ST 2 L, respectively. Let #( ) be a function to indicate the number of squared terms. Obviously #(ST 1 L ) = (M + 1) L. The suffix on the summation path does not affect the number of the terms, which implies that the number of the squared terms is determined by the number of b im in alph. Let us assume that b M+1 = 1 in Eq (39) in order not to count it as a parameter. Under this assumption, #(ST 1 L ) = (M +1) L 1 because the term for the alphabet y = (M + 1)(M + 1)(M + 1) in Eq (39) vanishes i.e. { path aa path a a } 2 = 0. Then, the numbers of alph and b im in ST 1 L 1 are completely the same as those of M alph and b im in ST 2 L except for the first term M alph b2 1. Therefore it is easy to confirm that M #(ST 2 L ) = #(ST 1 L 1 ) + #( b 2 1) alph = (M + 1) L M, (50) which proves Lemma 1-2. (End of Proof) [1] Christophe Andrieu, Nando de Freitas, Arnaud Doucet, and Michael I. Jordan. An introduction to MCMC for machine learning. Machine Learning, 50(1-2):5 43, [2] R. E. Kalman. A new approach to linear filtering and prediction problems. J. Basic Engineering, 82:35 45, [3] Karim Lari and Steve J. Young. The estimation of stochastic context-free grammars using the insideoutside algorithm. Computer Speech and Language, 4:35 56, [4] L. R. Rabiner. A tutorial on hidden Markov models and selected applications in speech recognition. Proceedings of the IEEE, 77(22): , [5] M. Stone. Cross-validatory choice and assessment of statistical predictions. Journal of the Royal Statistical Society, Series B, 36: , [6] S. Watanabe. Algebraic analysis for non-identifiable learning machines. Neural Computation, 13 (4): , [7] K. Yamazaki, M. Aoyagi, and S. Watanabe. Asymptotic analysis of Bayesian generalization error with Newton diagram. Neural Networks, to appear. [8] Keisuke Yamazaki and Sumio Watanabe. Algebraic geometry and stochastic complexity of hidden Markov models. Neurocomputing, 69(1-3):62 84,

75 2009 Technical Report on Information-Based Induction Sciences 2009 (IBIS2009) Measurement Error Model Estimation by Kernel Markov Chain Monte Carlo Method Shotaro Akaho Yukito Iba Abstract: Measurement error models are the statistical models in which random noise is added to input variables as well as output variables. We consider the estimation problem of the regression function in a reproducing kernel Hilbert space (RKHS) for the measurement error models. We apply Markov chain Monte Carlo approach to estimate the posterior of the function. To deal with the infinite dimensionality of RKHS, we introduce a trick to exchange the order of sampling of the hidden variable and the function. Keywords: Bayesian framework, Markov chain Monte Carlo, reproducing kernel Hilbert space, Gaussian process, measurement error model 1 [1, 2] x = z + ɛ x, y = f(z)+ɛ y (1) z x y (Measurement Error Model) [3] f [4] [5] (RKHS: Reproducing Kernel Hilbert Space) (GP: Gaussian Process) [6, 7], , tel , s.akaho@aist.go.jp, The National Institute of Advanced Industry and Scientific Technology, Central 2, Umezono, Tsukuba, Ibaraki , , tel (), iba@ism.ac.jp, Institute of Mathematical Statistics, 10-3 Midoricho, Tachikawa, Tokyo (MAP ) (Representer ) (MCMC) [8, 9] z MCMC 62

76 2 x, y, z p(y z) =N[f(z),σ 2 y], p(x z) =N[0,σ 2 x]. (2) N[μ, σ 2 ] 0, σ 2 z σ x,σ y x y f k(z,z ) 0 E[f(z)] = 0, Cov[f(z),f(z )] = k(z,z ) (3) k(z,z )=exp( β(z z ) 2 ) (4) n D =(x 1,y 1 ), (x 2,y 2 ),...,(x n,y n ) f ( (x i,y i ) z i ) f,z 1,...,z n p(f,z 1,...,z n D) exp ( (y i f(z i )) 2 2σ 2 y (z i x i ) 2 2σ 2 x 1 2 f 2 )(5) f k(z,z ) 3 [10] Metropolis-Hastings Gibbs D z 1, z 2,..., z n f 1 z =(z 1,z 2,...,z n ) i =1, 2,...,n 1. z f 2. f z i z i f f Gibbs z i Metropolis-Hastings 3.1 f z f f Representer f z i f(z i ) z i z i z i z i f(z i ) f(z i ) z i z i z i f(z i ) f(z i ) z i z i f f : f f(z 1 ),..., f(z n ), f(z i ) : z i f f(z 1 ),..., f(z n ), f(z i ) 63

77 z i x i z i p(z i) exp ( 12 ) (z i x i ) 2 (6) z i f z i z i z i f f(z i ),f(z i ) (6) z i z i Metropolis-Hastings α =exp ( (y i f(z i ))2 (y i f(z i )) 2 ) (7) 2σ 2 y min(1,α) z i 3.3 f f f f f u 1,u 2,...,u m f n +1+m z =(z 1,...,z n,z i,u 1,...,u m ) T (8) f f =(f(z 1 ),...,f(z n ),f(z i),f(u 1 ),...,f(u m )) T (9) z f p(f z, D) ( exp 1 2σy 2 (y + f) T J(y + f) 1 ) 2 f T K 1 f (10) y + =(y T, 0,...,0) T, y =(y 1,...,y n ) T, (11) ( ) I n O J = (12) O O K z n, 1,m K 1 k 2 K 3 K = k T 2 k 3 k T 4 (13) K3 T k 4 K 5 z i z i k 2,k 3, k 4 n+1+m K 5 K 1,K 3 z i k 2,k 3, k 4 z i (i i ) f µ, V 1 z f (= MAP ) (z 1,y 1 ),...,(z n,y n ) f(z) = n i=1 α ik(z i,z) z µ = K 1 k T 2 K T 3 (K 1 + σ 2 yi n ) 1 y (14) Representer (10) 2 2 ( V = K J) σy 2 (15) K 0 K Cholesky K = L T L ( V = L T I n+1+m + 1 ) 1 σy 2 LJL T L (16) L K Cholesky µ V N[µ,V] f 64

78 z 1,...,z n ((6) )(13) K K 1,K 3,K 5 2. i =1, 2,...,n (a) z i ((6) ) (b) K k 2,k 3, k 4 (c) µ,v ((14),(16) ) (d) N[µ,V] f y Target Regression PostMean PostStd (e) (7) z i z i z i x 3. 2 f 4 ( 6, 6) z n =20 z i σx 2 x i y i f(z i )=exp( zi 2 /5) sin(z i ) (17) σy 2 β =1 ( 6, 6) 50 σx 2 = σy 2 =0.1 2 ( 1) 100 burn-in ( ) ( 6, 6) () 2 25 % 50 1: σx 2 = σy 2 =0.1 2 ({u i } m i=1 ) f(u i) (Target): f, (Regression):, (PostMean): (PostStd): ± 3 σx 2 = σy 2 = ,5, [4]

79 MSE y Target Regression PostMean PostStd iteration x 2: (σx 2 = σy 2 =0.3 2 ) 4: (σ 2 x = σ 2 y =0.3 2 ) proposed regression MSE iteration 3: (50, σx 2 = σy 2 =0.1 2 ) 5: (σ 2 x = σ 2 y =0.3 2 ) 66

80 proposed [8] A. J. Smola, S. V. N. Vishwanathan, and T. Hofmann. Kernel methods for missing variables. In Proc. of 10th Int. Workshop on AI& Statistics 2005, pp , [9],.. PRMU , Vol. 108, No. 327, pp , [10],,,,,. II. 6., regression 6: (50, σx 2 = σy 2 =0.3 2 ) [1] K. Nakae, Y. Iba, and T. Aoyagi. Statistical estimation of phase response curves and application in neural science. In Int. Soc. for Bayesian Analysis 9th World Meeting, [2],,.. 11 (IBIS), [3] P. J. Bickel, C. A. J. Klaassen, Y. Ritov, and J. A. Wellner. Efficient and Adaptive Estimation for Semiparametric Models. Springer-Verlag, [4],. 2?, Vol. 6, No. 2, pp , [5] S. M. Berry, R. J. Carroll, and D. Ruppert. Bayesian Smoothing and Regression Splines for Measurement Error Problems. J. of the American Statistical Association, Vol. 97, No. 457, pp , [6]..., [7] C. E. Rasmussen and C. K. I. Williams. Gaussian Processes for Machine Learning. MIT Press,

81 2009 Technical Report on Information-Based Induction Sciences 2009 (IBIS2009) Curve fitting in the space of one-dimensional normal distribution Jun Fujiki Shotaro Akaho Abstract: We propose a method for extracting a one-dimensional structure from a set of parameters of one-dimensional normal distributions. In this paper, the one-dimensional structure is represented as a curve, which contains a linear parameter, on the manifold of one-dimensional normal distributions. And the fitting error is measured by metric tensor, which is the second order approximation of e-divergence and/or m-divergence. In this formulation, the estimation of curve fitting is represented by the framework of Jacobian kernel principal component analysis, which is the extension of kernel principal component. Keywords: information geometry, manifold fitting, Euclideanization, kernel principal component analysis, Jacobian kernel e- m- m- e- [3]1 principle component analysis PCAPCA PCA 1 1 [4, 5]PCA 1 2, , jun-fujiki@aist.go.jp, The National Institute of Advanced Industrial Science and Technology (AIST) Neuroscience Research Institute, Central 2, Umezono, Tsukuba, Ibaraki s.akaho@aist.go.jp 1 2 kernel PCA KPCA[9] Jacobian kernel PCA JKPCA [8] 2 e- m- [3] S 4 µ σ> 0 µ = (µ, σ) p(x; µ) = 1 2πσ exp { } (x µ)2 2σ 2 (1) 2 σ > 0 ξ = (µ, V ) V = σ 2 68

82 1 p(x; ξ) = 1 } (x µ)2 exp { 2πV 2V (2) 2 V > 0 p(x; µ) = { 1 exp 1 2πσ 2 2σ 2 x2 + µ } σ 2 x µ2 2σ 2 F 1 (x) = x 2, F 2 (x) = x θ 1 = 1 2σ 2 = 1 2V, θ = (θ 1, θ 2 ) θ 2 = µ σ 2 = µ V p(x; θ) = exp{θ 1 F 1 (x) + θ 2 F 2 (x) ψ(θ)} (3) θ 1 S e- e- θ 1 < 0 η i = E θ [F i (x)] η 1 = µ 2 + σ 2 = µ 2 + V, η 2 = µ η = (η 1, η 2 ) θ m-m- η 1 > η 2 2 θ η θ(η), η(θ) e- m- 1 θ 1 (η) = 2(η 1 η 22 ), θ η 2 2(η) = η 1 η 2, 2 ( ) 2 θ2 η 1 (θ) = 1, η 2 (θ) = θ 2 2θ 1 2θ 1 2θ S e- m- e- e- θ θ 1 θ 2 e- θ(t) = tθ 1 + (1 t)θ 2, t T R m- m- η η(t) = tη 1 + (1 t)η 2 S 2 e- aθ 1 + bθ 2 + c = 0, m- pη 1 + qη 2 + r = 0 S flat e- m-e- M M 2 e- M m- m- S [3] 1 1 e- m- 1 m- e- 2 m- e- θ 1 θ {θ [d] } D d=1 m- θ θ m θ m = 1 D D d=1 θ [d] e- m- η 1 η {η [d] } D d=1 e- η η e η e = 1 D D d=1 η [d] e- θ aθ 1 + bθ 2 + c = 0 µ a 2bµ 2cσ 2 = 0 µ µ µ θ = (µ, σ 2, 1) a µ θ = 0 ξ e- ξ ξ θ = (µ, V, 1) a ξ θ = 0 η e- η η θ = (η 1 η 2 2, η 2, 1) a η θ = 0 1 Akaho[3] e- m-e- m- 69

83 m- η aη 1 + bη 2 + c = 0 µ a(µ 2 + σ 2 ) + bµ + c = 0 µ µη = (µ, µ 2 + σ 2, 1) a µη = 0 ξ m- µ 2 1 ξ ξη = (µ, µ 2 + V, 1) a ξη = 0 θ m- θ θη = (2θ 1 θ 2 2, θ 1 θ 2, 1) a θη = a F (ϕ) = 0 ϕ f(ϕ; a) = a F (ϕ) = µξθη ( ) ( ) Gµ = σ 2, G 0 1 ξ = 1 2V 0 2V 2, 0 1 ( ) G θ = 1 θ2 2 θ 1 θ 1 θ 2 2θ1 3, θ 1 θ 2 θ1 2 ( ) Gη = G 1 θ = 1 1/2 η 2 (η 1 η2 2)2 η 2 η 1 + η2 2 [4] 4.4 p M e- m- p M m- q q m- D(p q) = ψ(θ(p)) + ϕ(η(q)) θ(p) η(q) q p M e- q q e- D(q p) = ψ(θ(q)) + ϕ(η(p)) θ(q) η(p) q e-m- 3 D(p p + dq) 1 2 dη Gηdη = 1 2 dθ G θ dθ D(p + dp p) e- m- 2 5 ϕ G ϕ f(ϕ; a) = a F (ϕ) = 0 (4) e- µ aµ + bσ 2 + c = 0 µ µ θ a µ θ = 0 ξ ξ θ η θ 5.1 D {ϕ [d] } D d=1 ϕ [d] ϕ [d] (4) ϕ [d] ϕ [d] = δϕ [d] f( ϕ [d] ; a) f(ϕ; a) + f ϕ (ϕ [d]; a) δϕ [d] = 0 (5) F : ϕ F (ϕ) J F = F ϕ ϕ [d] J F J F[d] f ϕ (ϕ [d]; a) = a J F[d] δϕ [d] (5) a J F[d] δϕ [d] = a F [d] a [ J F[d] δϕ [d] δϕ [d] J F [d] ] [ ] a = a F [d] F[d] a, (6) F [d] = F (ϕ [d] ) r [d] G [d] = G ϕ[d] δϕ [d] G [d] δϕ [d] = δf [d] G [d] δf [d] (7) 70

84 G [d] = J + F [d] G [d] J + F [d] 2 (6) r 2 [d] = ] a [F [d] F[d] a a G + [d] a (8) ϕ E(a) = D d=1 ] a [F [d] F[d] a a G + [d] a (9) a 5.2 (9) [6, 10] 1 Akaho[1] f(ϕ; a) a f/ ϕ a â λ [d] Λ λ [d] = â G + [d] â, Λ = diag{λ [1],..., λ [D] }, F = (F [1] F [D] ) 3 E(a) a [ FΛ 1 F ] a a = UnitMinEigenVec [ FΛ 1 F ] [k] k [0] (1) {λ [0] [d] }D d=1 (2) (a) (b) (a) â [k+1] := UnitMinEigenVec[F(Λ [k] ) 1 F ] (b) λ [k+1] [d] := (â [k+1] ) G + [d] (â[k+1] ). 2 X + X 3 UnitMinEigenVec[X] X 6 0 Akaho[1] LS LS {λ [0] [d] = 1}D d=1 E(a) a [FF ]a â [1] = UnitMinEigenVec[FF ] LS LS Euclideanization of metric [7]. Fujiki and Akaho[7] equidirectional projectionedp [7] n k k 1 n k 2 n ϕ F (ϕ) R N J F = F ϕ ϕ F (ϕ) 2 F dϕ 1 F dϕ 2 ϕ 1 ϕ 2 = = det (J F J F ) dϕ 1 dϕ 2 1 Det G G = J + F G ϕ J + F Det X X 4 2 (Det G) 1/2 (Det G) 1/4 LS (Det G) 1/2 LS 4 2 G 2 71

85 r 2 = dϕ G ϕ dϕ = df (J +T F G ϕ J + F )df = df G df 0 3 r 2 = {f(ϕ; a)}2 a a = {f(ϕ; a)}2 a I N a 0 r 2 = {f(ϕ; a)} 2 a {(Det G + ) 1/n I N }a, 1 r 2 = {f(ϕ; a)}2 a G + a n 5 F (ϕ) R N G + (Det G + ) 1/n I N 0 LS [9, 11] F (ϕ) S n F (ϕ) n (10) rank G + = n G + Udiag {d 1,, d n }U 0 G + (d 1 d n) 1/n UU F (ϕ) n n Gaussian kernel kernel curve 7.1 [8] k(x, y) = F (x) F (y) Jacobian kernel[8] k(x, y) (m 1) = k(x, y) x = J F (x) F (y) a F [d] a (n 1) = p α [d] F [d] = F α (n D) (D 1) 6 D D K ) (K) ij = k(x [i], x [j] ), K = (K [1] K [D] D m K [d] k [i][j] = k(x [i], x [j] ), ) K [d] = (k [d][1] k [d][d] K [d] = F F [d], K [d] = F J F[d] (9) [ ] a F [d] F[d] a = α K [d] K[d] α, a G + [d] a = α K [d] G 1 [d] K [d]α (9) F E (α) = D α K [d] K [d] α α K [d] G 1 (10) d=1 [d] K [d]α D ( ) F (x) 6 [2] a = Φα + β [d] a x d=1 [d] K(x, y) = J F (x) J F (y) metric kernel 72

86 α (9) α α λ [d] Λ λ [d] = α K [d] G 1 [d] K [d] α, Λ = diag{λ [1],..., λ [D] }, E (α) α [ KΛ 1 K ] α α = UnitMinEigenVec [ KΛ 1 K ] (1) {λ [0] [d] }D d=1 7 (2) (a) (b) (a) α [k+1] := UnitMinEigenVec[K(Λ [k] ) 1 K] (b) λ [k+1] [d] := ( α [k+1] ) K [d] G 1 [d] K [d]( α [k+1] ). 8 F = (F [1] F [D] ) ϕ F (ϕ) sample feature map K : ϕ K(ϕ) = F F (ϕ) R D K(ϕ) R D sample feature space K = F J F SLS α K(ϕ) = 0 (10) 1 r 2 = (α K [d] ) 2 α K [d] G 1 [d] K [d] α r 2 (α K [d] ) 2 = α {(Det K [d] G 1 [d] K [d]) 1/n I D }α, 7 [9] 0 α 9 F (ϕ) PCA References [1] S. Akaho, Curve fitting that minimizes the mean square of perpendicular distances from sample points, In Proc. of SPIE93, Vision Geometry II, Vol.2060, (1993). [2],, D-II, vol. J86-D-II, no. 7, pp , [3] S. Akaho, The e-pca and m-pca: dimension reduction by information geometry, In Proc. of Int. Joint Conf. on Neural Networks (IJCNN2004), pp , [4] S. Amari, Differential Geometrical Methods in Statistics, Springer-Verlag, [5] S. Amari, H. Nagaoka, Methods of Information Geometry, AMS and Oxford university press, [6] W. Chojnacki, M. J. Brooks, A. van den Hengel and D. Gawley, On the fitting of surfaces to data with covariances, IEEE Trans. Patt. Anal. Mach. Intell., vol.22, no.11, pp , Nov [7] J. Fujiki and S. Akaho, Small hypersphere fitting by Spherical Least Square, In Proc. of ICONIP05, pp , (2005). [8],,, vol. 108, no. 327, pp , Nov [9] B. Schölkopf, A. Smola and K.-R. Müller, Nonlinear component analysis as a kernel eigenvalue problem, Neural Computation, vol. 10, pp , [10] G. Taubin, Estimation of planar curves, surfaces and, non-planar space curves defined by implicit equations with applications to edge and rage image segmentation, IEEE Trans. Patt. Anal. Mach. Intell., vol.13, no.11, pp , [11],, D-II, vol.j82-d-ii, no.4, pp , Apr

87 2009 Technical Report on Information-Based Induction Sciences 2009 (IBIS2009) Linear Programming Boosting by Column and Row Generation Kohei Hatano Eiji Takimoto Abstract: We propose a new boosting algorithm based on a linear programming formulation. Our algorithm can take advantage of the sparsity of the solution of the underlying optimization problem. In preliminary experiments, our algorithm outperforms a state-ofthe-art LP solver and LPBoost especially when the solution is given by a small set of relevant hypotheses and support vectors. Keywords: 1 l 1 l 1 [12, 5]l 1 [6,8,1,14] l 1 LPBoost Demiriz l 1, , tel , hatano@i.kyushuu.ac.jp, Department of Informatics, Kyushu University, 744 Motooka, Nishi-ku, Fukuoka, Japan, , tel , eiji@i.kyushuu.ac.jp Department of Informatics, Kyushu University, 744 Motooka, Nishi-ku, Fukuoka, Japan [5]LPBoost LPBoost [7] X m (x 1,y 1 ),..., (x m,y m )(x i X, y i { 1, +1}) n h 1,...,h n (h j : X [ 1, +1]). LPBoost (1) t d t h t (2) {h 1,...,h t } d t+1 m n. u ij = y i h(x i )(i =1,...,m,j =1,...,n). l 1 LPBoost LPBoost Column Generation [11]. LPBoost l 1 74

88 , Sparse LPBoost Sparse LPBoost ε>0 Sparse LPBoost γ ε γ Sparse LPBoost LPBoost l 1 Sparse LPBoost Warmuth Entropy Regularized LPBoost [16] LPBoost Entropy Regularized LPBoost l 1 O(log(m/ν)/ε 2 ) LPBoost Ω(m) [15] LPBoost Mangasarian [10] Sra [13] Bregman [3] Bradley Mangasarian [2] LPBoost 2 X S =((x 1,y 1 ),...,(x m,y m )) x X y i 1 +1 (i =1,...,m) H h H X [ 1, +1] k P k k P k = {p [0, 1] k k : i=1 p i =1} α P n n (x,y) y i j=1 α jh j (x i ) d P m h H m i=1 y id i h(x i ) γ d (h) 2.1 l 1 l 1 [5, 16] max ρ,α,ξ ρ 1 ν m ξ i (1) i=1 sub.to y i α j h j (x i ) ρ ξ i (i =1,...,m), j γ d (h j )= i α P n, ξ 0, d i y i h j (x i ) γ (j =1,...,n), d 1 ν 1, d Pm, min γ (2) γ,d sub.to (1) (2) (1) (ρ, α, ξ ) (2) (γ, d ) ρ 1 m ν i=1 ξ i = γ KKT d i y i α j h j(x i ) ρ + ξi =0 (i =1,...,m). j d i 0, y i α j h j(x i ) ρ + ξi 0 (i =1,...,m). ξ i (1/ν d i )=0, j ξ i 0, d i 1/ν (i =1,...,m). 75

89 y i j α j h j(x i ) >ρ d i =0 0 <d i < 1/ν n y i j α j h j(x i )=ρ ξ i > 0 d i =1/ν. (x i,y i ) d i (x i,y i ) ρ ξi > 0 ν i d i > 1 γ d (h j ) <γ α j =0. α j h j 3 (2) 3.1 LPBoost LPBoost [5] d 1 LPBoost t (i) LPBoost d t γ t +ε h t (ii) γ d (h t ) γ (i). 1 LPBoost (2) 1. LPBoost γ ε. LPBoost γ T max h H γ dt (h) ε d t (2) max h H γ dt (h) γ γ T γ ε Algorithm 1 LPBoost(S,ε) 1. d 1 S γ 1 = 1 2. For t =1,..., (a) d t γ t + ε h t (b) H T = t 1 (c) (2) {h 1,...,h t } (γ t+1, d t+1 ) (γ t+1, d t+1 )=arg min γ,d P m γ sub. to γ d (h j ) γ (j =1,...,t) d 1 ν f(x) = T t=1 α th t (x) α t (t =1,...,T) (2) Lagrange 3.2 Sparse LPBoost LPBoost d t α t Sparse LPBoost Sparse LPBoost 2 Sparse LPBoost 2. Sparse LPBoost γ ε. C = S S T γ T, d T γ T + ε γ T + ε max h H γ dt (h) d T P ST S T (2) max h H γ dt (h) γ γ T γ ε d T =(d T, 0,...,0) P S C. (γ T, d T ) S 76

90 Algorithm 2 Sparse LPBoost(S,ε) 1. () ν S 1 S 1 d 1 f 1 (x) =0ρ 1 =1γ 1 = 1, H 1 = 2. For t =1,..., (a) f t ρ t S t (b) S t T = t 1 break. (c) S t+1 = S t S t (d) For t =1,..., i. d t γ t + ε H t ii. H t break. iii. H t +1 = H t H t. iv. (2) S t H t +1 (γ t +1, d t +1) f t+1 (x) = h H t +1 α h h(x) ρ t+1 α h (2) Lagrange 3. f T (x) = h H T α h h(x) KKT. 3.3 Sparse LPBoost γ t+ε ρ t / / - d t γ t + ε Ĥt f t ρ t Ŝt K L K =min{ Ĥt, 2t }L =min{ Ŝt, 2 t } ν =0.2m ρ m k (k ) / Sparse LPBoost ν ν t k > t k d t = νk+1 1 =Ω(m k+1 ) k +1 t=1 t=1 / -Sparse LPBoost cm (0 <c 1) log(cm) t=1 (ν +2 t ) k = = = log(cm) s=0 s=0 t=1 k ( ) k ν s 2 t(k s) s s=0 log(cm) )ν s 2 t(k s) k ( k s t=1 k ( ) k ν s 2 (k s)(log(cm)+2) s k s=0 ( ) k ν s (c m) k s s =(ν + c m) k =(0.2+c ) k m k = O(m k ). / - Sparse LPBoost / - Sparse LPBoost 77

91 4 LPBoost Sparse LPBoost Xeon 3.8GHz CPU 8Gb C++, CPLEX m = X = { 1, +1} n f(x) =x 1 + x x k + b x 1,...,x k x k b f x ( 1 or+1) f(x) n = 100, k =10andb =5 0% 5% n +1 n n h j (x) =x j (j =1,...,n) +1 ν ν =1andν =0.2m LPBoost and Sparse LPBoost ε = CPU Sparse LPBoost ν ν Sparse LPBoost m (2) m =10 6 LP LPBoost Sparse LPBoost, ν =1 d ν =0.2m d ν 4.2 Reuters ,RCV1[9],news20 Reuters modified Apte ( ModApte ) acq 30, RCV1 news20 LIBSVM tools [4] m =20, 242, n =47, 236, m =19, 996, n =1, 355, LPBoost and Sparse LPBoost ε =10 4 Reuters RCV1, news20 ν =0.2m 2 Sparse LPBoost Sparse LPBoost 0.6m 0.8m Sparse LPBoost Sparse LPBoost 30, ε>0 l 1 ε LPBoost Sparse LPBoost

92 ν =0() ν =0.2m () m () #(d i > 0)(%) #(α j > 0)(%) () #(d i > 0)(%) #(α j > 0) (%) 10 4 LP LPB (69.3) (9.9) SLPB (23.0) 25(98) (47) 9.9(10.9) 10 5 LP LPB (61.4) (10.9) SLPB (18.9) 20.8(80) (58.7) 10.9(59.4) 10 6 LP n/a n/a n/a n/a LPB n/a n/a n/a n/a SLPB (25.2) 15.8(71.3) (46.5) 90(91) 1: ν =1ν =0.2m) #(d i > 0) #(α j > 0) d, α LPBoost (LPB) Sparse LPBoost (SLPB) #(d i > 0)(%) #(α j > 0)(%) Reuters LP (m=10,170,n=30,839) LPB (1.67) SLPB (68) 1.52(1.96) RCV1 LP (m=20,242,n=47,237) LPB (4.0) SLPB (63.4) 3.9(4.6) news20 LP (m=19,996,n=1,355,193) LPB (0.090) SLPB (68.1) 0.088(0.117) 2: #(d i > 0) #(α j > 0) d, α LPBoost (LPB) Sparse LPBoost (SLPB) (B) [1] N. Balcan, A. Blum, and N. Srebro. A theory of learning with similarity functions. Machine Learning, 72(1-2):89 112, [2] P. S. Bradley and O. L.Mangasarian. Massive data discrimination via linear support vector machines. Optimization Methods and Software, 13(1):1 10, [3] Y. Censor and S. A. Zenios. Parallel Optimization: Theory, Algorithms, and Applications. Oxford University Press, [4] C. C. Chang and C. J. Lin. Libsvm: a library for support vector machines. Software available at libsvm, [5] A. Demiriz, K. P. Bennett, and J. Shawe-Taylor. Linear programming boosting via column generation. Machine Learning, 46(1-3): , [6] T. Graepel, R. Herbrich, B. Schölkopf, A. Smola, P. Bartlett, K. Müller, K. Obermayer, and R. Williamson. Classification on proximity data 79

93 with LP-machines. In International Conference on Artificial Neural Networks, pages , [7] A. J. Grove and D. Schuurmans. Boosting in the limit: Maximizing the margin of learned ensembles. In Proceedings of the fifteenth National Conference on Artificial Intelligence (AAAI-98), pages , [8] M. Hein, O. Bousquet, and B. Schölkopf. Maximal margin classification for metric spaces. Journal of Computer ans System Sciences, 71: , [9] D. D. Lewis, Y. Yang, T. G. Rose, and F. Li. Rcv1: A new benchmark collection for text categorization research. Journal of Machine Learning Research, 5: , [10] O. Mangasarian. Exact 1-norm support vector machines via unconstrained convex differentiable minimization. Journal of Machine Learning Research, 7: , [11] S. Nash and A. Sofer. Linear and Nonlinear Programming. Macgraw-Hill, [12] R. E. Schapire, Y. Freund, P. Bartlett, and W. S. Lee. Boosting the margin: a new explanation for the effectiveness of voting methods. The Annals of Statistics, 26(5): , [13] S. Sra. Efficient large scale linear progamming support vector machines. In Machine Learning: ECML 2006, pages , [14] L. Wang, M. Sugiyama, C. Yang, K. Hatano, and J. Fung. Theory and algorithms for learning with dissimilarity functions. Neural Computation, 21(5): , [15] M. Warmuth, K. Glocer, and G. Rätsch. Boosting algorithms for maximizing the soft margin. In Advances in Neural Information Processing Systems 20, pages , [16] M. Warmuth, K. Glocer, and S. V. N. Vishwanathan. Entropy regularized LPBoost. In Proceedings of the 19th International Conference on Algorithmic Learning Theory, pages ,

94 2009 Technical Report on Information-Based Induction Sciences 2009 (IBIS2009) Multiple Kernel Learning for Object Classification Shinichi Nakajima Alexander Binder Christina Müller Wojciech Wojcikiewicz Marius Kloft Ulf Brefeld Klaus-Robert Müller Motoaki Kawanabe Abstract: Combining information from various image descriptors has become a standard technique for image classification tasks. Multiple kernel learning (MKL) approaches allow to determine the optimal combination of such similarity matrices and the optimal classifier simultaneously. Most MKL approaches employ an l 1 -regularization on the mixing coefficients to promote sparse solutions; an assumption that is often violated in image applications where descriptors hardly encode orthogonal pieces of information. In this paper, we compare l 1 -MKL with a recently developed non-sparse MKL in object classification tasks. We show that the non-sparse MKL outperforms both the standard MKL and SVMs with average kernel mixtures on the PASCAL VOC data sets. Keywords: multiple kernel learning, support vector machine, image classification, sparsity. 1 Introduction Data fusion is an important topic in computer vision. Images can be represented by a multiplicity of features capturing certain aspects, including color, textures, and shapes. Unfortunately, the importance of different types of features varies with the tasks; color information, for instance, substantially increases the detection of stop signs while coloring is almost irrelevant for finding cars in images. Techniques for appropriately combining relevant features for a task at hand are therefore crucial for state-of-the-art object recognition systems. From a machine learning view, different representations give rise to different kernel functions. Kernels define (possibly nonlinear) similarities between data points and allow to abstract learning algorithms from data. Thus, kernel machines have been successfully applied to many practical problems in various fields [19]. Given a task at hand, designing an appropriate kernel is essential for achieving good generalizations, for instance by incorporating prior assumptions and domain knowledge [9, 28]. However, in the absence of prior knowledge one has to resort to alternatives. For object recognition tasks, combining information from Nikon Corporation, nakajima.s@nikon.co.jp, Fraunhofer Institute FIRST, binder@first.fhg.de, Technische Universität Berlin, muechr@first.fraunhofer.de, Technische Universität Berlin, wojcikie@informatik.hu-berlin.de, Technische Universität Berlin, mkloft@cs.tu-berlin.de, Technische Universität Berlin, brefeld@cs.tu-berlin.de, Technische Universität Berlin, klaus-robert.mueller@tu-berlin.de, Fraunhofer Institute FIRST, nabe@first.fhg.de, various image descriptors into several kernels K 1,..., K m has become a standard technique. Unfortunately, the choice of the right kernel mixture is often a matter of trial and error. As a remedy, uniform mixtures of normalized kernels [14, 26] or brute-force approaches [2] are employed frequently. However, the former approach may lead to suboptimal kernels and the latter is computationally infeasible if many kernels are to be combined. Recently, multiple kernel learning (MKL) [13, 1, 20, 18, 27] was applied to object classification tasks involving various image descriptors [24]. Compared to uniform mixtures and brute-force approaches, MKL has the appealing property of always finding the optimal kernel combination and converges quickly as it can be wrapped around a regular support vector machine (SVM) [20]. MKL aims at learning the optimal kernel mixture and the model parameters simultaneously. More specifically, MKL approaches find a linear mixture of the kernels, that is K = j β jk j. To support the interpretability of the solution, many MKL approaches promote sparse mixtures by incorporating an l 1 -norm constraint on the mixing coefficients. However, it has often been observed that l 1 -norm MKL is outperformed by the average-sum kernel K = j K j. An explanation is that enforcing sparse mixtures may lead to degenerate models if the optimal kernel mixture is non-sparse. A remedy might be recently developed non-sparse variants of MKL promoting non-sparse kernel mixtures [10]. In this contribution, we empirically compare sparse and 81

95 non-sparse MKL approaches to object classification tasks. We employ candidate kernels obtained from many different image descriptors including the 30 color SIFT features by the VOC2008 winner [22]. Our empirical results on image data sets from the PASCAL visual object classification (VOC) challenge 2007 and 2008 [8] show that the nonsparse MKL significantly outperforms the uniform mixture and l 1 -norm MKL. This paper is organized as follows. In Section 2, we briefly review the underlying techniques, including sparse and non-sparse MKL. Section 3 discusses similarities between the prepared kernels. Based on this analysis, we precompute averages of similar kernels and apply MKL with a substantially reduced sets of kernels. We discuss our empirical results in Section 4 and Section 5 concludes. 2 Preliminaries 2.1 Support Vector Machines In the supervised learning setting, we are given n training samples {(x i, y i )} n i=1, where x i X is the input vector and y i Y. For instance, in object recognition, inputs x are frequently histograms of some image features and Y is a discrete set of objects that are to be identified in the images. Inputs are often annotated with several labels as different objects can occur in the same image. To account for these multi-label scenarios, we take a one-vs-all approach and focus on binary classification settings. That is, we have y i {+1, 1}, where y i = +1 denotes that at least one object from the actual category is included in the i-th image and y i = 1 otherwise. Support vector machines (SVMs) originate from linear classifiers and maximize the margin between sample clouds of both classes. Introducing a feature mapping ψ from the input space X to a reproducing kernel Hilbert space (RKHS) H, linear classifiers in H of the form f(x) = w ψ(x) + b (1) provide a rich set of flexible classifiers in X. The parameters (w, b) are determined by solving the optimization problem min w,b,ξ 1 2 w C n ξ i, (2) i=1 s.t. i, y i { w ψ(x i ) + b } 1 ξ i ; ξ i 0, where 2 denotes the l 2 norm and C > 0 is a regularization constant. Notice that the spanned RKHS can be infinite-dimensional, however, translating the above formulation into the equivalent dual optimization problem prevents from dealing with features in H explicitly. min α s.t. n α i 1 2 i=1 n α i α l y i y l k(x i, x l ) (3) i,l=1 0 α i C, i; n y i α i = 0. i=1 The above dual depends only on inner products (similarities) of inputs which can be alternatively computed by means of kernel functions k, given by k(x, x) = ψ(x), ψ( x) H. Once, optimal parameters are found, these are used as plugin estimates and the final decision function can be written as f(x) = n α i k(x i, x) + b. i=1 Note that only a small fraction of the α s usually take nonzero values which are often called support vectors. The threshold b is determined by saturated support vectors with α = C. Finally, we remark that we need to use different regularization constants C + and C for the positive and negative examples, respectively, to compensate the unbalanced sample sizes of the two classes [3]. 2.2 Multiple Kernel Learning Let K 1,..., K m be m kernel matrices with K t = [k t (x i, x j )] i,j=1,...,n, obtained from different sources or features. The multiple kernel learning (MKL) framework extends the regular SVM formulation by additionally learning a linear mixture of the kernels, i.e. K β = m β j K j. j=1 Thus, the model in Equation (1) is extended to f(x) = m β j w j ψ j (x) + b. j=1 A common approach is to rephrase the above expression by incorporating the mixing coefficients into the parameter vector w β = ( β 1 w 1,..., β m w m ) and the feature mapping ψ β (x i ) = ( β 1 ψ 1 (x i ),..., β m ψ m (x i )). The corresponding optimization problem maximizes the generalization performance by simultaneously optimizing the parameters w, b, ξ, and β. We obtain the common l 1 -norm 82

96 MKL for p = 1 [1, 20, 18, 27], and non-sparse MKL for p > 1 [10]. min β,w,b,ξ 1 2 w β C n i=1 s.t. i : y i ( w β, ψ β (x i ) + b) 1 ξ i (4) ξ i ξ 0; β 0; β p 1 Note that we resolve the regular SVM optimization problem in Equation (2) for learning with only a single kernel m = 1. Irrespectively of the actual value of p, the above optimization problem can be translated into a semi-infinite program [20, 10] which can be interpreted as a dualized variant of the optimization problem (4). We arrive at, min λ,β λ s.t. λ n α i 1 2 i=1 n α i α l y i y l i,l=1 m j=1 β j k j (x i, x l ), α R n (5) n 0 α i C, i; y i α i = 0; i=1 β j 0, j; β p 1 Initializing β with a uniform kernel mixture, the semi-infinite program can be optimized efficiently by interleaving the following two steps: 1. For the actual mixture β, the solution of the regular SVM generates the most strongly violated constraint (Equation (5)). 2. With respect to set of active constraints, the optimal values of β and λ are identified by solving the corresponding optimization problem for β. The actual optimization problems for the mixing coefficients, however, differ with varying values of p. For instance, for p = 1, one obtains a linear program that can be solved with standard techniques. For p = 2, the l 2 -norm gives rise to a quadratically constrained quadratic program (QCQP) that can also be optimized with off-the-shelf QPsolvers. For different values of p, things get a bit tricky because there is hardly an l p -norm solver. Nevertheless, one can approximate the l p -norm constraint by a second-order Taylor expansion around the current estimates β old given by β p p 1 p(3 p) 2 p(p 1) + 2 j (p 2 2p) j (β old j ) p 2 β 2 j. (βj old ) p 1 β j Using the above approximation, one obtain a QCQP, which can again be optimized with standard techniques [10]. 2.3 Kernel Alignment In the remainder, we will need to analyze the similarity of kernel matrices. For this purpose, we now introduce kernel target alignment [5] as an adequate measure of similarity or hyper kernel [17]. Let K 1 = [k 1 (x i, x j )] i,j=1,...,n and K 2 = [k 2 (x i, x j )] i,j=1,...,n be the Gram matrices of kernel functions k 1 and k 2 for x 1,..., x n. The alignment between k 1 and k 2 is defined as the cosine of the angle between the two matrices K 1 and K 2 given by A(K 1, K 2 ) := K 1, K 2 F K 1 F K 2 F, (6) where K 1, K 2 F denotes the standard inner product K 1, K 2 F := n i,j=1 k 1(x i, x j )k 2 (x i, x j ) and K 1 F is the Frobenius norm in matrix space defined as K 1 F := K 1, K 1 1/2 F. It is important to center the kernels before computing the alignment as many classifiers, including SVMs, are invariant against mean shifts in the RKHSs. The centering in the respective feature spaces is achieved by multiplying the matrix H, given by H := I 1 n 11 to the kernels K 1 and K 2 from both sides, where I is the identity matrix of size n and 1 is a column vector with all elements 1. Thus, the resulting alignment for centered kernels can be computed by A(HK 1 H, HK 2 H) = HK 1H, HK 2 H F HK 1 H F HK 2 H F, (7) where HK 1 H, HK 2 H F = tr(hk 1 HK 2 ), because H is a projection matrix. 3 Experiments 3.1 VOC data sets In order to show the advantage of our procedure, we compare the performance of the different MKL procedures to 83

97 1 0.5 SIFT_g SIFT_o 0.7 SIFT_no SIFT_nrg SIFT_rgb 0.2 PHoG SIFT_g1 SIFT_o SIFT_no SIFT_nrg SIFT_rgb PHoG (a) (b) 1: Similarity between the 35 prepared kernels: (a) hyper kernel and (b) graphical representation of the similarities within the first two eigen directions. In the panel (a), 6 groups are SIFT g1, SIFT o, SIFT no, SIFT nrg, SIFT rgb, and PHoG, while 6 elements within SIFT color channel consists of 3 pyramid levels (level 0, 1, y3) for dense grid and interest points. In the panel (b), the color channels are specified as black= g1, red= o, magenda= no, green= nrg and blue= rgb, while the markers discriminates the pyramid levels and sampling scheme for SIFT plus PHoG (triangle), i.e. circle= dense level0, square= dense level1, diamond= dense y3, plus= interest level0, X-mark= interest points level1, star= interest points y3. SVMs using the average-sum kernel. We experiment on the VOC 2007 and VOC 2008 classification data sets [8]. The VOC 2007 data set consists of 9963 images (2501 training, 2510 validation and 4952 test) annotated with 20 object classes. The VOC 2008 data set contains 8780 images categorized into the same 20 object classes as in the VOC 2007 data. The latter is split into train, validation and test sets by the organizers (2113 for train, 2227 for validation, and 4340 for test). The ground-truth of the test set is yet disclosed by the organizers who agreed to evaluate test performance on request. We split the multi-label problem into 20 binary classification problems using the one-vs-all strategy. That is, for each class, we define an auxiliary label y i = +1 if at least one object from the actual class is included in the i-th image, and y i = 1 if there is no such object in the image. 1 The evaluation is based on precision-recall (PR) curves and the principal quantitative measure is the average precision (AP) over all recall values. We employ model selection for the SVM/MKL tradeoff parameter C and for the parameter p which controls the sparseness of the MKL. We used p = λ, where λ = {, 5, 4, 3, 2, 1, 0, 1, }. We resolve p = 1 for λ = and obtain the unweighted-sum kernel for 1 Hardly detectable objects are indicated by y i = 0 by the organizers. Since these are omitted in the final evaluation we simply excluded them from the training process. p =. Furthermore, we optimized the parameter p based on the cross-validation score either jointly with all classes (l p -joint) or individually for each category (l p -single). The final classifiers are obtained by re-training the respective approaches on all available data (i.e., training and holdout sets) using the previously determined optimal parameters. We report on average AP scores over 10 repetitions with different training, holdout, and test sets. The baselines SVM and l 1 -norm MKL are implemented using the Shogun library [20]. 3.2 Image Features and Base Kernels In our experiments, we employed the following two sets of image features. The first category contains 30 histograms of visual words (HoW) representations [6] based on color SIFT descriptors [15] which are almost the same as those applied by the winner of VOC 2008 [22]. As sampling schemes, we use a dense grid with 6 pitches and interest points from gray-scale images by the scale invariant detector [25]. For both cases, we calculated the base SIFT descriptors in 10 color channels: g1 (grey), o1 (opponent color 1), o2, no1 (normalized o1), no2, nr (normalized red), ng (normalized green), r, g, b. For prototype calculation and visual word assignment, the color SIFTs are combined into the following 5 groups: g1, o=[o1,o2,g1], no=[no1,no2], nrg=[nr,ng], rgb=[r,g,b]. For each case, we created

98 visual words for the dense grid (800 for the interest points) by using k-means clustering. 2 Finally, we also consider three levels of the image pyramid representation [14]: for each image, its visual words are summarized into histograms for the whole image (level 0), for 4 quarter images (level 1) and for 3 horizontal stripes (y3). In total, we prepared 5 (colors) 2 (sampling) 3 (pyramid levels) = 30 kernels. The second category of our image features is the pyramid histogram of oriented gradient (PHoG) [7, 2]. For each of the 5 color channels, which are same as in the first category, we compute the PHoG representations of level 2 where the 3 pyramid levels are merged by a default scheme without any adaptation. In sum, we computed 5 PHoG kernels. We used the χ 2 kernel, which has proved to be a robust similarity measure for bag of words histograms [26], where the band-width is set to the mean χ 2 distances between all pairs of training samples [12]. Although our MKL implementations are throughout efficient, simply storing all 35 kernels exceeds 1.2GB. We therefore pre-combine kernels based on a similarity analysis using kernel target alignment [5] before applying MKL. Figure 1 (a) shows the kernel alignment score (7) between the 30 SIFT + 5 PHoG kernels. We can see: (i) the kernels within the same colors are mostly similar, (ii) g1 and rgb kernels are also similar and (iii) the PHoG and SIFT kernels are less similar. In order to assure our findings, we plotted the kernels in a 2-dimensional space spanned by the first and second eigenvectors of the hyper kernel obtained by a principal component analysis (PCA) and spectral clustering [16] (Figure. 1(b)). Based on this similarity analysis, we averaged 6 SIFT kernels with uniform weights within each color. By doing this, we reduced the number of base kernels to 10. We obtain 5 pre-combined SIFT and 5 PHoG kernels which are plugged into the MKL. 3.3 Result 1: Significance Test for 10 Random Splits of VOC 2008 Before we use the official VOC 2008 data split to compare our outcomes to already published results in Section 3.4, we investigate statistical properties of the performances of the different methods. We therefore draw 2111 training, 1111 validation, and 1110 test images randomly from the labeled pool (i.e., official training and holdout split). We report on APs and standard deviations over 10 repetitions 2 We use only 800 visual words for the interest points as about 1/5 of the descriptors are extracted per image. with distinct training, holdout, test sets. To test on the significance of the differences in performance, we conduct a Wilcoxon signed-ranks test for each method and class and additionally for the average AP over all classes. Table 1 shows the results. 3 The methods whose performance are not significantly worse than the best score are marked in bold face. The l p -single MKL is always among the best performing algorithms. Its jointly-optimized counterpart l p -joint, performs similarly and attains the second best performance. Uniform weights and l 1 -MKL are significantly outperformed by the two nonsparse MKL variants for several object classes. The result is however not really surprising as l p -single is optimized class-wise. Figure 2 shows the resulting kernel weights, averaged over the 10 repetitions. We see that the solutions of l p - joint distribute some weight on each kernel, achieving nonsparse solutions. The average p for l p -joint is Furthermore, Figure 2 implies that PHoW features carry more relevant information than PHoG. Since the PHoG features do not seem to play a great role in the classification, a natural question is whether PHoG do contribute to the accuracy at all. Table 2 shows the average gain in accuracy for using PHoW kernels alone and PHoG & PHoW kernels together, respectively. The result shows that the PHoG kernels absolutely contribute to the final decision. We observe a significant gain in accuracy by incorporating PHoG kernels into the learning process for all but the average-sum kernel. 2: Average gain in accuracy by adding PHoG features. uniform l 1 l p -joint l p -single PHoW 45.4± ± ± ±1.0 PHoW&G 45.2± ± ± ± Result 2: Results for the Official Splits of VOC 2007 and VOC 2008 In our second experimental setup, we evaluated the performance of the approaches for the official splits of the VOC 2007 and 2008 challenges. The winners of VOC2008 [21] reported an average AP of 60.5 on VOC 2007 and achieved an AP of 54.9 on VOC2008. Their result is based 3 Since creating a codebook and assigning descriptors to visual words is computationally demanding, we apply the codebook created with the training images of the official split. This could result in slightly better absolute test errors, since some information of the test images might be contained in the codebook. However, our focus in this Section lies on a relative comparison between different classification methods, and this computational shortcut does not favor any of these approaches. 85

99 1: Average precisions on the test images of our 10 splits. For each column, the best method and comparable ones based on a Wilcoxon signed-rank test at the significance level of 5% are marked in bold faces. average aeroplane bicycle bird boat bottle bus uniform 45.2± ± ± ± ± ± ±10.8 l ± ± ± ± ± ± ±10.0 l p -joint 46.9± ± ± ± ± ± ±11.2 l p -single 46.9± ± ± ± ± ± ±9.3 car cat chair cow diningtable dog uniform 53.0± ± ± ± ± ±3.0 l ± ± ± ± ± ±4.8 l p -joint 54.7± ± ± ± ± ±4.5 l p -single 54.4± ± ± ± ± ±3.4 horse motorbike person pottedplant sheep sofa uniform 48.2± ± ± ± ± ±7.4 l ± ± ± ± ± ±8.5 l p -joint 48.0± ± ± ± ± ±9.0 l p -single 49.3± ± ± ± ± ±9.0 train tvmonitor uniform 60.4± ±5.9 l ± ±5.6 l p -joint 61.6± ±6.4 l p -single 61.1± ± aeroplane bicycle bird boat bottle bus car cat chair cow diningtable dog horse motorbike person pottedplant sheep sofa train tvmonitor aeroplane bicycle bird boat bottle bus car cat chair cow diningtable dog horse motorbike person pottedplant sheep sofa train tvmonitor : Selected weights by MKL: l 1 (left) and l p -joint (right) on color descriptors [22], kernel codebook [23], and kernel discriminant analysis [4]. Table 3 shows the resulting average APs for our MKL approaches. 4 The non-sparse MKL increases the accuracy of the basic color descriptors (uniform only PHoW) of about 2%. Furthermore, [21] reports a loss in accuracy of less than 1% if a SVM is substituted for the kernel discriminant analysis. Taking the different code books into account, we conjecture that except for the code book non-sparse multiple kernel learning is on par or better as the winner of last years VOC challenge. We will address the validity of our assumption in future work. 4 APs for VOC2008 have been kindly evaluated by the organizers. 3: Average APs for VOC 2007/2008 using official splits. VOC2007 VOC2008 uniform (only PHoW) uniform 55.0 l l p -joint l p -single Discussion In contrast to anecdotal reports, we observed l 1 -MKL to outperform the average-sum kernel for PHoW and PHoG kernels (see Table 1). Nevertheless, carefully adjusting the norm p for boosts the performance of non-sparse MKL which 86

100 performed best throughout all our experiments. The optimal choice of the norm p thereby depends on the actual set of kernels. As a rule of thumb, large values of p work out in cases where all kernels encode a similar amount of independent information while smaller values of p are best if some kernels are less informative or redundant. As an illustrative example, consider a simple experimental setup where we deployed MKL together with the following 12 kernels: level-2 PHoW with grey and hue channels with 10 pixels pitch dense grid and 1200 vocabulary (3 pyramid levels 2 colors), PHoG of grey channel (3 pyramid levels), and the pyramid histograms of intensity with hue channel (3 pyramid levels). Table 4 shows the results. The sparse l 1 -MKL yields a similar accuracy as the average-sum kernel. As suspected, both approaches are significantly outperformed by non-sparse MKL. 4: A simple case where the performance of l 1 -norm MKL deteriorates. uniform l 1 l p -joint l p -single mean AP 40.8± ± ± ±0.9 5 Conclusions When measuring data with different measuring devices, it is always a challenge to combine the respective device uncertainties in order to fuse all available sensor information optimally. In this paper, we revisited this important topic and discussed machine learning approaches to adaptively combine different image descriptors in a systematic and theoretically well founded manner. While MKL approaches in principle solve this problem, it has been observed that the standard l 1 -norm based MKL can rarely outperform SVMs that use an average of a large number of kernels. One hypothesis why this seemingly unintuitive results may occur, is that the sparsity prior may not be appropriate in many real world problems. A close inspection reveals that most kernels contain useful structural information and should therefore not be omitted. A slightly less severe method of sparsification is to use another norm for optimization, namely the l p -norm. We tested whether this hypothesis holds true for computer vision and applied the recently developed non-sparse l p -norm MKL algorithms to object classification tasks. By choosing p as a hyperparameter which controls the degree of non-sparsity from a set of candidate values with the help of a validation data, we showed that l p -MKL significantly improves SVMs with averaged kernels and the standard sparse l 1 -norm MKL. Similar accuracy gain has been observed by controlling p in one-class MKL [11]. Future work will incorporate further modeling ideas of the VOC 2008 winner, e.g. the kernel code book, which we have so far not even employed. The test result with the official splits shown in this paper implied that our method is highly competitive to the winners solution. Furthermore, a combination of mid-level features by MKL will be an interesting research direction. [1] F. Bach, G. Lanckriet, and M. Jordan. Multiple kernel learning, conic duality, and the smo algorithm. International Conference on Machine Learning, [2] Anna Bosch, Andrew Zisserman, and Xavier Muñoz. Representing shape with a spatial pyramid kernel. In Proceedings of the 6th ACM international conference on Image and video retrieval (CIVR 07), pages , [3] U. Brefeld, P. Geibel, and F. Wysotzki. Support vector machines with example dependent costs. In Proceedings of the European Conference on Machine Learning, [4] D. Cai, X. He, and J. Han. Efficient kernel discriminant analysis via spectral regression. In ICDM, [5] N. Cristianini, J. Shawe-Taylor, A. Elisseeff, and J. Kandola. On kernel-target alignment. In Advances in Neural Information Processing Systems, volume 14, pages , [6] G. Csurka, C. Bray, C. Dance, and L. Fan. Visual categorization with bags of keypoints. In Workshop on Statistical Learning in Computer Vision, ECCV, pages 1 22, Prague, Czech Republic, May [7] N. Dalal and B. Triggs. Histograms of oriented gradientsfor human detection. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition, volume 1, pages , San Diego, USA, June [8] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 87

101 2008 (VOC2008) Results. = workshop/index.html, [9] Tommi Jaakkola and David Haussler. Exploiting generative models in discriminative classifiers. In Advances in Neural Information Processing Systems, volume 11, pages , [10] M. Kloft, U. Brefeld, S. Sonnenburg, A. Zien, P. Laskov, and K.-R. Müller. Efficient and Accurate l p -Norm MKL. In Advances in Neural Information Processing Systems 22, to appear. [11] M. Kloft, S. Nakajima, and U. Brefeld. Feature Selection for Density Level-Sets. In Proc. of ECML, [12] C. Lampert and M. Blaschko. A multiple kernel learning approach to joint multi-class object detection. In DAGM, pages 31 40, [13] Gert R.G. Lanckriet, Nello Cristianini, Peter Bartlett, Laurent El Ghaoui, and Michael I. Jordan. Learning the kernel matrix with semidefinite programming. Journal of Machine Learning Research, pages 27 72, [14] S. Lazebnik, C. Schmid, and J. Ponce. Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition, volume 2, pages , New York, USA, June [15] D. Lowe. Distinctive image features from scale invariant keypoints. International Journal of Computer Vision, 60(2):91 110, [16] A.Y. Ng, M.I. Jordan, and Y.Weiss. On spectral clustering: Analysis and an algorithm. In NIPS, [17] C. Ong, A. Smola, and R. Williamson. Hyperkernels. In NIPS, volume 15, pages , [18] A. Rakotomamonjy, F. Bach, S. Canu, and Y. Grandvalet. More efficiency in multiple kernel learning. In ICML, pages , [20] S. Sonnenburg, G. Rätsch, C. Schäfer, and B. Schölkopf. Large scale multiple kernel learning. Journal of Machine Learning Research, 7: , [21] M. Tahir, K. van de Sande, Jasper Uijlings, Fei Yan, Xirong Li, Krystian Mikolajczyk, Josef Kittler, Theo Gevers, and Arnold Smeulders. Surreyuva srkda method. [22] Koen E. A. van de Sande, Theo Gevers, and Cees G. M. Snoek. Evaluation of color descriptors for object and scene recognition. In IEEE Conference on Computer Vision and Pattern Recognition, Anchorage, Alaska, USA, June [23] J.C. van Gemert, J.M. Geusebroek, C.J. Veenman, and A.W.M. Smeulders. Kernel codebooks for scene categorization. In ECCV, [24] M. Varma and D. Ray. Learning the discriminative power-invariance trade-off. In Proceedings of the IEEE 11th International Conference on Computer Vision (ICCV 07), pages 1 8, [25] J. Zhang, M. Marszalek, S.Lazebnik, and C. Schmid. Scale and affine invariant interest point detectors. International Journal of Computer Vision, 60(1):63 86, [26] J. Zhang, M. Marszalek, S.Lazebnik, and C. Schmid. Local features and kernels for classification of texture and object categories: A comprehensive study. International Journal of Computer Vision, 73(2): , [27] Alexander Zien and C. Ong. Multiclass multiple kernel learning. In ICML, pages , [28] Alexander Zien, Gunnar Rätsch, Sebastian Mika, Bernhard Schölkopf, Thomas Lengauer, and K.-R. Müller. Engineering support vector machine kernels that recognize translation initiation sites. Bioinformatics, 16(9): , [19] B. Schölkopf and A. J. Smola. Learning with Kernels. MIT Press, Cambridge, MA,

102 2009 Technical Report on Information-Based Induction Sciences 2009 (IBIS2009) Phase Diagram Study on Variational Bayes Learning of Bernoulli Mixture Daisuke Kaji Sumio Watanabe Abstract: Variational Bayes learning is widely used in statistical models that contain hidden variables, for example, normal mixtures, binomial mixtures, and hidden Markov models. Although it is reported that variational Bayes learning of mixtute model has a phase transition structure which depends on hyperparameters of mixture ratio, the detail behavior and relation of hyperparameters concerning the phase transition have not yet known. In the present paper, we experimentally investigate the phase diagram concerning the hyperparameters by using the Bernoulli mixture and show the guidance to set the hyperparameters. Keywords: variational Bayes, phase transition, phase diagram, hyperparameter, Bernoulli mixture 1 EM [3, 8] [1, 12] /, , tel , kaji@cs.pi.titech.ac.jp Tokyo Inistitute of Technology, 4259 Nagatsuda, Midoriku,Yokohama, Japan (), , tel , daisuke.kaji@konicaminolta.jp Konicaminolta Medical & Graphic,INC, 2970 Ishikawa-machi, Hachioji-shi,Tokyo, , Japan, , swatanab@cs.pi.titech.ac.jp, PI Lab., Tokyo Institute of Technology, 4259 Nagatsuta Midoriku, Yokohama, , Japan 2 2 [3, 10] 1 M B(x µ) = µ x i (1 µ) (1 xi), i=1 x = (x 1,, x M ) T µ = (µ 1,, µ M ) T M K p(x π, µ) = π k B(x µ k ), k=1 π B(x µ k ) K x 89

103 z x z = (0,, 1,, 0) z x Z = (z 1,, z N ), X = (x 1,, x N ), π θ p(x Z, θ) = p(θ) = N p(z π) = n=1 k=1 m N K n=1 k=1 π z nk k, ( K M znk θ x nm km (1 θ km) (1 xnm)), p(π) = Γ(Ka) Γ(a) K K M k=1 m=1 K k π a 1 k, ( ) Γ(2b) Γ(b) 2 θb 1 km (1 θ km) b 1 (a, b) p(π)p(θ) Y X q(y ) p(y X) F (X) = F [q(y )] + KL(q(Y ) p(y X)), F, F Kullback-Leibler KL F (X) = log p(x, Y )dy = log p(x), q(y ) F [q(y )] = q(y ) log p(x, Y ) dy, q(y ) KL(q(Y ) p(y X)) = q(y ) log p(y X)) dy. q(y ) F [q(y )] q(y ) p(y X) Kullback- Leibler w q(y ) q(y X) = q(z, w X) = q 1 (Z X)q 2 (w X) F [q(y X)] q 1 q 2 Z q 1(Z X) = 1, q 2 (w X)dw = 1 log q 1 (Z X) = E q2 [log P (X, Z, w)] + C 1, (1) log q 2 (w X) = E q1 [log P (X, Z, w)] + C 2, (2) C 1, C 2 (1) (2) 3.2 (1)(2 2) VB e-step ( K ) log ρ nk = ψ(α k ) ψ α k + r nk = VB m-step ρ nk K k=1 ρ nk N k = n=1 N r nk, n=1 k M G(η km, η km) m=1 a k = a + N k N N η km = b + r nk x nm, η km = b + r nk (1 x nm ) G(η km, η km) n=1 = x nm ψ(η km ) x nm ψ(η km)+ψ(η km) ψ(η km + η km) ψ ψ(a) d da log Γ(a) = Γ (a) Γ(a) q 1 (π) = Dir(π α), q 2 (θ) = K M Beta(θ km η km, η km) k=1 m=1 90

104 4 [Theorem1:K.Watanabe,S.Watanabe] K 0 K ˆF n M = 3 p (x) = 0.8 (0.9 x x ) (0.1 x x ) 2 1 λ 1 log n + nk n (ŵ) + c 1 < ˆF n S n < λ 2 log n + c 2 S n K n (ŵ) ŵ c 1, c 2 λ 1 λ 2 M = M+1 2 λ 1 = { (K 1)a + M 2, (a M ) MK+K 1 2, (a > M ) λ 2 = { (K K 0 )a + MK0+K0 1 2, (a M ) MK+K 1 2, (a > M ) a a = M a M 0 a > M a = M [1] [12] 1: () 1 0 x i (i = 1, 2) a = = N N = S i (i = 1,, 8) t S (t) i (t = 1, 2, 3)S 1 S 8 P 1 P 8 91

105 VB e-step log ρ Si k = Ψ α (k) + S (1) i Ψ 1 (k) + (1 S (1) i )Ψ 1(k) + Ψ 2(k) + S (2) i Ψ 1 (k) + (1 S (2) i )Ψ 1(k) + Ψ 2(k) + S (3) i Ψ 1 (k) + (1 S (3) i )Ψ 1(k) + Ψ 2(k) ρ Si k r Si k = 4 k=1 ρ S i k VB m-step N k = 4 NP i r Si k, i=1 a k = a + N k η 1k = b + r S1 knp 1 + r S2 knp 2 + r S3 knp 3 + r S4 knp 4 η 2k = b + r S1kNP 1 + r S2kNP 2 + r S5kNP 5 + r S6kNP 6 η 3k = b + r S1 knp 1 + r S4 knp 4 + r S5 knp 5 + r S7 knp 7 η 1k = b + r S5 knp 5 + r S6 knp 6 + r S7 knp 7 + r S8 knp 8 η 2k = b + r S3kNP 3 + r S4kNP 4 + r S7kNP 7 + r S8kNP 8 η 3k = b + r S2kNP 2 + r S3kNP 3 + r S6kNP 6 + r S8kNP 8 ( K ) Ψ α (k) = ψ(α k ) ψ α k, k Ψ 1 (k) = ψ(η k1 ) ψ(η k1) + ψ(η k1) ψ(η k1 + η k1), Ψ 2 (k) = ψ(η k2 ) ψ(η k2) + ψ(η k2) ψ(η k2 + η k2), Ψ 1(k) = ψ(η k1) ψ(η k1 + η k1), Ψ 2(k) = ψ(η k2) ψ(η k2 + η k2) 5.2 K = 4 2 a, b (log ) () π 1,, π 4 z = π π z 0 () a = M = = 2 b 2: a b z = π π A:2 B:A C C: () a b 0.5 b = 0.5 b = 1 a b 1 0 [11] a > 2.0 b B A 92

106 C b = 0.5 a < 2.0 C A 4 M = 2 () / 6 M = 2, M = 3 3: (:), (:), () A,B,C [1] K. Watanabe and S. Watanabe. Stochastic complexities of general mixture models in Variational Bayesian Approximation. Neural Computation, Vol. 18, No. 5, pp , [2] S. Nakajima and S. Watanabe. Variational Bayes Solution of Linear Neural Networks and its Generalization Performance. Neural Computation, Vol. 19, No. 4, pp , [3] C. M. Bishop. Pattern Recognition and Machine Learning. Springer, [4] S. Watanabe. Algebraic analysis for singular statistical estimation. Proc. of International Journal of AlgorithmicLearningTheory Lecture Notes on Computer Sciences,1720, pp.39-50, : M = 2 () () a, b [5] S. Watanabe. Algebraic Analysis for Nonidentifiable LearningMachines. Neural Computation, Vol.13, No.4, pp , 2001 [6] S. Watanabe. Learning efficiency of redundant neural networksin Bayesian estimation. IEEE Transactions on NeuralNetworks, Vol.12, No.6, pp ,

107 [7] H. Attias. Inferring parameters and structure of latent variable models by variational Bayes, In Proc. of Uncertainty in Artificial Intelligence(UAI 99),1999. [8] M. J. Beal. Variational Algorithms for approximate Bayesian inference. PhD thesis, University College London, [9] Z. Ghahramani and M. J. Beal. Graphical Models and Variational Methods. In Advanced Mean Field. Methods. MIT Press, 2000 [10] P. F. Lazarsfeld and N. W. Henry. Latent structure analysis. Houghton Mifflin, 1968 [11] D. Kaji and S. Watanabe. Optimal Hyperparameters for Generalized Learning and Knowledge Discovery in Variational Bayes. To appear in Proc. of ICONIP, 2009 [12],.. NC, January

108 ¾¼¼ ÌÒÐ ÊÔÓÖØ ÓÒ ÁÒÓÖÑØÓÒ¹ ÁÒÙ¹ ØÓÒ ËÒ ¾¼¼ ÁÁ˾¼¼µ ØÖÑÒ Ø ÁÑ ËÑÒØØÓÒ Ý Ù Ó ÊÓÒ¹ ÀÒ ÎÖÐ Ë ÅÝÓ Ý Å ØÓ Ç ØÖØ ÁÒ Ñ ÔÖÓ Ò Ú Ý Ò ÒÖÒ ÓÒ ÅÊ ÑÓÐ ÒØÖÓÙØÓÒ Ó Ò ÚÖÐ «ØÚ ØÓ ÔÖ ÖÚ Ò Ø Ñº Ï ÖÚ Ø Ñ ¹ ÑÒØØÓÒ ÐÓÖØÑ ÓÒ Ø ÈÓØØ ¹ ÔÒ¹ØÝÔ ÖÓÒ¹ Ò ÚÖÐ Ò Ø ÚÖØÓÒÐ ÑØÓº Ì ÐÓÖØÑ ÔÔÐ ØÓ ÓØ Ø ÝÒØ Þ Ñ ÓÒØÑÒØ Ý Ù Ò ÒÓ Ò Ø ÒØÙÖРѺ ÜÔÖÑÒØÐ Ö ÙÐØ ÓÛ Ø «ØÚÒ Ò ÖÓÙ ØÒ º ÃÝÛÓÖ ÅÊ Ñ ÑÒØØÓÒ ÖÓÒ¹ ÚÖØÓÒÐ ÑØÓ ½ ÅÊ ½ ¾ ÅÊ ¾ ÅÊ ÃÒÑÙÖ ÌÔÔÒ ÓÔ ¹¼ Øк ¼¹ ¹½½½ ¹ÑÐ ÑÝÓ ÔÙºÒ ¹ ÙººÔ ÙÐØÝ Ó ÒÒÖÒ ËÒ ÃÒ ÍÒÚÖ ØÝ ¹ ¹ ÑØ¹Ó ËÙØ Ç ¹¼ ÂÔÒ Ý ¾¹½ ¹½¹ ½¹¼½ ¾¹½ ¹ÑÐ ÓºÙ¹ØÓÝÓººÔ ÖÙØ ËÓÓÐ Ó ÖÓÒØÖ ËÒ Ì ÍÒÚÖ ØÝ ÓÌÓÝÓ ¹½¹ à ÛÒÓ Ã Û ¾¹½ ÂÔÒ ÊÁÃÆ ÖÒ ËÒ ÁÒ ØØÙØ ¾¹½ ÀÖÓ Û ÏÓ ËØÑ ½¹¼½ ÂÔÒ ÅÊ ¾ ¾ ½¼ ¾ ÑÒ ÅÊ 95

109 ÖØ ÓÐ ½½ ¾ Ü Ü ½Æ à ½Æ Æ ½ ¾ ½ ¼¼µ Ì ¼ ½¼µ Ì ¼ ¼½µ Ì ½µ Ô Ü Ãµ Ü Ô Ãܵ Õ Ãµ Ä Õ Ãµµ Õ ÃµÐÒ Ã Ô Ü Ãµ Õ Ãµ ¾µ Õ Ãµ Ô Ãܵ ¹ ÃÄ Õ ÃµÔ Ãܵµ Ã Õ ÃµÐÒ Ô Ãܵ Õ Ãµ ÐÒ Ô Üµ Ä Õ Ãµµ ÃÄ Õ ÃµÔ Ãܵµ µ µ µ à ¹ Ô Ãܵ Õ Ãµ Ä Õ Ãµµ Õ Ãµ Õ Ãµ Õ Ãµ Ä Õ Ãµµ ¾ Õ µ Ä Õ Ãµµ Õ µ ÐÒ Õ µ ÐÒ Ô Ü Ãµ ÓÒ Øº µ Õ ½µ Ü Ãµ Ð Ñ Ü Ð Ü Ñ µ ¾ ½ Ð Ñ µ ÐÑ µ È ÐÑ Ð Ñ Ü Ð Ü Ñ µ ¾ Ô Ü Ãµ Ô Ü Ãµ ½ ÜÔ ¾ Ü Ãµ Ô Ü Ãµ Ö Ð Ñ µ Æ µ ܼ à µ ½ µ Ö µ ½ ÜÔ ½¼µ ½ ¾ Æ Ü¼ à µ ½ µ ¼ à µ ½ Ã È ¼ ¾Æ µ µ ½ µ µ ÓØÖÛ ½½µ Æ µ µ µ ÜÔ È ¾Æ µ È ½ ÜÔ È ¾Æ µ ¾ Ü Ü µ ¾ ¾ Ü Ü µ ¾ µ ½¾µ Õ Ãµ Õ µ µ ½¾µ 96

110 ½ ÈËÆÊ ¾ ¾ ½ ¼¼¾ ½¾µ ½ ¼ ¾ ÈËÆÊ ½ ½ ½ ¼¼¾ ½¼ ½ ½ ÈËÆÊ ¾ ¾ ½ ÈËÆÊ ½ ½ ¾ ¾ 30 Frequency Intensity Frequency Intensity ½¼ ½ ½½ 97

111 ½¾½ ½ ¼¼¼½ ½¾µ ½ ¼ Ô Ãܵ ½¾½ ½ ¼ ¾ ¼ ½ ½ ËØÒ º Ä ÅÖÓÚ ÊÒÓÑ Ð ÅÓÐÒ Ò ÁÑ ÒÐÝ ÌÖ ØÓÒµ ËÔÖÒÖ ¾¼¼º ¾ ¾¼¼º º ź ÓÔ ÈØØÖÒ ÊÓÒØÓÒ Ò ÅÒ ÄÖÒÒ ËÔÖÒÖ ¾¼¼º ú ÌÒ ËØØ ØйÑÒÐ ÔÔÖÓ ØÓ Ñ ÔÖÓ Ò Âº ÈÝ º Åغ Òº ʽßʽ¼ ¾¼¼¾º ½½ ź º ÌÔÔÒ Ò º ź ÓÔ Ý Ò Áѹ ËÙÔÖ¹Ö ÓÐÙ ÓÒ ÚÒ Ò ÆÙÖÐ ÁÒÓÖ¹ ÑØÓÒ ÈÖÓ Ò ËÝ ØÑ ½ ½¾ß½¾ ¾¼¼ º ½¾ ½ ½ ¾ º ÃÒÑÙÖ Ëº Å Ò Ëº Á ËÙÔÖÖ Ó¹ ÐÙØÓÒ ÛØ ÓÑÔÓÙÒ ÅÖÓÚ ÖÒÓÑ Ð Ú Ø ÚÖØÓÒÐ Å ÐÓÖØÑ ÆÙÖÐ ÆØÛÓÖ Ò ÔÖ µ ˺ ÑÒ Ò º ÑÒ ËØÓ Ø ÖÐÜØÓÒ ØÖÙØÓÒ Ò Ø Ý Ò Ö ØÖÓÖØÓÒ Ó Ñ Á ØÖÒ º ÈÅÁ µ ¾½ß½ ½º º ÑÒ Ëº ÑÒ º ÖÆÒ Ò Èº ÓÒ ÓÙÒÖÝ ØØÓÒ Ý ÓÒ ØÖÒ ÓÔØÑÞ¹ ØÓÒ Á ØÖÒ º ÈÅÁ ½¾ µ ¼ß¾ ½¼º ½ ½ ÅÊ Æ¹½ ½º ½¼ ƺ ʺ ÈÐ Ò Ëº ú ÈÐ ÖÚÛ Ó Ñ ÑÒØØÓÒ ØÒÕÙ ÈØØÖÒ ÊÓÒØÓÒ ¾ µ ½¾ß½ ½ º ½½ º ÖØ ÓÐ Ò Å ËÐÐ ÁÑ ÖÐÜØÓÒ Ý Ù Ó Ø ÈÓØØ ÑÓÐ ÛØ Ø ØÖÑÒ Ø ÑØÓ Âº ÇÔغ ËÓº Ѻ ½ µ ½¼ ß½¼ ½º 98

112 2009 Technical Report on Information-Based Induction Sciences 2009 (IBIS2009) Experiments and Considerations on Data Mining Based on Cofactor Implication Niki Katsuya Minato Shin-ichi Abstract: We report some unknown relations among the items in the database can be extracted by focusing on cofactor implication. Cofactor implication can be found by using ZDD which is efficient algorithm for processing data. However, as the means has strict restrictions for irregular data, some meaningful relations might be left unfound. In this paper, we propose the way for extracting such linkage, and studied what can be extracted, especially from the database which consists of one-hot type items. Keywords: BDD, ZDD, data mining, transaction database, cofactor implication, sets of combinations 1 Frequent Itemset Mining Agrawal [2] Apriori VLSI CAD BDD: Binary Decision Diagrams [3] BDDZDD: Zero-suppressed BDD [4] ZDD ZDD [1] a, , niki@mx-alg.ist.hokudai.ac.jp Algorithm Laboratory, Graduate School of Information Science and Technology, Hokkaido University, Sapporo, Japan. minato@ist.hokudai.ac.jp b a b mushroom [7] n [11] a b 2 ZDD 2.1 ZDD BDD 1 99

113 3: 1: BDDZDD 2: ZDD 01 0-/1-2 0-/1-1 2 BDD n BDD BDD ZDD [4] ZDD BDD ZDD 1 BDD 3 ab ba 2.2 [1] ab a b a b S ab, S ab, S ab, S ab 4a (S ab ) b (S ab ) S ab S ab a b a b b c a c S ab a b a b S S = {bc, acd, ce, ac, abe, bd, bcd} S ab = {ce} S ab = {c, d, cd} S ab = {cd, c} S ab = {e} 100

114 6: S ab S ab 4: n [1] [5] VSOP [6] 3 5: S ab S ab a b S ab S ab b a 4 S ab S ab S ab S ab 5 mushroom [7] n n(n 1) ZDD 6 S ab S ab S ab S ab α ab S ab S ab β ab S ab S ab γ ab α ab = S ab S ab, β ab = S ab \ S ab, γ ab = S ab \ S ab α ab = γ ab = S ab = α ab γ ab = S ab S ab a b S ab = {c, d, cd} S ab = {cd, c} α ab = {c, cd} β ab = {d} γ ab = { } γ a b α ab = α ba, β ab = γ ba γ ab α ab β ab γ ab 101

115 4 7: β ab γ ab α ab β ab γ ab VSOP [6] α ab VSOP α ab β ab γ ab α ab β ab γ ab α ab β ab γ ab α ab β ab γ ab x x α ab a A b B S ab B S ab A α ab mushroom(fimi [7] ) mushroom α xy (x, y)

116 8: mushroom α βγ several solitary 2GHz Intel Core2 Duo4GB 1067MHz DDR α 337 mushroom βγ α α β γ β γ 9: 10: BMS-Web-View BMS-Web-View-1(FIMI [7] ) α α βγ α 6 103

117 mushroom 7 Thomas Zeugman [1] :,, SIG-FPAI-A603-10, pp Mar [2] R. Agrawal, H. Mannila, R. Strikant, H. Toivonen and A. I. Verkamo, Fast Discovery of Association Rules, In Advances in Knowledge Discovery and Dat Mining, MIT Press, Boolean function manipulation, IEEE Transactions on Competers, Vol. C-35, No. 8, pp [4] S. Minato: Zero-Suppressed BDDs for Set Manipulation in Combinatorial Problems, In Proc. of 30th ACM/IEEE Design Automation Conference (DAC 93), pp Jun [5] S.Minato and K.Ito: Symmetric Item Set Mining Method Using Zero-suppressed BDDs and Aplication to Biological Data, Trans. of the Japanese Society of Artificial Intelligence, Vol. 22, No. 2, pp Feb [6] : VSOP BDD,, Vol. 105, No. 72, COMP , pp May [7] B. Goethals, M. Javeed Zaki (Eds.), Frequent Itemset Mining Dataset Repository, Frequent Itemset Mining Implementations (FIMI 03), (2003) [8], :,, Vol. J89-D, No. 2, pp Feb [9] : BDD,, Vol. 106, No. 566, COMP pp Mar [10] : BDD,,, Vol. 107, No. 78, AI2007-6, pp May [11] :, 4 (SIG-DMSM), SIG-DMSMJuly [3] R. E. Bryant : Graph-based algorithms for 104

118 2009 Technical Report on Information-Based Induction Sciences 2009 (IBIS2009) MFCC Text-Independent Speaker Verification using Optimized Linear Combination of Local MFCC Features Shunsuke Sakai Keisuke Kameyama Abstract: In recent years, studies of speaker verification have been conducted as a means for biometric person authentication. However, because of the overall verification performance, only few actual implementations exist. This paper focuses on the text-independent speaker verification system. We propose an effective method for speaker verification by adaptive weighting of local Mel Frequency Cepstrum Coefficient (MFCC) features. For a given set of registered persons, an optimal linear weightings of multiple speech frames are searched based on the likelihood ratio error, generalizing the scheme of the conventional use of Δ parameters [1]. It was observed that using the proposed adaptive parameters, superior verification performance was achieved compared with the cases using conventional features. Keywords: text-independent speaker verification, biometric authentication, adaptive feature weighting, inter-frame feature. 1 Introduction In recent years, as a technology to verify individuals, studies for authentication using human biometrics have been conducted actively [2]. In password authentication conventionally often used, there are problems that the users forget the phrase and impostor can be easily verified due to leakage or theft. Therefore, biometrics technology verifying the individuals using physical information such as fingerprint, vein pattern, iris, face, and speech is in the spotlight [3]. Speaker verification technology to verify the speaker by speech features can be useful as it does not require special verification, , tel ext. 8480, sakai@adapt.cs.tsukuba.ac.jp, Graduate School of Systems and Information Engineering, University of Tsukuba, Tennodai, Tsukuba, Ibaraki , Japan, , tel ext. 8480, Keisuke.Kameyama@cs.tsukuba.ac.jp, Graduate School of Systems and Information Engineering, University of Tsukuba, Tennodai, Tsukuba, Ibaraki , Japan hardware, is less stressful for the users, and can be used from remote places across the telephone network. However, actual use of speaker verification technology is still less common. The main reason is due to the fact that the verification performance is still low when compared with the use of other modalities. This study proposes a framework of speaker modeling to use the inter-frame dynamics in addition to perframe Mel Frequency Cepstrum Coefficient (MFCC) speech feature aiming to improve the verification precision. While ΔMFCC feature [1] is known to take a similar approach, it uses fixed weights of local MFCC features. In contrast, the proposed feature employs adaptive weights optimized for improving the verification performance. Also, some methods that present mel-cepstral analysis method and its adaptive algorithm [4], and method which optimizes the weights for each likelihood such that the overall expected loss can be minimized [5], have been reported. In this work, we examine the improvement of speaker discrimination ability by using an optimized linear combination 105

119 of relatively long-term features. 2 Text-independent speaker verification The methods of speaker verification is divided into three main groups [6]. Text-dependent system specifies the verification text in beforehand, text-independent system does not limit the text, and text-prompted system prompts the text at each verification. In general, it is known that text-dependent and text-prompted systems indicate higher performance, because the system can use the speaker information depending on phonological line describing the text [7]. Recently, the study of text-independent speaker verification having an advantage of not limiting the speech text, has been the mainstream research topic [6]. This paper discusses text-independent verification as well. 2.1 Procedure The general flow of text-independent speaker verification system is shown in Figs. 1 and 2 [6]. At first, in the modeling phase, the model of each speaker λ C and the background model λ C are obtained based on the speech signals. The speaker model λ C uses a collection the authentic speech, whereas the background model (Universal Background Model [8]) uses speeches by various speakers (average feature) in the training. In the test phase, the log-likelihood ratio of input speech feature vectors to the claimed speaker model λ C and the background model λ C is calculated, and this value is compared to the predetermined threshold value θ. The speaker is accepted if the value is higher than the threshold value, and is rejected otherwise. The loglikelihood ratio is defined as Λ C (X) =logp(x λ C ) log p(x λ C). (1) Here, if x(i) is a feature vector in frame i (i =1, 2,..., T ), p(x λ C ) indicates the likelihood that speech X = {x(1),..., x(t )} is from the claimed speaker giving the model λ C. On the other hand, p(x λ C) indicates the likelihood that speech X is not from the claimed speaker. Additionally, the likelihood that the model of the claimed speaker gives the input speech collection X is X C 1: Modeling phase p X λ ) ( C λ C p( X λ ) C λ C 2: Test phase Λ C (X ) ( Λ θ ) ( Λ < θ ) defined as log p(x λ C )= 1 T log p (x(t) λ C ). (2) T t=1 As a speech feature, Mel Frequency Cepstrum Coefficient (MFCC) is used commonly [9]. Here, MFCC feature of D component cepstrum coefficients is described as x =[c 1,c 2,..., c D ]. 2.2 Gaussian mixture speaker model Text-independent speaker verification does not have a limitation about the speech contents by the speaker. Therefore, in the speaker modeling, likelihood function of speech feature is modeled as a density function, for example with Gaussian Mixture Model (GMM) [9]. Here, if x is a D-dimensional feature vector, likelihood function of a registered speaker s (s =1,..., N) isdefined as M p(x λ s )= w sj b sj (x). (3) j=1 This is a linear combination of M Gaussian functions b sj (x), each computed as 1 b sj (x) = (2π) D 2 Σ sj 1 2 { exp 1 } 2 (x µ sj) (Σ sj ) 1 (x µ sj ), (4) which is determined by a mean vector µ sj,acovariance matrix Σ sj,andweightsw sj (j =1,..., M). Here, the parameter set of speaker s is denoted as λ s = {(w s1, µ s1, Σ s1 ),..., (w sm, µ sm, Σ sm )}. Thespeaker modelparametersareestimated using the Expectation- Maximization (EM) algorithm [10]. 106

120 3 Adaptive weighting of local MFCC features It has been reported that, by adding inter-frame dynamic information to short-time per-frame speech feature as MFCC, there are cases that the verification performances are improved [1]. This feature known as the ΔMFCC regression coefficient of change along the time axis of MFCC, is sometimes used together with MFCC [6]. However, it only uses restrictive weights of local MFCC features. We propose an adaptive method for weighting the local MFCC features that searches the optimal linear weightings of multiple speech frames based on the likelihood ratio error, generalizing the strategy taken by the Δ parameters employing restrictive weights. i i s Σ si i s 3: Extraction of weighted feature c Fsi. Each horizontal bar denotes the period for a single frame. Δ X u X s a s a s s 3.1 Generalization of ΔMFCC feature The inter-frame regression coefficient known as ΔMFCC is computed as l k= l Δc i (m) = k c i(m + k) l, (i =1,..., D) (5) k= l k2 λ UBM λ s where l is the frame range that regression coefficient is calculated, c is the cepstrum coefficient [1], and m is the frame index. In this work, we propose to generalize Eq. (5), searching for an arbitrary dynamic feature to improve verification precision among linear combinations of cepstrum coefficients in neighboring frames. Therefore, 2l + 1 parameters {a s ( l),..., a s (l)} in c Fsi (m) = l a s (k)c i (m + k) (i =1,..., D) (6) k= l are adjusted. Here, in the case of ΔMFCC, parameter a s (k) amounts to the special case of k a Δ (k) = l. (k = l,..., l) (7) k= l k2 Figure 3 shows the schematic for calculating the feature c Fsi. 3.2 Coefficient search based on likelihood ratio error The verification precision can be evaluated by loglikelihood ratio. Therefore, in the proposed method, the coefficient parameter vector for each speaker a s =[a s ( l),a s ( l +1),..., a s (l)] R 2l+1 (8) d Λ(X ) Δa u s 4: The training process. This modification is applied to the GMM parameter set λ s and coefficient vector a s of each registrant s (s =1,...,N), iteratively. is updated by steepest descent method to minimize the error in log-likelihood ratio for the teacher signal. Besides, each speaker has his/her own individual parameter vector. Figure 4 shows the whole procedure of parameter vector modification. For a speech X u attributed to speaker u, the T frames in X u is converted to a feature vector array g s (1),...,g s (T ) R 2D, using the parameters a s and λ s of a registrant s. The feature vector g s (t) which concatenates MFCC features x =[c 1,c 2,..., c D ] and F- MFCC features y s =[c Fs1,c Fs2,..., c FsD ], is denoted as g s (t) =[x, y s ] = M(t)a s + b(t) R 2D. (9) 107

121 M(t) = c 1 (t l) c 1 (t + l)..... c D (t l) c D (t + l) R 2D (2l+1) and (10) σ(λ) g s (t) = = 1 T 1 g s (t) T T log p(g s (i) λ k ) i=1 g s (t) log p(g s(t) λ k ), (19) g s (t) log p(g s(t) λ s )= M j=1 c sj(t){g s (t) µ sj } (Σ sj ) 1 M j=1 c sj(t) (20) b(t) =[c 1 (t),..., c D (t), 0,..., 0] R 2D. (11) Then, the likelihoods for the background model and the corresponding speaker model are calculated by Eq. (3), and the log-likelihood ratio Λ(X u ) is obtained as Eq. (1). Additionally, Λ(X u ) is converted to a value in (0, 1) by the sigmoid function defined as 1 σ(λ) = (12) 1+exp( β Λ) considering the convergence performance in the next training phase. Here, β is the slope parameter. The initial a s will be set identical to ΔMFCC coefficients {a Δ ( l),..., a Δ (l)}. Next, the coefficient parameter vector a s is modified to minimize the error between the log-likelihood ratio σ(λ) and teacher signal. The teacher signal d is set as, { 1 (s = u) (Authentic speaker) d = (13) 0 (s u) (Impostor)). Vector a s is updated as (τ +1) a s = a (τ s ) +Δa s, (14) by the amount of correction defined as Δa s = η E LLR a s = η E LLR σ(λ) σ(λ) a s, (15) where, η is the learning coefficient. The log-likelihood ratio error E LLR is defined as E LLR = 1 2 (d σ(λ))2. (16) Each partial differentiation is computed as follows based on the definition above. E LLR σ(λ) = 1 σ(λ) 2 (d σ(λ))2 = (d σ(λ)), (17) σ(λ) a s = T t=1 ( σ(λ) g s (t) ) g s (t), (18) a s where,c sj (t) =w sj b sj (g s (t)), and (21) g s (t) = {M(t)a s + b(t)} = M(t). (22) a s a s Coefficient vector a s foreachregisteredspeakerisupdated using the amount of correction Δa s in Eq. (15) by the following steps. Training procedure * Initialization 1) Train the background model (λ UBM )and all the speaker model (λ s ) using feature vector (MFCC+ΔMFCC). 2) Calculate the log-likelihood ratio Λ(X u ) using feature vector (MFCC+ΔMFCC) derived from speech X u. * Training of individual weight a s and GMM 3) Calculate Δa s to minimize the error E LLR based on the teacher signal d, and update a s. 4) Train the speaker model (λ s ) using speech X s again. 5) Verify using the new feature vector (MFCC+F-MFCC) for X u by the updated model, and calculate the log-likelihood ratio. 6) Repeat 3-5 for the whole training set for a predetermined time. 4 Experiment We evaluated the proposed method in four experiments. In Experiment 1, the optimal number of Gaussian component for our dataset is determined. In Experiment 2, the iteration number of learning a s is determined. Experiment 3 compares the verification performance for the conventional method and the proposed method, and Experiment 4 is the same comparison using telephone speech. 108

122 4.1 Experimental condition Allspeechdatausedintheexperimentsaresampled at 16kHz, in 16 bit. The speech data is extracted from the ASJ Continuous Speech Corpus by the Acoustical Society of Japan [11]. The number of clients are 10 (5 male, 5 female), and the speech length for speaker modeling is 60sec. The size of the training set is 100 trials (10 authentic, 90 impostor). The size of the test set is trials (500 authentic, 9500 impostor), with the speech length of 3sec. There are 50 speech clips from each speaker. The speech for the training and testing are from different texts. In the training and test phases, if one speaker is set to a client, all other speakers are set to impostors. The frame number l in Eq. (8) is set to l =2,and the slope β in Eq. (12) is set to β = 1.0. All the experiments were conducted on a 3.6 GHz Intel Xeon computer with 3.0 GB of RAM, running Windows XP. 4.2 Experiment 1: Optimal number of Gaussian components Procedure In this experiment, we determined the optimal number of Gaussian component for our dataset. The background model and the speaker model with 16 to 512 Gaussian components were trained and evaluated using the same training and test data with clean ( db) speech. Equal Error Rate (EER) and the detection error tradeoff (DET) curve [12] are used as the index for evaluation Result The comparison of each case is shown in Table 1 and Figure and 256 Gaussian components achieved the best among all in EER, and 128 appears slightly better than all others in DET curves. These results show that more Gaussian components are not always better. From this result, we use 128 Gaussian components in the following experiments. 4.3 Experiment 2: Change in MSE with the iteration number of learning Procedure In order to determine the appropriate training iteration, the change of error E LLR during training was 1: Exp.1 Comparison of EER by Gaussian component Gaussian component EER (%) : Exp.1 Comparison of DET curves by Gaussian component invetigated. Mean-squared error (MSE) E LLR for the training and test sets were recorded at each training round (100 trials). Normal speech in Experiment 3 and telephone speech in Experiment 4 are used, these include the same three cases of S/N as Experiment 3 and Result Figure 6 shows the case of normal speech in db test set and Figure 7 shows the case of telephone speech in db test set. These results show that the MSE of the test sets is minimum at the early stage in contrast to the monotonically decreasing training set error. The same tendency of these was seen in 30dB and 20dB test sets. From this result, we set the iteration number of learning a s varying GMM parameters at the same time to 400 (4 rounds of 100) in Experiment 3 and to 200 (2 rounds of 100) in Experiment 4. The value 400 was also used in Experiment

123 : Exp.3 Comparison of EER (%) MSE training set test set db 30dB 20dB MFCC MFCC+ΔMFCC MFCC+F-MFCC iteraon number (round of 100 trials) 6: Exp.2 Change in MSE for training and test sets (normal, db) MSE iteraon number (round of 100 trials) training set test set 7: Exp.2 Change in MSE for training and test sets (phone, db) 4.4 Experiment 3: Comparison of the verification performance Procedure The conventional feature of MFCC, MFCC with ΔMFCC, and the proposed feature of MFCC with F- MFCC are compared in terms of the verification performance. The verification speech include three cases of S/N, namely, db which is the ideal case, and levels common in usual verification use (30dB and 20dB). EER and DET curves are used as the index for evaluation as Experiment Result The comparison using the EER(%) measure is shown in Table 2. Figs show the DET curves obtained for each S/N. In the case of training and testing with clean speech, the proposed method is superior to the conventional ones for verification speech with db. Additionally, in the case of training and testing with 8: Exp.3 DET curves in db test set. noisy speech, the proposed feature indicates higher performance than the conventional ones. Thus, the superiority of adaptively choosing the weighting of local MFCC features (MFCC+F-MFCC) has been verified. 4.5 Experiment 4: Comparison of the verification performance (telephone speech) Procedure The same procedure as Experiment 3 is conducted with telephone speech. The frequency range of Hz was extracted from the same dataset, and was used as the simulated voice through a telephone line. The simulated telephone speech were used in both training and testing Result The comparison using the EER(%) measure is shown in Table 3, and the DET curves are shown in Figs Here, the results also imply the superiority of using the proposed feature MFCC+F-MFCC in verification 110

124 9: Exp.3 DET curves in 30dB test set. 10: Exp.3 DET curves in 20dB test set. 3: Exp.4 Comparison of EER (%) db 30dB 20dB MFCC MFCC+ΔMFCC MFCC+F-MFCC through the telephone line. 5 Conclusion In this paper, we proposed a method for speaker verification using adaptive weighting of local MFCC features. The core idea was to determine the optimal frame coefficient parameter to minimize the verification error. In the experiments, it was shown that this mechanism gives lower verification error than the conventional methods under clean and certain level of noise environment, including authentication via a telephone line. In future works, we consider methods that the present linear combination of local MFCC is extended to nonlinear transformation and indices other than the likelihood ratio error are used. [1] S. Furui, Comparison of speaker recognition methods using static features and dynamic features, IEEE Transactions on Acoustics, Speech and Signal Processing, vol. 29, no. 3, [2]A.K.Jain,P.Flynn,andA.Ross,Handbook of Biometrics. Springer, [3] A. K. Jain, A. Ross, and S. Prabhakar, An introduction to biometric recognition, IEEE Transactions on Circuits and Systems for Video Technology, vol. 14, no. 1, pp. 4 20, [4] T. Fukada, K. Tokuda, T. Kobayashi, and S. Imai, An adaptive algorithm for mel-cepstral analysis of speech, ICASSP-92, vol. 1, pp , [5] Y.-H. Chao, W.-H. Tsai, H.-M. Wang, and R.- C. Chang, Improving the characterization of the alternative hypothesis via minimum verification error training with applications to speaker verification, Pattern Recognition, vol. 42, no. 7, pp , [6] F. Bimbot, J.-F. Bonastre, C. Fredouille, G. Gravier, I. Magrin-Chagnolleau, S. Meignier, T. Merlin, J. Ortega-Garcia, D. Petrovska- Delacretaz, and D. A. Reynolds, A tutorial on text-independent speaker verification, EURASIP Journal on Applied Signal Processing, vol. 2004, no. 4, pp , [7] T. Matsui and S. Kuroiwa, Speaker recognition technology : A review and perspective, Institute of Electronics, Information, and Communication Engineers, vol. 87, no. 4, pp ,

125 11: Exp.4 DET curves in db test set. [8] D. A. Reynolds, T. F. Quatieri, and R. B. Dunn, Speaker verification using adapted Gaussian mixture models, DigitalSignalProcessing, no. 10, pp , [9] D. A. Reynolds, Speaker identification and verification using Gaussian mixture speaker models, Speech Communication, no. 17, pp , [10] A.P.Dempster, N.M.Laird, and D.B.Rubin, Maximum likelihood from incomplete data via the EM algorithm, Journal of the Royal Statistical Society (B), vol. 39, no. 1, pp. 1 38, [11] S. Itahashi, M. Yamamoto, T. Takezawa, and T. Kobayashi, Development of ASJ continuous speech corpus Japanese newspaper article sentences (JNAS), Proceedings of COCOSDA 97, [12] A. Martin, G. Doddington, T. Kamm, M. Ordowski, and M. Przybocki, The DET curve in assessment of detection task performance, Proceedings of European Conference on Speech Communication and Technology, pp , : Exp.4 DET curves in 30dB test set. 13: Exp.4 DET curves in 20dB test set. 112

126 2009 Technical Report on Information-Based Induction Sciences 2009 (IBIS2009) Online Multiscale Dynamic Topic Models Tomoharu Iwata Takeshi Yamada Yasushi Sakurai Naonori Ueda Abstract: We propose an online topic model for sequentially analyzing the time evolution of topics in document collections. Topics naturally evolve with multiple timescales. For example, some words may be used consistently over one hundred years, while other words emerge and disappear over periods of a few days. Thus, in the proposed model, current topic-specific distributions over words are assumed to be generated based on the multiscale word distributions of the previous epoch. Considering both the long-timescale dependency as well as the short-timescale dependency yields a more robust model. We derive efficient online inference procedures based on a stochastic EM algorithm, in which the model is sequentially updated using newly obtained data; this means that past data are not required to make the inference. We demonstrate the effectiveness of the proposed method in terms of predictive performance and computational efficiency by examining collections of real documents with timestamps. Keywords: Topic model, Time series analysis, Stochastic EM algorithm 1 [1, 4, 6, 10, 13, 17, 18, 19] bag-of-words Probabilistic Latent Semantic AnalysisPLSA[8] Latent Dirichlet AllocationLDA[5] [5] [9] [11] (Multiscale Dynamic Topic Model: MDTM) NTT NTT Communication Science Laboratories iwata@cslab.kecl.ntt.co.jp EM [1, 4, 10, 18] 113

127 1: Notation Symbol Description D t number of documents at epoch t N t,d number of words in the dth document at epoch t W number of unique words w t,d,n nth word in the dth document at epoch t, w t,d,n {1,, W } Z number of topics z t,d,n topic of the nth word in the dth document at epoch t, z t,d,n {1,, Z} S number of scales θ t,d multinomial distribution over topics for the dth document at epoch t, θ t,d = {θ t,d,z } Z z=1, θ t,d,z 0, P z θ t,d,z = 1 φ t,z multinomial distribution over words for the zth topic at epoch t, φ t,z = {φ t,z,w} W w=1, φ t,z,w 0, P w φt,z,w = 1 ˆω (s) t,z multinomial distribution over words for the zth with scale s topic at epoch t, ˆω (s) t,z = {ˆω t,z,w} (s) W w=1, ˆω (s) t,z,w 0, P w ˆω(s) t,z,w = 1 Multiscale Topic Tomography Model (MTTM)[13] MTTM LDA[5] [14] [8] t d w t,d = {w t,d,n } N t,d n=1 t LDA LDA θ t,d w t,d,n z t,d,n θ t,d φ t,zt,d,n 1(a) LDA θ t,d φ t,z LDA t z φ t,z t 1 { ˆω (s) t 1,z }S s=1 ˆω (s) t 1,z t 1 s z φ t,z φ t,z Dirichlet( S s=0 λ t,z,s ˆω (s) t 1,z ), (1) λ t,z,s > 0 s = 0 ˆω (s=0) t,z,w = W t 1 { ˆω (s) t 1,z }S s=1 t 2.3 ˆω (s) t,z t 2s t 2 LDA θ t,d α t = {α t,z } Z z=1 α t,z Gamma(γα t 1,z, γ), (2) α t 1,z α t 1,z /γ γ t 1 ˆΩ t 1 = {{ ˆω (s) t 1,z }S s=0} Z z=1 α t 1 = {α t,z } Z z=1 t W t = {w t,d } D t d=1 114

128 α t t-1 t α α α θ θ θ θ z z z z w N D w N D w N D w N D φ φ Z ω^ β φ λ N^ S+1 Z φ λ N^ λ S+1 Z S+1 Z (a) LDA (b) MDTM (c) online MDTM 1: Graphical models of (a) latent Dirichlet allocation, (b) the multiscale dynamic topic model, and (c) its online inference version. w t-8 w ˆω (4) t-1,z t-4 w ˆω (3) t-1,z t-2 w ˆω (2) t-1,z ˆω (1) t-1,z t-1 λ t, z,4 λ t, z,3 λ t, z,2 φ t,z λ t, z,1 λ t, z,0 (0) ˆω t-1,z 2: Illustration of multiscale word distributions at epoch t with S = 4. Each histogram shows ˆω (s) t 1,z, which is a multinomial distribution over words with timescale s. 1. For each topic z = 1,, Z: (a) Draw word probability φ t,z Dirichlet( s λ t,z,s ˆω (s) t 1,z ), (b) Draw topic proportion prior α t,z Gamma(γα t 1,z, γ), 2. For each document d = 1,, D t : (a) Draw topic proportions θ t,d Dirichlet(α t ), (b) For each word n = 1,, N t,d : i. Draw topic z t,d,n Multinomial(θ t,d ), ii. Draw word w t,d,n Multinomial(φ t,zt,d,n ). 1(b) 2.2 EM [2] w t W t ˆΩ t 1 P (W t, Z t, α t α t 1, γ, ˆΩ t 1, Λ t ) = P (Z t α t )P (W t Z t, ˆΩ t 1, Λ t )P (α t α t 1, γ),(3) Z t = {{z t,d,n } N t,d n=1 }Dt d=1 Λ t = {{λ t,z,s } S s=0} Z z=1 {φ t,z } Z z=1 P (Z t α t ) = Dt d=1 P (zt,d θ t,d )P (θ t,d α t )dθ t,d {θ t,d } Dt d=1 ( Γ( z P (Z t α t ) = α ) D t,z) z Γ(α t,z) d z Γ(N t,d,z + α t,z ) Γ(N t,d + z α t,z), (4) Γ( ) N t,d,z t d z N t,d = z N t,d,z {φ t,z } Z z=1 P (W t Z t, ˆΩ t 1, Λ t ) = Γ( s λ t,z,s) z w Γ( s λ t,z,s ˆω (s) t 1,z,w ) w Γ(N t,z,w + s λ t,z,s ˆω (s) t 1,z,w ) Γ(N t,z + s λ, (5) t,z,s) N t,z,w t w z N t,z = w N t,z,w 115

129 (2) P (α t α t 1, γ) = z γ γα t 1,z α γα t 1,z 1 t,z exp( γα t,z ). Γ(γα t 1,z ) (6) Collapsed [7] j = (t, d, n) z j z j (3) P (z j = k W t, Z t\j, α t, ˆΩ t 1, Λ t ) N t,d,k\j + α t,k N t,d\j + z α t,z N t,k,wj\j + s λ t,s,k ˆω (s) N t,k\j + s λ t,s,k t 1,k,w j \j j, (7) Λ t α t (3) [12] Λ t λ t,z,s λ t,z,s w ˆω(s) t 1,z,w M t,z,w M t,z, (8) M t,z,w = Ψ(N t,z,w + s λ t,z,s ˆω (s ) t 1,z,w ) Ψ( s λ t,z,s ˆω (s ) t 1,z,w ), M t,z = Ψ(N t,z + s λ t,z,s ) Ψ( s λ t,z,s ). Ψ( ) Ψ(x) = log Γ(x) x α t α t,z γα t 1,z 1 + α t,z d (Ψ(N t,d,z + α t,z ) Ψ(α t,z )) γ + d Ψ(N t,d + z α t,z ) Ψ( z α t,z ) (9) (7) (8)(9) 2.3 ω (s) t,z,w EM ω (s) t,z,w t 2s t z w ˆN (s) t ˆN ˆω (s) t,z,w t t,z,w = = =t 2 s 1 +1 t,z,w (s) t w ˆN ˆN, t,z,w w t =t 2 s 1 +1 t,z,w (10) (s) ˆN t,z,w t 2s t z w ˆN t,z,w t 116 ˆN t,z,w = N t,z ˆφt,z,w ˆφ t,z,w φ t,z,w ˆφ t,z,w = N t,z,w + s λ t,s,z ˆω (s) N t,z + s λ t,s,z t 1,z,w, (11) (10) ˆN t,z,w N t,z,w ˆω (s=1) t,z,w φ t,z,w ˆN ˆω (s=1) t,z,w t,z,w = w ˆN = ˆφ t,z,w, (12) t,z,w (s) (s) ˆN t,z,w ˆN t 1,z,w 2 s 1 ˆN (s) t,z,w ˆN (s) t 1,z,w + ˆN t,z,w ˆN t 2 s 1,z,w. (13) (s) ˆN t,z,w t 2S 1 t 1 ˆN t,z,w O(2 S 1 ZW ) 3 ˆN (s) t,z,w 2s 1 (s) ˆN t 1,z,w (s) ˆN t,z,w ˆN t,z,w O(SZW ) (s) 4 ˆN t,z,w ˆN t,z,w t (s) ˆN t,z,w 1(c) (1) S S λ t,z,s ˆω (s) t 1,z,w = s=1 = λ t,z,s s=1 t 1 t 1 ˆN t =t 2 s 1 t,z,w t 1 ˆN w t =t 2 s 1 t,z,w t =t 2 S 1 λ t,z,t ˆφ t,z,w, (14)

130 (1) 1: ˆN t,z,w ˆN t,z,w 2: for s = 2,, S do 3: if t mod 2 s 1 = 0 then (s) 4: ˆN t,z,w 5: else 6: (s) ˆN 7: end if 8: end for t,z,w ˆN (s 1) t,z,w + ˆN (s) t 1,z,w ˆN (s 1) t 1,z,w 3: Algorithm for the approximate update of s=3 s=2 s=1 t= t= t= t= : Illustration of approximate updating t = 4 to t = 8 with S = 3. λ t,z,t = S s= log 2 (t t +1)+1 w 7 t= λ t,z,s w ˆN t,z,w t 1 t =t 2 s 1 ˆN (s) t,z,w ˆN (s) t,z,w from ˆN t,z,w (15) Λ t O(2 S 1 Z) O(SZ) O(2 S 1 ZW ) O(SZW ) 3 NIPSPNASDiggAddresses 4 NIPS NIPS (Neural Information Processing Systems) ,740 14,036 PNAS Proceedings of the National Academy of Sciences ,477 20,534 Digg Digg ( ,356 23,494 Addresses [18] 6,413 6,759 NIPSPNASAddresses 1 Digg 1 MDTM DTMLDAallLDAoneLDAonline 4 DTM MDTM S = 1 LDAallLDAone LDAonline LDA LDAall LDAone LDAonline LDA [3] Z = 50 S = log 2 T + 1 T γ = 1 α t,z % MDTM MDTM DTM LDAall LDAonline LDAone 6 7 Xeon GHz CPU () MDTM MDTM 117

131 2: Average perplexities over epochs. The value in the parenthesis represents the standard deviation over data sets. MDTM DTM LDAall LDAone LDAonline NIPS (41.3) (37.2) (36.4) (44.0) (41.5) PNAS (122.0) (146.8) (159.7) (268.7) (149.1) Digg (37.7) (46.4) (27.1) (43.4) (43.6) Addresses (56.5) (49.7) (75.3) (70.9) (62.0) perplexity perplexity MDTM LDAall LDAone epoch epoch (a) NIPS (b) PNAS perplexity perplexity MDTM LDAall LDAone epoch epoch (c) Digg (d) Addresses 5: Perplexities for each epoch with MDTM, LDAall, and LDAone. LDAall 8 λ t,z,s 1 4 [15, 16] [1] L. AlSumait, D. Barbara, and C. Domeniconi. On-line lda: Adaptive topic models for mining text streams with applications to topic detection and tracking. In ICDM 08, pages 3 12, [2] C. Andrieu, N. de Freitas, A. Doucet, and M. I. Jordan. An introduction to MCMC for machine learning. Machine Learning, 50(1):5 43, [3] A. Banerjee and S. Basu. Topic models over text streams: A study of batch and online unsupervised learning. In SDM 07, [4] D. M. Blei and J. D. Lafferty. Dynamic topic models. In ICML 06, pages , [5] D. M. Blei, A. Y. Ng, and M. I. Jordan. Latent Dirichlet allocation. Journal of Machine Learning Research, 3: , [6] K. R. Canini, L. Shi, and T. L. Griffiths. Online inference of topics with latent Dirichlet allocation. In AISTATS 09, volume 5, pages 65 72, [7] T. L. Griffiths and M. Steyvers. Finding scientific topics. Proceedings of the National Academy of Sciences, 101 Suppl 1: ,

132 perplexity #scales perplexity #scales perplexity #scales perplexity (a) NIPS (b) PNAS (c) Digg (d) Addresses #scales 6: Average perplexity of MDTM with different numbers of scales all one online all one online all one online all one online (a) NIPS (b) PNAS (c) Digg (d) Addresses 7: Average computational time (sec) of MDTM per epoch with different numbers of scales, LDAall, LDAone, and LDAonline. lambda scale lambda scale lambda scale lambda scale (a) NIPS (b) PNAS (c) Digg (d) Addresses 8: Average normalized weight λ with different scales estimated in MDTM. [8] T. Hofmann. Probabilistic latent semantic analysis. In UAI 99, pages , [9] T. Hofmann. Collaborative filtering via Gaussian probabilistic latent semantic analysis. In SIGIR 03, pages , [10] T. Iwata, S. Watanabe, T. Yamada, and N. Ueda. Topic tracking model for analyzing consumer purchase behavior. In IJCAI 09, [11] T. Iwata, T. Yamada, and N. Ueda. Probabilistic latent semantic visualization: topic model for visualizing documents. In KDD 08, pages , [12] T. Minka. Estimating a Dirichlet distribution. Technical report, M.I.T., [13] R. Nallapati, W. Cohen, S. Ditmore, J. Lafferty, and K. Ung. Multiscale topic tomography. In KDD 07, pages , [14] S. Papadimitriou, J. Sun, and C. Faloutsos. Streaming pattern discovery in multiple time-series. In VLDB 05, pages , [15] L. Ren, D. B. Dunson, and L. Carin. The dynamic hierarchical Dirichlet process. In ICML 08, pages , [16] Y. W. Teh, M. I. Jordan, M. J. Beal, and D. M. Blei. Hierarchical Dirichlet processes. Journal of the American Statistical Association, 101(476): , [17] C. Wang, D. M. Blei, and D. Heckerman. Continuous time dynamic topic models. In UAI 08, pages , [18] X. Wang and A. McCallum. Topics over time: a non- Markov continuous-time model of topical trends. In KDD 06, pages , [19] X. Wei, J. Sun, and X. Wang. Dynamic mixture models for multiple time-series. In IJCAI 07, pages ,

133 2009 Technical Report on Information-Based Induction Sciences 2009 (IBIS2009) Observational Reinforcement Learning Jaak Simm, Masashi Sugiyama, and Hirotaka Hachiya Abstract: We introduce an extension to standard reinforcement learning setting called observational RL (ORL) where additional observational information is available to the agent. This allows the agent to learn the system dynamics with fewer data samples, which is an essential feature for practical applications of RL methods. We show that ORL can be formulated as a multitask learning problem. A similarity-based and a component-based multitask learning methods are proposed for learning the transition probabilities of the ORL problem. The effectiveness of the proposed methods is evaluated in experiments of grid world. 1 Introduction Recently, there is an increasing interest for methods of planning and learning in unknown and stochastic environments. These methods are investigated in the field of Reinforcement Learning (RL) and have been applied to various domains, including robotics, AI for computer games, such as tetris, racing games and fighting games. However, one of the main limiting factors for RL methods has been their scalability to large environments, where finding good policies requires too many samples, making most RL methods impractical. 1.1 Transfer Learning in RL One of the approaches for solving the scalability problem is to reuse the data from similar RL tasks by transferring data or previously found solutions to the new RL task. These methods have been a focus of the research lately and are called transfer learning methods. The transfer learning methods can be separated into value-based and model-based transfer learning methods, depending on what is being transferred between the RL tasks. In value-based transfer learning the value functions of previously solved RL tasks are transferred to the new task at hand. A popular approach for transferring value functions is to use the previously found value functions as initial solutions for value function of the new RL task. These methods are called startingpoint methods, for example see the temporal-difference learning based approach by Tanaka and Yamamura [4] and a comparative study of these methods by Taylor et al. [5]. For successful transfer, a good mapping of states and actions between the RL tasks is required. When a poor mapping is used the transfer can result in worse performance than doing the standard reinforcement learning without a transfer. Department of Computer Science, Tokyo Institute of Technology, Tokyo, Japan On the other hand, model-based transfer learning methods transfer the transition models and reward models from the solved RL tasks to new RL tasks. Similarly to the value-based transfer, the mapping between states and actions of the learned RL tasks and the target RL task is required. However, the requirements for the mapping are weaker than those in the case of value-based transfer and, thus, the transfer is also possible between less similar tasks. The reason is that the transition model and reward model only depend on a single transition from the current state whereas the value function depends on a sequence of rewards (and thus transitions) starting from the current state. This difference can be seen from an example of transferring knowledge from a previous task where the agent is able to obtain a big positive reward after opening the door and moving around in the room behind the door. However, if the new RL task gives a negative reward after the agent enters the room, the value-based transfer is not useful, probably even worsening the performance. On the other hand, model-based transfer could transfer the knowledge that the opening of the door allows to enter the room and if the agent has already learned that the room contain negative rewards in the new task it can infer the negative value of the actions that open the door and enter that room. In summary, the advantage of model-based transfer over value-based transfer is in cases where actions in different tasks have similar results, e.g., the same action opens the door, but the value of the action is different between the tasks. A model-based transfer method called was proposed by Wilson et al. [6] that successfully estimates the probabilistic prior of tasks. If the model of the new task is similar to previously encountered tasks, the data from the previous tasks can be used to estimate the transition and reward model for the new task. Thus, the new task can be learned with fewer samples. A similar approach has also been applied to partially observable environments [2]. However, these model-based and value-based transfer approaches still require almost full learning of at 120

134 least one initial task. That is previous tasks, which are used in transfer learning of new tasks, should have been learned with sufficient accuracy. If the tasks have large state spaces, then the initial learning will require a huge amount of data, which is not realistic. This kind of setting where the tasks are ordered is called transfer learning. In contrast, multitask learning is a setting where there is no initial task and all tasks are solved simultaneously. Another issue with the above reviewed methods is that the advantage of transferring between large RL tasks is problematic because a good mapping between them is usually not available. 1.2 Proposed Observational Idea To tackle the above mentioned problems we propose a setting where the sharing does not occur between different RL tasks but between different regions (parts) of the same RL task. This is accomplished by allowing the agent to access additional observational data about the regions of state-action space of the RL task. The usefulness of the observational data is that it identifies the regions of the task that participate in the multitask learning. Moreover, the strength of the sharing between different regions depends on the similarity of their observations. The more similar the observations are, the stronger the sharing is. This kind of observational data is often available in practice, e.g., in the form of camera data or sensor measurements. A motivating example for our observational framework is a mobile robot moving around on a ground, where there are two types of ground conditions: slippery and non-slippery. The robot knows its current location and thus, can model the environment using a standard Markov decision formulation, predicting the next location from the current location and the movement action (e.g., forward and backward). However, if the robot has access to additional sensory information about the ground conditions at each state, it could use that additional observation to share the data between similar regions and models of the environment more efficiently even when only a small amount of transition data is available. We call this kind of RL setting Observational RL. In our observational setting there is no order for solving the tasks, meaning that all regions are solved simultaneously, i.e., as a multitask learning setup. Additionally, since the sharing takes place between regions of the whole problem, the mapping is essentially between smaller parts of the problem. Therefore, the problem of finding a good mapping is often mitigated. In our proposed setting, the model-based sharing is more natural than the value-based sharing, as the value of the states often depends on the global location of the region, and thus the value of similar regions is not expected to be same. In the mobile robot example described above, the probabilities of moving forward would be similar in locations with similar ground conditions, but the value of going forward in these locations depends on where the robot makes a transition to after executing the forward action. For this reason, from here we only focus on the model-based multitask learning in the setting of ORL. 1.3 Outline In the next section we formally introduce the setting of ordinary RL. The notions of observations and similarity will be formalized in Section 3. After that we propose two methods for solving the Observational RL problem in Section 4. Their performance is evaluated experimentally in Section 5. Finally, we conclude in Section 6. 2 Ordinary RL The goal of reinforcement learning is to learn optimal actions in unknown and stochastic environment. The environment is specified as a Markov Decision Problem (MDP), which is a state-space-based planning problem defined by S, P I, A, P T, R and γ. Here S denotes the set of states, P I (s) defines the initial state probability, A is the set of actions, and 0 γ < 1 is the discount factor. The state transition function P T (s s, a) defines the conditional probability of the next state s given the current state s and action a. At each step the agent receives rewards defined by function R(s, a, s ) R. The goal of RL is to find a policy π : S A that maximizes the expected discounted sum of future rewards when the transition probabilities P T and/or the reward function R is unknown. The discounted sum of future rewards is t=0 γt r t, where r t is the reward received at step t. In this paper we focus on the case where the transition probabilities are unknown, but the reward function is known, due to space constraints. The extension of the proposed methods to an unknown reward function is straight-forward. 3 Observational RL In this section we formulate the setting of Observational RL (ORL). For better understandability, we first start with a simpler framework that already includes the main idea. Then, later extend it to a more general setting. 3.1 Basic Idea The Observational RL setting extends the ordinary RL setting by allowing the agent to access additional observational information about the state-action space. For the basic case, consider that the agent has observations about each state 1. This means that for each 1 Observations are separated from the state information because they do not necessarily satisfy the Markov property. 121

135 state s S the agent has some observation o O, where O is the set of observations. Thus, formally the observational information can be defined as a function ϕ(s) O mapping each state to its observation. For example, in the case of the mobile robot these observations could be sensor measurements about ground conditions at each location. The general idea of ORL is to use these additional observations for speeding up the learning, thus, requiring fewer samples to find good policies. ORL will be effective if the states that have similar observations have similar transition structure. If the transition structure has nothing in common applying ORL-based methods will not be able to improve the performance. On the other hand, if similar observations imply similar transition structure, then ORL-methods should have strong advantages. The current paper focuses on the model-based RL approach [3], which consists of following two steps: 1. Estimate the transition probabilities P T (s s, a) using transition data. 2. Find an optimal policy for the estimated transition model by using a dynamic programming method, such as value iteration. More specifically, the transition data consists of, possibly non-episodic 2, samples {(s t, a t, s t)} T t=1, where s t and a t correspond to the current state and action of the t-th transition and s t is the the next state. Thus, the idea is to use observational data expressed by ϕ to have more accurate estimates of the transition probabilities P T. To take advantage of observational information we have to require that the agent assumes a common parameterization for the transition models for all states. In other words, transition probabilities for all states are modelled with the same parametric form P T (s s, a; β s ) where β s is the parameter for the transition model for state s. For example, in the case of discrete MDPs, we can use a multinomial parameterization. This common parameterization implicitly defines the mapping between the actions and next states of different states. Thus, it is similar to the mappings used in other transfer learning methods discussed in Section 1.1. Similarly to other transfer learning methods the choice of mapping (in the case of ORL the choice of the parameterization) greatly affects the performance. Use of improper parameterization will negate all advantages of data sharing and could even worsen the performance, depending on what method is used for solving the ORL problem. Next we formalize the ORL framework that extends the described basic idea. 2 Non-episodic means that there is no requirement that the next state of the t-th transition sample (i.e., s t ) has to equal to the starting state of the (t + 1)-th transition sample (i.e., s n+1 ). 3.2 Formulation of ORL In the previous formulation the observations were just connected to single states. It is useful to extend the formulation by connecting the observations to regions (i.e., subsets) of the state-action space S A. Let u denote a region an observation is connected to. We call u an observed region and as it is a subset of stateaction space u S A. Thus, the basic ORL idea described above was just a special case when u S. There are two motivations for this extension. Firstly, it allows us to work with structural problems where one observation is connected to several states, e.g., a manipulation task of various objects by a robotic arm, where an observation is connected to an object, and thus to all states involved in the manipulation of that object. Secondly, this extension means that the observations are now also connected to actions. This allows one to have different observations for different actions and the sharing can depend on actions. For example, in the mobile robot case the movement actions (forward and backward) could participate in the sharing, whereas some other actions, such as picking up an object, could be left out from the sharing. Now the observations function ϕ : U O where U contains all observed regions. If there are N observations then, the observational data is {(u n, o n )} N n=1 where observation o n O corresponds to region u n S A. In this case the set of observed regions is U = {u n } N n=1. Additionally, we require that states can belong at most to a single observed region, this means that u i u j =, for i j. However, there is no requirement that all state-action pairs belong to an observed region. The state-action pairs that do not belong to any observed region do not benefit from the observational information. This extension allows the agent to consider models where all regions of the state-action space are not equipped with observations or certain parts of the state space are different, e.g., there is a maze with corridors and rooms and the agent only has observations about the rooms. Next we propose two methods for solving the ORL problem. 4 Proposed Methods First of the methods is based on the similarity idea and the second one comes from the mixture-ofcomponents multitask learning ideas. 4.1 Similarity-based ORL The idea of similarity-based ORL method is to add data from similar tasks directly to the likelihood function of the models for every observed region. Consider 122

136 the single task estimation of maximum (log) likelihood for observed region u βˆ u = argmax log P T (s s, a; β u ), (1) β u (s,a,s ) D u where D u is a set of transition data from observed region u. A straight-forward extension of the single task estimation (1) is to add data from other tasks and weight them according to the similarity of the other tasks to the current task at hand. This can be expressed by βˆ u = argmax k u (v) log P T (s s, a; β u ), β u v U (s,a,s ) D v (2) where k u (v) [0, 1] is the similarity of the observed region v to observed region u. Thus, data from observed regions that have high similarity k u (v) have a big effect on the estimation of the model of region u. In the case of a mobile robot, consider the estimation of the model for a region of slippery states u (e.g., an icy region). If the similarity function k u assigns high similarity to other regions of slippery states (e.g., other icy regions or wet regions) and a low similarity value for non-slippery states then the similarity-based ORL method will provide an accurate estimate for β u even if region u has few or no samples. A practical option for the similarity function is to just use the Gaussian kernel between the observations of the regions, expressed as k u (v) = exp( ϕ(u) ϕ(v) 2 /σ 2 ), (3) where σ is the width of the kernel. This parameter could be chosen using cross-validation and it controls how much multitask effect distant tasks have on the current task at hand. The only constraint on k u is that it should give value 1 for the region itself, i.e., k u (u) = 1. No other properties are required. Thus, we also allow non-symmetric and non-positive definite similarities. One disadvantage of similarity-based ORL is it suffers from the curse of dimensionality if the observations are high-dimensional. In this case it means that all tasks will become dissimilar to each other due to the high-dimensionality of observations. Therefore, next we will introduce a more sophisticated method that is based on mixture-of-components, which uses a probabilistic framework to model the multitasking problem of ORL and thus could be expected to mitigate the above mentioned problem. 4.2 Component-based ORL In this section we introduce a component-based multitask learning method for learning transition probabilities P T (s s, a) for ORL. Consider again the example of a mobile robot that is moving along a difficult terrain that has obstacles and varying ground conditions. The robot knows its location and speed at each step. That knowledge allows the robot to learn the state transition probabilities for each action. However, if the robot has access to additional observations about the states (using sensors or a camera), then using such observational information could allow the robot to estimate the transition probabilities in fewer samples than by just using robots location and speed. Recall that in ORL the agent has access to observations, i.e., the agent knows function ϕ(u) O. For example, for the mobile robot the set of observations could contain measurements about the ground type (e.g., gravel or tarmac) or visual information about the obstacles around a particular location. As already mentioned, in terms of multitask learning an observed region u U is a task and ϕ(u) specifies its features. Here we introduce the idea of component-based multitask learning where the role of task features is to a priori determine the component the task belongs to. Let there be M components, then P (m ϕ(u)) denotes the probability that the task u with features ϕ(u) belongs to the component m (where m {1,..., M}). Let (s, a) be a state-action pair and u U be such that (s, a) u, then the sharing between elements of U is formulated as a mixture of components for the transition probability: P T (s s, a) = M P T (s s, a, m)p (m ϕ(u)), (4) m=1 where P T (s s, a, m) is the transition probability to state s under component m for state-action pair (s, a) and P (m ϕ(u)) is the component membership probability mentioned above. In the example of a mobile robot, these components would comprise of states that have similar transition dynamics, e.g., one component could be a group of states where a certain moving action fails due to difficult ground conditions and another component represents states where the moving action succeeds. Given the number of components M and data about transitions and observations, we want to find the maximum likelihood estimate for (4). To do that we first need to assume a parametric form for its elements. The parameterized version of (4) is given by P (s s, a, β, α) = M P (s s, a, β m )P (m ϕ(u), α), (5) m=1 where β m is the parameter for the transition model of component m and α is the parameter for component membership probabilities. The estimates of both of these parameters will be determined by maximum likelihood estimation. It should be noted that any parameterization will work as long as its maximum likelihood 123

137 estimation is tractable. The choice of parameterization for P (m ϕ(u), α) depends on the type of observations, O. For discrete observations an option is to use a Naive Bayes model: P (m o, α) = α m,0 K k=1 α m,k,ok, (6) where o is observation, i.e., o = ϕ(u) = (o 1,..., o K ) T. Parameter α m,0 controls the overall probability of component m and α m,k,ok controls the probability of component m for regions whose observation s k-th dimension is equal to o k. Since parameters are multiplied together, the model assumes that each dimension independently affects the component probability. For continuous observations following parameterization can be used: P (m ϕ(u), α) = exp( α m, ϕ(u) ) m=1 exp( α m, ϕ(u) ), (7) where ϕ(u) denotes the observation for u, α m R K, i.e., observations are K-dimensional real values, and, is inner product. This parameterization corresponds to multi-class logistic regression problem. Because of its complicated form, the maximum likelihood estimate for (5) cannot be found using straightforward optimization. A standard approach doing maximum likelihood estimation on such problems is to use an EM-based method [1]. To do that we introduce a latent indicator variable z {0, 1} M, (8) which denotes the true component for u. Thus, only one of the elements of z is equal to one and all others are equal to zero. Using z we can rewrite the mixture (5) as P (s, z s, a, β, α) M = z m P (s s, a, β m )P (m ϕ(u), α) (9) = m=1 M [P (s s, a, β m )P (m ϕ(u), α)] zm, (10) m=1 where the summation form is transformed into a product form, which allows us to easily handle the log likelihood. This latent variable formulation allows us to use the EM algorithm for determining a maximum likelihood solution for β and α. The outline of the EM-method is 1. Start with initial values for parameters β and α. 2. Calculate the posterior probabilities of the latent variables, given the parameters β and α (E-step). 1: KL-divergence of the estimated transition probabilities from the true model, for the slippery grid world experiment with 2-dimensional observations. For each method the mean and standard deviation of its KLdivergence averaged over 50 runs are reported, for different data sizes N = 50, N = 100, N = 150, and N = 200. Bolded values in each column show methods whose performance is better than others using t-test with 0.1% confidence level. Method N = 50 N = 100 Comp(1) ± ± Comp(2) ± ± Comp(3) ± ± Comp(CV) ± ± Sim(fixed) ± ± Sim(CV) ± ± Single task ± ± (a) N = 50 and N = 100 Method N = 150 N = 200 Comp(1) ± ± Comp(2) ± ± Comp(3) ± ± Comp(CV) ± ± Sim(fixed) ± ± Sim(CV) ± ± Single task ± ± (b) N = 150 and N = Find β and α that maximize the expectation of the regularized data likelihood (M-step). 4. If the solution has converged stop, otherwise go to step 2. Due to space restriction we leave out the details of E-step and M-step and only present the conclusions. E-step can be performed analytically by just applying the Bayes law. The M-step for transition models can be performed analytically for discrete and Gaussian models and M-step for observation-based component membership parameter α can be effectively computed by convex optimization based methods. We follow standard approach for implementing the EM method. This includes using several restarts to the EM procedure to avoid local optima and using crossvalidation to choose the number of components (M). 5 Experimental Results In this section we present experimental results from two simulated domains: grid world with slippery ground conditions. 5.1 Slippery Grid World We conducted experiments on a mobile robot task with discrete state and action space. The size of the 124

138 state space of the grid world is and there are 4 movement actions: left, right, up and down. There are two types of states, one type is slippery, where all movement actions fail with probability 0.8, keeping the robot at the same spot and the other type is nonslippery having probability of failure The goal of the agent is to reach the goal state from the initial state. An example of the grid world is shown in Figure 1. The goal of the robot is to reach the goal state denoted with G starting from bottom left state S. White squares are non-slippery and colored squares are slippery states. If the robot moves at the edge squares it receives a negative reward of 1 and is reset to the starting state. The robot receives reward +1 when it reaches goal state, after which it is again reset to the initial position. Other states do not give any reward. The observations about each state are twodimensional real values of sensor measurements. The first dimension shows the measurement of the depth of the water layer covering the ground at that location and the second dimension the amount of loose gravel. Both measurements are noisy and for the experiments are generated randomly from two Gaussian distributions, one for slippery states and another for non-slippery states. The two Gaussians are quite separated, as can be seen from Figure 2. The average performance over 50 runs for the component-based and the similarity-based ORL methods is reported in Table 1. The table reports average KL-divergence values of the estimated transition probabilities from the true transition probabilities. Methods named Comp(n) are component-based methods with n components. Thus, Comp(1) actually just merges all observed regions as a unified task. The results in Table 1 use transition data that is collected uniformly over the state and action space, this allows us to compare the pure performance of different methods without the side effects of non-uniform data collecting policy. Secondly, in this experiment the methods used manually-chosen parameters to show the performance of the methods without the problem of choosing optimal parameters. For component-based methods, Comp(2) and Comp(3), we manually chose the regularization parameter of the logistic regression to be For similarity-based method Sim(fixed) the Gaussian kernel with a fixed width σ = 2.5 was used. The single task method that does not use observations and Comp(1) do not have any extra parameters. Table 1 also reports the performance of methods using cross-validation(cv) for the choice of the parameters. The Comp(CV) is the component-based ORL that uses 5-fold CV to choose the regularization parameter for logistic regression from the set {10 3, 10 1, 10 0 } and the number of components. Similarly, Sim(CV) is the similarity-based ORL that uses 5-fold CV to choose the optimal width for the Gaussian kernel from the set {1.5, 3.0, 4.5, 6.0, 10.0}. Firstly, we can use the performance of the unified 1: Mobile robot in a grid-world with slippery and non-slippery states. Robot starts from an initial state at bottom left denoted with S and has to reach the goal state G. task ( Comp(1) ) as a good comparison point in Table 1, because unifying all tasks is not expected to provide good results when a large number of samples are available. All ORL methods outperform the Single task implying that the use of data sharing in this case is valuable, even with just 50 samples. As expected the component-based method using 2 components Comp(2) is performing the best overall with 100 or more samples. The performance of Comp(3) and Sim(fixed) is slightly worse than Comp(2), but still clearly outperforming the unified task and single task methods, validating their usefulness in this experiment. Also, as seen from Table 1 the cross-validation version of component-based method Comp(CV) is performing almost as well as the best fixed parameter version. Actually in the case N = 50 the CV method is outperforming the fixed methods, because the regularization that was used in the fixed cases (10 3 ) is too small, resulting in poor performance of the EM-based method, if only 50 samples are available. The effect of the regularization of logistic regression is depicted in Figure 3(a) for sample sizes 100 and 200. For both sample sizes if the regularization is not too big the component-based ORL has good performance. Similarly, the Sim(CV) method is very close to the fixed width case and the performance of similaritybased ORL is not very sensitive to the chosen Gaussian widths unless a too small width is chosen as seen from Figure 3(b). These results suggest that CV can be used for tuning the parameters of component-based and similarity-based ORL. Table 2 shows the value of the policies that were found from the transition probabilities learned by different methods. The two ORL methods have similar performance and obtain significantly higher value than unified task ( Comp(1) ) and single task. They 125

139 : Distribution of observations for non-slippery (blue circles) and slippery (green crosses) states. The horizontal axis displays the measured water level and the vertical axis displays the measured amount of loose gravel for each state N=100 N=200 2: Value of the the policy found by using the estimated transition probabilities, for the slippery grid world experiment with 2-dimensional observations. For each method the mean and standard deviation of its value averaged over 50 runs are reported, for different data sizes N = 50, N = 100, N = 150, and N = 200. Bolded values in each column show methods whose performance is better than others using t-test with 0.1% confidence level. Method N = 50 N = 100 Comp(CV) ± ± Sim(CV) ± ± Comp(1) ± ± Single task ± ± (a) N = 50 and N = 100 Method N = 150 N = 200 Comp(CV) ± ± Sim(CV) ± ± Comp(1) ± ± Single task ± ± (b) N = 150 and N = 200 KL error Grid World with High-dimensional Observations Regularization for logistic regression (a) Dependence of component-based ORL on the regularization of logistic regression. KL error Kernel width N=100 N=200 (b) Dependence of similarity-based ORL on Gaussian width. 3: Average KL-divergence from the true distribution in slippery grid world tasks with two-dimensional observations for sample sizes N = 100 and N = 200. The averages and standard deviations were calculated from 50 runs. are quite close to the value of optimal policy, which is in this task. The good performance of Comp(1) is explained be the fact that in their nature the slippery and non-slippery states are similar, because all 4 actions result in similar outcomes, just the probabilities of these outcomes differ. We also tested the grid world example width high dimensional observations. Now the observations were 10- dimensional. The first two dimensions were exactly the same as before, containing useful information about the states as depicted in Figure 2. The new 8 dimensions did not contain any information, i.e., the observations for slippery and non-slippery states were generated from the same distribution, which was a single 8-dimensional Gaussian with zero mean and identity covariance. The results of high-dimensional grid world experiments for component-based and similarity-based ORL methods with CV are shown in Table 3. The sets of model parameters used by CV are the same as in the previous experiment. For comparison the results for Comp(1) and Single task, are also presented in the table and as they do not use observations, we just report again their performance from the previous experiment. Comparing Table 3 to Table 1 shows that the performance of both ORL methods is degraded compared to the problem with low-dimensional observation. As expected, the performance of the similarity-based approach, Sim(CV), has worsened more than the performance of the component-based approach, Comp(CV). The similarity-based approach just slightly outperforms the unified task Comp(1) for sample sizes N = 150 and N = 200. Although component-based ORL also has weaker performance compared to the low-dimensional observation case, it is performing still 126

140 3: KL-divergence of the estimated transition probability from the true model, for the slippery grid world experiment with 10-dimensional observations. For each method the mean and standard deviation of its KL-divergence averaged over 50 runs are reported, for different data sizes N = 50, N = 100, N = 150, and N = 200. Bolded values in each column show methods whose performance is better than others using t-test with 0.1% confidence level. Method N = 50 N = 100 Comp(CV) ± ± Sim(CV) ± ± Comp(1) ± ± Single task ± ± (a) N = 50 and N = 100 Method N = 150 N = 200 Comp(CV) ± ± Sim(CV) ± ± Comp(1) ± ± Single task ± ± (b) N = 150 and N = 200 4: Value of the the policy found by using the estimated transition probabilities, for the slippery grid world experiment with 10-dimensional observations. For each method the mean and standard deviation of its value averaged over 50 runs are reported, for different data sizes N = 50, N = 100, N = 150, and N = 200. Bolded values in each column show methods whose performance is better than others using t-test with 0.1% confidence level. Method N = 50 N = 100 Comp(CV) ± ± Sim(CV) ± ± Comp(1) ± ± Single task ± ± (a) N = 50 and N = 100 Method N = 150 N = 200 Comp(CV) ± ± Sim(CV) ± ± Comp(1) ± ± Single task ± ± (b) N = 150 and N = 200 rather well and clearly outperforms other methods for N = 150 and N = 200. Table 4 shows the value of the policies that were found from the transition probabilities learned by different methods for high-dimensional observations case. As expected, compared to the case of low-dimensional observations (see Table 2) both ORL methods have weaker performance. The component-based method slightly outperforms similarity-based method, significantly only for N = 100. This suggests that although the KL-error of the similarity-based method is much higher than the component-based method, it still captures useful structure in the transition probabilities resulting in almost similar performance in the grid world task. In summary, both ORL methods show good performance in the grid world task and the curse of dimensionality has mild effect on their performance. 6 Conclusions The results of the grid world task show that the proposed ORL framework is suitable in cases useful observations are available about the state-action space. The two proposed method were shown to effectively employ the additional observations to speed up the learning of the transition probabilities. Our next step is to apply the proposed methods to a more challenging task of object lifting by robotic arm where the robot has observations about the objects. Additionally, our future work is to investigate the relationship of ORL to the studies of Bayesian RL and Partially Observable MDP (POMDP). [1] Christopher M. Bishop. Pattern Recognition and Machine Learning. Information Science and Statistics. Springer, [2] Hui Li, Xuejun Liao, and Lawrence Carin. Multitask reinforcement learning in partially observable stochastic environments. The Journal of Machine Learning Research, 10: , [3] Richard S. Sutton and Andrew G. Barto. Reinforcement Learning. MIT Press, Cambridge, MA, USA, [4] Fumihide Tanaka and Masayuki Yamamura. Multitask reinforcement learning on the distribution of MDPs. In Computational Intelligence in Robotics and Automation, 2003, volume 3, pages , July [5] Matthew E. Taylor, Peter Stone, and Yaxin Liu. Value functions for RL-based behavior transfer: A comparative study. In Proceedings of the Twentieth National Conference on Artificial Intelligence, pages , July [6] Aaron Wilson, Alan Fern, Soumya Ray, and Prasad Tadepalli. Multi-task reinforcement learning: a hierarchical Bayesian approach. In ICML 07: Proceedings of the 24th International Conference on Machine learning, pages , New York, NY, USA, ACM. 127

141 2009 Technical Report on Information-Based Induction Sciences 2009 (IBIS2009) Matching between Piecewise Similar Curve Images Kazunori Iwata Akira Hayashi Abstract: Matching between curve images in two dimensions is frequently performed in shape analysis. We concentrate on a specific but meaningful deformation of curve images defined by a piecewise similar relation. We present a curve matching algorithm for dealing with the deformation, together with a way of sampling points from each curve image. Our algorithm is unique in that it considers not only matching between curve images, but also sampling points. Using several experiments, we explain how to implement the algorithm for digital images of line drawings, and show that it is effective even when the number of sample points is relatively small. Keywords: shape analysis, curve matching, line drawing recognition 1 Introduction Matching between curve images in two dimensions is often performed in shape analysis for digital image processing, line drawing interpretation, and character handwriting recognition. A curve image is represented as a set of points. The number of points in the set can be very large, and hence, to reduce the computational cost of processing the image, a curve image is usually re-parameterized as a reduced set of sampled points [1 7]. In this case, curve matching involves finding correspondences between the sampled points of two curve images. Shape analysis relies on these correspondences. A number of curve (or shape) matching algorithms have been proposed [1 9]. The difference between these algorithms lies in their choice of matching cost function (MCF) for matching curve images. The MCF for curve images is used to quantify dissimilarities between the images by exploiting some of their geometric attributes. Almost all the MCFs used in the curve matching algorithms are designed to be somewhat effective for certain kinds of deformations. However, which MCF to select for a particular application of curve matching remains a puzzle, since the MCFs are neither optimal nor have theoretical guarantees for all the kinds of deformations with respect to curve matching. Although the MCFs will be practically meaningful, here we are not concerned with considering certain of Graduate School of Information Sciences, Hiroshima City University, Hiroshima, , Japan. tel , kiwata@hiroshima-cu.ac.jp Graduate School of Information Sciences, Hiroshima City University, Hiroshima, , Japan. tel , akira@hiroshimacu.ac.jp the kinds of deformations concurrently. In this paper, we concentrate on a specific but meaningful deformation defined by a piecewise similar relation. We present a curve matching algorithm with a novel MCF, together with a way of sampling points from each curve image. Unlike most algorithms with sample points, such as [1 6, 10], our algorithm is also unique in that it considers not only matching between curve images, but also sampling points. The algorithm has an asymptotic guarantee for finding correspondences from the sample points of a curve image to those of a piecewise similar deformation thereof. The guarantee will act as a useful guide in judging whether or not the algorithm is appropriate for an application. Using several experiments, we explain concretely how to implement the algorithm for digital images, and show that it is effective even when the number of sample points is relatively small. The organization of this paper is as follows. We introduce the piecewise similar relation, and formulate curve matching in Section 2. We describe our curve matching algorithm in Section 3. The experimental results of the algorithm are shown in Section 4. We conclude with a summary in Section 5. 2 Preliminaries Let Z be the integers and R the real numbers. The nonnegative and positive elements in Z are denoted by Z + 0 and Z +, respectively. For any i, j Z, Z j i denotes the integers from i to j. The nonnegative and positive elements in R are denoted by R + 0 and R+, respectively. ( ) represents the number of elements in a finite set, and denotes the 128

142 norm of a vector in Euclidean space. 2.1 Piecewise Similar Relation We define a curve, curve segment, and piecewise regular curve to introduce a similarity of curve images. Definition 1 (Curve). Let I =[a, b] R be a closed interval, where a<b. A plane curve is a continuous map C I : I R 2, with C I (t) (x I (t),y I (t)). (1) When a time-parameter t I increases from a to b, we obtain the directed trajectory of C I (t), C I (I) { C I (t) t I }, (2) where the ordering of points in the curve image C I (I) preserves that of t in I, that is, for all t, t I where t<t, C I (t) precedes C I (t ) in C I (I). The curve image which is an ordered set of points with respect to t is simply called an image. A plane curve C I that 1. is twice differentiable on (a, b) I, and 2. satisfies dc I (t)/dt 0for all t (a, b), is said to be regular, and its image is called a regular image. Definition 2 (Curve Segment). For any interval [a, b] I, the segment of curve C I with respect to [a, b] is described as a continuous map C I [a, b] :[a, b] R 2. The image of a segment is also called an image. Definition 3 (Piecewise Regular Curve). Let C I be a curve for any I =[a, b]. If there exists a partition of I, such that a = k 0 <k 1 < <k N 1 <k N = b, (3) 1. N is a finite integer, and 2. segment C I [k i,k i+1 ] is regular for all i Z N 1 0, then C I is called a piecewise regular curve and C I (I) is called a piecewise regular image. The total length of a piecewise regular image is calculated as the sum of all the segment image lengths. Definition 4 (Image Set). The set of piecewise regular images with positive length is denoted as S. When an image is uniformly magnified or reduced, the resulting image is similar to the original image in the following sense. Definition 5 (Similarity). Let C I (I) and C J (J) be any images in S. If there exist a map ζ : C I (I) C J (J) and a constant λ R + such that, for all c 1, c 2 C I (I), ζ(c 1 ) ζ(c 2 ) = λ c 1 c 2, (4) then C I (I) and C J (J) are similar images and we write C I (I) C J (J). Similarity plays an important role in human recognition of images, because similar images appear to have the same shape, even though they may differ in scale. For example, a small image of the letter S and a large image thereof are recognized as the same letter. Analogous to similarity is piecewise similarity according to Definition 6. It plays the same role as similarity in human recognition. Definition 6 (Piecewise Similarity). Let C I (I) and C J (J) be images in S, wherei =[a, b] and J =[a,b ].Ifthere exist partitions of I and J, such that a = k 0 <k 1 < <k N 1 <k N = b, (5) a = l 0 <l 1 < <l N 1 <l N = b, (6) 1. N is a finite integer, 2. for all i Z N 1 0, C I [k i,k i+1 ]([k i,k i+1 ]) and C J [l i,l i+1 ]([l i,l i+1 ]) are regular images in S, and 3. for all i Z N 1 0, C I [k i,k i+1 ]([k i,k i+1 ]) C J [l i,l i+1 ]([l i,l i+1 ]), (7) then C I (I) and C J (J) are piecewise similar and we write C I (I) P C J (J). The points C I(k 0 ),...,C I (k N ) are called segment endpoints on C I (I). Example 1 (Piecewise Similarity). On the left in Fig. 1 is an image of the letter S. In the center of the figure, the original image has been deformed by uniformly making the upper part of the letter smaller, while on the right, the image has been further deformed by uniformly making the lower part larger. Accordingly, these images are piecewise similar to each other and can be recognized by humans as representing the same letter S. 129

143 and the components of the i-th element are expressed as (x i,y i ) p i. (13) For all i Z N 1 0, the finite difference at p i is defined as Fig. 1: Piecewise similar deformation of images. Most of the raw data available from a database of shapes, line drawings, and characters are not drawn to scale. If some of the raw data in the same classes have a similar relation, it is relatively easy to make them the same size by preprocessing and then to find the correspondences between them. However, we have rarely seen such data in practice. Most of the data have a piecewise similar relation, since they appear to represent the same shape. In general, it is difficult to find correspondences between them. Accordingly, in this paper, we concentrate on the piecewise similar deformation of images. 2.2 Curve Matching A curve image is often re-parameterized as a reduced set of sampled points. This is described with Definitions 7 and 8. Definition 7 (Sample Points). For any interval I =[a, b] and any N Z +,let γ N (I) { { t 0,t 1,...,t N 1,t N } I N+1 a = t 0 <t 1 < <t N 1 <t N = b}. (8) For any sequence T N = { t 0,...,t N } γ N (I), C I (T N ) { C I (t i ) C I (I) i Z N 0 }, (9) are called the sample points of C I (I). The ordering of the sample points in C I (T N ) preserves the ordering of t i I. Definition 8 (Re-parameterization). We define { C I (T N ) C I (I) N+1 T N γ N (I) }. Γ N (C I (I)) (10) For any sequence T N = { t 0,...,t N } γ N (I), thesample points on the image are simply denoted as P N C I (T N ). (11) For all i Z N 0,thei-th element of P N is denoted by p i C I (t i ), (12) Δp i =(Δx i, Δy i ), (14) (x i+1 x i,y i+1 y i ). (15) For all i Z N 2 0, the second-order finite difference at p i is expressed as Δ 2 p i = ( Δ 2 x i, Δ 2 ) y i, (16) (Δx i+1 Δx i, Δy i+1 Δy i ). (17) For all i Z N 1 0, the unit tangent and unit normal vectors at p i are defined as ( e (1) Δxi P N (p i ) Δp i, ( e (2) P N (p i ) Δy i Δp i, ) Δy i Δp i Δx i Δp i, (18) ), (19) respectively. For all i Z N 2 0, the curvature at p i is defined as κ PN (p i ) Δx iδ 2 y i Δ 2 x i Δy i Δp i 3. (20) For simplicity, an image C I (I) is denoted by C. Also,we sometimes describe the unit vectors using angles. Definition 9 (Angle). Let C be an image in S. For any P N Γ N (C), we define θ PN such that for all i Z0 N 1, the unit tangent and unit normal vectors at p i P N are e (1) P N (p i )=(cosθ PN (p i ), sin θ PN (p i )), (21) e (2) P N (p i )=( sin θ PN (p i ), cos θ PN (p i )). (22) For all i Z N 1 0, the finite difference at p i is defined as Δθ PN (p i ) θ PN (p i+1 ) θ PN (p i ). (23) Now, curve matching is formulated using sample points. Definition 10 (Curve Matching). Let C and C be images in S. We say that C matches C with P N Γ N (C) and Q M Γ M (C ), respectively, if there is a correspondence from each element in P N to an element in Q M. A correspondence is represented by a many-to-one map f : Z N 0 that satisfies the following two conditions: Z M 0 1. f(0) = 0 and f(n) =M, and 130

144 p 0 C p 1 p 2 C p 3 C C p N p N 1 p N 2 q 0 q 1 q 2 q 3 p 0 p 1 p 2 Fig. 2: Matching map f. Fig. 3: Part of an image and sample points. 2. f(i) f(i +1)for all i Z N 1 0, where f(i) =j denotes the correspondence from p i P N to q j Q M. This is called a matching map. Example 2 (Matching Map). Fig. 2 illustrates a matching map. Let C and C denote the upper and lower curve images in Fig. 2, respectively. The points on the curve images represent the sample points on the images. The arrows depict correspondences from the sample points on C to those on C, which are expressed as f(0) = 0, f(1) = 0, f(2) = 2, and f(3) = 3. 3 Curve Matching Algorithm We start with a definition for equipartition sample points. Definition 11 (Equipartition Sample Points). Let C be an image in S. Letp i denote the i-th element of P N Γ N (C). If for all i Z0 N 1, the finite difference at p i satisfies Δp i = r N > 0, (24) then P N is referred to as the equipartition sample points on C. For any N Z +, the set of such sample points on C is simply denoted as Γ N (C) { P N C N+1 Δp i = r N,i Z N 1 } 0. (25) Note that r N depends only on N, and not on i. The following curvature-based measure plays an important role in quantifying the difference between images in terms of piecewise similarity. Definition 12 (Curvature-based Measure). Let C be an image in S. LetC Cdenote a part of the image in S. For any P N Γ N (C), the measure α PN : S R is defined as α PN (C) κ PN (p i ), (26) p i C P N,i Z N 2 0 where κ PN is the curvature defined in (20). Example 3 (Curvature-based Measure). Fig. 3 depicts a part C of an image C and sample points P N according to Definition 12. In this case, because C P N = { p 2,...,p N 2 }, (27) the measure α PN (C) is calculated as α PN (C) =κ PN (p 2 )+ + κ PN (p N 2 ). (28) We introduce a convenient notation to indicate a part of an image together with its sample points. Definition 13 (Image with Sample Points). Let C be an image in S. For any p i and p i in P N Γ N (C), C [p i,p i ] denotes a part of C such that 1. it exists in S, 2. it contains all sample points between p i and p i,but does not include the other elements of P N. We should note that C [p i,p i ] contains a single sample point p i, but its length is positive because it is in S,andthat for all images with the same sample points, the curvaturebased measure gives the same value. Example 4 (Image with Sample Points). The part C of C in Fig. 3 can be expressed as C [p 2,p N 2 ]. Definition 14 (Dissimilarity Measure). Let C and C be images in S. LetC Cand C C denote their parts in S. For any P N Γ N (C) and Q M Γ M (C ), their dissimilarity measure μ PN,Q M : S S R + 0 is defined as ( μ PN,Q M C, C ) ( α PN (C) α QM C ). (29) It is simple to compute the dissimilarity measure, since it requires only the curvatures. The dissimilarity measure is invariant for all translations, reflections, and rotations, since the curvatures in α PN and α QM are invariant for these. Using the dissimilarity measure, we describe an MCF that computes the cost of obtaining correspondences from the sample points of one image to those of another. 131

145 Definition 15 (Matching Cost Function). Let C and C be images in S. For any N, M Z +,letp N Γ N (C) and Q M Γ M (C ) denote the respective sample points. For any matching map f : Z N 0 Z M 0,let N { } N { } I f min i I f (i) i, I f max i I f (i) i, i=0 where for all i Z N 0, i=0 (30) I f (i) { i Z N 0 f(i) =f (i ) }. (31) Since ( ) ( ) ( ) ( ) I f = If holds, let L = If = If. For all n Z L 1 0,leti n and i n be the (n +1)-th smallest elements in I f and I f, respectively. Given a matching map f : Z N 0 Z M 0, the matching cost function (MCF) for C and C under P N and Q M is described as d PN,Q M (C, C f) L 1 ( [ ] μ PN,Q M C pin,p in, C [ ]) q jn 1+1,q jn, (32) n=0 where index j n is defined as 1, if n = 1, j n f ( ) (33) i n, otherwise. The best matching maps are described as f argmin d PN,Q M (C, C f). (34) f Example 5 (Matching Cost Function). Consider the matching map f given in Example 2. According to (31), we have I f (0) = { 0, 1 }, I f (1) = { 0, 1 }, I f (2) = { 2 } and I f (3) = { 3 }. Hence, I f = { 0, 2, 3 } and I f = { 1, 2, 3 }. The matching cost for the matching map f is written as d P3,Q 3 (C, C f) =μ P3,Q 3 (C [p 0,p 1 ], C [q 0,q 0 ]) + μ P3,Q 3 (C [p 2,p 2 ], C [q 1,q 2 ]) + μ P3,Q 3 (C [p 3,p 3 ], C [q 3,q 3 ]). (35) We now describe our algorithm incorporating the MCF. Algorithm 1 (Curve Matching). Perform the steps given below. 1. Extract sample points P N and Q M from images C and C, respectively, such that constraints 1a, 1b, and 1c given below hold: (a) P N = { p 0,...,p N } Γ N (C), (b) Q M = { q 0,...,q M } Γ M (C ), and (c) for any ɛ R +,thereexistn 0 Z + and M 0 Z + such that for all i Z N 1 0,allj Z M 1 0, all N N 0 and all M M 0, 1 Δp i 1 Δq j <ɛ. (36) 2. Using P N and Q M obtained in the previous step, find the best matching maps f that give the minimum cost, using a search algorithm. 3. Express correspondences from P N to Q M, according to f. Although the MCF is simple, it is sufficient for our algorithm to find correspondences between piecewise similar curve images as shown in Theorem 1 and Corollary 1. Theorem 1. Let C and C be images in S. Let C C and C C denote the respective parts which are regular images in S. If 1. C C, 2. sample points P N and Q M are extracted from C and C, respectively, such that they satisfy constraints 1a, 1b, and 1c of the algorithm, and 3. N and M go to infinity such that for all i Z0 N 1, ( ν PN (C) ν QM C ) lim =0, (37) N, M Δp i where p i denotes the i-th point of P N, and ν PN : S R is defined by ν PN (C) Δθ PN (p i ), (38) p i C P N,i Z N 2 0 then lim μ ( P N,Q M C, C ) =0. (39) N, M The proof sketch is given in [11]. This theorem states that there is an asymptotic guarantee for coping with partially similar deformations of images under the constraints, because if two image parts are similar, then their dissimilarity measure tends asymptotically to zero. The dissimilarity measure confirms whether or not images can be similar by verifying the equation in (39). From Theorem 1, we readily obtain an asymptotic guarantee of the algorithm in Corollary 1. Interestingly, the algorithm finds the matching maps that give the minimum cost without knowing the segment endpoints or the scale of piecewise similar images in advance. 132

146 Corollary 1. Let C and C be any images in S. If C P C, then the algorithm finds a matching map for which the matching cost from C to C tends to zero as N and M. Proof. From Theorem 1, because C P C, there exists a matching map f : Z N 0 Z M 0 for which the matching cost described in (32) tends to zero as N and M. Hence, the minimum cost given by f tends to zero as N and M. Recall that P denotes the piecewise similar relation (see Definition 6). Proposition 1 implies that we can simplify constraint 1c of the algorithm when images are digitized. Proposition 1. Let C and C be images in S. Let p i and q j be the i-th and j-th sample points of P N Γ N (C) and Q M Γ M (C ), respectively. If for all i Z N 1 0 and all j Z M 1 0, 1. Δp i 1 and Δq j 1, and 2. for a given ɛ R +, then (36) holds. Δp i Δq j <ɛ, (40) The proof is routine. Note that digital images embedded in the pixel points of ( Z + ) 2 0 always satisfy the first condition of Proposition 1 if the same pixel point is not sampled more than once. In this case, Proposition 1 indicates that minimizing Δp i Δq j is sufficient to minimize 1/ Δp i 1/ Δq j. For the same reason, we do not need to take care the third condition (37) of Theorem 1 in implementation when images are digitized. 4 Experiments In this section, we show experimental results of the algorithm to explain concretely how the algorithm is implemented for digital images. This is because digital images are embedded in the pixel points of ( Z + ) 2, 0 but not R 2 [12]. Line Drawing Images We have implemented the algorithm for digital images of line drawings, examples of which are shown in Figs. 4 and 5. The images in the figures were drawn by hand with a pen on a touch panel 1. Hence, they 1 A drawing software is available at kiwata/panel/. are affected by hand oscillation and are a little distorted. Each example consists of three images of the same class. The center and right images in each example have been drawn so as to be piecewise similar to the left image. In Fig. 4, the center image has been deformed by uniformly reducing the upper part of the left image, while the right image has been further deformed by uniformly magnifying the lower part. The example shown in Fig. 5 is much more complicated in shape. In Fig. 5, the center image has been deformed by uniformly reducing the middle part of the left image, while the right image has been deformed by uniformly magnifying the starting spiral part of the left image. The left image in each example is called the query image. The center and right images, which are deformations of the query image, are called database images. For each of the examples, we use our algorithm to obtain correspondences from the sample points of the query image to those of a database image. In this section, let C be the query image and C a database image. These digital images are expressed as C = { } { } c 0,...,c N 1, C = c 0,...,c, (41) M 1 where c n and c m denote the n-th and m-th elements of C and C in the pixel points, respectively, and N and M denote the number of elements in C and C, respectively. For all n Z N 1 0, the length of a subset of a digital image C is given by where σ 0 (C) =0. σ n (C) = n 1 n =0 c n c n +1, (42) Implementation of Step 1 According to Proposition 1, we replace (36) with (40) in implementing step 1 of the algorithm. This results in the following procedure. For any N N 1, when segmenting a query image C with equipartition sample points P N on C, thei-th equipartition sample point p i of P N is the n i -th point c ni of C such that for all i Z N 0, n i = argmin σ n (C) σ N 1 (C) i N. (43) n Z N 1 0 Thus, we extract N +1equipartition sample points from C. In this case, because of constraint 1c rewritten as (40), the number of equipartition sample points Q M on the other image C is meant to be M = argmin z z Z N σ M 1 (C ) σ + N 1 (C). (44) 133

147 Then, Q M is obtained from C in the same way. Thus, we obtain P N and Q M which approximately satisfy the constraints in step 1 of the algorithm. Implementation of Step 2 In step 2 of the algorithm, there appear to be some matching maps with the same cost, because the curvatures at p N 1 and p N are ignored in (26). This would ordinarily be a problem for relatively small values of N. However, to avoid such a problem, we use instead α PN (C) κ PN (p i ), (45) p i C P N,i Z N 0 in computing the curvature-based measure in (32). Here the curvatures at p N 1 and p N are computed additionally using the pseudo finite differences, Δp N = 1 3 (2Δp N 1 +Δp N 2 ), (46) Δp N+1 = 1 5 (4Δp N 1 +Δp N 2 ). (47) The second-order finite difference in (17) can be defined additionally for i = N 1,N using these finite differences. Results We set N =24in all the examples. This means that there are 25 equipartition sample points on the query image in each example. Recall that the number of equipartition sample points on each database image is determined according to (44) when N is given. The resulting correspondences obtained by the algorithm are shown in Figs. 4 and 5. In cases where there were several best matchings providing the minimum cost, we have shown only one of these. In the figures, an x on the image represents an equipartition sample point. In each example, the sample points on the query image are labeled with successive numbers from 0 to 24. The numbering of sample points on the database images indicates correspondences from sample points with the same numbers on the query image. For example, the sample point labeled 0 in Fig. 4(a) corresponds to the sample points labeled 0 in Figs. 4(b) and 4(c). Unnumbered sample points on a database image have no correspondence from sample points on the query image. The figures confirm that the algorithm consistently provides correct correspondences from the sample points on a query image to those on an almost piecewise similar database image. It is somewhat surprising that correct correspondences are given even for such complicated images as in Fig. 5. The results also suggest that the algorithm performs well even with a relatively small number of sample points (a) S , 3 7 0, (b) S , (c) S 3. Fig. 4: Best matching map from the query image S 1 to the database images S 2 and S (a) G-clef , , (b) G-clef , , (c) G-clef 3. Fig. 5: Best matching map from the query image G-clef 1 to the database images G-clef 2 and G-clef 3. Next, we examine the effect of the constraints of the algorithm. Instead of using equipartition sample points, we employed sample points randomly extracted from the respective images in executing the algorithm. Clearly, such samples points do not adhere to the constraints. The results of the correspondences on the same example of images as Fig.4areshowninFig.6. Inthefigure,anxonanimage denotes a randomly extracted sample point. Comparing Figs. 4 and 6, we confirm that the algorithm failed to find the correct correspondences. It follows that the constraints provide an outstanding method for using sampling points in the algorithm. Thus, it is effective to consider both sampling points and matching. 5 Conclusion We explained that our algorithm gives the best matchings between piecewise similar images without knowing the segment endpoints or the scale of the images in advance. The most important use for the best matchings is as a foundation for shape analysis. It may be necessary to select a few 134

148 (a) S (b) S 2. 0, , , , , 19 (c) S 3. Fig. 6: Best matching map from the query image S 1 to the database images S 2 and S 3 using randomly extracted sample points. of the best matchings according to application dependent properties. For example, in character handwriting recognition, we sometimes need to select matchings by examining the difference between left and right derivatives at each segment endpoint, because not all piecewise similar images represent the same character. However, even in such a case, the algorithm is still effective in retrieving a possible small set of correspondences before embarking on more accurate matching. In this paper, we discussed a piecewise deformation given by a similarity relation. We presented a curve matching algorithm for coping with the deformation of images, together with a way of distributing sample points on the respective images. We confirmed through several experimental results that the algorithm is effective even with a relatively small number of sample points. Acknowledgments This work was supported in part by Grants-in-Aid and for scientific research from the Ministry of Education, Culture, Sports, Science, and Technology, Japan. References [1] L. Younes, Computable elastic distances between shapes, SIAM Journal on Applied Mathematics, vol.58, no.2, pp , [2] M.J.D. Powell, An optimal way of moving a sequence of points onto a curve in two dimensions, Computational Optimization and Applications, vol.13, no.1 3, pp , April [3] T.B. Sebastian, P.N. Klein, and B.B. Kimia, On aligning curves, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.25, no.1, pp , Jan [4] A. Srivastava, S.H. Joshi, W. Mio, and X. Liu, Statistical shape analysis: Clustering, learning, and testing, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.27, no.4, pp , April [5] S. Belongie, G. Mori, and J. Malik, Matching with shape contexts, in Statistics and Analysis of Shapes, pp , Birkhäuser, Boston, [6] S. Manay, D. Cremers, B.W. Hong, A.J. Yezzi Jr., and S. Soatto, Integral invariants for shape matching, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.28, no.10, pp , Oct [7] C. Xu, J. Liu, and X. Tang, 2D shape matching by contour flexibility, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.31, no.1, pp , Jan [8] Y. Wang, K. Woods, and M. McClain, Informationtheoretic matching of two point sets, IEEE Transactions on Image Processing, vol.11, no.8, pp , Aug [9] C. Grigorescu and N. Petkov, Distance sets for shape filters and shape recognition, IEEE Transactions on Image Processing, vol.12, no.10, pp , Oct [10] Y. Gdalyahu and D. Weinshall, Flexible syntactic matching of curves and its application to automatic hierarchical classification of silhouettes, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.21, no.12, pp , Dec [11] K. Iwata and A. Hayashi, Sampling curve images to find similarities among parts of images, Proceedings of the 15th International Conference on Neural Information Processing, LNCS, vol.5506, pp , Springer, July [12] D. Coeurjolly and R. Klette, A comparative evaluation of length estimators of digital curves, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.26, no.2, pp , Feb

149 ¾¼¼ ÌÒÐ ÊÔÓÖØ ÓÒ ÁÒÓÖÑØÓÒ¹ ÁÒÙ¹ ØÓÒ ËÒ ¾¼¼ ÁÁ˾¼¼µ Ð Ö ÓÖ ËÕÙÒØÐ ÈØØÖÒ ÓÒ ÅÓÖØÝ ÎÓØÒ Å ÖÓ ÍÃÍÌÇÅÁ Ý ÃÓ ÇÏÊ Ý Ó Æ Ý Ë ÍÀÁ ØÖØ ÁÒ Ø ÔØÖ Û ÔÖÓÔÓ ÒÓÚÐ ÑØÓ ÓÖ ÖÓÒÞÒ ÕÙÒØÐ ÔØØÖÒ Ò Û ÐÓÐ Ð Ö ÔÖÔÖ ÓÖ ÚÖÝ ÑÔÐÒ ÔÓÒØ Ò ØÒ Ø ÛÓÐ ÔØØÖÒ ÖÓÒÞ ÓÒ ÑÓÖØÝ ÚÓØÒ Ó Ø Ð ÐÐ ÚÒ Ý Ø ÐÓÐ Ð Ö º ÇÒ ÑÔÓÖØÒØ ÑÒ Ñ Ó Ø ÔÖÓÔÓ ÑØÓ ØØ ÔÖ Ó ÑÔÐ ÔÓÒØ Ö ÓÖ ØÓ Ú Ø Ñ Ð ÐÐ ØØ Ø Ð ÐÐ Ö ØÖÑÒ ÒÓØ Ò Ò ÒÔÒÒØ ÑÒÒÖ ÙØ Ò ÔÖØÐÐݹÔÒÒØ ÑÒÒÖº Ì Ð ÐÐ ÒÑÒØ ÔÖÓÐÑ Ò ÓÐÚ ÆÒØÐÝ Ý Ø ÖÔ ÙØ ÐÓÖØÑ ÛØ ÔÓÐÝÒÓÑйÓÖÖ ÓÑÔÙØØÓÒ º Ì ÔÖÓÔÓ ÑØÓ Û ÔÔÐ ØÓ Ò ÓÒÐÒ ÖØÖ ÖÓÒØÓÒ Ø Ò ÓÖÖ ØÓ ÚÐÙØ Ø ÆÒÝ ÕÙÐØØÚÐÝ Ò ÕÙÒØØØÚÐݺ ÃÝÛÓÖ ÕÙÒØÐ ÔØØÖÒ ÖÓÒØÓÒ ÑÓÖØÝ ÚÓØÒ ÓÒÐÒ ÖØÖ ÖÔ ÙØ ½ ½ È ¾ ÀÒ ÅÖÓÚ ÅÓÐÀÅÅ ½Æ Æ µ µ ½ß¼ Øк ¼¾¹¼¾¹ ¹ÑÐ ÙÙØÓÑÙÑÒº ºÝÙ Ù¹ ÙººÔ ÖÙØ ËÓÓÐ Ó ÁÒÓÖÑØÓÒ ËÒ Ò ÐØÖÐ ÒÒÖ¹ Ò ÃÝÙ Ù ÍÒÚÖ ØÝ ÅÓØÓÓ Æ ¹Ù ÙÙÓ¹ ½ß¼ ÂÔÒ Ý ÙÐØÝ Ó ÁÒÓÖÑØÓÒ ËÒ Ò ÐØÖÐ ÒÒÖÒ ÃÝÙ Ù ÍÒÚÖ ØÝ Æ ¾ Æ Æ µ 136

150  µ Æ Ú½ Ú Ú µ ½µ ½ Æ ¾» ¾ ¾ ¾º½ ½ ¾ Æ Æ ½ Æ ½ Ý Ú Ú ½ ¾ µ ½¾ ½¼½ ½ Ú Ú ½Æµ Ú ¾¼ ½ Ú Ú µ ½  µ ½ Æ µ ½ Ú Ú Ý Úµ Ú Úµ Ú Ú µ Ú Ù Ù Úµ  µ Ú ¾º¾ ¾ Ù Ú ÙÚ Ù Ú µ ½µ  µ Æ Ú Ú µ Ú½ ÙÚµ¾ ÙÚ Ù Ú µ ¾µ ¾ Ù Úµ ¾ Ù Ú ÙÚ Ù Ú µ ¼ Ù Ú µ ¼ ÓØÖÛ ¾ Ù Ú ÙÚ Ù Ú µ 137

151 y2 (2nd sample point) y1 (1st sample point) (a) (b) 図 * + &. (c) (d) 多数決型アルゴリズムによる識別境界 * (e) 4 &+ 平滑化項なし * + * + *5 "+ *5 "+ 5 *" * + *5 "+ 4 5 *" *" 5+ 5 * + *5 "+ 4 *" 5+ 4 * + * + * + 図 * + ). 多数決型アルゴリズムによる識別境界 * 4 )+ それぞれ ) 視点から表示 * + 5 * + * + 4 正定数 * + * + 4 の組で異なる正定数 の * + の組について一定値でも良いし 何らかの学習 ルの割り当て方のうち * + が最小となるものを探索 法によりそれぞれの * + について適切な値に定めても する問題となる 総当り探索は簡単であるが の増 よい 加を考えると現実的ではない 特にこの探索処理は事前 各サンプル点に割り当てられるラベルは & つに１つで 通り存 えられる毎に実施する必要があるために この膨大な計 通りのラベ 算量は致命的である 以下 ) 節においては より洗練さ あるため パターン全体としての割り当ては & 在する 式 *&+ の最小化問題は これら & に実施しておけばよいものではなく 入力パターンが与 138

152 ¾º Æ ¾ Æ ¾ ¾ Ú ½ ¾ ¼ ¾ ¾µ Ì ½ µ Ì Ý ½ Ý ¾ µ Ì Ú Ú µ Ú ¼µ Ý Ú ¾µ ¾ Ú ½µ Ý Ú µ ¾ ¾ µ ½ ¾ µ µ¹ µ ½¾ ¼ ½µ ½¾ ½ ¼µ ½ µ  µ µ ½ ¾ Æ Ú Ú µ Æ ¾ Æ ¾ ¾ µ µ µ ¾ µ µ ¾ µ  µ ½ Ú Æ ¾¼ ½ µ µ ¾ ½ ½ µ ½ ½ µ ¾ ¾µ  µ Æ ¾ Æ ¾µ ÙÑÓÙÐÖØÝ ÙÚ ¼ ¼µ ÙÚ ½ ½µ ÙÚ ¼ ½µ ÙÚ ½ ¼µ Æ µ ¾µ ε Î Î Î Æ Ú ¾ Ø Ú Ú Ú Ø ¾µ 139

153 µ µ µ µ ¾µ Úµ Ú Øµ Ù Úµ Ú ½µ Ú ¼µ ÙÚ ¼ ½µ ¾º  µ ¾ ËØ ¾ Ì Î ËÌ Ë Ì Î Ë Ì µ  µ ½¼ ½½ Ú Ú ¼Ø ½ È ¾  µ È ¾ È ¾ ÙÚ Ù Ú µ ¾ Ù Úµ ¾ Ù Úµ ÙÚ ¼ ½µ ÙÚ ½ ¼µ ¾ Ù ¼ Ú ¼ µ ¾ ÙÚ ¼ ½µ ÙÚ ½ ¼µ ÙÚ ¼ ½µ ÙÚ ½ ¼µ ¼ 140

154 º¾ ½º Ù Úµ ¼ ¾º º Ù Úµ Ù Úµ Ù Úµ ½ ¾ º Ù Úµ ½ ¾ ½ ¾ º ÙÚ ¼ ¼µ ÙÚ ½ ½µ ¼ º½ ¾ ØÑ ÐÔÝÒ Ø ¼¹ ¼¼ ¼¼ ¼¹ ¾ Æ Æ ¼¼ µ ¾ µ µ ¾ ½¼ È ¼ ½ ½Ñ ¾ ÙÚ ¼ ½µ ¼ ¾ ¹ ¹ ½ ¹ ¹ º µ µ º ¹ µ ¹ ½¼ 141

155 ½ ± µ µ ¹ ¼¾ ¾ ¹ ½ recog. rate[%] with learned h uv recog. rate[%] with constant h uv v µ ¹ u v µ ¹ u µ µ µ ½¼ µ ½¼ µ ¾ ½º ¾º µ µ ½¼ ¹ µ µ ¹ ½ ¹ ½ µ 142

156 ¾ ½ ¾ ¾ µ ĺ ʺ ÊÒÖ Ò º Àº ÂÙÒ Ò ÒØÖÓÙØÓÒ ØÓ Ò ÅÖÓÚ ÑÓÐ Á ËËÈ Åº ÚÓк ÔÔº ß½ ½º ˺ Í Ò Ãº ÑÑÓØÓ ÖÐÝ ÖÓÒØÓÒ Ó ÕÙÒØÐ ÔØØÖÒ Ý Ð Ö ÓÑÒØÓÒ ÈÖÓº ÁÈÊ Ì̺ ¾¼¼º κ ÃÓÐÑÓÓÖÓÚ Ò Êº ÏØ ÒÖÝ ÙÒ¹ ØÓÒ Ò ÑÒÑÞ Ú ÖÔ ÙØ Á ÌÖÒ º ÓÒ ÈÅÁ ÚÓк¾ ÒÓº¾ ÔÔº½¹½ ¾¼¼º ˺ ÃÙÑÖ Ò Åº ÀÖØ ÖÑÒØÚ Ð ÓÖ ÑÓÐÒ ÔØÐ ÔÒÒ Ò ÒØÙÖРѹ ÈÖÓº ÆÁÈË ¾¼¼ º º ÖÑÖ Ò Äº ÖÝ ËØØ ØÐ ÔÖÓÖ ÓÖ ÆÒØ ÓÑÒØÓÖÐ ÓÔØÑÞØÓÒ Ú ÖÔ ÙØ ÈÖÓº Î ÔÔº ¾ ¹¾ ¾¼¼º º ÓÝÓÚ Å¹Èº ÂÓÐÐÝ ÁÒØÖØÚ ÖÔ ÙØ ÓÖ ÓÔØÑÐ ÓÙÒÖÝ ÖÓÒ ÑÒØØÓÒ Ó ÓØ Ò Æ¹ ÁÑ ÈÖÓº ÁÎ ÚÓк ½ ÔÔº ½¼¹½½¾ ¾¼¼½º º ÒÙÐÓÚ º Ì Ö Îº ØÐ Ú º ÃÓÐÐÖ º ÙÔØ º ÀØÞ Ò º Æ ¹ ÖÑÒØÚ ÐÖÒÒ Ó ÑÖÓÚ ÖÒÓÑ Ð ÓÖ ÑÒØØÓÒ Ó Ò Ø ÈÖÓº ÎÈÊ ÔÔº º½¹½ ¾¼¼º ½¼ ̺ ÓÖÑÒ º Ä Ö ÓÒ Êº ÊÚ Ø ¾ ½º ½½ ÚÓк ¹Á ÒÓº ¾ ÔÔº ¹ ¾¼¼ º ½ ¾¼¼¹ÎÁŹ½¹ ¾µ ¾¼¼º ¾ È ÈÊÅ;¼¼¹½ ¾¼¼º 143

157 2009 Technical Report on Information-Based Induction Sciences 2009 (IBIS2009) On Deterministic Annealing and Partial Optimization of Hyperparameters for Variational Bayes Algorithm Kenji Nagata Kentaro Katahira Kazuo Okanoya Masato Okada Abstract: Variational Bayes (VB) algorithm is widely used as an approximation method of Bayesian learning. In a recent study, the deterministic annealing VB algorithm has been proposed to overcome the local optimal problem. In this study, we propose a new deterministic annealing method and a partial optimization method of hyperparameter for VB algorithm by introducing two types of temperature parameters to variational free energy. We also apply the proposed methods to Gaussian mixture model to show the effectiveness of the proposed methods. Keywords: Variational Bayes algorithm, Determinisitc Annealing, Optimization of Hyperparameter, Gaussian Mixture Model 1 [11] (VB) Expectation Maximization (EM) [1], , nagata@mns.k.u-tokyo.ac.jp, Graduate School of Frontier Sciences, The University of Tokyo, 5-1-5, Kashiwanoha, Kashiwa, Chiba, , Japan ERATO katahira@mns.k.utokyo.ac.jp Japan Science Technology Agency, ERATO, Okanoya Emotional Information Project, Graduate School of Frontier Sciences, The University of Tokyo. ERATO okanoya@brain.riken.jp Japan Science Technology Agency, ERATO, Okanoya Emotional Information Project, RIKEN Brain Science Institute., okada@k.u-tokyo.ac.jp Graduate School of Frontier Sciences, The University of Tokyo, RIKEN Brain Science Institute. VB [10] VB [5][8] Hara LDPC [3] [6] 144

158 2 X n = {x 1,, x n } θ p(x θ) X n θ p(θ X n ) = 1 Z(X n ) p(xn θ)ϕ(θ) (1) p(x n θ) ϕ(θ) Z(X n ) p(x θ) p(x X n ) = p(x θ)p(θ X n )dθ (2) MAP [11] 3 X n Y n = {y 1,, y n } p(x, y θ) p(y n, θ X n ) q(y n, θ) q(y n, θ) = Q(Y n )r(θ) (3) q(y n, θ) p(y n, θ X n ) D(q p) = q(y n q(y n, θ), θ) log p(y n, θ X n dθ (4) ) Y n (4) D(q p) 0 (1) p(y n, θ X n ) log Z(X n ) Y n Y n q(y n, θ) log q(y n, θ)dθ q(y n, θ) log p(x n, Y n, θ)dθ F (q) (5) Y n Y n q(y n, θ) F (q) Q(Y n ) r(θ) Q(Y n ) VB-E r(θ) VB-M (6)(7) Q(Y n ) exp log p(x n, Y n θ) r(θ) (6) r(θ) ϕ(θ) exp log p(x n, Y n θ) Q(Y n ) (7) r(θ) Q(Y n ) r(θ) Q(Y n ) EM EM β p(y n, θ X n, β) {p(x n, Y n θ)ϕ(θ)} β (8) 4 β 1 β 2 p(y n, θ X n, β 1, β 2 ) p(x n, Y n θ) β1 ϕ(θ) β2 (9) 145

159 F β1,β 2 (q) = q(y n, θ) log q(y n, θ)dθ Y n (prior) β 2 β 1 q(y n, θ) log p(x n, Y n θ)dθ β 2 Y n r(θ) log ϕ(θ)dθ (10) 1.0 F β1,β 2 (q) VB-E VB-M β (likelihood) Q(Y n ) exp β 1 log p(x n, Y n θ) r(θ) r(θ) ϕ(θ) β 2 exp β 1 log p(x n, Y n θ) Q(Y n ) β 1 = 1.0β 2 = 1.0 F (q) E M F (q) F β1,β 2 (q) β 1, β 2 β 1 = β 2 = β β 1 β 2 [] 1. β 1 β 2 2. r(θ) 3. VB-EM. VB-E : Q(Y n ) exp β 1 log p(x n, Y n θ) r(θ) VB-M : r(θ) ϕ(θ) β 2 exp β 1 log p(x n, Y n θ) Q(Y n ) 4. β 1 5. β 1 = β VB-EM 1: [VB03] 1.0 F β1,β 2 (q) β 2 F β1,β 2 (q) 8. β 2 < β ( 3-5) β 1 ( 6-8) 146

160 F β1,β 2 (q) β 2 F β1,β 2 (q) β 2 ϕ(θ) ϕ(θ) β 2 β 2 β 1 β x M K a = {a k } K k=1 µ = {µ k} K k=1 S = {S k } K k=1 a Dirichlet : ϕ(a) = Dir(a; φ 0 ) K k=1 a φ 0 1 k, (13) S k µ k : ϕ(µ k S k ) = N (µ k ; ν 0, (ξ 0 S k ) 1 ), (14) S k Wishart : ϕ(s k ) = W(S k ; η 0, B 0 ) S k (η 0 M 1)/2 exp { 1 } 2 Tr(S kb 0) (15) {φ 0, ν 0, ξ 0, η 0, B 0 } β 2 φ 0 = β 2 (φ 0 1) + 1, ν 0 = ν 0 ξ 0 = β 2 ξ 0, η 0 = β 2 (η 0 M 1) + M + 1 B 0 = β 2 B 0 β 1 = 1.0 β 1 VB- M a r(a) Dirichlet : r(a) = Dir(a; {φ k } K k=1), (16) p(x θ) = K k=1 a k N (x; µ k, S 1 k ) (11) S k µ k : N (x; µ, S 1 ) µ S 1 M y p(x, y θ) = K k=1 { ak N (x; µ k, S 1 k )} y k (12) y k x k y k = 1 y k = 0 r(µ k S k ) = N ( µ k ; µ k, (ξ k S k ) 1), (17) S k Wishart : r(s k ) = W(S k ; η k, B k ) (18) φ k = β 1 n k + φ 0, µ k = β1n k x k + ξ 0ν 0, β 1n k + ξ 0 147

161 ξ k = β 1n k + ξ 0, η k = β 1n k + η 0, B k = B 0 + β 1 C k + β1n kξ 0 ( x β 1n k + ξ 0 k ν 0)( x k ν 0) T, n k = n ȳi k x k = 1 n ȳi k x i, n k C k = i=1 i=1 n ȳi k (x i x k )(x i x k ) T i=1 VB-E Q(Y n ) y k i ȳk i ȳi k = yi k Q(Y = exp ( ) β 1 γi k n ) ( ), (19) K j=1 exp β 1 γ j i K γi k = Ψ(φ k ) Ψ 1 2 log B k j=1 φ j M ( ηk + 1 i Ψ 2 i=1 ) M 2ξ k η k 2 Tr ( B 1 k (x i µ k )(x i µ k ) T ) Ψ( ) β 1 β [ VB ] β 1 β 2 β 1 β 2 β i (0) = 0.01 β 1 β 2 β i (t+1) = 2β i(t) 1+β i (t) [2] t β i = 1.0 t = 10 β 1 (t) = 1.0 β 2 β 2 > 1.0 β 2 (t + 1) = 1.25 β 2 (t) 15 2.[ VB ] Katahira β [ VB ] β(0) = 0.01 β(t+1) = 2β(t) 1+β(t) t = 10 β(t) = : 3.[ VB ] µ k k 1 k 2 µ k1 µ k2 5.2 X n 5 M = 2 2 [4][5] n 200 K = 5 ϕ(θ) φ 0 = 1.0, ν 0 = 1 n n x i, ξ 0 = 0.01, i=1 η 0 = M + 1 = 3.0, B 0 = I M I M M M r(θ) ν k B k B k = B 0 n k = n/k φ k = n k + φ 0, ξ k = n k + ξ 0, η k = n k + η 0 148

162 min: min: Frequency Frequency Frequency min: Free Energy Free Energy Free Energy [ VB ] [ VB ] [ VB ] 3: min: min: Frequency Frequency Frequency min: Free Energy Free Energy Free Energy [ VB ] [ VB ] [ VB ] 4: [ VB ] [ VB ] [ VB ] [ VB ] [ VB ] 5.3 The Saccharomyces Cerevisiae Morphological Database (SCMD)[9] [7] 4 n = 4718M = 4 K = 10 ϕ(θ) φ 0 = 1.0, ν 0 = 1 n x i, ξ 0 = 1.0, n i=1 η 0 = M + 1 = 5.0, B 0 = I M r(θ) t = 15 β(t) = [ VB ] [ VB ] 149

163 Frequency min:22317 Frequency min:22317 Frequency min: Free Energy Free Energy Free Energy [ VB ] [ VB ] [ VB ] 5: [ VB ] [ VB ] [ VB ] 6 (1) [VB01] [VB02] (19) β 1 k ȳi k Katahira [5] Sato VB [8] VB (2) 2 5 [ VB ] β 2 [ VB ] [ VB ] [ VB ] VB 2 150

164 The Saccharomyces Cerevisiae Morphological Database (SCMD) SCMD [8] I. Sato et. al., Quantum Annealing for Variational Bayes Inference, in Proc. 25th Conf. on Uncertainty in Artificial Intelligence, [9] The Saccharomyces Cerevisiae Morphological Database (SCMD), [10] K. Watanabe, and S. Watanabe, Stochastic Complexities of Gaussian Mixtures in Variational Bayesian Approximation, The Journal of Machine Learning Research, Vol. 7, pp , [11] S. Watanabe, Algebraic Analysis for Non- Identifiable Learning Machines, Neural Computation, vol. 13, pp , [1] H. Attias, Inferring Parameters and Structure of Latent Variable Models by Variational Bayes, in Proc. 15th Conf. on Uncertainty in Artificial Intelligence, pp.21-30, [2] Z. Ghahramani, G. E. Hinton, Variational Learning for Switching State-Space Models, Neural Computation, vol.12, pp , [3] S. Hara et. al. LDPC Decoding Dynamics from a PCA Viewpoint, Interdisciplinary Information Sciences, Vol. 13, No. 1, pp , [4] EM I [5] K. Katahira, K. Watanabe, and M. Okada, Deterministic Annealing Variant of Variational Bayes Method, Journal of Physics: Conference Series, Vol. 95, , [6] Y. Ogata, A Monte Carlo Method for an Objective Bayesian Proceure, Ann. Inst. Stat. Math., Vol. 42, No. 3, pp , [7] Y. Ohya et.al., High-dimensional and large-scale phenotyping of yeast mutants, Proc. Natl. Acad. Sci. USA, Vol. 102, No. 52, pp ,

165 2009 Technical Report on Information-Based Induction Sciences 2009 (IBIS2009) Radon Image restoration for the Medical Images using Radon Transform Hayaru Shouno Masato Okada Abstract: We propose an image reconstruction algorithm using Bayes inference for the Radon transformed observation data, which is usually used in medical imaging such like CT/PET. Thorough our Bayesian reconstruction method, we introduced several hyperparameters for the prior and the observation process. The quality of the reconstructed image is influenced of the estimation accuracy of those hyper-parameters. Hence, we also propose the inference method of those hyper-parameters using marginal likelihood maximization principle. We show a better reconstruction result rather than that of a conventional method. Keywords: Radon Transform, Bayes Inference, Image Reconstruction, Hyper-parameter Inference 1, (Image Reconstruction from Projection). Computed Tomography (CT) CT X X CT, PET (Positron Emission Tomography) Radon Radon, , tel , shouno@ice.uec.ac.jp, Dept. of Information and Communication Engineering, University of Electro-Communications, Chofu-ga-oka 1-5-1, Chofu, , Japan., , okada@k.u-tokyo.ac.jp, Graduate School of Frontier Science, The University of Tokyo, Kashiwa-no-ha 5-1-5, Kashiwa, , Japan. Bayes Radon Radon X Radon Radon 1 1 (x, y) (s, t) s 1 1 θ (s, t) (x, y) : ( ) ( ) ( ) s cos θ sin θ x =. t sin θ cos θ y 152

166 t ξ(x, y) y s θ τ(s, θ) x θ {τ(s, θ)} Fourier ξ( x, ỹ) FBP Fourier ( (3)) x, ỹ s, θ : σ(x, y) = d xdỹ σ( x, ỹ) e 2πj(x x+yỹ) (6) 1 Radon : s θ ξ(x, y) X CT (x, y) X PET Positron s τ(s, θ) : τ(s, θ) = dt ξ(x, y) = dt ξ (x(s, t), y(s, t)) (1) Radon 2.2 τ(s, θ) Fourier FBP (Filtered Back Projection) FBP σ(x, y) Fourier : σ( x, ỹ) = dxdy σ(x, y) e 2πj(x x+yỹ) (2) σ(x, y) = d xdỹ σ( x, ỹ) e 2πj(x x+yỹ) (3) Radon τ(s, θ) s Fourier, (x, y) (s, t) : τ( s, θ) = ds τ(s, θ)e 2πjs s (4) = ξ( s cos θ, s sin θ). (5) ξ( s cos θ, s sin θ) = π dθ 0 d s s σ( s cos θ, s sin θ) e 2πjs s (7) (7) s Fourier : g(s, θ) = d s s σ( s cos θ, s sin θ) e 2πjs s (8) g(s, θ) = d s s τ( s, θ) e 2πjs s (9) θ τ(s, θ) s Fouier s Fourier (Filtered Image) g(s, θ) : g(s, θ) = du h(u) τ(s + u), (10), h(u) s Fourier g(s, θ) s = x cos θ + y sin θ σ(x, y) = π 0 dθ g(x cos θ + y sin θ, θ) (11) σ(x, y) g(s, θ) FBP(Filtered Back Projection) CT h(u) Ramachandran-Lakshminarayanan s Logan- Shepp [1][2] 2 153

167 Ramachandran-Lakshminarayanan Frequency Domain Logan-Shepp Frequency Domain [3] p(τ σ) exp( H n (τ σ)) (15) Frequency Index Spatial Domain Frequency Index Spatial Domain Planchrel ( p(τ σ) exp 4π 2 γ dθ d s τ s,θ σ s,θ 2 ) (16) Posion Index Posion Index : Ramachandra Logan Shepp [1][2] 2.3 Poisson Radon ξ(x, y) n p (x, y) τ(s, θ) Radon τ(s, θ) = dt (σ(x, y) + n p (x, y)) (12) = dtσ(x, y) + N p (s, θ) (13) τ s,θ = τ( s, θ), σ s,θ = σ( s cos θ, s sin θ) Bayes H pri (σ) H pri (σ) = β dx dy σ(x, y) 2 + 4π 2 h dx dy σ(x, y) 2 (17) (17) 1 MRF 2 (17) p(σ) exp( H pri (σ)) (18) ) = exp ( 4π 2 dθ d s (β s 2 + h) s σ s,θ 2 (19) N p (s, θ) N p (s, θ) = dt n P (x, y) H n (τ σ) [3][5]: H n (τ σ) = 4π 2 γ π 0 dθ ds ( 2 τ(s, θ) dt σ(x, y)). (14) γ N p (s, θ) p(σ τ ) = p(τ σ)p(σ) σ p(τ σ)p(σ) (20) (19) (16) (20) ( π ) p(σ τ ) exp 4π 2 dθ d s σ s,θ F s γ F s 2 τ s,θ π exp ( 4π 2 dθ 0 0 ) ) d s γ (1 γ F s τ s,θ 2,. F s = (β s 2 + h) s + γ (21) 154

168 σ p(τ σ)p(σ) ( 3 ) s, θ ( s k, θ l ) s k = k s, θ l = l θl θ = π/n θ [0, π] N θ s s s Fourier (4) L N s Fourier (4) τ( s k, θ l ) = ds τ(s, θ)e 2πjs s (22) N s 1 k=0 s τ(s k, θ l )e 2πjs k s k (23) 1/2 s [ 1/2 s, 1/2 s ] N s s = 1/N s s (4) Fourier N s 1 τ( s k, θ l ) s τ(s k, θ l ) k=0 1 N s 1 N s k=0 k k 2πj τ k,l e Ns = s τ k,l (24) k k 2πj τ k,l e Ns (25) {{τ k,l }, { τ k,l }} (20) d s Ns 1 k s ( ) γ N s p(σ k,l τ ) = N σ k,l τ k,l, F k 8π 2 (26) θ F k N (x µ, S) x µ, S σ k,l τ k,l, F k σ k,l = σ s k,θ l, τ k,l = τ s k,θ l, F k = F s k 2.4 = (β s 2 k + h) s k + γ σ(x, y) Fourier 3 y~ Δs ~ Δθ x~ ~ ~ (s k,θ l ) : σ(x, y) = π 0 2πj s(x cos θ+y sin θ) dθ d s s σ s,θ e (27) σ(x, y), Fourier { σ s,θ } σ s,θ σ s,θ = σ R π σ s,θ e 4π2 0 dθ R d s F s σ s,θ γ F s τ s,θ 2 σ e 4π2 R π 0 dθ R d s F s σ s,θ γ F s τ s,θ 2 (28) (28) σ k,l = γ F k τ k,l (29) (26) MAP (Maximum A Priori) { σ k,l } FBP g(s, θ) (s k, θ l ) g k,l = g(s k, θ l ) g k,l = 1 N s 1 N s d s s σ s,θl e 2πjs k s k=0 k k 2πj s k σ k,l e Ns (30) g k,l (x, y) σ(x, y) s = x cos θ + y sin θ s s k s < s k +1 k v = s s k s g(s, θ l ) = (1 v) g k,l + v g k +1,l (31) 155

169 3.0 x x 10 4 Histogram of pixels 1 β, h ( (17) ) γ ( (14) ) 1.0 x Logan-Shepp Logan-Shepp CT/PET R (x, y) σ(x, y) σ(x, y) = π 0 dθ g(x cos θ + y sin θ, θ) (32) N θ 1 θ g(x cos θ l + y sin θ l, θ l ) (33) l=0 2.5 β, h, γ ln p(τ β, h, γ) = ln Z post (β, h, γ) ln Z n (γ) ln Z pri (β, h). (34) 3 Z post, Z n, Z pri Z pri (β, h) = σ Z n (γ) = τ Z post (β, h, γ) = σ e H pri(σ β,h) e H n(τ σ,γ) e H pri(σ β,h) H n (τ σ,γ) (35) (36) (37) β h γ : ln Z pri (β, h) = N θ 2 ln Z n (γ) = N θn s 2 N s 1 k=0 ln Z post (β, h, γ) = 4π2 θ N s N θ 2 N s 1 k=0 ln(β s 2 k + h) (38) ln γ (39) N s 1 k=0 ( ) γ 1 τ k,l γ Fk 2 ln F k. (40) 0 (34) β, h, γ βt+1 h t+1 = βt h t γ t+1 γ t + η ln p(τ β, h, γ) (41) 3 Logan-Shepp N x N y L [ L/2, L/2] [ L/2, L/2] L/2, θ l s 1 s = 0 (x, y) s 156

170 FBP Reconstrucon Bayes Reconstrucon Noise S.D. = 0 Noise S.D. = 1.0 Noise S.D. = 2.0 Noise S.D. = FBP Bayes. PSNR [db] 40 Bayes FBP Noise S.D , 1.0, 2.0, f 7 PSNR PSNR N x = N y = N θ = N s = 256 L = 1. 4 Logan-Shepp [0, 6] Radon n p (x, y) n p (x, y) (41) 157

171 β, h γ 5 Ramachandran-Lakshminarayanan ( 2 ) 6 6 FBP FBP Ramachandran-Lakshminarayanan [1] 6 Bayes FBP PSNR (Peak Singal to Noise Ratio) 7 7 PSNR PSNR Bayes PSNR [db] FBP 1.5 PSNR 27.7 [db] FBP MAP σ(x, y) β, h, γ [1] G. N. Ramachandran and A. V. Lakshminarayanan: Three-dimensional reconstruction from radiographs and electron micrographs, Proc. Natl. Acad. Sci., vol.68, pp , [2] L. A. Shepp and B. F. Shepp: The Fourier reconstruction of a head section, IEEE Trans. Nucl. Sci., vol.ns-21, pp.21-43, [3] K. Tanaka and J. Inoue: Maximum likelihood hyperprameter estimation for solvable Markov random field model in image restoration, IEICE Trams. Inf. Syst., vol. E85-D, No.3, [4] K. Tanaka: Statistical-mechanical approach to image processing, J. Phys. A, vol. 35, No.37, pp , [5] K. Tanaka, H. Shouno, M. Okada and D. M. Titterington: Accuracy of the Bethe Approximation for Hyperparameter Estimation in Probabilistic Image Processing, J. Phys. A, vol. 37, No.36, pp , pp , Radon Bayes MRF p(σ) p(τ σ) 158

172 2009 Technical Report on Information-Based Induction Sciences 2009 (IBIS2009) MapReduce MapReduce Particle Filter by Using Cloud Computing Service Tsukasa Ishigaki Kazuyuki Nakamura Yoichi Motomura Abstract: Particle filter is a filtering method for state estimation of probabilistic latent variables. For the correct estimation, a large number of particles are needed, however, the use of a large number of particles is forced the high computational cost. The present paper describes parallel computing implementations of particle filters in the framework of MapReduce. The MapReduce is a platform that enables parallel processing without task managements by user. We propose two MapReduce algorithms for the particle filter and vilify the performance of the algorithms with respect to the number of using particles and the number of CPU by using cloud computing service. Keywords: MapReduce algorithm, particle filter, parallel processing, cloud computing, 1 [1, 2, 3, 4] [5] [6] [7] [8] [9, 10, 11] [12] Monte Carlo FilterBootstrap Filter ConDensation, tel , ishigaki-tsukasa@aist.go.jp, National Institute of Advanced Industrial Science and Technology, Aomi, Koto-ku, Tokyo, , Meiji University, Higashi-Mita, Tama-ku, Kawasaki-shi, Kanagawa, , National Institute of Advanced Industrial Science and Technology, Aomi, Koto-ku, Tokyo, N log(n) Merging Particle Filter[13, 14] Gaussian Particle Filter[15] 159

173 Input data... Partitioned data Master computer [Map step] Mapper 1 Mapper 2 Mapper 3... Mapper M Task control [Shuffle step] [Reduce step] Reducer 1 Reducer 2... Reducer R 1: MapReduce map : (key m,value m ) list(key s,value s ) shuffle : list(key s,value s ) {key r,list(value r )} reduce : {key r,list(value r )} list(value) Output 2: MapReduce key value [16] MapReduce [17] MapReduce Map Reduce Map 2 MapReduce MapReduce MapReduce 2 2 MapReduce 2.1 MapReduce MapReduce MapShuffleReduce 3 3 (key) (value) (key,value) MapReduce Fig.1 MapShuffleReduce Fig.2 MapReduce Mapper Map Map (key m,value m ) list(key s,value s ) Shuffle {key r,list(value r )} Reducer Reduce Reduce Reducer list(value) 2.2 MapReduce MapReduce Hadoop [18]Hadoop Google Google File System MapReduce Java Hadoop Streaming RubyPerl PythonPHPRC++ MapReduce Map Reduce MapReduce WordCount [17] Map 1 Shuffle Reduce Statistical Query Model k-means 160

174 7GB 20 ECU : Amazon Elastic MapReduce EM MapReduce [19] [20] 2.3 MapReduce Hadoop Amazon.com, Inc. Amazon web services MapReduce Amazon Elastic MapReduce Amazon web services Amazon EC2 Amazon S3 MapReduce EC2 S3 Amazon Elastic MapReduce Mapper Reducer S3 MapReduce 3 EC2 MapReduce S3 1 High- CPU Extra Large Instance 64bit t y t x t x t f h v t q(v) w t r(v) x t = f t (x t 1, v t ), (1) y t = h t (x t, w t ) (2) {t 0,, t 1 } t 2 x t2 t 1 {y 1,, y j } y 1:j y 1:t x t p(x t y 1:t ) [] p(x t y 1:t 1 ) = p(x t x t 1 )p(x t 1 y 1:t 1 )dx t 1, (3) [] p(x t y 1:t ) = α 1 p(y t x t )p(x t y 1:t 1 ), (4) α = p(y t x t )p(x t y 1:t 1 )dx t Kalman Filter [21] 161

175 ## initialization step ## Generate initial particles x (i) 0 0 p 0(x), where p 0 is an initial distribution for t = 1 to T do ## Prediction step ## for i = 1 to N do x (i) t t 1 = f(x(i) t 1 t 1, v(i) t ) = likelihood of particle i w (i) t end for W t = N i=1 w(i) t ## Filtering step ## for i = 1 to N do x (i) t t = filterd particles from {x(1) t t 1 x(n) t t 1 } by sampling with replacement in proportion to w (i) t /W t end for end for 4: Kalman Filter Extended Kalman Filter Non-Gaussian Filter[22] p(x t y 1:t ) 1 N p(x t y 1:t 1 ) 1 N N i=1 N i=1 δ(x t x (i) t t ), (5) δ(x t x (i) t t 1 ), (6) (3),(4) 4 N T Do initialization step ## In Map step ## Do partition particles for t = 1 to T do Do prediction step Do filtering step end for list(key s,value s ) (random number, x (i) T T ) Do Shuffle step ## In Reduce step ## list(value) {key r,list(value r )} 5: MapReduce MRPF Do initialization step for t = 1 to T do ## In Map step ## Do partition particles Do prediction step Do filtering step list(key s,value s ) (random number, x (i) t t ) Do Shuffle step ## In Reduce step ## list(value) {key r,list(value r )} particles at next time step list(value) end for 6: MapReduce MRPF [11] 4 MapReduce MapReduce MapReduce MRPF(MapReduce particle filter) 1 MapReduce MRPF MapReduce MRPF MapReduce MapReduce MRPF 2 MRPF MapReduce MRPF 5 MapReduce MRPF MapReduce Map t = 1 T 162

176 Initial particles Partitioned particles... [Map step] Particle filter 1... Particle filter M Shuffle [Reduce step] Integration of particles Output From (key = 1, value = x (i) 1 1) to (key = T, value = x (i) T T) 7: MapReduce MRPF Initial particles Partitioned particles... [Map step] Prediction &filtering 1... Prediction & filtering M Shuffle [Reduce step] Integration of particles Output of each steps (key = ID, value = x (i) t t) Iteration for time evolution 8: MapReduce MRPF Mapper (t, x (i) t t ) t Reduce 1 T x (i) 1 Map Reduce 4.2 MapReduce Mapper t (random number, x (i) t t ) ID Reduce t x (i) t t t + 1 t + 1 MapReduce MapReduce [1, 2, 23] x t = 1 2 x t x t cos(1.2t) + v 1+x 2 t t 1 y t = x2 t 20 + w t x 0 N(0, 5), v t N(0, 1), w t N(0, 10) x t y t t N(0, σ 2 ) σ 2 163

177 1: MapReduce MRPF [] C=1 C=2 C=4 C=8 C=16 10K K M ,2,4,8, MapReduce T = 100MapReduce T = 1 MapReduce 1 T = 1 T Mapper [1, 2, 23] 5.2 MapReduce MRPF MapReduce MRPF T = 100 MapReduce MRPF 1 10K50K1M C=n n Sped up 9: MapReduce MRPF 2: MapReduce MRPF [] C=1 C=2 C=4 C=8 C=16 10K K M MapReduce MRPF T = 1 MapReduce MRPF 2 3 MapReduce MRPF T =

178 6 3: [] 0.1M 0.5M 1M MapReduce MRPF MapReduce MRPF 2 MapReduce MRPF MapReduce MapReduce 7 MapReduce 2 MapReduce MapReduce IT [1] N. J. Gordon, D. J. Salmond and A. F. M. Smith, Novel approach to nonlinear/non- Gaussian Bayesian state estimation, The Proceedings of IEE F, Vol. 140, No. 2, pp , 1993 [2] G. Kitagawa, Monte Carlo filter and smoother for non-gaussian nonlinear state space model, Journal of Computational and Graphical Statistics, Vol. 5, No. 1, pp [3] A. Doucet, J. de Freitas and N. Gordon, Sequential Monte Carlo Methods in Practice, Springer- Verlag, 2001 [4],,, Vol.88, No.12, pp , 2005 [5] M. Isard and A. Blake, CONDENSATION- Conditional Density Propagation for Visual Tracking, International Journal of Computer Vision, Vol29, No.1, pp , 1998 [6] S. Thrun, W. Burgard and D.Fox, Probabilistic Robotics, MIT Press, 2005 [7] K. Nakamura, N. Hirose, B.H. Choi and T. Higuchi, Particle Filtering in Data Assimilation and its Application to Boundary Condition of Tsunami Simulation Model, Data Assimilation for Atmospheric, Oceanic and Hydrologic Applications, pp , S.K. Park and L. Xu (ed.), Springer, 2009 [8] S. Nakano and T. Higuchi, Estimation of a longterm variation of a magnetic- storm index using the merging particle filter, IEICE Trans. on Information and Systems, Vol.E92-D, No.7, pp , 2009 [9] M. Nagasaki, R. Yamaguchi, R. Yoshida, S. Imoto, A. Doi, Y. Tamada, H. Matsuno, S. 165

179 Miyano and T. Higuchi, Genomic Data Assimilation for Estimating Hybrid Functional Petri Net from Time-Course Gene Expression Data, Genome Informatics, Vol.17, No.1, pp.46-61, 2006 [10] R. Yoshida, M. Nagasaki, R. Yamaguchi, S. Imoto, S. Miyano and T. Higuchi, Bayesian learning of biological pathways on genomic data assimilation, Bioinformatics, Vol.24, No.22, pp , 2008 [11] K. Nakamura, R. Yoshida, M. Nagasaki, S. Miyano and T. Higuchi, Parameter Estimation of In Silico Biological Pathways with Particle Filtering Towards a Petascale Computing, The Proceedings of 14th Pacific Symposium on Biocomputing, pp , [12],,, Vol.38, No.1, pp.1-19, 2008 [20] C. Ji, C. Vecchiola and R. Buyya, MRPGA: An Extension of MapReduce for Parallelizing Genetic Algorithms, 4th IEEE International Conference on escience, pp , 2008 [21] R. E. Kalman, A new approach to linear filtering and prediction problems, Transaction of ASME - Journal of Basic Engineering, Vol.82 pp , 1960 [22] G. Kitagawa, Non-Gaussian State-Space Modeling of Nonstationary Time Series, Journal of the American Statistical Association, Vol.82, No.400 pp , 1987 [23],,, Vol.53, No.2, pp , 2005 [13] Merging Particle Filter Vol.56 No.2pp [14] S. Nakano, G. Ueno and T. Higuchi, Merging particle filter for sequential data assimilation, Nonlinear Processes in Geophysics, Vol.14, No.4, pp , 2007 [15] J.H. Kotecha and P.M. Djuric, Gaussian particle filtering, IEEE Transactions on Signal Processing, Vol.51, No.10, pp , 2003 [16] BP,, BP, 2009 [17] J. Dean and S. Ghemawat, MapReduce: Simplified Data Processing on Large Clusters, OSDI 04: Sixth Symposium on Operating System Design and Implementation, 2004 [18] D. Borthaku, The Hadoop Distributed File System: Architecture and Design, Retrieved from lucene.apache.org/hadoop, [19] C-T. Chu, S.K. Kim, Y-A. Lin, Y.Y. Yu, G. Bradski, A.Y. Ng and K. Olukotun, Map-Reduce for Machine Learning on Multicore, Advances in Neural Information Processing Systems 19,

180 2009 Technical Report on Information-Based Induction Sciences 2009 (IBIS2009) Virutual Concept Drift RBFNN Model selection for RBFNN under Virtual Concept Drift Environments Abstract: In this research, a model-selection criterion for RBFNN under virtual concept drift environments is proposed. Under such environments, the prior distribution of learning samples is changing over time so that online learning tasks usually cause catastrophic forgetting. Such environments are parts of covariate shift. First of all, a statistical model of such environments is constructed. Then, we applied the learning strategies under covariate-shift using the statistical model. The method also provides the model selection criterion. Moreover, several strategies for reducing the computational complexity are also discussed. Keywords: Virutal Concept Drift, RBFNN, t, 1 P (x, y) = P (y x)p (x) (x b, y b ) (b = 1, 2, ) x y P (y x) Radial Basis Function Neural Network(RBFNN) online (i.i.d) x P (x) online Catastrophic forgetting Virtual Concept Drift [1] [2 9]. one-pass-learning, , tel , yamauchi@cs.chubu.ac.jp, Chubu University Department of Information Science, 1200, Matsumoto-cho, Kasugai-shi, Aichi, , JAPAN (e.g [10] [11]). (Covariate Shift) [12,13] RBFNN [14]

181 6 7 2 (.1)Radial Basis Function Neural Network(RBFNN) ( ) [2,3,5 7,9] f θ (x) Append new instances (xnew, F (xnew )) (x 1, F (x 1 )) (x 2, F (x 2 )) Buffer Rehearsal x RBFNN 1: RBF RBF RBFNN f θ (x) RBFNN f θ (x) = ( ) M w j exp x u j 2 2vj 2, (1) j=1 M (Kernel) E = (F (x) f θ (x)) 2 q(x)dx, (2) F (x) q(x) q(x) P (x) 3 E = b W (x t ) { y t f θ (x t ) } 2 (3) (x t, y t ) W (x) ( ) λ q(x) W (x) = (4) P (x) P (x) q(x) 0 λ 1 q(x) q(x) P (x) 3.1 q(x) N q(x) q(x) ( ˆq(x) ) ˆq(x) ˆq(x) = P (x S)P (S x 1, x 2, x N )ds, (5) S q(x) ˆq(x) N 1 Student s-t [15] ˆq(x) = Γ [(N 1 + p)/2] ((N 1)π) p/2 Γ [(N 1)/2] Σ 1/2 [ 1 + (x u)t Σ 1 (x u) N 1 ] (N 1+p)/2 (6), p = dim(x), u = E[x] Σ P (x) N ML ˆP (x) P (x) ˆP (x) N (Σ, u), Σ = 1 N N b=1 (x b u)(x b u) T u = 1 N N b=1 x b, ˆq(x)/ ˆP (x) ˆq(x) ˆP (x) = ( 2 ) p/2 Γ[(N 1 + p)/2] N 1 Γ[(N 1)/2] 168

182 [ ] 1 + (x u)t Σ 1 (N 1+p)/2 (x u) N 1 exp ( 1 2 (x u)t Σ 1 (x u) ). (7) Student s-t N N ˆq(x)/ ˆP (x) 1 P (x) q(x) Student s t i.i.d () 3.2 q(x) ( S i (i = 1, 2, ) ) ( 2 ) S i q(x S 3 ) p 33 p 13 S 3 space of x p 11 S 1 p 32 2: ( ) S i p 23 p 21 p 12 p22 ˆq(x) ˆq(x) i S2 q(x S i )p(s i ) (8) 1 ˆq(x) 1 S i ˆP (x) ˆP (x) i P (x S i )p(s i ) (9) ˆq(x) ˆP (x) = i q(x S i)p(s i ) i P (x S i)p(s i ) (10) P (x S i ) q(x S i ) P (x S i ) Student t ˆq(x) P (x) q(x S j)p(s j ) P (x S j )p(s j ) = q(x S j) P (x S j ) (11) j = arg max P (x S i ) (12) i p(x S i ) Expectation and Maximization (EM) [16] () AIC [17] q(x S i ) p(x S i ) Student-t W (x) (11) W (x) = { ( ) p/2 2 Γ[(N i + p 1)/2] N i 1 Γ[(N i 1)/2] [1 + (x u i) T Σ 1 (x u i i ) N i 1 exp ( 1 2 (x u i) T Σ 1 i (x u i ) ) ] (Ni +p 1)/2 λ,(13) 1 i = arg max (14) j (2π) p/2 Σ j 1/2 ( exp (x u ) j) T Σ 1 j (x u j ). 2 N i, Σ i, u i i, W (x) W (x) W (x) 169

183 4 3.2 W i (x) RBFNN RBFNN (3) Weighted Least Squre (WLS) Moody [18] k k-means Shimodaira IC w RBFNN Moody [18] fuzzy k-means WLS fuzzy k-means, fuzzy k-means [19] u j 1 u (n+1) j := N b=1 W (x b )x b exp( x b u (n) j 2 /c 2 ) ĉ w j exp( x b u (n) j 2 /c 2 ), (15) ĉ w = N b=1 W (x b) c B k, x j, fuzzy k-means Hidden Unit σ 2 j = κ min j j u j u j 2, (16) κ ( 1) [20] WLS Eq(3) w ML = (w 1, w 2,, w M ) T w i i w ML = (ΦW T Φ) 1 Φ T WF, (17) F (F = (F (x 1 ), F (x 2 ),, F (x N ))) T ) W W bb = W (x b ) (b = 1, 2,, N) Φ design matrix Φ bj = exp( x b u j 2 /(2σj 2 )) Eq.(3) W (x) 4.2 λ λ M Shimodaira(2000) [10] IC w regression IC w [10] IC w := + 2 N b=1 { ˆq(x b ) ˆε 2 b ˆP (x b ) ˆσ 2 + log(2πˆσ2 ) { N b=1 ˆq(x b ) ˆP (x b ) } ˆε 2 b ˆσ 2 ĥb + W (x b) 2ĉ w (18) ( ) ˆε 2 2 } b ˆσ 2 1, ˆε b x b ˆσ 2 = N b=1 W (x b)ˆε 2 b /ĉ w ĉ w = N b=1 W (x b) (b = 1, 2,, N) hat matrix ĥb ĥ = Φ(Φ T W T Φ) 1 Φ T W. (19) IC w (λ, M ) λ M IC w 5 P (x) RBFNN λ (AIC/ IC w ) V RBFNN 1 AIC IC w 170

184 V 6 (recording phase) (rehearsal phase) 1 recording, rehearsal phase recording phase WRBFNN, org-rbfnn org-rbfnn [2 7, 9] org-rbfnn λ = 0 WRBFNN W (x) W (x) 1 (x, y) = (x, 1.5) x 1 2 N ( 20, 2) + 1 N (20, 2) (20) 2 50 Eq. (13) N (10, 5) W(x) (x,y) : W (x) x RBFNN 6.2 λ M (λ, M ) WRBFNN M org- RBFNN UCImachine learning repository cpu-performance 2 5 -Quick Mixture of Gaussian AIC org AIC Quick recording-, rehearsal-phase 1 recording, rehearsal-phase recording phase B λ (x, y) = (MSE W RBF NN, MSE org RBF NN ) MSE 1 N total (F [x b ] f θ (x b )) 2, (21) N total b=1 y = x WRBFNN org-rbfnn W (x) 10 4 (x, y) = (MSE W RBF NN, MSE org RBF NN ) λ M λ = 0 WRBFNN org-rbfnn 2 [14] [14] 171

185 y = x λ > 0 y = x cpu-performance y = x AIC-Quick AIC-org 7 RBFNN () RBFNN RBFNN [14] 3.2 Virtual Concept Drift Virtual Concept Drift Virtual Concept Drift [1] A. Tsymbal. The problem of concept drift: definitions and related work. Technical Report TCD-CS , Department of Computer Science, Trinity College Dublin, [2] Takao Yoneda, Masashi Yamanaka, and Yukinori Kakazu. Study on optimization of grinding conditions using neural networks a method of additional learning. Journal of the Japan Society of Precision Engineering/Seimitsu kogakukaishi, 58(10): , October [3] Hiroshi Yamakawa, Daiki Masumoto, Takashi Kimoto, and Shigemi Nagata. Active data selection and subsequent revision for sequential learning with neural networks. World congress of neural networks (WCNN 94), 3: , [4] Stefan Schaal and Christopher G. Atkeson. Constructive incremental learning from only local information. Neural Computation, 10(8): , November [5] Koichiro Yamauchi, Nobuhiko Yamaguchi, and Naohiro Ishii. Incremental learning methods with retrieving interfered patterns. IEEE transactions on neural networks, 10(6): , November [6] Robert M. French. Pseudo-recurrent connectionist networks: An approach to the sensitivity stability dilemma. Connection Science, 9(4): , [7] Bernard Ans and Stephane Roussert. Neural networks with a self-refreshing memory: knowledge transfer in sequential learning tasks without catastrophic forgetting. Connection Science, 12(1):1 19, [8] Nikola Kasabov. Evolving fuzzy neural networks for supervised/unsupervised online knowledge-based learning. IEEE Transactions on Systems, Man, and Cybernetics, 31(6): , December [9] Seiichi Ozawa, Soon Lee Toh, Shigeo Abe, Shaoning Pang, and Nikola Kasabov. Incremental learning of feature space and classifier for face recognition. Neural Networks, 18: , [10] Shimodaira Hidetoshi. Improving predictive inference under covariate shift by weighting the log-likelihood function. Journal of Statistical Planning and Inference, 90(2): , [11] Masashi Sugiyama, Shinichi Nakajima, Hisashi Kashima, Paul von Bunau, and Motoaki Kawanabe. Direct importance estimation with model selection and its application to covariate shift adaptation. In Twenty-First Annual Conference on Neural 172

186 AIC-quick: AIC-org: y=x : cpu-performance (AIC-org AIC-quick ) Information Processing Systems (NIPS2007), December [12] Koichiro Yamauchi. Covariate shift and incremental learning. In Advances in Neuro-Information Processing 15th International Conference, ICONIP 2008, Auckland, New Zealand, November 25-28, 2008, Revised Selected Papers, Part I, pages , November [13].. Technical Report NC ,, [14] Koichiro Yamauchi. Optimal incremental learning under covariate shift. Memetic Computing, page Accepted, [15].., [16] A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximum likelihood from incomplete data via the em algorithm. Journal of the Royal Statistical Society, B 39(1):1 38, [17] Hirotugu Akaike. A new look at the statistical model identification. IEEE Transactions on Automatic Control, AC-19(6): , December [18] J. Moody and C. J. Darken. Fast learning in neural networks of locally-tuned processing units. Neural Computation, 1: , [19] J.C. Bezdek. A convergence theorem for the fuzzy isodata clustering algorithms. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2:1 8, [20] John Platt. A resource allocating network for function interpolation. Neural Computation, 3(2): ,

187 2009 Technical Report on Information-Based Induction Sciences 2009 (IBIS2009) HTML Utilizing Similarities of HTML Structures in Splog Detection by Machine Learning Taichi Katayama Takayuki Yoshinaka Takehito Utsuro Yasuhide Kawada Tomohiro Fukuhara Abstract: Spam blogs or splogs are blogs hosting spam posts, created using machine generated or hijacked content for the sole purpose of hosting advertisements or raising the PageRank of target sites. Among those splogs, this paper focuses on detecting a group of splogs which are estimated to be created by an identical spammer. We especially show that similarities of html structures among those splogs created by an identical spammer contribute to improving the performance of splog detection. In measuring similarities of html structures, we extract a list of blocks (minimum unit of content) from the DOM tree of a html file. We show that the html files of splogs estimated to be created by an identical spammer tend to have similar DOM trees and this tendency is quite effective in splog detection. Keywords: spam blog, machine learning, HTML structure, confidence measure, SVM 1, , {katayamataichi2008, nlp.iit.tsukuba.ac.jp Graduate School of Systems and Information Engineering,University of Tsukuba, Tsukuba, , Japan, , cdl.im.dendai.ac.jp Graduate School of Engineering, Tokyo Denki University, Tokyo, , Japan (), , Navix Co., Ltd., Nishi-Gotanda, Shinagawa-Ku Tokyo , Japan, , race.u-tokyo.ac.jp Research into Artifacts, Center for Engineering, University of Tokyo Kashiwa, Chiba , Japan Technorati 1 BlogPulse 2 kizasi.jp 3 blogwatcher 4 [1] Globe of Blogs 5 Best Blogs in Asia Directory 6 Blogwise 7 () [2,3,4,5,6] () 4 ()

188 1: / 1 (a) / CC SS (b) CC SS ID [4] 88% 75%[3, 7] [5] TREC 8 Blog06 / [4, 6] BlogPulse [8, 9, 10, 4] HTML Support Vector Machines [11] (SVM) SVM SVM [12] HTML SVM 8 1: HTML DOM DOM 2 / / / [13, 14] / / [13, 14] ID ID 1 CC ID=1 SS ID= 2, 3, 4 [13, 14] 2 HTML 175

189 3 HTML 3.1 HTML DOM [15] HTML DOM 1 HTML s HTML HTML P DIV P DIV P DIV [15] BODY P DIV BODY HTML [15] SCRIPT STYLE HTML HTML s DOM dm(s) 3.2 DOM HTML s t DOM dm(s) dm(t) DP DP 1 2 DP edit distance (dm(s),dm(t)) st DOM Rdiff(s, t) Rdiff(s, t) = edit distance (dm(s),dm(t)) dm(s) + dm(t) 1 HTML DOM 3.3 DOM HTML S T HTML s S t T DOM DOM HTML s S HTML T t T Rdiff(s, t) 10 AvMinDF 10 (s, T ) 9 AvMinDF 10 (s, T ) = T Rdiff(s, t T ) 10 t Rdiff(s, t) (ID=1, CC ) (ID=2, 3, 4, SS ) DOM 2 ID (ID=1) S T S s AvMinDF 10 (s, T ) (ID=1, CC ) ID=1 S CC T S s AvMinDF 10 (s, T ) SC SS 2(a) CC S T S s AvMinDF 10 (s, T ) 2(b)(c)(d) SS ( 2(b)(c)(d) ) 2(a),(c),(d) 2(a) AvMinDF 10 (s, T ) AvMinDF k (s, T ) k =1,...25 k =10 176

190 (a) (ID = 1CC ) (b) (ID = 2SS ) (c) (ID = 3SS ) (d) (ID = 4SS ) 2: DOM 2(b) ID=2, AvMinDF 10 (s, T ) ID HTML DOM 4 SVM 4.1 DOM 3 HTML DOM s 3.3 AvMinDF 10 (s, T ) log AvMinDF 10 (s, T ) T 1. T DOM () 2. T DOM ( ) 4.2 [16, 17] / URL / HTML URL URL i) HTML URL ii) HTML 2 URL 177

191 URL u URL log u u u [18, 13, 14] / 10 / w w φ 2. w w freq(, freq(, w )=a w )=b freq( freq(, w) =c φ 2 (, w) =, w) =d (ad bc) 2 (a + b)(a + c)(b + d)(c + d) log ( ) φ 2 (, w) w w URL / URL URL () w s AncfB(w, s) AncfW (w, s) s w Ancf B(w, s) = URL 10 ( sourceforge.jp/) ipadic s w AncfW (w, s) = URL AncfB(w, s) 2 s URL w URL t t URL log w ( s ) Ancf B(w, s) AncfB(w, t) AncfW (w, s) 2 s URL w URL t t URL log ( ) AncfW (w, s) AncfW (w, t) w s SVM SVM TinySVM (

192 5.2 SVM [12] 11 LBD p LBD n (a) CC SS / ( 3) / ( 4) /CC 408 SS 552 / CC SS / / LBD p LBD p LBD p LBD n LBD n 11 ( [19, 12, 20] ) LBD n DOM () DOM () +DOM () DOM 3 4 (a-1)(a-2) 2 DOM () URL DOM () 3 4 (b-1)(b-2) 2 DOM () URL URL DOM () 3 (a-1) (b-1) (a-2) (b-2) DOM () (a-2) DOM () 4 DOM () DOM () DOM ()+DOM () DOM ()+DOM () 4/5 +DOM ()+DOM () 179

193 (a-1) (CC ) (a-2) (CC ) (b-1) (SS ) (b-2) (SS ) 3: / (a-1) (CC ) (a-2) (CC ) (b-1) (SS ) (b-2) (SS ) 4: / 180

194 DOM () 2 DOM () DOM ( ) 7 HTML SVM HTML SVM DOM [21] HTML DOM DOM [1] T. Nanno, T. Fujiki, Y. Suzuki, and M. Okumura. Automatically collecting, monitoring, and mining Japanese weblogs. In WWW Alt. 04: Proc. 13th WWW Conf. Alternate Track Papers & Posters, pp , [2] Z. Gyöngyi and H. Garcia-Molina. Web spam taxonomy. In Proc. 1st AIRWeb, pp , [3] Wikipedia, Spam blog. blog. [4] P. Kolari, A. Joshi, and T. Finin. Characterizing the splogosphere. In Proc. 3rd Ann. Workshop on the Weblogging Ecosystem: Aggregation, Analysis and Dynamics, [5] C. Macdonald and I. Ounis. The TREC Blogs06 collection : Creating and analysing a blog test collection. Technical Report TR , University of Glasgow, Department of Computing Science, [6] P. Kolari, T. Finin, and A. Joshi. Spam in blogs and social media. In Tutorial at ICWSM, [7] Y.-R. Lin, H. Sundaram, Y. Chi, J. Tatemura, and B. L. Tseng. Splog detection using self-similarity analysis on blog temporal dynamics. In Proc. 3rd AIRWeb, pp. 1 8, [8].. Letters, Vol. 6, No. 4, pp , [9].. Web (WebDB Forum)2008., [10] P. Kolari, T. Finin, and A. Joshi. SVMs for the Blogosphere: Blog identification and Splog detection. In Proc AAAI Spring Symp. Computational Approaches to Analyzing Weblogs, pp , [11] V. N. Vapnik. Statistical Learning Theory. Wiley- Interscience, [12] S. Tong and D. Koller. Support vector machine active learning with applications to text classification. In Proc. 17th ICML, pp , [13],,,,,,.. DEWS2008, [14] Y. Sato, T. Utsuro, T. Fukuhara, Y. Kawada, Y. Murakami, H. Nakagawa, and N. Kando. Analysing features of Japanese splogs and characteristics of keywords. In Proc. 4th AIRWeb, pp , [15],.., Vol. 8, No. 1, pp , [16],,,,,.. DEIM, [17] T. Katayama, Y. Sato, T. Utsuro, T. Yoshinaka, Y. Kawada, and T. Fukuhara. An empirical study on selective sampling in active learning for splog detection. In Proc. 5th AIRWeb, pp , April [18] Y.M. Wang, M. Ma, Y. Niu, and H. Chen. Spam double-funnel: Connecting web spammers with advertisers,. In Proc. 16th WWW, pp , [19] D. D. Lewis and W. A. Gale. A sequential algorithm for training text classifiers. In Proc. 17th SIGIR, pp. 3 12, [20] G. Schohn and D. Cohn. Less is more: Active learning with support vector machines. In Proc. 17th ICML, pp , [21] J. Suzuki, H. Isozaki, and E. Maeda. Convolution kernels with feature selection for natural language processing. In Proc. 42nd ACL, pp ,

195 2009 Technical Report on Information-Based Induction Sciences 2009 (IBIS2009) Loopy belief Propagation Yusuke Watanabe Kenji Fukumizu Abstract: In this paper, we show a new formula which connects the Hessian of the Bethe free energy and the multivariable graph zeta function. Utilizing this formula, we show new methods for analyses of Loopy Belief Propagation (LBP) algorithm. We mainly prove and discuss the formula in the case of binary pairwise model. First, we give a sufficient condition that the Hessian of the Bethe free energy is positive definite, which shows nonconvexity for graphs with multiple cycles. The formula clarifies the relation between the local stability of a fixed point of LBP and local minima of the Bethe free energy. We also propose a new approach to the uniqueness of LBP fixed point, and show various conditions of uniqueness. Finally, we discuss the extension to more general class of graphical models including multinomial models and Gaussian models. Keywords: Loopy Belief Propagation, Bethe free energy, graph zeta function 1 Pearl Belief Propagation, BP [1] [2] (Loopy Belief Propagation, LBP) LBP BP LBP [3] LBP, , tel , watay@ism.ac.jp,fukumizu@ism.ac.jp Institute of Statistical Mathematics, 4-6-7, Minami-Azabu, Minato-Ku, Tokyo, LBP LBP LBP Heskes [4] LBP Heskes LBP LBP 1 attractive LBP 182

196 2 LBP G = (V, E) V E N M x i {±1} x = (x i ) i V : p(x) = 1 ψ ij (x i, x j ) ψ i (x i ), (1) Z ij E i V Z ψ ij ψ i ψ i (x i ) = exp(h i x i ), ψ ij (x i, x j ) = exp(j ij x i x j ) p i (x i ) := x\{x i } p(x) p ij (x i, x j ) := x\{x i x j } p(x) Pearl BP [1] BP LBP) LBP µ i j (x j ) µ new i j(x j ) x i ψ ji (x j, x i )ψ i (x i ) k N i \j µ k i (x i ), (2) N i i V x i =±1 µ i j(x j ) = 1 {µ i j (x j)} p i (x i ), p ij (x i, x j ) b i(x i) ψ i(x i) k N i µ k i(x i), b ij(x i, x j) ψ ij(x i, x j)ψ i(x i)ψ j(x j) µ k i(x i ) µ k j(x j ), (3) k N i \j k N j \i x i,x j b ij (x i, x j ) = 1, x i b i (x i ) = 1 (2) (3), x j b ij (x i, x j ) = b i (x i ) b ij (x i, x j ) > 0 (1) p(x) = argminˆp F Gibbs (ˆp) (x i ) i V F Gibbs (ˆp) F Gibbs (ˆp) = KL(ˆp p) log Z ˆp KL(ˆp p) = ˆp log(ˆp/p) ˆp p x i,x j b ij (x i, x j ) = 1, x j b ij (x i, x j ) = b i (x i ), b ij (x i, x j ) > 0 {b i (x i ), b ij (x i, x j )} {b i (x i ), b ij (x i, x j )} pseudomarginal d i := N i F (b) := b ij (x i, x j ) log ψ ij (x i, x j ) ij E x ix j b i (x i ) log ψ i (x i ) i V x i + b ij (x i, x j ) log b ij (x i, x j ) x ix j ij E + i V (1 d i ) x i b i (x i ) log b i (x i ). (4) F pseudomarginal {b i (x i ), b ij (x i, x j )} LBP [3] LBP m i = E bi [x i ], χ ij = E bij [x i x j ] pseudomarginals : b ij (x i, x j ) = 1 4 (1 + m ix i + m j x j + χ ij x i x j ), b i (x i ) = 1 2 (1 + m i). (5) F { L(G) := {m i, χ ij } R N+M ; 1 + m i x i + m j x j + χ ij x i x j > 0 ij E } x i, x j = ±1. F 2 F {m i, χ ij } N + M L(G) 2 F J ij h i 183

197 3 3.1 G E E = 2M e E o(e) V e t(e) V e e E ē [e] = [ē] E G (e 1,..., e k ) t(e i ) = o(e i+1 ), e i ē i+1 (i = 1,..., k 1), t(e k ) = o(e 1 ), e k ē 1 c = (e 1,..., e k ) m c m = (e 1,..., e k, e 1,..., e k,......, e 1,..., e k ) c multiple multiple P u = (u e ) e E [5] ζ G (u) := p P(1 g(p)) 1. p = (e 1,..., e k ), g(p) := u e1 u ek u e C 1. G ζ G (u) = 1 N C N (e 1, e 2,..., e N ) (ē N, ē N 1,..., ē 1 ) ζ CN (u) = (1 N l=1 u e l ) 1 (1 N l=1 u ē l ) 1 C 2M C( E) C( E) M 1 e ē o(e) = t(e ), M e,e := 0 (6) 1 ([5], Theorem 3). ζ G (u) = det(i UM) 1, (7) U U e,e := u e δ e,e 3 2 (). V C(V ) C(V ) ( ˆDf)(i) ( := e E t(e)=i (Âf)(i) := e E t(e)=i u e uē 1 u e uē u e ) f(i), f(o(e)). (8) 1 u e uē f C(V ) det(i UM) = det(i + ˆD Â) (1 u e uē). (9) [e] E. O, T, ι : (Of)(e) := f(o(e)), (T g)(i) := g(e), (ιg)(e) := g(ē), e E,t(e)=i f C(V ), g C( E). M = OT ι ( ) det(i UM) = det I T (I + Uι) 1 UO det(i + Uι) n m, m n A, B det(i n AB) = det(i m BA) ι I + Uι (e, ē) [ ] 1 u e uē 1 det(i + Uι) = [e] E (1 u euē) T (I + Uι) 1 UO = f C(V ), ( T (I + Uι) 1 UOf = e E,t(e)=i = e E,t(e)=i = e E,t(e)=i ) (i) ( (I + Uι) 1 UOf 1 1 u e uē 1 1 u e uē = (Âf)(i) ( ˆDf)(i) Â ˆD ) (e) ( ) (UOf)(e) u e (UOf)(ē) ( ) u e f(o(e)) u e uēf(o(ē)) 184

198 e E u e = u [6] ζ G (u) 2 ζ G (u) 1 = (1 u 2 ) M det(i+ u2 1 u 2 D u A) (10) 1 u2 (10) D A (Df)(i) := d i f(i), 3.2 (Af)(i) := e E,t(e)=i f(o(e)), f C(V ). 3 (). L(G) : det(i UM) = det( 2 F ) i V ij E x i,x j =±1 b ij (x i, x j ) b i (x i ) 1 di 2 2N+4M (11) x i =±1 b ij, b i (5) u i j := χ ij m i m j 1 m 2 j (12). (E,E)- (V,E)- (E,V)- X det X = 1 [ Y 0 X T ( 2 F )X = ( 0 2 F χ ij χ kl ) 1 + (χ ik m i m k ) 2 1 m 2 k N i i (1 m 2 i )(1 m2 i m2 k +2m im k χ ik χ 2 ik ) (Y ) i,j = : i = j, χ ik m i m k : A i,j 1 m 2 i m2 j +2m im j χ ij χ 2 ij u j i = χ ij m i m j I 1 m 2 N + ˆD Â = i Y W Â, ˆD (8) W W i,j := δ i,j (1 m 2 i ) det(i UM) = det(y ) i V = (11) ] (1 m 2 i ) (1 u e uē) [e] E 2 3 det(i UM) UM LBP LBP 4 LBP Pakzad [7] Heskes [8] X, Spec(X) C X ρ(x) 4. M (6) {m i, χ ij } L(G) U (12) Spec(UM) C \ R 1 2 F {m i, χ ij }. t [0, 1] m i (t) := m i, χ ij := tχ ij + (1 t)m i m j {m i (t), χ ij (t)} L(G), {m i (1), χ ij (1)} = {m i, χ ij } U(t) 2 F (t) {m i (t), χ ij (t)} U(t) = tu Spec(UM) C \ R 1 det(i tum) 0 t [0, 1] 3 det( 2 F (t)) 0 2 F (0) 2 F (t) t 2 F (1) u i j u j i χ ij m i m j β i j = β j i := {(1 m 2 i )(1 m2 j )}1/2 = Cov bij [x i, x j ] {Var bi [x i ]Var bj [x j ]} 1/2 (13) u i j u j i = β i j β j i β i j = β j i β i j β ij β ij < 1 Z, B (Z) e,e := δ e,e (1 m 2 t(e) )1/2, (B) e,e := δ e,e β e BM = Z UMZ 1 Spec(UM) = Spec(BM) 185

199 pseudomarginals 1. α M L α 1(G) := {{m i, χ ij } L(G); β e < α 1 e E} 2 F L α 1(G). β e < α 1 ρ(bm) < ρ(α 1 M) = 1 ([9] Theorem ) Spec(BM) R 1 = ϕ α 1 1 ζ G (u) = 1G ζ CN (u) = (1 u N ) 2 α 1 1 β e < 1 L α 1(G) = L(G) F L(G) [8] [9] min i V d i 1 α max i V d i 1 2. t < 1 {m i (t) := 0, χ ij (t) := t} L(G) lim t 1 det( 2 F (t))(1 t) M+N 1 = 2 M N+1 (M N)κ(G), κ(g) M N 1F L(G). [10] u 1 F L(G) 5 LBP LBP Heskes [4] LBP µ i j (x j ) η i j LBP η = (η e ) e E C( E), (2) T LBP {η C( E); T (η ) = η } η T T [11] η Spec(T (η )) {λ C; λ < 1} LBP (damp) T ϵ := (1 ϵ)t + ϵi 0 ϵ < 1 I Spec(T (η )) {λ C; Reλ < 1} LBP T (η ) Furtlehner 5 ([12], Proposition 4.5). u i j LBP η (3), (5), (12) T (η ) UM P UM = P T (η )P 1 3 det(i T (η )) = det(i UM) 3 4 LBP Spec(T (η )) C\R 1 {λ C; Reλ < 1} C\R 1 Heskes [4] Spec(T (η )) C \ R 1 {λ C; Reλ < 1} LBP attractive J ij 0 6. attractive {ψ ij (t), ψ i (t)} t ψ ij (t) = exp(t 1 J ij x i x j ), ψ i (t) = exp(t 1 h i x i ) t LBP 186

200 t t = t 0 t = t 0. attractive u i j [11] h i = 0 m i = 0 6 LBP LBP LBP minmax [8] [13, 14]Gibbs [15] 3 7. q ( F ) 1 (0) det 2 F (q) 0 sgn ( det 2 F (q) ) = 1, q: F (q)=0 x > 0 sgn(x) = 1x < 0 sgn(x) = q F ( F ) 1 (0) LBP LBP q n L(G) F (q n ) L(G) L(G) R N+M F 7 1 LBP LBP LBP 1: LBP β ij (13) β ij tanh( J ij ), sgn(β ij ) = sgn(j ij ). (3) θ i, θ j b ij (x i, x j ) exp(j ij x i x j + θ i x i + θ j x j ) θ i = 0 θ j = [14] [14] 3 ([14]). ρ(j M) < 1 LBP J J e,e = tanh( J e )δ e,e. β ij tanh( J ij ) ρ(bm) ρ(j M) < 1 det(i BM) = det(i UM) > 0 LBP 3 4 {J ij, h i } {J ij, h i } (s i) {±1} V J ij = J ijs i s j h i = h is i x i x i s i LBP 4. G ( M N + 1 = 2) attractive LBP 2. V := {1, 2, 3, 4}, E := {12, 13, 14, 23, 34} {h i } { J 12, J 13, J 14, J 23, J 34 } J ij β 13, β 23, β 14, β 34 < 1 1 < β 12 0 det(i 187

201 2: 2 3: Ĝ x α α F i V = (x i ) i α E α x i E i ϕ α (x α ), ϕ i (x i ) r α, r i ϕ α i α ϕ i ϕ α (x α ) = (ϕ i1 (x i1 ),..., ϕ ik (x ik ), ˆϕ α (x α )) α = {i 1,..., i k } 4: BM) > 0 G Ĝ ( 3) det(i BM) = det(i ˆB ˆM) ˆβ e1 = β 12 β 23, ˆβe2 = β 13, ˆβ e3 = β 34 1 < ˆβ e1 0 0 ˆβ e2, ˆβ e3 < 1 det(i ˆB ˆM) = (1 ˆβ e1 ˆβe2 ˆβ e1 ˆβe3 ˆβ e2 ˆβe3 2 ˆβ e1 ˆβe2 ˆβe3 )(1 ˆβ e1 ˆβe2 ˆβ e1 ˆβe3 ˆβ e2 ˆβe3 + 2 ˆβ e1 ˆβe2 ˆβe3 ) > 0 Ĝ 4 attractive LBP [11, 14] LBP J ij G = (V F, E) e E o(e) t(e) G (e 1, e 2,..., e k ) i Z/kZ t(e i ) o(e i+1 ), t(e i ) t(e i+1 ) G {r i } i V, r j r i {u α i j } i,j α ζ G (u) := 1 [C] P det(i u o(e k) t(e k 1 ) t(e k ) uo(e 1) t(e k ) t(e 1 ) ) C = (e 1, e 2,..., e k ) 1 p(x α ) E α x p(x α\i α) E i LBP LBP η i = E bi [ϕ i ] η α = E bα [ϕ α ] (η α ) i = η i α i LBP 8 (). {η i, η α } ζ G(u) 1 = det( 2 F ) det(var bα [ϕ α ]) bi [ϕ i ]) α F i Vdet(Var 1 d i, u α i j := Var b j [ϕ j ] 1 Cov bα [ϕ j, ϕ i ] r j r i fractional belief propagation [16] 8 Bartholdi [17] 8 8 Generalized Belief Propagation (GBP) Expectation Propagation GBP LBP LDPC Koetter [18] pseudo-codewords, Johnson [19] 188

202 [20, 21] LBP Acknowledgements (20-993) ( ) References [1] J. Pearl. Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. Morgan Kaufmann Publishers, [2] K. Murphy, Y. Weiss, and M.I. Jordan. Loopy belief propagation for approximate inference: An empirical study. UAI, 15: , [3] J.S. Yedidia, W.T. Freeman, and Y. Weiss. Generalized belief propagation. NIPS, 13:689 95, [4] T. Heskes. Stable fixed points of loopy belief propagation are minima of the Bethe free energy. NIPS, pages , [5] H.M. Stark and A.A. Terras. Zeta functions of finite graphs and coverings. Adv. in Math., 121(1): , [6] Y. Ihara. On discrete subgroups of the two by two projective linear group over p-adic fields. Journal of the Mathematical Society of Japan, 18(3): , [7] P. Pakzad and V. Anantharam. Belief propagation and statistical physics. CISS, [8] T. Heskes. On the uniqueness of loopy belief propagation fixed points. Neural Comput., 16(11): , [9] R.A. Horn and C.R. Johnson. Matrix analysis. Cambridge University Press, [10] K. Hashimoto. On zeta and L-functions of finite graphs. Internat. J. Math, 1(4): , [11] JM Mooij and HJ Kappen. On the properties of the Bethe approximation and loopy belief propagation on binary networks. J. Stat. Mech: Theor. Exp., (11):P11012, [12] C. Furtlehner, J.M. Lasgouttes, and A. De La Fortelle. Belief propagation and Bethe approximation for traffic prediction. INRIA RR-6144, [13] A.T. Ihler, JW Fisher, and A.S. Willsky. Loopy belief propagation: Convergence and effects of message errors. JMLR, 6(1): , [14] J. M. Mooij and H. J. Kappen. Sufficient Conditions for Convergence of the Sum-Product Algorithm. IEEE Trans. on Inf. Th., 53(12): , [15] S. Tatikonda and M.I. Jordan. Loopy belief propagation and Gibbs measures. UAI, 18: , [16] W. Wiegerinck and T. Heskes. Fractional belief propagation. Advances in Neural Information Processing Systems, pages , [17] L. Bartholdi. Counting Paths in Graphs. Enseign. Math., 45:83 131, [18] R. Koetter, W.C.W. Li, PO Vontobel, and JL Walker. Pseudo-codewords of cycle codes via zeta functions. IEEE Information Theory Workshop, [19] J.K. Johnson, V.Y. Chernyak, and M. Chertkov. Orbit-Product Representation and Correction of Gaussian Belief Propagation. ICML, [20] S. Ikeda, T. Tanaka, and S. Amari. Stochastic reasoning, free energy, and information geometry. Neural Computation, 16(9): , [21] S. Ikeda, T. Tanaka, and S. Amari. Information geometry of turbo and low-density parity-check codes. IEEE Transactions on Information Theory, 50(6): ,

203 2009 Technical Report on Information-Based Induction Sciences 2009 (IBIS2009) Submodularity Cuts and Applications Yoshinobu Kawahara Jeff A. Bilmes Kiyohito Nagano Koji Tsuda Abstract: Several key problems in machine learning can be formulated as submodular set function maximization. We present herein a novel algorithm for maximizing a submodular set function under a cardinality constraint the algorithm is based on a cutting-plane method and is implemented as an iterative small-scale binary-integer linear programming procedure. It is well known that this problem is NP-hard, and the approximation factor achieved by the greedy algorithm is the theoretical limit for polynomial time. As for (nonpolynomial time) exact algorithms that perform reasonably in practice, there has been very little in the literature although the problem is quite important for many applications. Our algorithm is guaranteed to find the exact solution in finite iterations, and it converges fast in practice due to the efficiency of the cutting-plane mechanism. Moreover, we also provide a method that produces successively decreasing upper-bounds of the optimal solution, while our algorithm provides successively increasing lower-bounds. Thus, the accuracy of the current solution can be estimated at any point, and the algorithm can be stopped early once a desired degree of tolerance is met. We evaluate our algorithm on sensor placement and feature selection applications showing good performance. Keywords: 1 [7, 11, 22] f max f(s) s.t. S k (1) V = {1, 2,..., n} k( n) (1) V S, T V f(s)+f(t ) f(s T )+f(s T ) f [2, NP 4] (1 1/e)( 0.63) [18, 3], , tel , kawahara@ar.sanken.osaka-u.ac.jp [7, 1, 11, 22], , nagano@is.titech.ac.jp, , koji.tsuda@aist.go.jp Dept. of Electriacal Engineering, Washington Univ., Seattle, WA USA, bilmes@u.washington.edu 190 S V

204 [16, 14] Nemhauser&Wolsey [16] [5, 19] C(θ) = 1 N i=1 N L(f θ(x i ), y i ) + Ω(θ) (1) f x y 2 BILP L Ω f f(x) = w T x + b Ω(w) = λ n i=1 α i w i q q = 1 lasso N 1 min α i=1 N L(f(x i), y i ) s.t. p i=1 α i ω i q γ λ γ Nemhauser&Wolsey L(f(x), y) = (y f(x)) 2 [16] 2 (V ar(y) 1 N (y f(x))2 )/V ar(y) ɛ > 0 [1] q = 0 ɛ (1) ɛ- l0 BILP (1) MIP NP l0 CPLEX l Krause ɛ [9, 11] 2 (1) [13] [7] 1 [10] [10]

205 1: Lovász (discrete) f : 2 V R S V f is submodular Eq.(2) (continuous) = ˆf : R n R Eq.(3) Thm.1 I S R n ˆf is convex H - H+ H y 1 d 1 v d2 P c* y 2 H* [20] 1: H Lovász ˆf f [12, 15] ˆf(I S ) = f(s) (S V ) (3) [21, 17, 8] (1) f : 2 V R {S V : S k} Lovász ˆf : R n R D 0 (1) D 0 = {x R n : 0 x i 1 (i = 1,, n), n i=1 x i k}, 3.1 Lovász Lovász (1) max { ˆf(x) : x D 0 } (1) P D 0 Lovász P = {x R n : A T j x b j, j = 1,..., m}a j Lovász b j 1 p R n p 3.2 m ˆp 1 > ˆp 2 > > ˆp m f : 2 V R Lovász ˆf : R n R. 3 (1) Lovász ˆf(p) = m 1 k=1 (ˆp k ˆp k+1 )f(u k ) + ˆp m f(u m ), (2) [8] U k = {i V : p i ˆp k } ˆf ˆf f 2 g : R n R P R n ˆf [12, 15] P 1 f : 2 V R Lovász P D 0 f ˆf γ e i i I S := i S e i {0, 1} n P V ). 4 I S S S S(P ) I S P V S(P ) P 3 Lovász V (P ) I S P 4 V = 6 S = {1, 3, 4} I S = P S(D (1, 0, 1, 1, 0, 0) 0 ) V (D 0 ) 192

206 1 Compute a subset S 0 s.t. S 0 k, and set a lower max{f(s ) : S S(P )} max{ ˆf(x) bound γ 0 = f(s 0 ). : x P }. (4) 2 Set P 0 D 0, stop false, i 1 and S = S 0. 3 while stop=false do ˆf 4 Construct with respect to S i 1, P i 1 and γ i 1 a P S S( P ) submodularity cut H i. f( S) 5 if S(P i 1) = S(P i 1 H ) i then max{ ˆf(x) 6 stop true (S is an optimal solution and : x P } γ i 1 the optimal value). 7 else (4) 8 Update γ i (using S i and other available (1) information) and set S s.t. f(s ) = γ i. Lovász 9 Compute S i S(P i), and set P i P i 1 H+ i and i i : ( ) γ P D 0 4 P D 0 γ v f(s ) = γ S V S S(P ) v = I s P H (6) 3 (γ ) g : R n R x R n γ t > 0 f(s ) γ for all S S(P H ). y R n d R n \ {0} x g γ θ R { } y = x + θd with θ = sup{t : g(x + td) γ} (5) S(P ) = S(P H ) γ = max{f(s) : S k} Lovász S(P ) = S(P H ) x R n γ- 4.2 v S(P H ) v V (P ) S S(P ) v = I S v / S(P H + ) P d 1,..., d n v K = K(v; d 1,..., d n ) K = {v + t 1 d t n d n : t l 0} l = 1,..., n ˆf d γ l v γ- P P H + y l = v + θ l d l 1 2 d 1,..., d n P K θ l > d l y l (l = 1,, n) (1) H = H(y 1,, y n ) 1 H = {x : e T Y 1 x = 1 + e T Y 1 v}. (6) e = (1,..., 1) T R n Y = ((y 1 v,, (y n v)) H H = {x : e T Y 1 x 1 + e T Y v} H + = {x : e T Y 1 x S(P ) γ S(P H ) S(P ) > S(P H + ). (7) 4 1 BILP e T Y v} 2 v 5 H kawahara/software.html Matlab 193

207 4.1 S 0 S(P 0 ) S 1 S(P 1 ) S opt-1 S(P opt-1 ) S opt S(P opt ) H 0 P 1 =P 0 H 0 H H opt-1 P opt =P opt-1 H opt-1 2: 1 Compute a subset S 0 s.t. S 0 k, and set a lower bound γ 0 = f(s 0 ). P D 0 2 Set P 0 D 0, stop false, i 1 and S = S 0. 3 while stop=false do 4 d 1,..., d n Construct with respect to S i 1, P i 1 and γ i 1 a 1 4 ). 6 submodularity cut H i. 5 Solve the BILP problem (9) with respect to A j S < k d 1,..., d n e l (l S) and b j (j = 1,, n k ), and let the optimal e l (l V \ S) S = k solution and value S i and c, respectively. 6 if c 1 + e T Y 1 v i 1 then S 7 stop true (S is an optimal solution and γ i 1 the optimal value). 8 else S (i,j) := (S \ {i}) {j} (i S, j V \ S) 9 Update γ i (using S i and other available information) and set S s.t. f(s ) = γ i. S (i,j) S V \S 10 Set P i P i 1 H + and i i + 1. I S(i,j) I S = e j e i S 2: BILP f(s (i,j) ) S (i,j ) k S k (n k) S i S (i,j) (i S, j V \ S) O(nk) f(s (i,j )) > γ S(P ) = S(P H ) S(P ) γ f(s (i,j )) S S (i,j ) P R n P γ f(s (i,j )) H H {d 1,..., d n } H = H H H v S(P ) = S(P H ) e j e il if l {1,..., k} LP d l = e jl e j if l {k + 1,..., n 1} (8) (6) e j if l = n. 7 2 c d 1,..., d n max Y 1 x : A x {0,1} n{et j x b j, j = 1,, m k }. (9) 6 (8) d 1,..., d n c 1 + e T Y 1 v S(P ) H K D 0 = {x R n : 0 x l 1 (l = 1,, n), c > 1 + e T Y 1 v (9) n l=1 x l k} x S(P \ H ) K(I S ; d 1,..., d n ) = {I S + t 1 d t n d n : t l 0} γ- (1 ) 5 ɛ- [10] ). 4.2 S(P ) = S(P H ) 6 P K ɛ- 194

208 Time (log-scale) [s] k = 5 Dimensionality (n) Time (log-scale) [s] k = 8 Dimensionality (n) 3: Nemhauser 4: & Wolsey [16] ɛ- (log-scale) Function value Time (log-scale) [s] θ = (5) Nemhauser&Wolsey θ = 10 6 [16] f : 2 V R (1) 6.1 [16] max η s.t. η f(s) + j V \S ρ j(s)y j (S V ), j V y j = k, y j {0, 1} (j V ) (10) ρ j (S) := f(s {j}) f(s) K- [16] MIP f(s) = m i=1 max j S c ij, Nemhauser&Wolsey C = c ij m n V = {1,, n} MIP n( m m = n + 1 ) C k = 5, 8 2 [16] ɛ- 3 n k = 5, 8 3 S i C MIP n = 45 k = 8 2 ɛ ɛ [16] 6.3 (k = 5, n = 45) Matlab Parallel CPLEX ver (8 ) 2.5GHz 64-bit (1) [16] (2) 2 n 1 n 2 (1) 195

209 Time (log-scale) [s] Cardinality ( k ) Improvement [%] Cardinality ( k ) 5: () (%)() 6.2 [10] [1] A. Das and D. Kempe, Algorithms for subset selection in NIMS linear regression, In R. E. Ladner and C. Dwork, editors, Proc. of the 40th Ann.. ACM Symp. on Theory of Computing (STOC 08), pages 45 54, V [6, 9] V = 86). [2] J. Edmonds, Submodular functions, matroids, and certain polyhedra, In R. Guy, H. Hanani, N. Sauer and [9] S V J. Shönheim, editors, Combinatorial Structures and Their Applications, pages 69 87, Gordon and Breach, f(s) = V ar( ) V ar(s) = 1 s n F s(s) 7 ɛ > 0 ɛ- [3] U. Feige, A threshold of ln n for approximating set cover, Journal of the ACM, 45: , [4] S. Fujishige, Submodular Functions and Optimization, Elsevier, second edition, F s (S) = σs 2 σs S 2 σ2 s S [5] I. Guyon and A. Elisseeff, An introduction to variable and S V s V feature selection, Journal of Machine Learning Research, 3: , ɛ = 0.05, 0.1, 0.2 [6] T. C. Harmon, R. F. Ambrose, R. M. Gilbert, J. C. Fisher, M. Stealey and W. J. Kaiser, High resokution river hydraulic and water quality characterization using rapidly de- (%) ployable, Tecnical report, CENS, [7] S. C. H. Hoi, R. Jin, J. Zhu and M. R. Lyu, Batch mode k = 5 active learning and its application to medical image classification, In Proc. of the 23rd Int l Conf. on Machine k Learning (ICML 06), pages , [8] R. Horst and H. Tuy, Global Optimization (Deterministic Approaches) (3rd eds.), Springer, [9] A. Krause, H. B. McMahan, C. Guestrin and A. Gupta, 6.3 Robust submodular observation selection, Journal of Machine Learning Research, 9: , Reuters [10] Y. Kawahara, K. Nagano, K. Tsuda and J. Bilmes, Submodularity cuts and applications, In Advanced in Neural Information Processing Systems, Vol. 22, MIT Press, Cambridge, 2009 (to 5,180 ( ) appear). 90 7,770 [11] A. Krause, A. Singh and C. Guestrin, Near-optimal sensor placements in Gaussian processes: Theory efficient algorithms and empirical studies, Journal of Machine Learn- 2 k = 5, 10 ing Research, 9: , [12] L. Lovász, Submodular functions and convexity, In ( A. Bachem, M. Grötschel and B. Korte, editors, Mathematical Programming The State of the Art, pages , ɛ = ) (tp/(tp + fp), tp; true positive, fp; false [13] J. Leskovec, A. Krause, C. Guestrin, C. Faloutsos, J. Vanpositive) (5,180 Briesen and N. Glance, Cost-effective Outbreak Detection in Networks, In Proc. of the 13th ACM SIGKDD 90 3,019 ) Int l Conf. on Knowledge Discovery and Data Mining (KDD 07), pp ,

210 2: [, ]. k greedy submodularity cuts 5 (tonn, agricultur,trade,pct, market )[2.59,0.53] ( week,tonn,trade,pct, washington )[2.66,0.58] 10 (...,week,oil,price, dollar, offici )[3.55,0.57] (...,price,oil, bank, produc, blah )[3.8,0.62] [14] H. Lee, G. L. Nemhauser and Y. Wang, Maximizing a submodular function by integer programming: Polyhedral results for the quadratic case, European Journal of Operational Research, 94: , [15] K. Murota, Discrete Convex Analysis, volume 10 of Monographs on Discrete Math and Applications. Society for Industrial & Applied Mathematics, [16] G. L. Nemhause and L. A. Wolsey, Maximizing submodular set functions: formulations and analysis of algorithms, In P. Hansen, editor, Studies on Graphs and Discrete Programming, Vol.11 of Annals of Discrete Mathematics, [17] G. L. Nemhause and L. A. Wolsey, Integer and Combinatorial Optimization, Wiley-Interscience, [18] G. L. Nemhauser, L. A. Wolsey and M. L. Fisher, An analysis of approximations for maximizing for submodular set functions I, Mathematical Programming, 14: , [19] S. Perkins, K. Lacker and J. Theiler, Grafting: Fast, incremental feature selection by gradient descent in function space, Journal of Machine Learning Research, 3: , [20] A. Singh, A. Krause, C. Guestrin, W. Kaiser and M. Batalin, Efficient planning of informative paths for multiple robots, In Proc. of the 21st Int l Joint Conf. on Artificial Intelligence, pp , [21] H. Tuy, Concave programming under linear constraints, Soviet Mathematics Doklady, 5: , [22] M. Thoma, H. Cheng, A. Gretton, J. Han, H. Kriegel, A. Smola, L. Song, P. Yu, X. Yan and K. Borgwardt, Nearoptimal supervised feature selection among frequent subgraphs, In Proc. of the 2009 SIAM Conf. on Data Mining (SDM 09), pages ,

211 2009 Technical Report on Information-Based Induction Sciences 2009 (IBIS2009) A Study on Multi-dimensional Path Following for Weighted Kernel Machines Masayuki Karasuyama Naoyuki Harada Ichiro Takeuchi Abstract: We propose multi-dimensional path following for weighted kernel machines. In some situations of data modeling, each data point has a weight which represents the importance or confidence of the data point. We can easily implement such weighting schema in kernel machines by introducing multiple regularization parameters. In this paper, we derive the piece-wise linear path of these parameters which extends the idea of well known regularization path. Conventional algorithm only deals with the change of single regularization parameter. On the other hand, our approach can handle the change of multiple parameters simultaneously. Experimental results show that proposed algorithm can efficiently update optimal parameters. Our approach is especially beneficial for adaptive learning or online learning of weighted model. Keywords: weighted kernel machines, support vector machines, path following. 1 [1] [2, 3] [4] n {(x i, y i )} n i=1 x i X R d y i {1, 1} 2 y i R (SVM)[5], , tel , krsym@ics.nitech.ac.jp, Nagoya Institute of Technology, Gokisocho, Showa, Nagoya, Japan, , tel , harada@goat.ics.nitech.ac.jp, Nagoya Institute of Technology, Gokisocho, Showa, Nagoya, Japan, , tel , takeuchi.ichiro@nitech.ac.jp, Nagoya Institute of Technology, Gokisocho, Showa, Nagoya, Japan Φ : X F f(x) = w T Φ(x) + b : min w,b 1 2 w C n L(y i, f(x i )). i=1 L(y, f(x)) C > 0 n C i > 0, i = 1,, n, : min w,b 1 2 w n C i L(y i, f(x i )). i=1 i = 1,, n, C i < C i+1 C i C i f(x) 198

212 C SVM [6] C f(x) (Path Following) (Parametic Programming) [7] f(x) C n C i [8] 2 SVM SVM 2 (y i {1, 1}) SVR 2.1 SVM KKT SVM L : L(y, f(x)) = max{0, 1 yf(x)}. : min w,b,{ξ i } n i=1 1 2 w n C i ξ i, i=1 s.t. y i f(x i ) 1 ξ i, ξ i 0, i = 1,, n. SVM Weighted SVM[9] Fuzzy SVM[10] α i, ρ i 0, i = 1,, n, : L = 1 n 2 w 2 + C i ξ i i=1 n α i {y i f(x i ) 1 + ξ i } i=1 n ρ i ξ i. (1) i=1 w, b, ξ i 0 L n w = 0 w = α i y i Φ(x i ), i=1 L n b = 0 α i y i = 0, i=1 L ξ i = 0 α i = C i ρ i, i = 1,, n, (1) : max {α i } n i=1 s.t. 1 2 n i=1 j=1 n α i α j y i y j K(x i, x j ) + n y i α i = 0, 0 α i C i, i=1 n i=1 K(x i, x j ) = Φ(x i ) T Φ(x j ) f(x) : f(x) = n α i y i K(x, x i ) + b. i=1 b y i f(x i ) 1 = 0 i y i f(x i ) 1 KKT α i {y i f(x i ) 1 + ξ i } = 0, i = 1,, n, ξ i (α i C i ) = 0, i = 1,, n, y i f(x i ) 1 > 0 ξ i = 0, α i = 0, y i f(x i ) 1 = 0 ξ i = 0, 0 α i C i, y i f(x i ) 1 < 0 ξ i > 0, α i = C i, : O = {i : y i f(x i ) > 1, α i = 0}, (2a) M = {i : y i f(x i ) = 1, 0 α i C i }, (2b) I = {i : y i f(x i ) < 1, α i = C i }. (2c) v = [v 1,, v n ] I = {I 1,, I I } [v I1,, v I I ] v I n n M M M,O M O α i 199

213 2.2 [6] C α = [α 1,, α n ], b C 1 c =., C n c (old) c (new) [6] O, M, I c c = [ C 1,, C n ] α, b α i = C i, i I, α i = 0, i O y i f(x i ) : y i f(x i ) = j M Q ij α j + j I Q ij C j + y i b, Q ij = y i y j K(x i, x j ) c c (2b) y i f(x i ) = 0 : j M Q ij α j + j I Q ij C j + y i b = 0, i M. (3) j M y j α j + j I y j C j = 0, (4) (3),(4) : [ ] [ b y I M + α M M = [ Q M,I 0 y M y M (5) [ ] [ b = M 1 α M Q M ] y I ] c I = 0, (5), Q M,I ] c I, (6) α O = 0, (7) α I = c I, (8) c I, M, O (2a)-(2c) : y i {f(x i ) + f(x i )} < 1, i I, (9a) 0 α i + α i C i + C i, i M, (9b) y i {f(x i ) + f(x i )} > 1, i O. (9c) Critical Region[8] Critical Region α, b (6)-(8) c η 0 c = η(c (new) c (old) ), (10) c c (old) c (new) η Critical Region (10) η (10) (6) [ ] b = ηϕ, (11) α M [ ] ϕ = M 1 y I Q M,I (c (new) I c (old) I ), (12) f(x i ) y f = [ ] [ ] b y Q :,M + Q :,I c I (13) α M = η ψ, (14) ψ ψ = [ y Q :,M ] ϕ + Q :,I (c (new) I c (old) I ), (11),(14) (9) η (6),(13) (9) c c Critical Region η η H η 200

214 C 2 path 3 c (old) c η d ηd c (new) borders of current Critical Region d = c (new) c (old) [13, 14] C 1 λ (0, 1) : 1: n = 2 c c (old) c (new) c η d c (new) ηd O, M, I Critical Region c (new) η = C 1 c + c c (new) η η η H η (c c (new) ) η min(h {η }), 1 2 )η η O, M, I Critical Region breakpoint O, M, I breakpoint (12) 1 [11, 12] O( M 2 ) (6)-(8) α, b c breakpoint Algorithm 1 : Algorithm 1 Multi-dimentional Path Following for Weighted SVM 1: given optimal α, b for c (old) 2: initialize M, O, I 3: calculate M 1 4: while c c (new) do 5: calculate ϕ, ψ using M 1 6: calculate H using ϕ, ψ 7: η min(h {η }) 8: update α, b, c using step length η 9: update M, O, I 10: update M 1 11: end while C i = Cλ n i, i = 1,, n. i = 1,, n 1 (x n+1, y n+1 ) : (x 1, y 1 ) (C 1 = 0 ) (x n+1, y n+1 ) C n+1 =C i = 2, 3,, n C i λc i : c (new) c (old) = 0 Cλ n 1. Cλ 2 Cλ C λ = 0.99 Cλ n 1 Cλ n 2. Cλ C 0. 2 : C i 1 C i λc i α i α, b SMO SVM 2 α i 2 Sequential Minimal Optimization (SMO) [15] 2 maximum violating pair [16, 17] KKT ε =

215 2: 2 t 2 t µ α alpha seeding [18] α, b K(x i, x j ) RBF : K(x i, x j ) = exp( x i x j 2 2σ 2 ). M t = 1,, n, ( 2): p(x t y t = +1) = N (0, I), p(x t y t = 1) = N (µ(t), 0.25I), [ ] cos(2πt/n) µ(t) = 2. sin(2πt/n) I R 2 2 n = 200, 300, 400, 500 t = 1,, n/2 n/2 1 n/21 x 0 1 C = 100RBF σ 2 = 1 3 n n 3 3: CPU (MD-Path: SD-Path: 1 ) 1: (msec) method \ n SMO(1e-6) : breakpoint method \ n : method \ n : CPU ratio \ n / / breakpoint breakpoint CPU breakpoint breakpoint breakpoint 1 C i breakpoint η (Algorithm )breakpoint 2 202

216 C 2 c (new) c (old) Multi-dimensional Path Single-dimensional Path borders of Critical Region 4: breakpoint 2 breakpoint3 1 2 breakpoint4 c (new) 1 1 C i 4 1 S m, S s breakpoint B m, B s 1 p 1 C 1 S m = B m + 1, (15) S s = B s + p, (16) p n/2+1 3 breakpoint n 2 T S T L T S, T L 1 B m (T S + T L ) + T S : B s (T S + T L ) + pt S, (17) (17) S m : S s T S T S + T L = 1, T L T S 3.2 Fisher river 1 (x R 21 ) (y {1, 1}) x 0 1 n = C {10 1, 10 2, 10 3, 10 4 }, σ 2 {10 1, 10 0, 10 1 } C σ C 1 SMO C σ SMO SMO C C [17] 5-7 breakpoint σ breakpoint σ 7 1 breakpoint c p c (new) 1 1 breakpoint breakpoint breakpoint (p ) breakpoint 5 breakpoint 8-10 C 1 1 C 1 1 p SVM C C i α i C i λc i α i C i StatLib 203

217 (a) σ 2 = 0.1 (b) σ 2 = 1 (c) σ 2 = 10 5: Fisher river CPU (MD-Path:SD-Path: 1 ) 5: σ 2 =0.1 breakpoint method \ C : σ 2 =1 breakpoint method \ C : σ 2 =10 breakpoint method \ C SVM α, b 1 SVM Solver [1] R. J. Carroll and D. Ruppert, Transformation and weighting in regression. London, UK, UK: Chapman & Hall, Ltd., [2] A. Refenes, Y. Bentz, D. Bunn, A. Burgess, and 8: σ 2 =0.1 method \ C : σ 2 =1 method \ C : σ 2 =10 method \ C A. Zapranis, Financial time series modelling with discounted least squares backpropagation, Neurocomputing, vol. 14, no. 2, pp , [3] L. Cao and F. Tay, Support vector machine with adaptive parameters in financial time series forecasting, IEEE Transactions on Neural Networks, vol. 14, no. 6, pp , [4] A. Kolcz and J. Alspector, SVM-based filtering of spam with content-specific misclassification costs, in In Proceedings of the TextDM 01 Workshop on Text Mining - held at the 2001 IEEE International Conference on Data Mining, [5] V. N. Vapnik, The Nature of Statistical Learning Theory. Springer-Verlag,

218 11: σ 2 =0.1 ratio \ C / / : σ 2 =1 ratio \ C / / : σ 2 =10 ratio \ C / / [6] T. Hastie, S. Rosset, R. Tibshirani, and J. Zhu, The entire regularization path for the support vector machine, Journal of Machine Learning Research, vol. 5, pp , [7] T. Gal, Postoptimal Analysis, Parametric Programming, and Related Topics. Walter de Gruyter, [8] E. N. Pistikopoulos, M. C. Georgiadis, and V. Dua, Process Systems Engineering: Volume 1: Multi-Parametric Programming WILEY-VCH, [9] X. Yang, Q. Song, and Y. Wang, A weighted support vector machine for data classification, International Journal of Pattern Recognition and Artificial Intelligence, vol. 21, no. 5, pp , Computer Sciences, vol. E86-A, no. 8, pp , [14] H. Funaya, Y. Nomura, and K. Ikeda, A support vector machine with forgetting factor and its statistical properties, in Proc. Intl. Conf on Neural Information Processing (ICONIP 08). [15] J. C. Platt, Fast training of support vector machines using sequential minimal optimization, in Advances in Kernel Methods Support Vector Learning (B. Schölkopf, C. J. C. Burges, and A. J. Smola, eds.), (Cambridge, MA), pp , MIT Press, [16] S. Keerthi, S. Shevade, C. Bhattacharyya, and K. Murthy, Improvements to platt s smo algorithm for svm classifier design, Neural Computation, vol. 13, no. 3, pp , [17] L. Bottou and C.-J. Lin, Support vector machine solvers, in Large Scale Kernel Machines (L. Bottou, O. Chapelle, D. DeCoste, and J. Weston, eds.), pp , Cambridge, MA.: MIT Press, [18] D. DeCoste and K. Wagstaff, Alpha seeding for support vector machines, in Proceedings of the International Conference on Knowledge Discovery and Data Mining, pp , [10] C.-F. Lin and S.-D. Wang, Fuzzy support vector machines, IEEE Transactions on Neural Networks, vol. 13, no. 2, pp , [11] J. R. Schott, Matrix Analysis For Statistics. Wiley-Interscience, [12] G. H. Golub and C. F. Van Loan, Matrix computations. Baltimore, MD, USA: Johns Hopkins University Press, [13] F. Liu, T. Zhang, and R. Zhang, Modified kernel RLS-SVM based multiuser detection over multipath channels, IEICE Transactions on Fundamentals of Electronics, Communications and 205

219 情 報 論 的 学 習 理 論 テクニカルレポート 2009 Technical Report on Information-Based Induction Sciences 2009 (IBIS2009) ベイズ 確 率 文 脈 自 由 文 法 のための 高 速 構 文 木 サンプリング 法 Split Position Slice Sampler * 武 井 俊 祐 牧 野 貴 樹 * 高 木 利 久 Shunsuke Takei * Takaki Makino Toshihisa Takagi * Abstract: We propose a new tree sampling algorithm for Bayesian probabilistic context-free grammar (PCFG) called Split Position Slice Sampler. Split Position Slice Sampler is developed based on Beam Sampling method that is a fast MCMC sampling algorithm for Bayesian Hidden Markov Model, and adapted to Bayesian PCFG. This tree sampling method can be combined with Metropolis-Hastings sampler to constitute an efficient grammar sampling algorithm for Bayesian PCFG. Because this algorithm does not involve any approximation, more efficient inference is achieved without losing accuracy. We evaluate our approach by comparing with an existing method in a small artificial corpus. Keywords: Bayesian inference, PCFG, MCMC 1 Introduction 本 研 究 は,ベイズ 拡 張 された 確 率 文 脈 自 由 文 法 (Bayesian PCFG)に 対 する 効 率 的 なサンプリングアル ゴリズムを 構 築 することで, 高 速 かつ 高 精 度 な 文 法 学 習 を 可 能 にすることを 目 的 とする.Bayesian PCFG モデルに 対 する 精 度 の 高 いパラメータ 推 定 手 法 とし ては,Johnson らの 動 的 計 画 法 を 利 用 したマルコフ 連 鎖 モンテカルロ 法 (MCMC)による 手 法 [1]が 知 ら れているが,モデルが 複 雑 になるにつれて 増 大 する 計 算 コストが 問 題 となる.そこで 我 々は,ベイズ 拡 張 された 隠 れマルコフモデル(Bayesian HMM)のた めの 高 速 な MCMC 法 として 知 られている Beam Sampling[2]を PCFG に 応 用 することで Johnson らの 手 法 を 高 速 化 する.Beam Sampling は, 動 的 計 画 法 と Slice Sampling[3]を 利 用 することにより,Bayesian HMM の 高 速 なパラメータ 推 定 を 可 能 にする 手 法 で あるが,これをそのままの 形 式 でベイズ PCFG の 枠 組 みへ 応 用 しようとするとき,HMM において 各 時 刻 において 割 り 当 てられている 補 助 変 数 をベイズ * 東 京 大 学 大 学 院 新 領 域 創 成 科 学 研 究 科 情 報 生 命 科 学 専 攻, 千 葉 県 柏 市 柏 の 葉 総 合 研 究 棟 609, tel , takei_shunsuke@cb.k.u-tokyo.ac.jp, Department of Conputational Biology, Graduate School of Frontier Science, The University of Tokyo, General Research Building 609, Kashiwa-no-ha, Kashiwa, Chiba, JAPAN 東 京 大 学 総 括 プロジェクト 機 構 情 報 システム 研 究 機 構 ライフサイエンス 統 合 データベースセン ター PCFG で 利 用 される 内 側 確 率 表 に 直 接 対 応 付 けるこ とができずアルゴリズムが 構 築 できない. 本 論 文 で は 内 側 確 率 表 における 終 端 位 置 と 分 割 位 置 に 補 助 変 数 を 対 応 付 ける 形 式 として Split Position Slice Sampler という 手 法 を 新 たに 提 案 することで Beam Sampler の 枠 組 みを 拡 張 し,これを Johnson らの 手 法 へ 組 み 込 むことによって 高 精 度 かつ 高 速 な Bayesian PCFG モデルのパラメータ 推 定 法 を 構 築 する Motivation ベイズモデルとは, 統 計 的 機 械 学 習 分 野 において 学 習 モデルにベイズ 統 計 の 手 法 を 導 入 するもので, ベイズ 化 された 学 習 モデルは, 与 えられたデータに 対 するモデルパラメータの 推 定 値 と 不 確 実 性 の 範 囲 を 確 率 分 布 として 表 現 する. 結 果 としてベイズ 統 計 における 学 習 は, 従 来 の 最 尤 法 による 点 推 定 に 比 べ 汎 化 性 能 に 優 れ, 過 学 習 の 問 題 も 生 じにくい.また, あらかじめ 推 定 したい 対 象 に 何 らかの 知 識 がある 場 合,それを 事 前 確 率 分 布 という 形 式 でモデルに 導 入 することでより 良 い 推 定 が 可 能 であるという 利 点 も ある.さらに, 近 年 注 目 を 集 めているベイズモデル の 拡 張 であるノンパラメトリックベイズモデルでは, 無 限 次 元 のパラメータ 空 間 を 仮 定 し,データを 表 現 する 最 適 なモデルを 推 定 することでモデル 選 択 の 問 題 を 解 決 することができるため, 様 々な 統 計 学 習 モ 206

220 デルに 導 入 されており[4][5][6], 今 後 より 広 範 な 問 題 に 応 用 されることが 期 待 されている. ベイズ 学 習 モデルにおいて 学 習 の 対 象 である 個 々 のパラメータは 確 率 変 数 で 表 現 されているため, 従 来 の 最 尤 法 とは 異 なりしばしば 計 算 コストの 高 い 高 2.1. PCFG 次 元 の 期 待 値 計 算 を 伴 う 方 法 でパラメータ 推 定 を 行 文 脈 自 由 文 法 (CFG)は 文 の 生 成 過 程 を 明 らかにす う. 特 に 高 い 精 度 が 要 求 されるような 場 合 において るモデルであり G = ( VN, VT, S, R) の 4-タプルで 定 義 は 高 精 度 なパラメータ 推 定 手 法 として 知 られる される. V は 終 端 記 号 集 合, V T MCMC 法 が 利 用 されるが, 膨 大 なサンプルを 必 要 と N は 非 終 端 記 号 集 合, R は 生 成 規 則 の 集 合, S V N は 開 始 記 号 である. するため 計 算 コストが 高 く,ベイズモデルの 応 用 に ここでは 生 成 規 則 についてチョムスキー 標 準 形,す おける 問 題 となっている. なわち A BC または A s の 形 式 の 生 成 規 則 の PCFG[9]は 計 算 機 科 学 分 野 だけでなくバイオイン みを 考 える.ここで A, B, C V N, s VT である. フォマティクス 分 野 など[10] 幅 広 い 分 野 で 活 用 され 確 率 文 脈 自 由 文 法 (PCFG)は,CFGの 各 々のルール ている 一 般 性 の 高 い 確 率 モデルであり,そのベイズ に 対 して 確 率 値 を 割 り 振 ったものであり, ( G, θ ) と モデルである Bayesian PCFG についても 期 待 が 高 い. 定 義 される. θ は R 次 元 の 実 数 ベクトルであり,θ 近 年, 変 分 ベイズ 法 [7][8]のような 近 似 に 基 づく 高 速 のそれぞれの 要 素 はθ パラメータ 推 定 手 法 が 提 案 され,さかんに 研 究 され r で 表 され,これは r R の 生 成 規 則 の 確 率 値 であることを 表 す.たとえば, ているものの, 高 精 度 な 推 定 を 必 要 とする 場 合 は 依 θ A BC ならば A BC の 規 則 の 確 率, θ A s ならば 然 として 計 算 コストの 高 い MCMC のような 手 法 が A s の 規 則 の 確 率 を 表 す.ここで, 確 率 の 定 義 か 用 いられている. らθ r 0 かつ = 1 Bayesian PCFG に 対 する MCMC 法 を 構 成 する 場 合, θa a である 必 要 がある. 単 純 には Gibbs Sampler のような 手 法 の 適 用 が 考 えら れるが, PCFG のパラメータ 間 に 強 い 相 互 依 存 があ りパラメータ 空 間 も 大 きいことから Gibbs Sampler で は 収 束 が 遅 く 効 率 が 悪 い.これに 対 し Johnson らは 構 文 木 のみのサンプリングによる Metropolis-Hastings Sampler を 構 築 し,より 効 率 的 なサンプリング 手 法 を 提 案 している[1].しかし Johnson らの 手 法 は 依 然 として 計 算 コストが 大 きく, 大 規 模 データに 適 用 す るのには 困 難 を 伴 う. 近 年 Bayesian HMM のための 高 速 なサンプリング 手 法 として Beam Sampler が 提 案 された.この 手 法 は Slice Sampling の 手 法 を 応 用 し, 隠 れ 状 態 間 の 遷 移 確 率 分 布 を 補 助 変 数 (スライサー)で 間 引 く ことで 計 算 の 高 速 化 を 図 っている.この 間 引 き の 処 理 は 近 似 等 の 処 理 ではなくサンプリングの 処 理 の 一 部 であるため,Beam Sampler は 精 度 を 損 なわず Bayesian HMM のパラメータ 推 定 を 高 速 に 行 うことができ る.HMM における 遷 移 確 率 分 布 は PCFG における 書 き 換 え 規 則 の 確 率 分 布 に 対 応 するため, 正 しいス ライス 手 法 が 与 えられれば, 同 様 な 高 速 化 を 達 成 で きることが 期 待 される. そこで 我 々は,Johnson らの 手 法 に Beam Sampler の 枠 組 みを 導 入 することで, 精 度 を 保 ったまま 高 速 に 動 作 する Bayesian PCFG のための MCMC 法 を 構 築 することを 目 指 す. 2 Background a V V V 2.2. Bayesian PCFG T N N Bayesian PCFG では, 生 成 規 則 のパラメータ 確 率 変 数 P(θ) として 扱 う.ベイズアプローチにおい て 離 散 パラメータは,その 扱 いやすさから Dirichlet 分 布 のような 共 役 な 確 率 分 布 によって 表 され, 本 論 文 における Bayesian PCFG についても Dirichlet 分 布 を 導 入 した PCFG を 仮 定 している.ここで Dirichlet P ( ) A R θ A 分 布 を D, Γ をガンマ 関 数, の 形 式 の 文 法 規 則 を, における の 形 式 の 文 法 規 則 A についてのパラメータベクトル θ A ハイパーパラメータベクトルを α A α θ P ( θ α) θ を,Dirichlet 分 布 の,α における の 形 式 の 文 法 規 則 についてのサブベクトルを A で 表 すと, の 事 前 分 布 D は P D ( α) = PD ( θ A α A ) A V N θ (1) のような Dirichlet 分 布 で 表 現 される.ここで, 1 α r 1 PD ( θ A α A ) = θ (2) r C( α ) A r RA Γ( α ) r R A A C( α A ) = (3) Γ( α ) r RA A 207

221 である. 一 般 にモデルパラメータは 未 知 でありデー タから 推 定 する 必 要 がある.ベイズアプローチにお ける 推 定 の 対 象 はデータ 観 測 後 の 事 後 確 率 分 布 であ り, P( θ w) P( w θ) P( θ α) = n i= 1 端 記 号 列 を 意 味 する. P( w i θ) P( θ α) の 左 項 を 求 めることである. w = ( w 1,..., w n ) は 終 端 記 号 列 からなるデータ 集 合 であり, が 個 別 の 終 2.3. Gibbs Sampler w i (4) Bayesian PCFG の 事 後 分 布 は 解 析 的 積 分 が 不 可 能 であり,その 推 定 には MCMC 法 や 変 分 ベイズ 法 な どの 手 法 が 用 いられる. MCMC 法 のもっとも 単 純 な 手 法 の 1 つである Gibbs Sampler を Bayesian PCFG のパラメータ 推 定 法 として 用 いる 場 合,パラメータ θ と 構 文 木 t を 交 互 にサンプリングすることでアル ゴリズムが 構 築 される.ここで, t = ( t 1,..., t n ) であ り,それぞれのt は 終 端 記 号 列 w に 対 する 構 文 木 を i 意 味 する. 詳 しい 導 出 は Johnson らの 論 文 に 譲 り, ここでは 具 体 的 な Gibbs Sampler のアルゴリズムを 直 接 示 す. 構 文 木 t において 文 法 規 則 r R が 使 われた 回 数 を f r (t), t における A の 形 式 の 文 法 規 則 が 使 われた 回 数 のベクトルを PCFG の Gibbs Sampler は, のサンプリング, A V N f A t(t) i とすると,Bayesian を 固 定 してθ について P( θ t,w,α) = PD ( θ A f A ( t) + α A ) (5) θ を 固 定 して t についてのサンプリング, P( t θ,w,α) = P( t i w i, θ) n i= 1 (6) ある 終 端 記 号 列 s = s,..., s ) が 与 えられたとき,そ の 内 側 確 率 を ( 1 n A pk k =θ (7), A s k A B C p, = θ p, p (8) 1, A BC R i j< k よるサンプルの 確 率 的 な 受 理 から 構 成 されるサンプ を 交 互 に 繰 り 返 すことになる.パラメータθ につい リングアルゴリズムを 提 案 している.この 手 法 では, てのサンプリングは, 構 文 木 t における 各 文 法 規 則 依 存 の 強 いθ のサンプリングは 回 避 されるため, の 使 用 回 数 から 容 易 に 計 算 できるが, 構 文 木 t の 具 Gibbs Sampler に 比 べ 速 い 収 束 が 見 込 まれる.さらに 体 的 なサンプリングについては 自 明 ではない. θ のサンプリングにかかる 計 算 コストも 不 要 である Johnson らは 構 文 木 のサンプリングについて, 動 的 ため, 効 率 的 な Bayesian PCFG のサンプリングアル 計 画 法 を 用 いた 効 率 的 な 手 法 を 提 案 している.まず, ゴリズムであるといえる. i k A BC のように 計 算 する.ここで, i j A p i, k j+ k は 非 終 端 記 号 A が 終 端 記 号 列 s を 生 成 する 確 率 を 意 味 する. 次 に, i,..., s k 作 られた 内 側 確 率 表 の 確 率 にしたがい 構 文 木 を 再 帰 的 にサンプリングする.これを 疑 似 コードで 示 すと, Function SAMPLE( A, i, k) If i = k return TREE( A, sk ) Else ( j, B, C) = MULTI( A, i, k) return TREE( A,SAMPLE( B, i, j),sample( C, j, k)) となる.ここで, 関 数 SAMPLE( A, i, k) はある 構 文 木 のノードにおいて 非 終 端 記 号 A が 最 終 的 に 終 端 記 号 列 s を 生 成 するとき,そのノード 以 下 の 構 文 i,..., s k 木 を 確 率 的 にサンプリングする 関 数 である.また, 関 数 MULTI( A, i, k) はどの 分 割 点 で 構 文 木 が 枝 分 か れするのか,その 分 割 位 置 j と 子 ノードの 非 終 端 記 号 B,C を 確 率 的 に 返 す 関 数 であり,その 確 率 は P( j, B, C) と 表 すことが 出 来 る. p p B C A BC i, j j+ 1, k = θ (9) A pi, k 2.4. Metropolis-Hastings Sampler PCFG における 各 々のパラメータ 間 には 強 い 依 存 があるため,パラメータ θ を 個 別 にサンプリングす る Gibbs Sampler は 十 分 なサンプルを 得 るために 多 大 な 時 間 が 必 要 になる.この 問 題 に 対 し Johnson らは Bayesian PCFG のパラメータ を 積 分 消 去 し, 構 文 木 のサンプリングと Metropolis-Hastings の 枠 組 みに θ 208

222 このアルゴリズムでは,Gibbs Sampler とは 違 いパ ラメータ θ のサンプリングを 行 わず, 構 文 木 をサン プリングしたのち,そのサンプルを 確 率 的 に 受 諾 も しくは 拒 否 する.いま, w = ( w 1,..., w n ) を 終 端 記 号 列 のs の 集 合 とし,それぞれの 終 端 記 号 列 wi に 対 応 する 構 文 木 を t = ( t 1,..., t n ) で 表 す. t' が 新 しくサン i プルされた 構 文 木, t が を 除 いた 構 文 木 サンプル i ti 群 とすると,そのサンプルの 受 理 確 率 は P( t' i wi, t i, α ) P( ti wi, θ' ) A( ti, t' i ) = min 1, P( t, t, i wi i α) P( t' i wi, θ' ) となる.ここで, P( t' i t i, α) P( ti = min 1, P( t t, i i α) P( t' i A V N A A w i, θ' ) wi, θ' ) C( α A + f A ( t)) P( ti t i, α) = (9) C( α + f ( t )) である. θ' は θr ' を 要 素 に 持 つベクトルである. θr ' は t とα が 与 えられたときのθ r の 期 待 値 で, i r ' R A i r r ' ( t i ) i f r ( t ) + α θ r ' = (10) f + α r ' (7) 我 々のアプローチは Johnson らの 手 法 に Beam として 計 算 される. Sampler の 枠 組 みを 導 入 することで 高 速 化 を 図 るも 以 上 が Johnson らの Bayesian PCFG のための 構 文 のである. Beam Sampler でサンプルとする 隠 れ 状 態 木 サンプリングの 枠 組 みである.この 手 法 はパラメ 列 は PCFG における 構 文 木 にあたり,Beam Sampler ータθ と 構 文 木 t を 交 互 にサンプリングする Gibbs で 用 いている 動 的 計 画 法 は Johnson らの 手 法 では 内 Sampler に 比 べ 効 率 的 ではあるが,パラメータ 空 間 側 確 率 を 計 算 し, 構 文 木 をサンプリングすることと が 大 きい 場 合, 内 側 確 率 の 計 算 に 多 大 な 計 算 コスト 等 価 である.ここから Slice Sampling を 内 側 確 率 の がかかり 依 然 として 大 規 模 データには 不 向 きである. 計 算 に 利 用 すれば Beam Sampler が PCFG へと 適 用 可 2.5. Beam Sampler Beam Sampler は,Bayesian HMM のための 高 速 MCMC アルゴリズムであり,HMM のパラメータ 推 定 にしばしば 用 いられる 動 的 計 画 法 (この 場 合 Forward-Backward)と Slice Sampling の 手 法 を 応 用 す ることで 高 速 化 をしている.HMM のための Gibbs Sampler は, 各 時 刻 の 隠 れ 状 態 を 交 互 にサンプリン グする 形 式 で 行 われるが,Beam Sampler は 各 時 刻 の 隠 れ 状 態 を 個 別 にサンプリングするのではなく, 動 的 計 画 法 を 利 用 し 隠 れ 状 態 列 をひとつのサンプルと する 形 式 で 行 われるため,Johnson らの 手 法 同 様 パラ メータ 間 の 依 存 の 問 題 が 解 決 される.また Slice Sampling に 基 づく 補 助 変 数 (スライサー)を 各 時 刻 について 導 入 し, 1. 前 回 サンプルされた 状 態 遷 移 列 に 従 いスライ サーをサンプリング 2. 状 態 遷 移 分 布 をスライスすることで, 各 時 刻 における 可 能 な 状 態 遷 移 を 間 引 き 3. 間 引 かれた 分 布 に 従 い 隠 れ 状 態 列 をサンプル という 手 順 でサンプリングを 行 う.ここでスライスとは, 対 象 となる 確 率 分 布 においてスライサーの 値 以 上 のもの のみを 考 慮 するよう 分 布 を 間 引 く 操 作 であり,スラ イスされた 分 布 は 等 確 率 でサンプリングされる.この 手 順 により 各 時 刻 で 考 慮 すべき 状 態 遷 移 数 が 減 るために 計 算 が 高 速 に 行 われる.Beam Sampler における 分 布 のス ライスおよびスライスされた 分 布 のサンプリングと いう 処 理 は 近 似 処 理 ではなく,Slice Sampling を 援 用 したもので,それ 自 体 がサンプリングの 手 続 きであ る.そのため Beam Sampler は 高 精 度 なパラメータ 推 定 というマルコフ 連 鎖 モンテカルロ 法 の 性 質 を 維 持 したまま 高 速 化 に 成 功 した 手 法 であるといえる. 3 Method 3.1. PCFG へのスライスサンプリングの 導 入 能 である.Johnson らのアルゴリズムが 1. 内 側 確 率 表 を 計 算 2. 構 文 木 をサンプリング 3. 構 文 木 を Accept/Reject という 手 順 で 構 成 されていたのに 対 し, 提 案 手 法 は 1. スライサーをサンプリング 2. 生 成 確 率 をスライスして, 内 側 確 率 表 の 各 セルにお ける 文 法 規 則 適 用 分 布 を 間 引 き 3. 構 文 木 をサンプリング 4. 構 文 木 を Accept/Reject 209

223 という 手 順 で 構 成 されることになる.しかし PCFG と HMM のモデルの 違 いから Beam Sampler をそのま ま 適 用 しようとしてもアルゴリズムは 構 築 できない. セル を 担 当 するスライサー 問 題 となるのは,Beam Sampler における 前 回 のサ ンプルにおける 各 時 刻 の 状 態 遷 移 確 率 からスライサ ーの 値 をサンプリングするという 手 続 きである. HMM では 各 時 刻 についてスライサーがそれぞれ 割 り 当 てられており(Figure 1),これを PCFG にその まま 適 用 しようとすると 内 側 確 率 表 の 各 セル,つま り 考 えうる 全 ての 構 文 木 のノード 位 置 についてスラ イサーを 導 入 するという 形 式 となる.この 場 合 前 回 のサンプルにあたるのが 構 文 木 であり,スライサー のサンプルに 必 要 なのは, 構 文 木 の 各 ノードで 用 い られた 文 法 規 則 の 確 率 値 である.しかしこの 場 合, 前 回 のサンプル,すなわちひとつの 構 文 木 からすべ てのセルのスライサーを 作 ることができない.なぜ なら,サンプルとして 得 られた 構 文 木 は, 内 側 確 率 表 全 てのセルに 一 対 一 対 応 するだけノード 数 を 保 持 しておらず, 従 ってこのような 配 置 でスライサーを 導 入 しようとした 場 合, 全 てのスライサーをサンプ を 提 案 する.この 形 式 におけるスライサーは 内 側 確 率 表 において, 最 終 的 に 終 端 記 号 を 出 力 する 終 端 点 を 担 当 するスライサー と, 構 文 木 の 分 割 の 2 種 類 に 分 けて 扱 わ れる(Figure 3). 終 端 セルに 対 応 する 構 文 木 上 のノー ドは 常 に 存 在 するため, については Beam Sampler 同 様 に, 前 回 の 構 文 木 サンプル 中 の 対 応 するノード において 用 いられた 文 法 規 則 の 確 率 値 をもとに 値 を 決 める. 一 方,それぞれの 終 端 記 号 の 間 を 分 割 点 と 定 めると, 構 文 木 の 終 端 ノード 以 外 のノード, すなわち 分 岐 ノードは 必 ず 1 つの 分 割 点 を 担 当 することがわかる.ここから, 分 割 点 を 担 当 す るスライサー は, 構 文 木 中 の 分 割 点 を 担 当 す る 分 岐 ノードの 文 法 規 則 の 確 率 値 から 作 られる (Figure 4).この 手 法 では 内 側 確 率 計 算 の 際, 終 端 セ ル 以 外 のセルにおける 計 算 に 複 数 の が 利 用 される ことになるが,サンプルされた 構 文 木 における 各 ノ ードでは,そのいずれかのスライサーにより 作 られ た 文 法 規 則 が 用 いられ, 構 文 木 全 体 で 見 たときにス ライサーの 重 複 はない.また, 導 入 すべきスライサ リングするには 情 報 が 足 りないからである(Figure 2). ーは 最 下 列 セル 数 n, 分 割 点 数 n-1 であり, 構 文 木 これは n を 文 中 の 単 語 数 とした 場 合, 構 文 木 は 2n-1 個 のノードしか 持 たず, 一 方 内 側 確 率 表 には n(n+1)/2 個 のセルがあるということからも 分 かる. の 持 つノード 数 と 一 致 するため, 必 ず 一 対 一 対 応 が 出 来 る.この 手 法 を 我 々は Split Position Slice Sampler と 呼 ぶ. 以 下 では Split Position Slice Sampler を 利 用 した 構 文 木 のサンプリングアルゴリズムの 具 体 的 3.2. Split Position Slice Sampler そのため 我 々はスライサーの 配 置 に 関 する 新 たな な 手 順 を 示 す. 終 端 記 号 列 s に 対 する 前 回 の 構 文 木 サンプルにお ルールとして,Split Position Slice Sampler という 形 式 いて, sk を 導 出 した 文 法 規 則 の 確 率 値 をπ k, 構 文 木 の 分 割 点 j における 文 法 規 則 の 確 率 値 を ρ j と 定 義 する. 関 数 I(C) は, 条 件 C が 真 のとき I( C ) = 1, Time 6 7 偽 の 時 I( C) = 0 となる 関 数 である. 8 9 v u u v v S 3 S 1 S 2 S 3 u 6 7 u 7 8 u 8 9 S A u 1,3 u 1,2 C A B u 1,1 u 2,2 u 3,3 Figure 1. HMM のための Beam Sampler におけるスライサー 上 段 が 1 つの 隠 れ 状 態 列 サンプル, 下 段 が 動 的 計 画 法 に 導 入 されるスライサー 列 を 表 し, 破 線 がその 対 応 を 示 している. 各 時 刻 間 の 状 態 遷 移 確 率 を 用 い, 動 的 計 画 法 の 各 セルに 対 応 する スライサーu i をサンプリングする. 観 測 列 は 省 略 している. w 1 w 3 w 2 Figure 2. PCFG へ Beam Sampler を 単 純 に 適 用 した 場 合 左 が 1 つの 構 文 木 サンプル, 右 が 内 側 確 率 表 に 導 入 さ れるスライサーを 表 し, 破 線 がその 対 応 を 示 している. 情 報 が 足 りずスライサーを 導 入 できないセルが 存 在 する 210

224 1. Sampling u, v step u Uniform(0, π ) (11) k ~ k j ~ Uniform(0, j v ρ ) (12) 2. Inside-Filtering Step p = I( u < θ ) (13) A k, k k A wk A B C p i, k = v j < A BC ) pi, j p j+ 1, k A BC R i j< k I( θ (14) 3. Tree-Sampling Step Function SAMPLE( A, i, k) If i = k return TREE( A, sk ) Else ( j, B, C) = MULTI( A, i, k) return TREE( A,SAMPLE( B, i, j),sample( C, j, k)) 関 数 MULTI における 確 率 計 算 は P( j, B, C) I( ) p B C A BC i, j j+ 1, k = θ (15) A pi, k と 置 き 換 えられる. このアルゴリズムは Johnson らの 手 法 における 構 p 文 木 のサンプリングステップについての 置 き 換 えで あり,その 他 の 変 更 は 無 い.つまり 本 手 法 は Gibbs Sampler に 組 み 込 んでθ のサンプリングと 組 みあわ せるか,Metropolis-Hastings Sampler と 組 み 合 わせて サンプリングされた 構 文 木 の 確 率 的 な 受 理 をするこ とで PCFG のサンプラーとして 機 能 する. 以 上 で Beam Sampling 法 の 枠 組 みを PCFG へと 適 用 した 高 速 な MCMC 法 である Split Position Slice Sampler が 構 築 された.この 手 法 は, 単 純 に 考 えた 場 合 に 対 応 のつかない, 形 の 違 う 構 文 木 同 士 に 対 応 をつけることで Beam Sampler の 枠 組 みを 拡 張 した. これにより,Beam Sampler の 枠 組 みは Bayesian PCFG のためのサンプリングアルゴリズムに 導 入 可 能 となり, 高 速 なサンプラーが 構 築 された. 4 Experiments 人 工 的 な CFG 文 法 から 小 規 模 なコーパスを 生 成 し, ノンパラメトリック PCFG モデル[11]による 教 師 無 し 文 法 学 習 により, 本 手 法 と 従 来 手 法 の 比 較 を 行 っ た.コーパスは 文 法 規 則 S xsy,s zsw,s SS, S εから 生 成 され, 平 均 文 長 9 単 語 の 100 文 を 訓 練 データ, 平 均 文 長 13 単 語 の 20 文 をテストデータと して 用 い, 従 来 の Johnson の 手 法 と 提 案 手 法 につい てそれぞれ 50 回 の 実 験 を 行 い, 計 算 速 度 とテストデ ータの 対 数 尤 度 について 評 価 を 行 った. v 1 v 2 v 1 v 2 u 1 u 2 u 3 v 1 v 2 Figure 3. 終 端 セルにスライサーu, 分 割 点 にスライサーv を 導 入. 結 果 と して 導 入 するスライサーの 総 数 は 構 文 木 のノード 数 になる. v 1 v 2 v 1 v 2 Figure 4. 分 割 点 と 分 岐 ノード,スライサーの 関 係 丸 が 分 岐 ノード, 四 角 が 終 端 ノードを 表 し,3 文 字 の 終 端 記 号 列 に 対 応 する 2 種 類 の 構 文 木 を 上 下 に 配 し, 黒 いノードが 担 当 する 分 割 点 を 実 線 矢 印 で 示 してい る.どのような 形 の 構 文 木 でも 必 ず 分 割 点 を 担 当 する 分 岐 ノードが 1 つ 存 在 し, 分 割 点 を 担 当 するスライサーを 作 ることができる. 211

225 -300 Time (second) Loglikelihood Figure5. 時 間 ごとの 対 数 尤 度 上 昇 従 来 手 法 提 案 手 法 従 来 手 法 提 案 手 法 Time (second) Figure6. 1 回 の Sampling Step ごとにかかる 時 間 Sampling Step -300 Sampling Step Loglikelihood Figure7. サンプリングステップごとの 対 数 尤 度 上 昇 従 来 手 法 提 案 手 法 212

226 Figure5 は 時 間 ごとに 見 たコーパスの 対 数 尤 度, Figure6 はサンプリング 1 ステップごとにかかる 時 間, Figure7 はサンプリング 1 ステップごとに 見 たテスト データの 対 数 尤 度 である.グラフは 50 回 の 実 験 によ る 結 果 の 平 均 値 であり, 誤 差 棒 は 95%の 信 頼 区 間 を 示 している.Figure5 より, 提 案 手 法 は 時 間 ごとの 対 数 尤 度 が 急 速 に 上 昇 しており, 従 来 手 法 に 比 べより 高 速 に 学 習 していることがわかる.Figure5 と Figure7 においてサンプリングの 初 期 に 対 数 尤 度 が 低 いのは,スライサーという, 新 たに 推 定 すべきパラ メータが 加 わったため,つまりモデルが 複 雑 になっ たためであると 考 えられる.しかし,Figure6 からは, 1 ステップの 計 算 速 度 が 従 来 手 法 に 比 べて 提 案 手 法 が 大 幅 に 高 速 化 されているために,その 影 響 を 補 っ て 余 りある 高 速 化 効 果 が 得 られ,その 結 果, 従 来 手 法 に 比 べ 計 算 速 度 の 向 上 がなされていることがわか った. 5 まとめ 本 論 文 では Beam Sampler の 枠 組 みを Split Position Slice Sampler へと 拡 張 し, 従 来 手 法 に 組 み 込 むこと で Bayesian PCFG のための 高 速 なパラメータ 推 定 手 法 を 構 築 し,それを 小 規 模 な 実 験 によって 評 価 した. 我 々の 手 法 では 一 切 の 近 似 をせず, 高 い 精 度 でのパ ラメータ 推 定 が 可 能 であるという MCMC 法 の 性 質 を 保 っているため, 高 速 かつ 高 精 度 な 手 法 であると いえる.このアルゴリズムは CKY アルゴリズムを 伴 えばどのような 問 題 にも 適 用 可 能 であり,ベイズ 拡 張 された PCFG のスーパーセットのようなモデル にも 適 用 可 能 であることが 期 待 できる. また,Beam Sampler は 元 々ノンパラメトリック 拡 張 された HMM において 提 案 されたものであるとい うことから, 実 験 でも 示 したとおり,ノンパラメト リック 拡 張 された PCFG のためのサンプリング 法 と しても 用 いることができる.ただし,PCFG ではノ ンパラメトリック HMM と 動 的 計 画 法 の 向 きが 逆 で あることなどが 原 因 で,いくつかの 変 更 が 必 要 とな る.しかしそれらは 提 案 手 法 とは 直 接 関 係 が 無 く, 性 能 評 価 にも 影 響 しないため, 本 論 文 では 省 略 した. 今 後 は PCFG のスーパーセットのためのサンプリ ングアルゴリズムへの 拡 張 といったアルゴリズム 的 な 応 用, 実 際 の 大 規 模 データへの 適 用 などの 実 用 面 での 応 用 の 可 能 性 を 探 るつもりである. 参 考 文 献 [1] Johnson, M., Griffiths, T. L. & Goldwater, S. (2007). Bayesian Inference for PCFGs via Markov Chain Monte Carlo. In Proceedings of North American Chapter of Association for Computational Linguistics Human Language Technologies. [2] Gael, J. V., Saatci, Y. Teh, Y. W. & Gharamani. Z. (2008) Beam Sampling for the Infinite Hidden Markov Model. In Proceedings of the 25 th International Conference on Machine Learning. [3] Neal, R. M. (2003). Slice sampling. The annals of Statistics.. [4] Teh, Y. W., Jodan, M. I., Beal, M. & Blei, D. (2006). Hierarchical Dirichlet processes. Journal of the American Statistical Association.. [5] Blei, M., Gharamani, Z. & Rasmussen, C. (2002). The infinite hidden Markov model. In Advances in Neural Information Proceeding Systems. [6] Liang, P., Petrov. S., Jordan., M. I. & Klein. D. (2007) The infinite PCFG using hierarchical Dirichlet processes. In Proceedings of the 2007 joint Conference on Empirical Methods in Natural language Processing and Computational Natural Language Learning. [7] Attias, H. (1999). Inferring Parameters and Structure of Latent Variable Models by Variational Bayes. In Proceedings of Uncertainty in Artificial Intelligence. [8] Kenichi, K., Yoshitaka, K., Taisuke, S. (2004). Variational Bayesian Approach to Probabilistic Context-Free Grammar based on Dynamic Programing. Journal of Information Processing Society, Japan. [9] Chaniak, E. (1996). Treebank grammars. Association for the Advancement of Artificial Intelligence. [10] Sakakibara, Y.; Brown, M.; Hughey, R.; Mian, I. S.; Sjölander, K.; Underwood, R. C.; and Haussler, D. (1994). Stochastic Context-Free Grammars for trna Modeling. Nuc. Acids Res. [11] Liang, P., Petrov. S., Jordan., M. I. & Klein. D. (2007) The infinite PCFG using hierarchical Dirichlet processes. In Proceedings of the 2007 joint Conference on Empirical Methods in Natural language Processing and Computational Natural Language Learning. 213

227 2009 Technical Report on Information-Based Induction Sciences 2009 (IBIS2009) VC Wishart VC Theory and a Concentration Inequality for Sums of Eigenvalues of Wishart Matrix Yasutaka Uwano Yohji Akama Abstract: Let d-dimensional column vectors x 1,..., x n be an i.i.d. sample drawn from the d-dimensional standard normal distribution. Let S be n i=1 x ix i /n. The left and the right tail probabilities for the sum of any k eigenvalues of S is uniformly evaluated non-asymptotically from above, by using upper bound of the VC dimensions of principal component analysis and by using a Vapnik s theorem of generalization errors in empirical risk minimization. For the right tail probability, we represent a subspace with the kernel of a linear mapping and then employ a concentration inequality for the chi square distributions. 1 Introduction Let x 1,..., x n be independently distributed, each subject to d-dimensional normal distribution N(0, Σ). Then the distribution of n i=1 x ix i is defined to be Wishart distribution, denoted by W (Σ, n). If Σ = E d, the identity matrix E d of size d, then so-called data covariance matrix 1 n n i=1 x ix i of the sample is subject to W (E d /n, n). Johnstone [6] proved that for a matrix subject to W (E d, n), if the largest eigenvalue is appropriately centered and scaled, then the distribution approaches to the Tracy-Widom law of order 1, as n, d goes to infinity with n/d fixed γ 1. On the other hand, for the data covariance matrix S = 1 n n i=1 x ix i, as n with d being fixed, the sum of any k eigenvalues of S tends to k almost surely, because the law of large numbers guarantees that S converges to the identity matrix E d almost surely. In terms of principal component analy-, , sa8m07@math.tohoku.ac.jp, Mathematical Institute, Tohoku University Sendai Miyagi JAPAN, sis (PCA), the sum of the largest k eigenvalues of S is the sum of variances of principal component and the sum of the square distances of data and the approximate affine subspace. Below, the left and the right tail probabilities for the sum of any k eigenvalues of the data covariance matrix S is uniformly evaluated non-asymptotically from above, by using upper bound of the VC dimensions of principal component analysis and by using a theorem [9, (5.43)] of Vapnik s statistical learning theory. For the right tail probability, we represent a subspace with the kernel of a linear mapping and then employ a concentration inequality [7, (5.1)] for the chi square distributions. Johnstone s result [6] is satisfied if the number n of observations and the dimensions p are large enough with n/d fixed. On the other hand, our results are useful in the case that n/d = Ω(n 1 2 +ϵ ), i.e. d/n = O(n ( 1 +ϵ) 2 ), where ϵ is any positive number, especially, in the case that d is fixed and n is large. For the largest eigenvalue λ 1 of a symmetric random matrix whose entries are independent random vari- 214

228 ables with absolute value bounded by 1, the sub-gaussian evaluation of the right tail of λ 1 is derived in [1] from Talagrand s inequality. Our result is for Gaussian random variables, which vary from to. This paper is organized as follows: In the next section, we review VC theory. In Section 3, we relate the sum of eigenvalues of the data covariance matrix to the statistical learning that formulates PCA. In Section 4, we provide an upper bound of VC dimension of the PCA. In Section 5, we present a concentration inequality for the sum of eigenvalues of the data covariance matrix for an i.i.d. sample from the multi-dimensional standard normal distribution. In the final section, we mention future work, which are hopefully related to concentration inequality and model selection. 2 VC theory In the framework of statistical learning theory [9], a learning model in general consists of (i) an unknown distribution F (z) of training data z drawn from a space Z, (ii) a class Λ of hypotheses, (iii) a loss function Q : Z Λ R. Here Q(z, α) stands for the loss of training data z against a hypothesis α. The risk of α Λ is defined to be R(α) = E z [Q(z, α)]. The goal of learning is to estimate α 0 Λ s.t. R(α 0 ) = min α Λ R(α) from training data z 1,, z n which are independently drawn from the distribution F. Proposition 1 ([9, (5.43)]). Let {Q(x, α) : α Λ} be any class of unbounded class of nonnegative, functions. Then for any α Λ, with probability greater than or equal to 1 η, it holds that R(α) R emp (α) < R(α)τ(p) Here p > 2 is such that ( 1 2 E x [Q(x, α) p ] 1/p sup α Λ E x [Q(x, α)] ( ) ) p 1 1/p p 1 ε. p 2 < τ(p), and {( ) } GΛ (2n) η := 4 exp ε2 n. n 4 G Λ (n) is the so-called growth function for Λ, and = n log 2 (n v) G Λ (n) v(log n v + 1) (n > v), where v is the VC dimension of the class of {x Z : Q(x, α) r} such that α Λ and r R. Let C be a nonempty class of subset of Z. We say a finite subset X of Z is shattered by C, if {X C : C C} is the class of subsets of X. By the VC dimension of the class C, we mean the supremum of the cardinality of a set X Z shattered by C. Important properties on the VC dimension in the study of empirical process(=statistical learning) are found in [5]. 3 Eigenvalues of data covariance matrix and empirical risks First, we relate the eigenvalues of the data covariance matrix S, to the empirical risk of a statistical learning. Put Λ to be the set of d k real matrices T such that T T = E k. We represent a (d k)-dimensional subspace H by any T Λ such that H = ker T. For any x R d and any T Λ, we define a loss function Q(x, T ) to be dist(x, ker T ) 2 = T x 2. On the other hand, we represent a k-dimensional subspace K by any T Λ such that K = Im T. For any x R d and any T Λ, we define another loss function Q (x, T ) to be dist(x, Im T ) 2. The empirical risks caused by T Λ are R emp (T ) = 1 n R emp (T ) = 1 n = 1 n n T x i 2, i=1 n x i 2 R emp (T ) i=1 n x i (E d T T )x i. i=1 If T consists of the k orthonormal eigenvectors of the data covariance matrix S, then R emp (T ) is the sum X 215

229 of the k corresponding eigenvalues λ 1,..., λ k of S and R emp (T ) is 1 n n i=1 x i 2 X. of semi-algebraically connected components of the realizations of all realizable sign conditions of P over Q. We write b i (d, m, L, s) for the maximum of b i (Q, P) 4 VC dimension of PCA formulated as statistical learning over all Q, P where Q and P are finite subsets of R[x 1,..., x m ], whose elements have degree at most d 1, the cardinality of P is s, and the algebraic set {x Put Ck d be the class of {x Rd : dist(x, H) < r} such R m : g(x) = 0 for all g Q} has real dimension L. that H is any k-dimensional affine subspace and the r is any positive real number. Theorem 1. There exists a positive constant c such that the VC dimension of C d k k + 1). is less than c(k + 1)(d Corollary 1. Let D d k denote the class of {x Rd : dist(x, H) < r} such that H is any k-dimensional subspace and the r is any positive real number. Then the VC dimension of D d k is less than c(k + 1)(d k + 1), where c is an absolute positive constant. The proof of the theorem uses a fact that any linear subspace is represented as a kernel and an image, as well as rather a standard evaluation of the number of sign sequences arising from an algebraic variety. We prove this proposition by following Basu-Pollack- Roy s argument [2]: For an element a R, 0 if a = 0, sgn(a) := 1 if a > 0, 1 if a < 0. Let Q and P be finite subsets of R[x 1,..., x m ]. A sign condition on P is an element of {0, 1, 1} P. The realization of the sign condition σ over Q, R(σ, Q), is the real semi-algebraic set {x R m : g(x) = 0 for all g Q, and sgn (P (x)) = σ (P ) for all P P}. Let b i (σ, Q) denote the i-th Betti number of R(σ, Q), i.e., the dimension of the i-th singular homology group of R(σ, Q) as a Q vector space, and let b i (Q, P) = σ b i (σ, Q). Especially, b 0 (σ, Q) is the total number Proposition 2 ([2]). L i ( ) s b i (d, m, L, s) d(2d 1) m 1 4 j. j j=0 Let (C d k ) be the class of open sets {x R d : dist(x, H) < r} C d k such that the k-dimensional affine subspace H intersects with the (d k)-dimensional subspace x 1 = = x k = 0 at exactly one point. Note that VCdim(C d k ) = VCdim((Cd k ) ), because if C d k shatters a finite set then (C d k ) does the set by appropriate perturbation. Lemma 1. Let L = (k+1)(d k)+1. Then, there exist a positive integer m 2L, an L-dimensional smooth submanifold V in R m defined by m L quadratic equations in m variables, and Φ: V Ck d with the following properties: (a) VCdim(Ck d ) = VCdim(Φ(V )); and (b) for each p R d, there exists a quadratic m- variate real polynomial f p such that for all x V, f p (x) > 0 if p is in Φ(x), while f p (x) < 0 if p is not in the closure of Φ(x). Proof. First, we consider the case k d/2. Let m = (d k)(d + 1) + 1. Then it is indeed m 2L. For (F, b, r) R m where F is a d (d k) real matrix, b R d k and r R, we consider a system of (d k) 2 = m L quadratic equations F ui F uj δ ij = 0 (1 i j d k), u F i+k, j = 0 (1 i < j d k). This defines an L-dimensional smooth submanifold V of R m by the implicit function theorem. 216

230 For (F, b, r) V where F R d (d k), b R d k and r R, define Φ(F, b, r) Ck d to be the set of points whose distance from a k-dimensional affine space { z R d : (F )z = b } (1) is less than r. Then Φ satisfies the property (a), since Φ(V ) Ck d contains (Cd k ). Moreover, for p R d, define f p by f p (F, b, r) = r 2 (F )p b 2. This satisfies the property (b), because (F )p b 2 is equal to the square of the distance from p to the affine subspace (1). Next we consider the case k < d/2. Let m = dk + d + 1. Then it is indeed m 2L. For (E, t, r) R m where E is a d k real matrix, t R d and r R, we consider a system of k + k 2 = m L quadratic equations, consisting of k equations t u E uj = 0 (1 j k) (2) u and k 2 equations E ui E uj δ ij = 0 (1 i j k), u E ij = 0 (1 i < j k). The system defines an L-dimensional smooth submanifold V of R m, by the implicit function theorem. For any (E, t, r) V with E R d k, t R d, r R, define Φ(E, t, r) to be the set of points whose distance from { Ex + t : x R k } (3) is less than r. Then Φ satisfies the property (a), since Φ(V ) contains (Ck d). Moreover, for p R d, define f p by f p (E, t, r) = r 2 p t 2 + (p )E 2. Then f p is clearly quadratic. By (2), we have p t 2 (p )E 2 = p t 2 (p t) E 2, which is equal to the square of the distance from p to the affine subspace (3). Thus we have the property (b). Now we will complete the proof of the upper bound. Proof of Theorem 1. Let m, L, V, Φ be as in the previous lemma. Take a set Q consisting of quadratic m- variate real polynomials g 1,..., g m L so that equations g 1 = = g m L = 0 define V. Let {p 1,..., p s } R d be a set shattered by Ck d. By (a) of the previous lemma, it is shattered by Φ(V ). If s m, then because the previous lemma implies m 2L, we have s m 2L as desired. If s > m, then put P := {f p1,..., f ps }. Because {p 1,..., p s } is shattered, 2 s #{σ { 1, 1} P : R(σ, Q) }. Then 2 s b 0 (Q, P) b 0 (2, m, L, s) by the definition. From Proposition 2, we have 2 s d(2d 1) m 1 L j=0 4j( s j) which is less than or equal to ) 36 L( 2 3 2L 1 4 L L ( ) s L. es j=0 j L This gives 2 s/l 36e(s/L), or s/l c where c is large enough. 5 The concentration inequalities Let x 1,..., x n be an i.i.d. sample drawn from N(0, E d ) and let λ 1,..., λ k be eigenvalues of the data covariance S = 1 n n i=1 x ix i. Let T Rd k consist of the corresponding orthonormal eigenvectors, as in Section 3. For the loss functions given there, the risks caused by T are R(T ) = E[ T x 2 ] and R (T ) = E[x (E d T T )x]. Because T T is an orthogonal projection of rank k, the loss functions are random variables subject to chi square distributions: T x 2 χ 2 k, x 2 T x 2 χ 2 d k, (4) where χ 2 m is the chi square distribution with degree m of freedom. So R(T ) = k and R (T ) = d k. By this and the last paragraph of Section 3, and k (λ λ k ) = R(T ) R emp (T ), (5) (λ λ k ) k = R (T ) R emp (T ) + ( 1 n ) n x i 2 d. (6) i=1 By applying Proposition 1 to R and R, we have inequalities for left and right tail probabilities of the sum of any k eigenvalues of S. But for the last term in the 217

231 inequality (6), we use a following inequality [7, (5.1)] for right tail probability of the chi square distribution: ( ( ) ) 2 P Y d + 2y e y (Y χ 2 d, y > 0), (7) which is proved by using Gaussian logarithmic Sobolev inequality [7, Theorem 3.4]. The p-th noncentral moment of the chi square distribution of degree k of freedom is written as m(k, p), which is k(k + 2)(k + 4) (k + 2p 2). Theorem 2. Let x 1,..., x n be an i.i.d. sample drawn from the d-dimensional standard normal distribution, λ 1,..., λ k (k d) be any eigenvalues of the data covariance d d matrix ( 1 n n i=1 x ) ix i, p > 2, ε > 0 and δ > 0. Then, the left tail probability of k i=1 λ i satisfies the following: ( k ( ) ) p 1 1/p m(k, p) p 1 P k λ i ε 2 p 2 i=1 {( GD d d k 4 exp (2n) ) } ε2 n. n 4 In particular, if n > v/2 with v being c(k + 1)(d k + 1) where c is an absolute positive constant, then the inequalities can be made concrete by replacing the two growth functions G D d d k (2n) and G D d k (2n) in the inequalities with v(log 2n v + 1). Proof. As for the left tail probability, in Proposition 1, as the loss function Q(T, x) is subject to χ 2 k by (4), we can take τ(p) = m(k, p) 1/p /k and thus k k i=1 λ i, which is R(T ) R emp (T ) by (5), exceeds ( ( ) ) p 1 1/p m(k, p) p 1 ε, 2 p 2 with probability at most (( ) ) GD d d k 4 exp (2n) ε2 n. (8) n 4 As for the right tail probability, in Proposition 1, as the loss function Q (T, x) is subject to χ 2 d k by (4), we can take τ(p) = m(d k, p) 1/p /(d k) and thus ( k ) i=1 λ i k ( 1 n n i=1 x i 2 d ), which is R (T ) R emp (T ) by (6), exceeds a := ε ( m(d k, p) 2 ( ) ) p 1 1/p p 1, p 2 (( ) ) with probability at most 4 exp G D d k (2n)/n ε 2 /4 n. But by taking Y = n i=1 x i 2 χ ( 2 nd and y = nd ) 2 + nδ nd /2 in (7), we have 1 n n i=1 x i 2 d δ with probability at most ) 2 b := exp ( 1 nd + δd 2 1. Therefore either k i=1 λ i k a or 1 n n i=1 x i 2 d δ holds with probability at most The right tail probability of k i=1 λ ( i satisfies the following: n k ( ) ) 1 n P λ i k x i 2 d a i=1 i=1 ( k ( ) ) ( ) p 1 1/p 1 n P m(d k, p) p 1 λ i k ε + δ + P x i 2 d δ. n 2 p 2 i=1 i=1 {( GD d k 4 exp (2n) ) } But the former summand is less than or equal to (8) ε2 n n 4 with k replaced by d k, while the latter is less than ( ) 2 + exp 1 2 nd 1 + δ or equal to b. d Future work: concentration and model selection Some mathematicians may be interested in how our approach is related to (1) papers of the local theory of Banach spaces on concentration of measure that is directly relevant (e.g. [8]), and (2) to the work on Talagrand s work on concentration of measure. Talagrand s inequalities for concentration of measure are recently employed in [7, Chapter 8], for statistical learning problems with the class of loss functions being uniformly bounded, as follows:

232 1. Bousquet s version of Talagrand s concentration inequality for empirical process is used to derive a new general upper bound of the difference between the expected risk and the empirical risk. 2. A concentration inequality is used to analyze Vapnik s structural risk minimization [9], a model selection method in terms of VC dimensions. PCA has the unbounded class of loss functions x R d dist(x, H) 2 where H is any k-dimensional affine subspace. We hope similar concentration inequalities which improves (1) previous Theorem for PCA and (2) model selection (i.e., selecting k) for PCA. This research is encouraged by a researcher who majors in concentration inequality and/or consistency of principal component analysis. We read in [7], Since the impressive works of Talagrand, concentration inequalities have been recognized as fundamental tools in several domains such as geometry of Banach spaces or random combinatorics. They also turn out to be essential tools to develop a nonasymptotic theory in statistics, exactly as the central limit theorem and large deviations are known to play a central part in the asymptotic theory. An overview of a non-asymptotic theory for model selection is given here and some selected applications to variable selection, change points detection and statistical learning are discussed.... We hope our work is connected to such applications and so on. References [1] Noga Alon, Michael Krivelevich, and Van H. Vu. On the concentration of eigenvalues of random symmetric matrices. Israel J. Math., Vol. 131, pp , [2] Saugata Basu, Richard Pollack, and Marie- Françoise Roy. On the Betti numbers of sign conditions. Proc. Amer. Math. Soc., Vol. 133, No. 4, pp (electronic), [3] Saugata Basu, Richard Pollack, and Marie- Françoise Roy. An asymptotically tight bound on the number of connected components of realizable sign conditions. To appear in Combinatorica. [4] Anselm Blumer, Andrzej Ehrenfeucht, David Haussler, and Manfred K. Warmuth, Learnability and the Vapnik-Chervonenkis dimension, J. Assoc. Comput. Mach. 36 (1989), MR MR (91f:68178) [5] R. M. Dudley, Uniform central limit theorems, Cambridge Studies in Advanced Mathematics, vol. 63, Cambridge University Press, MR MR (2000k:60040) [6] Iain M. Johnstone. On the distribution of the largest eigenvalue in principal components analysis. Ann. Statist., Vol. 29, No. 2, pp , [7] Pascal Massart. Concentration inequalities and model selection, Vol of Lecture Notes in Mathematics. Springer, Berlin, Lectures from the 33rd Summer School on Probability Theory held in Saint-Flour, July 6 23, 2003, With a foreword by Jean Picard. [8] S. Mendelson and R. Vershynin. Entropy and the combinatorial dimension. Invent. Math., Vol. 152, No. 1, pp , [9] Vladimir N. Vapnik, Statistical learning theory, Adaptive and Learning Systems for Signal Processing, Communications, and Control, John Wiley & Sons Inc., New York, NY, MR MR (99h:62052) 219

233 2009 Technical Report on Information-Based Induction Sciences 2009 (IBIS2009) 1 Singularities of One-dimensional Linear Dynamical Systems and its Effect on the Bayesian Generalization Error Takuto Naito Keisuke Yamazaki Abstract: Linear dynamical systems are widely used in such fields as system control and time-dependent data analysis. Such a system can be regarded as a statistical parametric model, where the coefficients of the state space equations are unknown and given as parameters. The properties of parameter learning have not yet been established, in spite of a wide range of applications. Therefore, this paper investigates the system from the viewpoint of learning theory. It is revealed that the system has singularities in the parameter space. The generalization error measured by the prediction accuracy for unseen data sequences is reduced, due to the presence of these singularities. Keywords: Kalman Filter, Bayesian Learning, Time-Series Data Analysis 1 Introduction Linear dynamical systems are widely used for modeling practical complex systems with hidden variables such as object tracking in image processing [4], and position detection in car navigation systems [6]. The system is described via state space equations containing both observable and hidden variables. The Kalman filter [5] is an algorithm to estimate the hidden variables from coefficients given preliminarily. It is important to be able to estimate coefficients using the observable data when the coefficients are unknown. The system is regarded as a parametric learning model, in which the coefficients correspond to parameters. As seen in Section 2, the system is expressed as a generative probability model of the data because the process and observation noises are taken into account.,, R2-5, tel , naitaku@cs.pi.titech.ac.jp, Department of Computational Intelligence and Systems Science, Tokyo Institute of Technology, R2-5, 4259 Nagatsuta, Midori- Ku, Yokohama, Kanagawa,, R2-5, tel , k-yam@pi.titech.ac.jp, Precision and Intelligence Laboratory, Tokyo Institute of Technology, R2-5, 4259 Nagatsuta, Midori-Ku, Yokohama, Kanagawa Parametric models generally fall into two types, regular and singular. If the relation between the parameter and the expressed probability function is one-toone, a model is referred to as regular. Otherwise, it is singular. Therefore, a singular model has a set of parameters indicating the same function, in which there are singularities. Because of the singularities, conventional analysis is not applicable; model selection criteria for regular models such as AIC [1] and BIC [7] are inappropriate. An algebraic geometrical method has been developed for Bayesian learning to reveal the asymptotic generalization error and the marginal likelihood for several singular models [8]. According to its application to several models, the presence of singularities results in unique properties of the learning process [3, 9]. In spite of a wide range of applications, properties of a linear dynamical system are still unknown in terms of a learning model. Therefore, the present paper investigates such a system both theoretically and experimentally. We confirm that the system is a singular model and analyze the Bayesian generalization error based on the algebraic geometrical method. Here, the error is defined as the prediction accuracy for unseen time-sequence data. This prediction is different from 220

234 that of the conventional Kalman situation in which the primary concern is the set of hidden variables rather than the observable sequences. Nevertheless, our analysis can also provide an insight into hidden variable estimation. The remainder of the paper is organized as follows. Section 2 formulates the system. Section 3 introduces Bayesian learning and summarizes the algebraic geometrical method. Section 4 contains our main contributions, deriving a theoretical upper bound of the generalization error and showing experimental results for the error. Section 5 contains a discussion and our conclusions. 2 Linear Dynamical Systems Linear dynamical systems can be described by state space models with hidden state variables: z t+1 = Az t + Dw t, (1) x t = Cz t + v t, (2) where z t R q is the hidden state vector at time t, x t R p is an output vector, w t R q and v t R p are process and observation noises, respectively. These noises are assumed follow a standard normal distribution. A R q q is the state matrix, C R p q is the output matrix and the elements of D R q q are the coefficients of the process noise. The Kalman filter is known as an efficient recursive filter that estimates hidden states from a series of outputs. In what follows, the notations ẑ n m and P n m represent the estimates of z at time n and its error covariance matrix, respectively, when observations from t = 1 to t = m are given. The Kalman filter has two phases: Predict and Update. The algorithms are described as follows: Predict ẑ t t 1 = Aẑ t 1 t 1 (3) P t t 1 = AP t 1 t 1 A + DD (4) Update K t = P t t 1 C ( I + CP t t 1 C ) 1 (5) ( ) ẑ t t = ẑ t t 1 + K t xt Cẑ t t 1 (6) P t t = (I K t C) P t t 1 (7) where I is a unit matrix and K t is called the Kalman gain. Firstly, the current state z t is estimated as ẑ t t 1 from the estimated state of the previous time t 1 (Eq.3). Then, a more refined value for ẑ t t is calculated on the basis of ẑ t t 1 after an observation x t is provided (Eq.6). From the viewpoint of machine learning, a linear dynamical system can be regarded as a learning model whose parameters are A, C, D and z 1. The variable z 1 indicates the initial state. Let X = (x 1, x 2,..., x T ) R p T be the vector of observations. The probability p(x w), where the parameters w = (A, C, D, z 1 ), can be calculated as follows: T p(x w) = p(x 1 w) p(x t x 1,..., x t 1, w). (8) t=2 Using the hidden state z t, p(x t x 1,..., x t 1, w) = p(x t z t, w)p(z t x 1,..., x t 1, w)dz t. (9) Let N ( µ, Σ) be a multivariate normal distribution with mean µ and covariance matrix Σ. By the definition of a linear dynamical system (Eq.2) and the derivation of the Kalman filter, p(x t z t, w) = N (x t Cz t, I), (10) p(z t x 1,..., x t 1, w) = N (z t ẑ t t 1, P t t 1 ). (11) Therefore, p(x t x 1,..., x t 1, w) is also a normal distribution described by p(x t x 1,..., x t 1, w) = N (x t Cẑ t t 1, I + CP t t 1 C ). (12) Eq.8 can be expressed as T p(x w) = N (x t Cẑ t t 1, I + CP t t 1 C ). (13) t=1 where we define ẑ 1 0 = z 1 and P 1 0 = 0. Let X n = (X 1, X 2,..., X n ) be a set of i.i.d. training samples. Each X i is a time sequence defined by X i = (x i 1, x i 2,..., x i T ). The likelihood of the parameter w = (A, C, D, z 1 ) can be calculated as n L(w) = p(x i w) = i=1 n i=1 t=1 T N (x i t Cẑt t 1 i, I + CP t t 1 i C ) (14) where ẑt t 1 i and P t t 1 i are evaluated using the Kalman filter. 221

235 3 Bayesian Learning and the Generalization Error This section describes Bayesian learning for time series data and the theoretical analysis of the generalization error. Let X n = (X 1, X 2,..., X n ) be a set of training samples taken independently and identically from the true distribution q(x), where n is the number of training samples. Each X i (i = 1,..., n) is a sequence whose length is T, i.e. X i = (x i 1,..., x i t,..., x i T ). Note that the sequence data X n are taken as i.i.d. whereas each sequence X i is not. Let p(x w) be a learning model, and ϕ(w) be an a priori probability distribution. The a posteriori probability distribution is defined by p(w X n 1 n ) = Z(X n ) ϕ(w) p(x i w) (15) i=1 where Z(X n ) is a normalizing constant. The Bayesian predictive distribution is defined by p(x X n ) = p(x w)p(w X n )dw. (16) The Bayesian generalization error G(n) is defined by G(n) = E X n[ q(x) log q(x) ] p(x X n ) dx, (17) which is the average Kullback information from the true distribution to the predictive distribution. The remainder of this section summarizes the algebraic geometrical method for deriving the asymptotic form of the error [8]. Let H(w) be the Kullback information from the true distribution q(x) to the learner p(x w), H(w) = q(x) log q(x) dx. (18) p(x w) The function ζ(z) of one complex variable z, defined by ζ(z) = H(w) z ϕ(w)dw, (19) is referred to as the zeta function. It is known that this zeta function is holomorphic in the region Re(z) > 0, and can be analytically continued to the meromorphic function on the entire complex plane. Then the poles are all real, negative and rational numbers. Let 0 > λ 1 > λ 2 >... be a sequence of poles, and m 1, m 2,... be the respective orders. The asymptotic form of the generalization error is expressed as G(n) = λ 1 n m 1 1 ( n log n + o 1 ) n log n (20) for n. In many cases, it is not straightforward to find the largest pole λ 1 and its order m 1 [3]. When a pole z = λ and its order m have been calculated, an upper bound is derived as G(n) λ n m 1 ( n log n + o 1 ). (21) n log n 4 Analysis of the Generalization Error This section analyzes the Bayesian generalization error for linear dynamical systems. In order to investigate the effect of redundant hidden states, we study an essential case, in which the learning model has a hidden variable and the true model generates i.i.d. sequences. This is the simplest setting for singularities to exist in the parameter space because the i.i.d. model can be regarded as a model with no hidden states. For simplicity, we assume that the output vector is one dimensional, where z t, x t, A, C, and D are all scalar. Moreover, we assume that the first hidden state is fixed as z 1 = 0. Formally, the learning model is defined as z t+1 = az t + dw t, (22) x t = cz t + v t, (23) where z t, x t R 1 and w t and v t are distributed from N ( 0, 1). The parameter is expressed as w = (a, c, d). The true model is a one-dimensional normal distribution N (x t 0, 1) for all t, i.e. x t = v t. Following Eq. 13, the true model is given by q(x) = T N (x t 0, 1). (24) t=1 4.1 Theoretical analysis Based on the algebraic geometrical method, the error has the following bound: Theorem 4.1 When the true model and a learning model are defined by Eq.24 and Eqs 22-23, respectively, the Bayesian generalization error is bounded above as follows: G(n) 1 2n 1 ( n log n + o 1 n log n ), (25) 222

236 where z 1 = 0 and the training sample size n is sufficiently large. Sketch of Proof: Because the parameter set {c = 0} attains p(x w) = q(x), there is a function f c (w) such that H(w) = c 2 f c (w). The set {d = 0} ensures the same property for H(w). Thus, there is a polynomial f(w) such that H(w) = c 2 d 2 f(w). We can find a limited support W of the parameter space, such that H(w) Cc 2 d 2. Here C is a positive constant. Considering the following zeta function ζ 1 (z) = {Cc 2 d 2 } z dw, (26) W the pole z = µ is a lower bound of z = λ 1 [8]. We can find a pole µ = 1/2 and its order m = 2. Combining with Eq. 21, we derive the following leading terms for the bound, which completes the proof. End of Proof 1 2n 1 n log n, (27) If the initial state is unknown and is regarded as a parameter such as w = (a, c, d, z 1 ), we can extend Theorem 4.1 as follows. Corollary 4.1 Under the same setting as Theorem 4.1, the error has an upper bound G(n) 1 ( 1 ) 2n + o. (28) n We omit the proof for lack of space. 4.2 Experimental results We experimentally evaluate whether the bound is valid when finite training data are given. Sampling from the a posteriori distribution, the predictive distribution is given by p(x X n ) 1 M M p(x w j ), (29) j=1 where (w 1,..., w M ) are sampled from p(w X n ). We use the Markov chain Monte Carlo (MCMC) method for the sampling technique [2]. The generalization error is approximated by [ 1 G(n) E X n N N q(x i ) ] log p(x i X n. (30) ) i=1 The experimental settings are as follows. The length of the time sequence is T = 10. The number of test data sequences is N = 1, 000. The number in the MCMC sample is M = 500. We obtain the expectation E X n[ ] over 100 sets of training data. The a priori distribution is a normal distribution for a, c and d. Figure 1-(a) describes an example of sampling from the a posteriori distribution in the parameter space (a, c, d). The vertical and horizontal planes indicate {c = 0} and {d = 0}, respectively. The points are located around the subspace {c = 0} {d = 0}, for which the parameters express the true model. Figure 1-(b) summarizes the error values corresponding to n = 250, 500, 750 and 1, 000. The horizontal and vertical axes describe the number of training data sequences and the error value, respectively. The heavy line depicts experimental values for G(n). The dotted line is the upper bound of Theorem 4.1. The upper bound is valid as seen in the graph. 5 Discussions and Conclusions First, let us discuss the upper bound of the generalization error. In the regular case, the error has the following asymptotic form, G(n) = dim w ( 2n + o 1 ), (31) n log n which means that λ 1 = dim w/2 and m 1 = 1. Note that even a singular model has this asymptotic form if the true and learning models have the same dimension of the hidden state vector. The asymptotic form indicates that the cost to fit all parameters determines the error as the dimension dim w appears. Comparing Theorem 4.1 with the regular case, we can derive the result that the error is much smaller, i.e. G(n) 1 2n 1 ( n log n + o < 3 ( 2n + o 1 n log n 1 ) n log n ), (32) which confirms that the fitting cost for redundant parameters is not strongly reflected in the error. Thus far, we have focused on prediction of the unseen observable data sequence X. Next, we consider estimation of the hidden states z t. According to the a posteriori distribution, there are two regions for the optimal parameters; one is around c = 0 and the other 223

237 3 2 1 d x 10 3 Result G(n) 1/(2n) 1/(n log n) a c (a) (b) Figure 1: An example of the a posteriori distribution and the generalization error. is around d = 0. They imply completely different behaviors of the hidden state. The former, c = 0, indicates that a and d can take any value, by which q(x) = p(x w). Thus, there are no constraints on the movement of the hidden state. By taking into account z 1 = 0, the latter, d = 0, contrarily implies that there is no movement because z t = 0 for all times t. If several hidden variables in the true model stop moving due to disorder in a practical situation, the desired estimation is d = 0. However, c = 0 can also be an estimated result; these variables move on the basis of arbitrarily-estimated a and d. This adverse estimation can occur along any dimension of the hidden state vector. Therefore, detection of hidden variable size is an essential problem to solve. Finally, we state our conclusions. The present paper establishes that linear dynamical systems are singular models. The singularities ensure that the upper bound of the Bayesian generalization error is small. The experimental results indicate that the bound is valid. Moreover, the a posteriori distribution implies that estimation of hidden states cannot be appropriate if there are redundant hidden variables. Acknowledgment This research was partially supported by the Ministry of Education, Science, Sports and Culture in Japan, Grant-in-Aid for Scientific Research References [1] H. Akaike. A new look at the statistical model identification. IEEE Trans. on Automatic Control, Vol. 19, pp , [2] Christophe Andrieu, Nando de Freitas, Arnaud Doucet, and Michael I. Jordan. An introduction to MCMC for machine learning. Machine Learning, Vol. 50, No. 1-2, pp. 5 43, [3] Miki Aoyagi and Sumio Watanabe. Stochastic complexities of reduced rank regression in Bayesian estimation. Neural Networks, Vol. 18, pp , [4] N. Funk. A study of the Kalman filter applied to visual tracking. Technical Report Project for CM- PUT 652, University of Alberta, [5] R. E. Kalman. A new approach to linear filtering and prediction problems. J. Basic Engineering, Vol. 82, pp , [6] D. Obradovic, H. Lenz, and M. Schupfner. Sensor fusion in siemens car navigation system. Proc. of MLSP 2004, pp , [7] G. Schwarz. Estimating the dimension of a model. Annals of Statistics, Vol. 6 (2), pp , [8] Sumio Watanabe. Algebraic analysis for nonidentifiable learning machines. Neural Computation, Vol. 13, No. 4, pp , [9] Keisuke Yamazaki and Sumio Watanabe. Algebraic geometry and stochastic complexity of hidden Markov models. Neurocomputing, Vol. 69, No. 1-3, pp , dec

238 2009 Technical Report on Information-Based Induction Sciences 2009 (IBIS2009) An Analysis of Transfer Learning between a Pair of Datasets with Different Qualities Shotaro Akaho Toshihiro Kamishima Abstract: It is often the case that we have a pair of datasets, where one consists of few high-quality data and the other consists of many low-quality data. Kamishima et al. has proposed a bagging-based learning algorithm to synthesize such a pair of data sets. However, it has not been clear when the learning algorithm improves the estimation obtained by using only the few high-quality dataset. We analyze a simple exponential family model, and prove that the bagging-based learning algorithm does not work for this model, but an appropriate modification improves the estimation drastically. Keywords: transfer learning, asymptotic analysis, statistical estimation, exponential family 1 [1], , tel , s.akaho@aist.go.jp, mail@kamishima.net The National Institute of Advanced Industry and Scientific Technology, Central 2, Umezono, Tsukuba, Ibaraki [6] [1] 2 x p T (x) T =(x 1,x 2,..., x T ) p T (x) 225

239 p N (x) p S (x) =αp T (x)+(1 α)p N (x) (1) ˆη T D ˆηTrBagg S = (x 1,x 2,...,x S ) T T S S α 1 x S, T p T (x) TrBagg () 1. T ˆp T (x) 2. S B 3. B ˆp B (x) 4. ˆp T (x) ˆp B (x) ˆp B (x) ˆp TrBagg (x) 1 ˆp TrBagg (x) T ˆp T (x) p T (x) S 1 TrBagg B η T αη T +(1 α)η N 1: TrBagg 3 TrBagg p T (x),p N (x) α 1 TrBagg ˆp T (x) 1/2 ( 1 ) p(x; θ) = exp(θ r(x) ψ(θ)+c(x)) (2) η =E p(x;θ) [r(x)] p T (x) p N (x) η T, η N x r(x) η η r(x) () T η T ˆη T η T S B p T (x) B T p N (x) B N B B k p T (x) k k = B α 2 B T η η BT η T 2 η N 226

240 B N η η BN η N B η η B k = B α η B = αη T +(1 α)η N, (3) p T (x) p N (x) S TrBagg p T (x) T ˆη T ˆη T D η B N[η B ] D D ˆη TrBagg = ηn[η B]dη D N[η (4) B]dη ˆη T η B ˆη T η B η T η B (3) η T η N α 1 T ˆη T η T ˆη T η T ˆη TrBagg ˆη T η T TrBagg p N (x) ˆp T (x) 1. T η ˆη T bε α σ T ˆμ T μ T ε μ N 1 α σ N 2: α = aε σ T = cε ˆμ T = μ T + bε d = μ T μ N 2. S r(x) ˆη T 3. r(x) ˆη Filter p T (x) r(x) p T (x) 4.2 ( 2) p T (x) N [μ T,σT 2 ] p N (x) N [μ N,σN 2 ] μ T T ˆμ T S ˆμ T ε ˆμ Filter μ T S ε ε ˆμ Filter ε ˆμ T μ T b ˆμ T = μ T + bε (5) 227

241 σ T = cε (6) ε ˆμ T μ T bε σ T T T b 0 [ ] c E[b 2 2 ] E (7) T T b<c S α α = aε (8) S ˆμ Filter = ˆμT +ε ˆμ T ε xp S (x)dx (9) ε ˆμ Filter = μ T + γbε + O(ε 2 ) (10) γ < 1 ˆμ Filter ˆμ T γ γ = φ(d, σ2 N )+ c 2 a φ(d, σn 2 )+ a 4 {erf( 2(b+1) 2c 4b {φ(b 1,c2 ) φ(b +1,c 2 )} ) erf( 2(b 1) 2c )} (11) φ(μ, v) μ, v φ(μ, v) = 1 ) exp ( μ2 (12) 2πv 2v erf erf(x) = 2 x exp( t 2 )dt (13) π d 0 d = μ N μ T (14) b 1 b Taylor γ = φ(d, σ2 N )+aφ(1,c2 ) φ(d, σ 2 N )+ a 2 erf( 2 2c )} + O(b2 ) (15) b 1 0 d σ N ( )c ( 1/3 ) 0 a/2 γ 2 φ(d, σ2 N ) a +2φ(1,c 2 ) (16) σ N p T (x) c γ 1 ˆμ T TrBagg TrBagg (p N (x) ) 4.3 μ ˆμ T R(μ) = log μ μ T ˆμ T μ T (17) 0 ˆμ T T 10 S 1000 ( p T (x) 100, p N (x) 900) μ T =0.0, σ 2 T =0.12 μ N =1.0, σ 2 N =

242 cut bag : R TrBagg ε =0.1 4: σ T R 100 TrBagg TrBagg TrBagg R 0 R ( 3) σ T (16) σ T c σ T 3 σ T k, k =0, 1, 2,...,5 σ T σ T ε ε k, k =0, 1,...,5 ε ε 3 ε : (ε) R σ T ε ε p N (x) α = aε S p T (x) 6 S = 1000 p T (x) 10 2 k, k =0, 1, 2,...,5 α S α α ε 229

243 [1],, :, ( 22 ), 2D1-3, [2],, :, 2008, p.88, : α R ( S p T (x) α S ) 5 p N (x) p T (x) p N (x) p N (x) p T (x) S p T (x) p T (x) p T (x) TrBagg [3] T. Kamishima, M. Hamasaki, S. Akaho: Baggtaming. learning from wild and tame data, in ECML/PKDD2008 Workshop: Wikis, Blogs, Bookmarking Tools. Mining the Web 2.0, [4] T. Kamishima, M. Hamasaki, S. Akaho: Personalized tag predition boosted by baggtaming.a case study of the hatena bookmark, in the 3rd Int. Workshop on Data-Mining and Statistical Science, [5] T. Kamishima, M. Hamasaki, S. Akaho: TrBagg: A Simple Transfer Learning Method and Its Application to Personalization in Collaborative Tagging, in Proc. IEEE Int. Conf. on Data Mining (ICDM2009), to appear. [6] S. J. Pan, Q. Yang: A Survey on Transfer Learning, Technical Report, Dept. of Computer Science and Engineering, Hong Kong Univ. of Science and Technology, HKUST-CS08-08,

244 2009 Technical Report on Information-Based Induction Sciences 2009 (IBIS2009) Supervised Dimensionality Reduction by Conditional Entropy Minimization Hideitsu Hino Noboru Murata Abstract:., Fisher Discriminant Analysis(FDA)., FDA,.,,.,.,. Keywords: dimensionality reduction, conditional entropy, classification, visualization 1,.,,,,,,.,,,., [1](Principal Component Analysis:PCA).,, Fisher [2](Fisher Discriminant Analysis:FDA). FDA,,,., FDA.,,, , tel , Waseda University, Ohkubo, Shinjuku, Tokyo , Japan FDA,, Local Fisher Discriminant Analysis(LFDA) [3]., FDA.,.,.,,, FDA.,. 2,,. 3,..,, FDA. 4, 231

245 , ( ). 5,,. 6,. 2 {x i } N i=1, x i R n, m f(x i ) = z i R m, (m < n)., f A z i = A T x i, A R n m. (1).,., (Shannon), X, H(X) = p(x) log p(x)dx (2). p X. PCA,.,, log p(x)., 1 N (x; µ, σ 2 ), p(x) log p(x)dx = 1 2 log(2πeσ2 ) (3), σ 2. PCA,,.,, x, FDA. FDA, {x i } N i=1 {y i } N i=1, y i {1, 2,..., C},. x R n (1), A T Σ b A A T Σ w A., Σ w = 1 N Σ b = 1 N C y=1 (x µ y )(x µ y ) T = x D y C N y (µ y µ)(µ y µ) T, y=1 C y=1 N y N Σ y,, D y y, µ y Σ y D y, µ., FDA, log A T Σ w A / A T Σ b A A.,., log,, A T Σ b A min A log AT Σ w A subject to A T Σ b A = const.. (4) Σ y = Σ c, y = 1,..., C, FDA Bayes. Σ w = Σ c, H(A T X Y ) = log(2π) m/2 e + C y=1 N y 2N log AT Σ y A = log(2π) m/2 e log AT Σ c A = log(2π) m/2 e log AT Σ w A FDA log A T Σ w A., PCA,, FDA,,.,. 3,, 232

246 . x z, I(X; Z) = H(Z) H(Z X) (5) [4]., y, y,, I(X; Z Y ) = H(Z Y ) H(Z X, Y ) (6)., f : x z H(Z X, Y ) 0, H(Z Y ),. H(Z Y ) f : x z., H(Z Y ) x f., f.,., ε, f D Ψ(f, D), εψ(f, D)., min H(Z Y ) + εψ(f, D) (7) f:x z. Ψ(f, D)., A R n m (7)., Gaussian.,, Leave- One-Out(LOO) [5]., D = {x i } N i=1, x ˆp(x; D, h) = 1 N N i=1 1 2πh 2 exp ( x x i 2 /2h 2) (8) 1.,. 1 f,,. h, Silverman s Rule of Thumb [6]. H(X) H(X) = E[log ˆp(X; D, h)] (9), ˆp(x; D, h) ˆp(x j ; D\{x j }, h), LOO H(X) Ĥ(X) = 1 N N log ˆp(x j ; D\{x j }, h) (10) j=1., ˆ.,.,,, z R m,, [4]., A l a l l H l (a T l X) = H l (Z l ) = p(z l ) log p(z l )dz l, H(Z) = m H l (Z l ) H(Z) (11) l=1, H l (a T l X) a l,., H(Z Y = y) = m l=1 H l(z l Y = y), ( ) H(Z Y ) H(Z Y ) = = C p(y)h(z Y = y) y=1 C m p(y) H l (Z l Y = y) (12) y=1 l=1.,. 3.1, A., A l a l, a T l x = 233

247 z l, l = 1, 2..., m, A. min A H(AT X Y ) + εψ(a, D) (13)., D = {x i } N i=1, y D y = {x j } Ny j=1, y = 1,..., C, H(A T X Y ) = C y=1 = N y N H(AT X Y = y) C y=1 1 N x j D y log ˆp(A T x j ; D y \{x j }, h). H(A T X) A l a l H(a T l X Y ) a l = 1 h 2 N C y=1 x j D y «x i D y \{x j } e at l x j at l x i 2 2h 2 x i D y \{x j } e v ji = (x j x i ) T a l (x j x i ) R n v ji «, 2h 2 at l x j at l x i 2,, (13) ,,. [7],, a T 1 x,..., a T mx., {x i } N i=1, A R n m, A T A I m F. I m m m, F., (13) Ψ(A, D) = Ψ(A) = A T A I m F., 2 : 1. A 3 2 A 1 2 AAT A. 2. A 1., A T A, E R m m A T A {d i } m i=1 D A T A = EDE T, step.1 A T A 1 4 (3A AAT A) T (3A AA T A) ( 9 = E 4 D 6 4 D2 + 1 ) 4 D3 E T. A d i [0, 1], A T A {h(d i )} m i=1 h(d i ) = 1 4 (9d i 6d 2 i + d3 i )., h(d i) d i = d i 4 {(d i 3) 2 4} 0 h(d i ) d i, A T A , 2 2, FDA. 2, FDA 1., A = a 2, FDA 1. FDA, H(a T X Y ) a. 2nd axis st axis LDA minh (a) Unimodal data. 2nd axis st axis LDA minh (b) Multimodal data. 1: Discriminative axes found by FDA and proposed method. (a), FDA., modality (b),. 4 H(Z Y ),., X [8]., J(Z) = H G (Z) H(Z), J(Z Y ) = H G (Z Y ) H(Z Y )., H G (Z), p(z)., z = A T x y 234

248 : I(Z; Y ) = H(Z) H(Z Y ) = {H G (A T X) H G (A T X Y )} {J(A T X) J(A T X Y )} = 1 2 log A T ΣA C y=1 AT Σ y A p(y) {J(A T X) J(A T X Y )}., Σ, Σ y, y, p(y)., H(Z Y ), H(Z Y ) = H(A T X) + {J(A T X) J(A T X Y )} 1 2 log A T ΣA C y=1 AT Σ y A p(y) = H G (A T X) J(A T X Y ) 1 2 log 3. A T ΣA C y=1 AT Σ y A p(y) (14) ,.,, (14)., (14),., H(Z Y ), (14)., FDA, C, FDA C 1, 3. UCI, Iris Soybeans., 4 3, (14) H G (A T X),.,,.,,., (),. 4.2 (a) Iris:FDA (c) Soybean:FDA (b) Iris:Proposal (d) Soybean:Proposal (), [8]. (14) J(A T X Y ),. (). 4.3 (14), (Heteroscedastic discriminant analysis;[9]), 2: Visualization result of multidimensional data., One-nearest-neighbor,, one-nearest-neighbor [10]., IDA 2. 1,,

249 2: Misclassification rate of linear methods. Data name min H(Z Y ) LFDA FDA PCA Euclidean banana 13.64(0.765)[2] 13.7(0.8) 38.34(3.966) 13.99(0.849)[2] 13.64(0.761) breast-cancer 33.90(4.704)[5] 34.7(4.3) 34.91(5.076) 40.71(7.085)[3] 32.73(4.824) diabetes 31.98(4.703)[7] 32.0(2.5) 31.32(2.813) 38.44(5.019)[4] 30.12(2.051) flare-solar 36.50(1.936)[4] 39.2(5.0) 36.42(1.875) 48.64(6.920)[5] 36.47(1.880) german 34.91(3.024)[7] 29.9(2.8) 32.03(2.577) 41.83(4.452)[2] 29.46(2.469) heart 27.68(3.909)[7] 21.9(3.7) 22.93(4.105) 46.27(23.894)[4] 23.16(3.735) image 5.72(1.712)[7] 3.2(0.8) 22.12(0.860) 37.33(9.546)[2] 3.381(0.540) ringnorm 20.25(1.303)[7] 21.1(1.3) 31.72(1.016) 28.04(5.075)[10] 35.03(1.362) splice 31.36(6.958)[2] 16.9(0.9) 20.35(0.783) 43.90(4.894)[2] 28.77(1.524) thyroid 4.674(2.535)[5] 4.6(2.6) 17.92(4.888) 9.05(4.366)[2] 4.36(2.210) titanic (1.107)[1] 33.1(11.9) 22.53(1.066) 26.41(8.392)[1] 22.50(1.057) twonorm 3.359(0.4241)[13] 3.5(0.4) 3.54(0.496) 7.55(18.770)[3] 6.68(0.718) waveform 24.76(3.167)[7] 12.5(1.0) 18.61(1.162) 31.69(18.714)[9] 15.83(0.654),, /. PCA 1: IDA data specifications. Data name dim train(test) data size set banana 2 400(4900) 100 breast-cancer 9 200(77) 100 diabetes 8 468(300) 100 flare-solar 9 666(400) 100 german (300) 100 heart (100) 100 image (1010) 20 ringnorm (7000) 100 splice (2175) 20 thyroid 5 140(75) 100 titanic 3 150(2051) 100 twonorm (7000) 100 waveform (1000) 100, 5, 5 fold cross validation. 2, one-nearestneighbor (% ). PCA D [D]., LFDA [3]., PCA, FDA. LFDA, titanic, waveform, splice,., (Euclidean) one-nearest-neighbor,,. 6,. FDA (Kernel Fisher Discriminant Analysis:KFDA) [11],.,,. 6.1 Kernel Fisher Discriminant Analysis 1, f(x) = a T x., a, 1. x Φ, f(x) = a T Φ(x)., a = N i=1 α iφ(x i ), Φ(x i ), Φ(x j ) = k(x i, x j ), f(x) = N α i k(x, x i ) (15) i=1. FDA KFDA, Σ b Σ w. K = [k(x i, x j )] ij R N N., 236

250 k y = 1 N y k = 1 N x i D y x i D ( k(x i, x 1 ),, k(x i, x N )) T R N, ( ) T k(x i, x 1 ),, k(x i, x N ) R N, V b V w V b = 1 N V w = 1 N C N y ( k y k)( k y k) T, y=1 C y=1 i D y (k i k y )(k i k y ) T. k i K i., KFDA log α T V w α / α T V b α, FDA α T V b α.,,,.,., V w + ζk. ζ., KFDA : min α log αt (V w +ζk)α subject to α T V b α = const. 6.2, KFDA. KFDA α = (α 1,..., α N ) T, f(x) = a T Φ(x) = N i=1 α ik(x, x i ) α.,,. KFDA α = α 0, KFDA, α., KFDA,,., KFDA α T 0 k y, y = 1,..., C, 1 N y x i D y N j=1 α jk(x i, x j ) = α T k y α T k 0 y., (7) Ψ(f, D) = C y=1 (αt k y α T k 0 y ) H(f(X) Y ) + ε C (α T ky α 0 ky ) 2 (16) y=1, KFDA α, α.,. (16) α, H(f(X) Y ) α = 1 h 2 N C y=1 x j D y α T (kj k i ) x 2 i D y \{x j } e 2h 2 v ji, x αt (k j k i ) 2 i D y\{x j} e 2h 2 v ji = (k j k i ) T α (k j k i ) R N, Ψ(f, D) = 2 ( ( k y ) T ky α α T 0 α k y k y). 6.3 One-nearest-neighbor KFDA, KFDA α 0 H(f(X) Y ) 3. KFDA Gaussian k(x j, x i ) = exp ( λ x j x i 2) (17), λ, 5 5 fold cross validation. KFDA 3., 2. KFDA α 0 (16),, 3 ringnorm, KFDA.,., KFDA cross validation. cross validation, KFDA,,.

251 3: KFDA. Misclassification rate by KFDA and modified Data name KFDA modified KFDA breast-cancer (4.9489) (5.1108) diabetes (2.2658) (2.2658) flare-solar (1.9819) (1.8159) german (2.2762) (2.4585) heart (3.7185) (3.7831) ringnorm 2.056(0.4546) 9.739(5.3771) waveform (0.7438) 11.61(0.7425) 7,. FDA.,.,, FDA PCA.,,, LFDA Locality Preserving Projection[12](LPP) Laplacian Eigenmap[13](LE). LFDA FDA,,. LPP LE,,,,.,,.,, LFDA., (),., 2,,.,,, [14], multiple kernel learning (MKL),., SVM MKL (e.g., [15]),, MKL. [1] K.Fukunaga. Introduction to statistical pattern recognition (2nd ed.). Academic Press Professional, Inc., San Diego, CA, USA, [2] R.A.Fisher. The use of multiple measurements in taxonomic problems. Annals Eugen., 7: , [3] M.Sugiyama. Dimensionality reduction of multimodal labeled data by local fisher discriminant analysis. J. Mach. Learn. Res., 8: , [4] T.M.Cover and J.A.Thomas. Elements of information theory. John Wiley and Sons, Inc., [5] J. Beirlant, E. J. Dudewicz, L. Györfi, and E. C. Meulen. Nonparametric entropy estimation: An overview. International Journal of the Mathematical Statistics Sciences, 6:17 39, [6] M. P. Wand and M. C. Jones. Kernel Smoothing. Chapman & Hall/CRC, December [7] A. Hyvärinen. Fast and robust fixed point algorithms for independent component analysis. IEEE Transactions on Neural Networks, 10(3): , [8] A.Hyvärinen, J.Karhunen, and E.Oja. Independent Component Analysis. J. Wiley, New York, [9] N. Kumar and A.G. Andreou. Heteroscedastic discriminant analysis and reduced rank HMMs for improved speech recognition. Speech Commun., 26(4): , [10] T. Cover and P. Hart. Nearest neighbor pattern classification. Information Theory, IEEE Transactions on, 13(1):21 27, [11] S. Mika, G. Rätsch, J. Weston, B. Schölkopf, and K. R. Müllers. Fisher discriminant analysis with kernels. In Neural Networks for Signal Processing IX, Proceedings of the 1999 IEEE Signal Processing Society Workshop, pages 41 48, [12] X. He and P. Niyogi. Locality preserving projections. In In Advances in Neural Information Processing Systems 16. MIT Press, [13] M. Belkin and P. Niyogi. Laplacian eigenmaps for dimensionality reduction and data representation. Neural Comput., 15(6): , [14] J. Shawe-Taylor and N. Cristianini. Kernel Methods for Pattern Analysis. Cambridge University Press, New York, NY, USA, [15] G. R. G. Lanckriet, N. Cristianini, P. Bartlett, L. E. Ghaoui, and M. I. Jordan. Learning the kernel matrix with semidefinite programming. J. Mach. Learn. Res., 5:27 72,

252 2009 Technical Report on Information-Based Induction Sciences 2009 (IBIS2009) Web Classification of Web page network by the methods of network science * * TEITO NAKAGAWA YASUHIRO SUZUKI Abstract: Recently, many kinds of network structure properties have been found in network science. But many studies of network science in existence have analyzed only a single network but not many networks to compare them and the relative property of networks is not clear. So, this study aims at collecting many web-page networks and comparing and classifying them by the methods of network science with less computational complexity and meaningful feature vector as against existing methods. As a result, the web-page networks were classified into two categories. A one of them has the structure like a complete graph and not scale-free. The other has scale-free tree like structure. Keywords: Complex Network, Link Mining, Graph Mining, SOM 1 Watts et al.[1]barabasi et al.[2] Web 2 2.1Web Web Web *, , tel nakagawa.teito@b.mbox.nagoya-u.ac.jp Page Rank[3] Web [4] [5] Web 2.2 [6]Web ,2,3 3 Web 239

253 2.3 Getoor and Diehl.[7] (Link Mining) (Graph Classification) {x1, x2, x3,., xn}{c1, C2, C3,,Ck} Kashima and Inokuchi.[8] Wilson et al.[9] Web Spam Challenge uk-07 uk Web index.html Web Web Web N M 1 N M 3.2 ( 1) 1. Web N-1 ( 2) 2. Web 0 ( 2) 2:()()() ( 3) 240

254 4. Newman-fast [10] Q Web ( 3) :()() ( 4) 6. jpeg mpeg 1 3 ( 4) Web :()()

255 (n=68) 15-1(n=85) () 15-1() ()() 7 KK [11] ()() Web [1] D.J.Watts.and S.H.Strogatz.: Collective dynamics of small-world networks, Nature, vol.393, pp , (1998) [2] A.L.Barabasi. and R.Albert.: Emergence of Scaling in Random Networks, Science, vol. 286, pp , (1999) [3] L Page, S Brin, R Motwani, and T Winograd.:The PageRank Citation Ranking: Bringing Order to the Web, Technical Report. Stanford InfoLab. (1999) [4] Reka Albert, Hawoong Jeong. And Albert-Lazlo Barabasi.:Diameter of the World-Wide Web, Nature vol.401, pp.130, (1999) [5] Eezsebet Ravasz and Albert-Lazlo Barabasi.:Hierarchical organization in complex networks, Physical Review, vol. E67, , (2003) [6] :, 5, (2009) [7] Lise Getoor. and Christopher P. Diehl.:Link Mining: A Survey, ACM SIGKDD Explorations Newsletter, vol.7, 2, pp. 3-12, (2005) [8] Hisashi Kashima. and Akihiro Inokuchi.:Kernels for Graph Classification, In ICDM Workshop on Active Mining, (2002) [9] Wilson, R.C., Hancock, E.R. and Bin Luo:Pattern vectors from algebraic graph theory, Pattern Analysis and Machine Intelligence, IEEE Transactions on, vol. 27, 7, pp [10] Aaron Clauset, M. E. J. Newman, and Crisopher Moore:Finding Community Structure in Very Large Networks, Physical Review, vol.e70, , (2004) [11] Tomihisa Kamada and Satoru Kawai. :An algorithm for Drawing General Undirected Graphs, Information Process Letters 31, 7-15, (1989) 242

256 2009 Technical Report on Information-Based Induction Sciences 2009 (IBIS2009) - Feature selection in chemical-protein binding activity space Satoshi Niijima Yasushi Okuno Abstract: In this paper, we address the issue of feature selection for chemical genomics. In particular, we propose an efficient feature selection algorithm for identifying chemical features that contribute to prediction of binding activity between chemicals and proteins. Notably, this algorithm allows feature selection in binding activity space, into which chemicals are mapped jointly with proteins by means of kernel methods. We apply the algorithm to a dataset on Cytochrome P450 (CYP), illustrating its capability of selecting a small subset of predictive features, which are also found to be indicative of CYP inhibitors. Although this study is directed toward the selection of chemical features within the context of chemical genomics, the proposed algorithm has the potential to find wide applications in real-world problems. Keywords: Kernel methods, feature selection, regression, chemical genomics (unpublished work) Hilbert-Schmidt Independence Criterion (HSIC) [6] [1, 15] HSIC - P450 [9] 2 -, , tel , niijima@pharm.kyoto-u.ac.jp, Graduate School of Pharmaceutical Sciences, Kyoto University, Yoshida Shimoadachi-cho, Sakyo-ku, Kyoto , , tel , okuno@pharm.kyoto-u.ac.jp, Graduate School of Pharmaceutical Sciences, Kyoto University, Yoshida Shimoadachi-cho, Sakyo-ku, Kyoto

257 explicit explicit c Φ(c) p Ψ(p) (c, p) Π(c, p) kernel-induced feature space Hilbert-Schmidt Independence Criterion (HSIC) Π(c, p) = Φ(c) Ψ(p) (1) HSIC BAHSIC Π(c, p) [6, 18] Φ(c) Ψ(p) Φ(c), Ψ(p) d c, d p explicit d c d p HSIC 0 x i y i n (x 1, y 1 ),..., (x n, y n ) HSIC x i, y i (i = 1,..., n) K, L Π(c, p) Π(c, p ) = ( Φ(c) Ψ(p) ) ( Φ(c ) Ψ(p ) ) IR n n 1 HSIC = Tr (KL) (4) = Φ(c) Φ(c ) Ψ(p) Ψ(p (n 1) 2 ) Tr K, L HSIC k chem (c, c ) Φ(c) Φ(c ), (2) - x i = k prot (p, p ) Ψ(p) Ψ(p ) (3) (c i, p i ) y i = 1, y i = 1 L ij = y i y j - K - k((c, p), (c, p )) Π(c, p) Π(c, p ) = k chem (c, c ) k prot (p, p ) explicit HSIC K = K chem K prot (5) (1) K chem, K prot IR n n (2), (3) 244

258 K (4), 1 (5) [4] HSIC K = (K chem K prot + λi n ) 1 (K chem K prot ) (6) 3.2 BAHSIC λ I n IR n n K chem K prot L (4) y i L ij = y i y j (4) HSIC Tr ( (K chem K prot + λi n ) 1 (K chem K prot ) L ) (7) BAHSIC leave-one-out HSIC HSIC (7) n n K chem K prot + λi n ( ) 1 (5) BAHSIC leave-one-out HSIC explicit (6) S ( ) ( ) (c1, p 1 ), y 1,..., (cn, p n ), y n L 1: L Repeat 2-4 until S = 2: I arg max I i I S HSIC (S \ {i}) 3: S S \ I 4: L L I leave-one-out HSIC (S \ {i}) S i HSIC leave-one-out BAHSIC (7) Tr ( (K chem K prot + λi n ) 1 L ) (K chem K prot + λi n ) 1 ( (Kchem f i f i ) K prot + λi n ) 1 (8) f i IR n i 4.1 HSIC P = K chem K prot + λi n IR n n HSIC Q = (f i g 1,..., f i g k ) IR n k HSIC HSIC K prot = GG, G = (g 1,..., g k ) IR n k (9) (8) Sherman-Morrison-Woodbury [5] K (5) P 1 + P 1 Q(I k Q P 1 Q) 1 Q P [18]

259 (9) k n i S n n k k I k Q P 1 Q 2 n = k 14 CYP [10] CYP [4] 3 (IC 50 ) 371CYP 5 DragonX [20] filter, wrapper, ( embedded [7] [18] [14, 23] HSIC [14, 23] 139 filter wrapper SVM- RFE [8] embedded 3 filter PROFEAT [13] RBF (Mismatch) [12] HSIC Kernel Target Alignment (KTA) [3] KTA (LA) [16] 2 2 HSIC [19] HSIC [22] (SVR) [21] (KRR) [17] [4] 6 14 n 798 P450 (CYP) RBF CYP 60 4 r 2 (n n i=1 = ŷiy i n n i=1 ŷi i=1 y i) 2 (n n CYP i=1 ŷ2 i ( n i=1 ŷi) 2 )(n n i=1 y2 i ( n i=1 y i) 2 ) ŷ i x i r 2 2 P 1 1 n = (9) 1 4 6:1 246

260 1 0.9 PROFEAT + RBF linear RBF Mismatch linear RBF LA linear RBF r r r Number of features Number of features Number of features 1: SVR 20 r SVR PROFEAT+RBF, Mismatch, LA CYP 20 RBF PROFEAT + RBF r 2-1, 2, 7, , RBF 0.60 CYP1A2 CYP3A4 [2, 11] 14 CYP CYP1A2, CYP3A4 34 % 8, 13 CYP3A [11] KRR r , RBF 0.58 SVR 247

261 1 0.9 PROFEAT + RBF linear RBF Mismatch linear RBF LA linear RBF r r r Number of features Number of features Number of features 7 2: KRR vances in Neural Information Processing Systems 14, , 2001 [1] J. Bajorath, Computational analysis of ligand relationships within target families, Curr Opin Chem Biol, 12, , 2008 [2] K. K. Chohan, S. W. Paine, J. Mistry, P. Barton, A. M. Davis, A Rapid Computational Filter for Cytochrome P450 1A2 Inhibition Potential of Compound Libraries, J Med Chem, 48, , 2005 [3] N. Cristianini, J. Shawe-Taylor, A. Elisseeff, J. S. Kandola, On Kernel-Target Alignment, Ad [4] K. Fukumizu, F. R. Bach, M. I. Jordan, Dimensionality Reduction for Supervised Learning with Reproducing Kernel Hilbert Spaces, J Mach Learn CYP Res, 5, 73 99, 2004 [5] G. H. Golub, C. F. Van Loan, Matrix Computations, 3rd edition, Johns Hopkins University Press, Baltimore, 1996 [6] A. Gretton, O. Bousquet, A. J. Smola, B. Schölkopf, Measuring statistical dependence with Hilbert-Schmidt norms, Proceedings of the Sixteenth International Conference on Algorithmic Learning Theory, 63 78, 2005 [7] I. Guyon, A. Elisseeff, An Introduction to Variable and Feature Selection, J Mach Learn Res, 3, , 2003 [8] I. Guyon, J. Weston, S. Barnhill, V. Vapnik, Gene Selection for Cancer Classification Using Support Vector Machines, Mach Learn, 46, , 2002 [9] L. Jacob, J.-P. Vert, Protein-ligand interaction prediction: an improved chemogenomics approach, Bioinformatics, 25, , 2008

262 [10] A. Kontijevskis, J. Komorowski, J. E. S. Wikberg, Generalized Proteochemometric Model of Multiple Cytochrome P450 Enzymes and Their Inhibitors, J Chem Inf Model, 48, , 2008 [11] J. M. Kriegl, T. Arnhold, B. Beck, T. Fox, Prediction of Human Cytochrome P450 Inhibition Using Support Vector Machines, QSAR Comb Sci, 24, , 2005 [12] C. S. Leslie, E. Eskin, A. Cohen, J. Weston, W. S. Noble, Mismatch string kernels for discriminative protein classification, Bioinformatics, 20, , 2004 [13] Z. R. Li, H. H. Lin, L. Y. Han, L. Jiang, X. Chen, Y. Z. Chen, PROFEAT: a web server for computing structural and physicochemical features of proteins and peptides from amino acid sequence, Nucleic Acids Res, 34, W32 W37, 2006 [14] S. Niijima, S. Kuhara, Gene subset selection in kernel-induced feature space, Pattern Recognit Lett, 27, , 2006 [15] D. Rognan, Chemogenomic approaches to rational drug design, Br J Pharmacol, 152, 38 52, 2007 [16] H. Saigo, J.-P. Vert, N. Ueda, T. Akutsu, Protein homology detection using string alignment kernels, Bioinformatics, 20, , 2004 [17] C. Saunders, A. Gammerman, V. Vovk, Ridge Regression Learning Algorithm in Dual Variables, Proceedings of the Fifteenth International Conference on Machine Learning, , 1998 [18] L. Song, J. Bedo, K. M. Borgwardt, A. Gretton, A. Smola, Gene selection via the BAHSIC family of algorithms, Bioinformatics, 23, i490 i498, 2007 [19] L. Song, A. Smola, A. Gretton, K. M. Borgwardt, J. Bedo, Supervised feature selection via dependence estimation, Proceedings of the Twenty-Fourth International Conference on Machine Learning, , 2007 [20] R. Todeschini, V. Consonni, M. Pavan, Dragon, Milano Chemometrics and QSAR Research Group, Milan, 2007 [21] V. N. Vapnik, Statistical Learning Theory, John Wiley & Sons, Inc., New York, 1998 [22] H. Xiong, M. N. S. Swamy, M. O. Ahmad, Optimizing the kernel in the empirical feature space, IEEE Trans Neural Netw, 16, , 2005 [23] L. Wang, Feature selection with kernel class separability, IEEE Trans Pattern Anal Mach Intell, 30, ,

263 2009 Technical Report on Information-Based Induction Sciences 2009 (IBIS2009) Large Geometric Margin Minimum Error Classification Hideyuki Watanabe Shigeru Katagiri Kouta Yamada Erik McDermott Atsushi Nakamura Shinji Watanabe Miho Ohsaki Abstract: The recent dramatic growth of computation power has resulted in increased interest in discriminative training methods for pattern recognition. Minimum Classification Error (MCE) training is especially attracting a great deal of attention, and it can be used to achieve minimum-error classification of various types of patterns. However, for increasing the robustness of classification, the conventional MCE framework has no practical optimization procedures like the geometric margin maximization in Support Vector Machine (SVM). To realize high robustness in a wide range of classification tasks, we derive the geometric margin for a general class of discriminant functions and develop a new MCE training method that increases the geometric margin value. We demonstrate the effectiveness of the new method by experiments using prototype-based classifiers and clarify relationships between the new method and such existing methods as SVM. Keywords: Minimum Classification Error, MCE, margin, geometric margin, robustness 1 [1, 2, 3] [4, 5, 6, 7, 8] MCE [4, 5] MCE MASTAR, , tel , hideyuki.watanabe@nict.go.jp Spoken Language Communication Group, MASTAR Project, National Institute of Information and Communications Technology, 3-5 Hikaridai, Seika-cho, Soraku-gun, Kyoto Japan, Graduate School of Engineering, Doshisha University, 1-3 Tatara Miyakodani, Kyotanabe City, Kyoto Japan NTT, NTT Communication Science Laboratories, NTT Corporation, 2-4 Hikaridai, Seika-cho, Soraku-gun, Kyoto Japan MCE [5, 9] [7] SVM[6, 7] SVM SVM MCE 250

264 SVM [4] X x J C j ; j =1,..., J 1 Λ N Ω N = {x n } N n=1 C j p Λ (C j x) J R(Λ) = p Λ (C y, x) y=1 X ( ) 1 p Λ (C y x) max j p Λ (C j x) dx (1) Λ [2] 1(P) P 1 0 (1) MCE R(Λ) J y=1 X p(c y, x)l ( d y (x, Λ) ) dx (2) [4]d y (x, Λ) p Λ (C j x) g j (x, Λ) ψ > 0 1/ψ d y (x, Λ) = g y (x, Λ) + log 1 e ψgj(x,λ) J 1 j,j y (3) x C y g j (x, Λ) x (3) d y (x, Λ) = g y (x, Λ) + max j,j y g j(x, Λ) (4) Λ ψ (3) (4) d y (x, Λ) < 0 d y (x, Λ) > 0 l( ) l ( d y (x, Λ) ) = exp ( ) (a >0) (5) ad y (x, Λ) Λ (1) 1( ) (1) (2) (3) (5) ψ a MCE Λ (2) Λ Λ (2) MCE [5] 2.2 (5) MCE Λ 0 [8] [10, 11, 12, 13] MCE

265 1: 2 2 [7] f(x) 2 C 1 C 1 x C j iff j = sgn ( f(x) ) sgn 1 1 x C u 1 z = uf(x) z >0 z<0 z z 2 x SVM f(x) =w x + b w b x C u r 0 = u(w x + b) w (6) L u u =1, 1 y y =1,..., J 2 SVM x φ(x) 2: (6) w SVM 1 w SVM 2 SVM 2 [3] SVM MCE MCE (5) MCE (6) r x 3 x C y g j (x, Λ) (j =1,..., J) x Λ 252

266 0 B y (Λ) = { x d y (x, Λ) = 0 } (7) C y ψ (4) B y (Λ) ψ B y (Λ) r B y (Λ) x minimize x x x 2 subject to d y (x, Λ) = 0 (8) x r = x x Lagrange λ J(x,λ) = x x 2 + λd y (x, Λ) (9) x 2(x x )+λ x d y (x, Λ) = 0 (10) d y (x, Λ) = 0 (11) x x x x d y (x, Λ) 0 (10) x x x d y (x, Λ) λ/2 r r = λ 2 xd y (x, Λ) (12) x d y (x, Λ) Taylor d y (x, Λ) = d y (x, Λ) + x d y (x, Λ) (x x ) +o ( x x ) (13) o(...) Landau (11) d y (x, Λ) = 0 (13) x = x x d y (x, Λ) (x x )=d y (x, Λ) + o(r) (14) (10) x d y (x, Λ) (x x )= λ 2 xd y (x, Λ) 2 (15) λ (12) d y (x, Λ) + o(r) r = x d y (x, Λ) (16) d y (x, Λ) x x 3: x x x (13) o(...) r = d y(x, Λ) x d y (x, Λ) (17) x o(...) x d y (x, Λ) x x (17) MCE (17) x (4) 253

267 MCE MCE λ > 0 R(Λ) + λ R(Λ) Λ R(Λ) (2) R(Λ) X B R(Λ) = J y=1 X B p(c y, x) x d y (x, Λ) 2 dx (18) d y (x, Λ) R(Λ) Tikhonov [14] λ MCE D y (x, Λ) = d y (x, Λ) x d y (x, Λ) (19) D y (x, Λ) MCE MCE D y (x, Λ) 5 HMM C j g j (x, Λ) = x p j 2 (20) p j C j C j x Λ C y x ψ (4) x best-incorrect C i d y (x, Λ) = x p y 2 x p i 2 (21) (19) D y (x, Λ) = x p y 2 x p i 2 2 p y p i (22) D y (x, Λ) (5) 0-1 j = y, i p j p j εl ( D y (x, Λ) ) pj D y (x, Λ) (23) ε > 0 C y C i py D y (x, Λ) = pi D y (x, Λ) = p y x p y p i d y (p y p i ) 2 p y p i 3 (24) x p i p y p i d y (p i p y ) 2 p y p i 3 (25) d y (21) UCI Machine Learning Repository 4 Glass Identification (Leave-One-Out ) MCE

268 3 4: 5: 5 (21) MCE (23) D y (x, Λ) d y (x, Λ) 4 (5) a a f(x) = w x + b 2 C u,u = 1, 1 d u (x, Λ) = u(w x + b) (2) 0-1 l( ) 2 (18) X B w w 2 SVM SVM 0-1 SVM SVM 2 SVM SVM 0-1 SVM [10] [15] LVQ 255

269 r d y (x, Λ) r = min x S g y (x, Λ) (26) S r [10]MCE µ = d y (x, Λ) g y (x, Λ) + max j,j y g j (x, Λ) (27) µ MCE [15] rµ d y (x, Λ) (4) (17) r µ r [10] [15] [11] HMM HMM (22) (22) 2 p y p i [11] 7 SVM MCE MCE MCE [10, 11, 12, 13] [9, 14, 16] MAS- TAR (B) [1] [2] R.O. Duda, P.E. Hart, and D.G. Stork, [3] C.M. Bishop, [4] B.-H. Juang and S. Katagiri, Discriminative learning for minimum error classification, IEEE Trans. Signal Processing, vol.40, no.12, pp , Dec [5] E. McDermott and S. Katagiri, A derivation of minimum classification error from the theoretical classification risk using Parzen estimation, Computer Speech and Language, vol.18, pp , April [6] V.N. Vapnik, The Nature of Statistical Learning Theory, Springer-Verlag, New York, [7] N. Cristianini and J.Shawe-Taylor [8] Y. Freund and R.E. Schapire, A decisiontheoretic generalization of on-line learning and an application to boosting, J. Comput. Syst. Sci., vol.55, no.1, pp , [9], (D-II)vol.J87-D-IIno.8 pp aug

270 [10] C. Liu, H. Jiang, and X. Li, Discriminative training of CDHMMs for maximum relative separation margin, Proc. ICASSP, pp.i , [11] H. Jiang, X. Li, and C. Liu, Large margin hidden Markov models for speech recognition, IEEE Trans. Audio, Speech, Lang., Process., vol.14, no.5, pp , Sept [12] J. Li, M. Yuan, and C.-H. Lee, Approximate test risk bound minimization through soft margin estimation, IEEE Trans. Audio, Speech, Lang., Process., vol.15, no.8, pp , Nov [13] D. Yu, L. Deng, X. He, and A. Acero, Largemargin minimum classification error training: a theoretical risk minimization perspective, Computer Speech and Language, vol.22, pp , Oct [14] C.M. Bishop, Training with noise is equivalent to Tikhonov regularization, Neural Computation, vol.7, no.1, pp , [15], (D-II)vol.J82- D-IIno.4pp April [16] T. Poggio and F. Girosi, Regularization algorithms for learning that are equivalent to multilayer networks, Science, vol.247, pp , Feb

271 2009 Technical Report on Information-Based Induction Sciences 2009 (IBIS2009) Estimation of the time-varying parameters characterizing point events Takeaki Shimokawa Shigeru Shinomoto Abstract: We selected a set of inter-event interval(iei) metrics that may efficiently characterize patterns of event occurrences and determined the function that may extract these characteristics. We found that the set of efficient metrics is the mean IEI and the mean log IEI, which represent the rate and the irregularity respectively, and the most suitable function is the gamma distribution function. We constructed Bayes method equipped with the gamma distribution function for estimating the instantaneous rate and irregularity of occurrence for a given event sequence. We confirmed that the Bayes method can capture the instantaneous rate and irregularity reasonably well even when a event sequence is generated from the log-normal and inverse-gaussian distributions. Keywords: Bayesian estimation, point process, rate, irregularity 1, (point process) {t i } n i=1 = {t (1, ) 1, t 2,, t n } [1] [2] Web (1, ) [3, 4] (firing-rate code) (spike-timing code) rate λ (inter-event interval; IEI) 1 1:,, , tel , shimokawa@ton.scphys.kyoto-u.ac.jp, Department of Physics, Kyoto University, Sakyo-ku, Kyoto , shinomoto@scphys.kyoto-u.ac.jp, regularity κ 258 (IEI)

272 IEI [9] IEI IEI 2.2 IEI p(t ) differential entropy IEI h = p(t ) log p(t )dt (2) = log λ g(λ) log g(λ)dλ. (3) 0 [5] p(t )dt = 1 E[Λ] = 1 h IEI 2 λ E[A(Λ)] = η [10] 2.1 g(λ) = exp [ {1 + a + bλ + ca(λ)}]. (4) renewal process T i = t i+1 t i λ η Fisher IEI p(t ) [ ] 2 E log p(t ) = 0. (5) IEI λ η IEI 1 b λ η + c η E[T A (λt )] = 0. (6) [ ] [6, 7, 8] λ = 1/E(T ) 1/ E log p(t ) = dt p(t ) = 0. (7) T p(t )dt λ λ 0 η 1 b IEI λ η + c η E[T A (λt )] + c η E[T A (λt )] = 0. (8) (6),(8) p(t )dt = g(λ)dλ, (1) η E[T A (λt )] = 0. (9) Λ λt IEI A(λT ) λ g 259 IEI 0 g(λ) = exp( Λ) A(λT ) log λt, (10)

273 (a) p(t) (b) κ = 0.5 κ = 1.0 κ = 5.0 irregular 0 1/λ 2/λ Τ regular 2: (a) κ = 0.5, 1.0, 5.0 (b) {κ(t)} {t i } n i=1 E[log λt ] = E[(T T ) 2 ] 2 T 2 E[(T T ) 3 ] 3 T 3 +, (11) p({t i } n i=1 {λ(t)}, {κ(t)}) = p(t i λ(t i ), κ(t i )). (13) IEI,1/E[T ] E[log λt ], p(t λ, κ) = λκ κ κ Γ(κ) T κ 1 exp ( λκt ). (12) η = log κ + ψ(κ) 2 (a) (b) (c) κ(t) λ(t) n 1 i=1 3 κ(t) [5] : (a) {λ(t)} {κ(t)} {λ(t)} {κ(t)}(3(a)) (b) (3(b)) (c) (3(c)) 95, Eq.(13) (12), {λ(t)}, {t i } n i=1 260 λ(t) time

274 {λ(t)},{κ(t)} p({λ(t)}, {κ(t)} {t i } n i=1) = p({t i} n i=1 {λ(t)}, {κ(t)}) p({λ(t)}) p({κ(t)}) p({t i } n i=1 ). (14) 4 τ 1 or τ 2 τ 2 or τ (Integrated squared Gaussian process prior λ(t) error; ISE) 4 κ(t) [ p({λ(t)}; γ λ ) = 1 Z(γ λ ) exp 1 T ( ) 2 (a) ISEλ dλ(t) dt] 2γλ 2, (15) 1 0 dt [ p({κ(t)}; γ κ ) = 1 Z(γ κ ) exp 1 T ( ) 2 dκ(t) dt] 2γκ 2, (16) 0 dt 0.1 γ λ γ κ p({t i } n i=1 ; γ λ, γ κ ) 0.01 gamma log-normal [11] inverse-gauss EM [12] 0.001, (14), λ(t) κ(t) (MAP ) [5] (b) 1 ISEκ 1/τ gamma log-normal [9] inverse-gauss /τ2 4: (a) 1/T T 0 (ˆλ(t) λ(t)) 2 dt(b) 1/T T 0 (ˆκ(t) κ(t))2 dt 4 λ κ λ = 1/E[T ], (17) log κ ψ(κ) = log E[T ] E[log T ]. (18) λ(t) = sin(t/τ 1 ), (19) κ(t) = sin(t/τ 2 + π/2), (20) 261

275 [1] S. Shinomoto, K. Shima, and J. Tanji (2003) Differences in spiking patterns among cortical neurons. Neural Computation, 15: [2] S. Shinomoto, H. Kim, T. Shimokawa et al. (2009) Relating neuronal firing patterns to functional differentiation of cerebral cortex. PLoS Computational Biology, 5: e [3] R. M. Davies, G. L. Gerstein, and S. N. Baker, (2006) Measurement of time-dependent changes in the irregularity of neural spiking. Journal of Neurophysiology, 96: [4] J. F. Mitchell, K. A. Sundberg, and J. H. Reynolds, (2007) Differential attention-dependent response modulation across cell classes in macaque visual area V4. Neuron, 55: [5] T. Shimokawa and S. Shinomoto (2009) Estimating instantaneous irregularity of neuronal firing. Neural Computation, 21: [6] S. Amari and H. Nagaoka (2000) Methods of information geometry. Oxford: Oxford University Press. [7] K. Ikeda (2005) Information geometry of interspike intervals in spiking neurons. Neural Computation, 17: [8] K. Miura, M. Okada, and S. Amari (2006) Estimating spiking irregularities under changing environments. Neural Computation, 18: [9] E. T. Jaynes (1957) Information theory and statistical mechanics. Physical Review, 106: [10] J. N. Kapur (1989) Maximum-entropy models in science and engineering. New York: Wiley. [11] D. J. C. MacKay (1992) Bayesian interpolation. Neural Computation, 4: [12] A. P. Dempster, N. M. Laird, and D. B. Rubin (1977) Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society, Series B, 39:

276 情 報 論 的 学 習 理 論 テクニカルレポート 2009 Technical Report on Information-Based Induction Sciences 2009 (IBIS2009) 生 物 学 情 報 への 機 械 学 習 解 析 の 応 用 (Toxicogenomics への 展 開 ) Application of machine learning to Biological data (Toxicogenomics) 武 藤 裕 紀 *, 松 下 智 哉, 芦 原 基 起 Hironori Mutoh, Tomochika Matsushita and Motooki Ashihara Abstract: Microarray technology has been widely utilized in the biological fields. To analyze the data derived from microarrays, bioinformatics technology is necessary because the amount of data is huge and biologically complicated. Toxicogenomics is an area of study where clarification and prediction of the mechanisms of toxicity are found by applying microarray technology. Here we applied Support Vector Machine (SVM) to predict a toxicity of the liver, the proliferation of bile ducts, using toxicogenomics database constructed by the Toxicogenomics Project in Japan (TGP). Keywords: Microarray, Bioinformatics, Toxicogenomics, Support Vector Machine 1 緒 言 マイクロアレイ 技 術 の 発 展 により 網 羅 的 な 遺 伝 子 発 現 データを 比 較 的 容 易 に 取 得 することが 可 能 となり 生 物 学 の 多 くの 分 野 で 活 用 されている マイクロアレイデ ータ 解 析 では 1サンプルにつき 数 万 の 遺 伝 子 発 現 値 が 得 られる これらのデータは 単 に 膨 大 なだけでなく 生 物 学 的 な 複 雑 さを 含 むため 統 計 解 析 やバイオインフォ マティクス 技 術 の 使 用 が 不 可 欠 である トキシコゲノミクスは マイクロアレイ 技 術 を 毒 性 学 の 領 域 に 応 用 し 薬 物 を 動 物 や 細 胞 に 暴 露 して 網 羅 的 に 遺 伝 子 発 現 解 析 を 行 うことにより 毒 性 発 現 メカニズム の 解 明 や 毒 性 予 測 を 行 う 学 問 領 域 である 従 来 の 毒 性 評 価 法 に 比 べ 創 薬 研 究 の 初 期 段 階 で 医 薬 品 候 補 化 合 物 の 毒 性 を 効 率 的 に 評 価 予 測 する 手 法 として 期 待 され 発 がん 肝 毒 性 等 のリスク 評 価 への 活 用 が 既 に 試 みられて いる[1-6] 毒 性 予 測 には あらかじめ 毒 性 の 発 現 が 確 認 されてい る( 毒 性 有 り) 化 合 物 と 毒 性 の 発 現 が 認 められていない ( 毒 性 無 し) 化 合 物 の 情 報 を 与 えて 予 測 モデルを 作 成 す る 教 師 付 き 機 械 学 習 の 手 法 が 有 用 である この 予 測 モデ ルの 構 築 には 多 数 のデータがリファレンスとして 必 要 と なるため 毒 性 予 測 モデルの 構 築 を 目 的 としたデータベ ースが 構 築 されている[7-8] 日 本 においては 独 立 行 政 法 人 医 薬 基 盤 研 究 所 国 立 医 薬 品 食 品 衛 生 研 究 所 および 製 薬 企 業 の 共 同 研 究 である トキシコゲノミクスプロジ * 中 外 製 薬 株 式 会 社 研 究 本 部 創 薬 資 源 研 究 部, 神 奈 川 県 鎌 倉 市 梶 原 200, tel , mutohrn@chugai-pharm.co.jp, Discovery Science & Technology Dept. Research division, Chugai Pharmaceutical CO., LTD. 200 Kajiwara, Kamakura, Kanagawa, JAPAN ェクト において 150 を 超 える 化 合 物 を 用 いた 暴 露 実 験 が 行 われ マイクロアレイのプラットフォームのひと つである Affymetrix 社 の GeneChip により 取 得 された 遺 伝 子 発 現 データおよび 付 帯 する 毒 性 関 連 情 報 が 格 納 された データベースが 構 築 された(TGP2; GeneChip 解 析 を 実 施 するうえでの 必 要 な 前 処 理 として 遺 伝 子 発 現 値 の 数 値 化 手 法 と 解 析 に 有 用 な 遺 伝 子 の 絞 り 込 み 手 法 の 二 つの 検 討 を 行 う 必 要 がある 数 値 化 手 法 に 関 しては Affymetrix 社 が 提 唱 している MAS5[9]をはじめとして いくつかの 数 学 的 処 理 が 知 ら れているが 処 理 法 によって 算 出 法 が 変 わるため 解 析 結 果 に 与 える 影 響 が 異 なることが 報 告 されている[10] 遺 伝 子 の 絞 込 み 手 法 に 関 しては 学 習 データに 基 づい て 機 械 的 に 絞 り 込 む 方 法 に 加 え 生 物 学 情 報 などにより 事 前 に 絞 り 込 む 方 法 などがある GeneChip にはプローブ 設 計 の 問 題 上 実 際 のターゲットとなる 遺 伝 子 発 現 を 捕 らえられていないデータも 存 在 するため これらを 事 前 に 取 り 除 き 毒 性 予 測 に 有 用 な 遺 伝 子 を 絞 り 込 むことが 重 要 となる 我 々は 毒 性 予 測 モデルの 構 築 を 目 的 として 前 述 し た 項 目 が 機 械 学 習 による 毒 性 予 測 モデルの 精 度 にどの ような 影 響 を 与 えるかについて 検 討 を 行 った リファレ ンスデータベースとして トキシコゲノミクスプロジェ クト で 構 築 されたデータベースを 用 い 機 械 学 習 アル ゴリズムには 近 年 未 学 習 データへの 汎 用 性 が 高 いこ とで 注 目 されている SVM(Support Vector Machine)を 用 いた また 毒 性 予 測 は 肝 臓 における 胆 管 増 生 を 対 象 として 実 施 した 263

277 2 データおよび 解 析 手 法 2.1. リファレンスデータベース リファレンスデータベースとして 独 立 行 政 法 人 医 薬 基 盤 研 究 所 国 立 医 薬 品 食 品 衛 生 研 究 所 および 製 薬 企 業 の 共 同 研 究 である トキシコゲノミクスプロジェクト に おいて 構 築 されたデータベースを 使 用 した(TGP2; データは Affymetrix 社 の GeneChip Rat Genome Array (14,562 遺 伝 子 31,099 プローブ)により 測 定 されたものを 使 用 した(Affymetrix 社 ; 学 習 用 データ 本 研 究 では 評 価 する 毒 性 として 肝 臓 における 胆 管 増 生 ( 病 理 所 見 名 Proliferation, Bile duct )をターゲット とした 毒 性 有 り 化 合 物 は Proliferation, Bile duct の 病 理 所 見 がいずれかの 条 件 において 確 認 された 化 合 物 を 選 択 した 毒 性 無 し 化 合 物 は Proliferation, Bile duct 以 外 の 所 見 の 確 認 された 化 合 物 の 一 部 を 選 択 した この 選 択 指 標 に 基 づいて リファレンスデータベースから 毒 性 有 り 化 合 物 6 化 合 物 および 毒 性 無 し 化 合 物 10 化 合 物 を 選 択 した (Table 1) これら 計 16 化 合 物 を 高 用 量 で 28 日 間 反 復 投 与 し 投 与 後 29 日 に 取 得 したデータ(n=3)および 各 化 合 物 の 溶 媒 のみを 投 与 したコントロールデータ(n=3)を 学 習 用 データセット( 計 96 サンプル)として 用 いた Table 1 List of Training data Compound Toxicity Vehicle Dose [mg/kg] acetamidofluorene Positive 0.5% MC 300 allyl alcohol Positive corn oil 30 lomustine Positive 0.5% MC 6 methapyrilene Positive 0.5% MC 100 naphthyl isothiocyanate Positive corn oil 15 thioacetamide Positive 0.5% MC 45 amiodarone Negative 0.5% MC 200 amitriptyline Negative 0.5% MC 150 clofibrate Negative corn oil 300 flutamide Negative corn oil 150 furosemide Negative 0.5% MC 300 hydroxyzine Negative 0.5% MC 100 imipramine Negative 0.5% MC 100 metformin Negative 0.5% MC 1000 omeprazole Negative 0.5% MC 1000 ticlopidine Negative 0.5% MC 評 価 用 データ リファレンスデータベースに 含 まれている 薬 物 投 与 を 行 った 全 実 験 データを 評 価 用 データセットに 用 いた 2.4. データ 処 理 および 解 析 手 法 GeneChip データは MAS5[9]および GCRMA[11]を 用 いて 数 値 化 し 底 を 2 として 対 数 化 を 行 った 溶 媒 のみを 投 与 したコントロールデータとの Ratio データを 生 成 する ため 各 化 合 物 に 対 し 溶 媒 のみを 投 与 したコントロール 実 験 データ(n=3)の 平 均 値 を 計 算 し 投 与 実 験 データ との 差 を 取 った( 数 式 1) ncontrol log2( ExpControl, j ) j= 1 (1) log2( Exptreated, i ) ncontrol 予 測 モデルの 構 築 には SVM(Support vector machine)を 用 い カーネル 関 数 は Linear kernel を 用 いた 変 数 の 機 械 的 な 絞 込 みには SVM-RFE(Recursive feature elimination)[12]を 用 い weight パラメータの 小 さい 変 数 5%を 繰 り 返 しごとに 消 去 評 価 関 数 の 値 が 下 がったところで 繰 り 返 しを 止 め た データセットは 同 一 化 合 物 の 反 復 実 験 を 含 むため クロスバリデーションは 化 合 物 単 位 の LOOCV(Leave one out cross validation) を 実 行 した(Fig.1) 評 価 関 数 には MCC(Matthews Correlation Coefficient)を 用 いた( 数 式 2) [13] また Sensitivity, Specificity は 下 記 に 示 す 数 式 3 4 を 用 いて 算 出 した TP TN FP FN (2) MCC = TP + FP TP + FN TN + FP TN + FN TP :TruePositive FP : FalsePositive ( ) ( ) ( ) ( ) FN : FalseNegaitive TN :TrueNegative TP Sensitivity = (3) TP + FN TN Specificity = (4) TN + FP すべての 解 析 は R version を 用 いて 行 った また 解 析 パッケージとして e1071, MASS および Bioconductor の affy, gcrma を 利 用 した(CRAN; (Bioconductor; Data Normalization (MAS5, GCRMA) Logarithmic processing Training Sample Data splitting Feature Selection using SVM-RFE with Compound-based LOOCV Model Construction Convert to Ratio Parameter tuning with Compound-based LOOCV Fig. 1 Analysis Flow ncontrol log2( ExpControl, j ) j= 1 log2( Exptreated, i ) ncontrol Compound-based LOOCV Test Sample Prediction using constructed Model 264

278 3 結 果 3.1. データ 前 処 理 検 討 GeneChip の 数 値 化 手 法 による 影 響 を 調 べるため MAS5[9]および GCRMA[11]により 数 値 化 を 行 い 両 者 の 精 度 の 比 較 を 以 下 の 要 領 で 行 った まず リフ ァレンスデータベースに 含 まれる 2 種 類 の 異 なる 実 験 プロトコルを 用 いて Duplicate で 取 得 したラット 初 代 培 養 肝 細 胞 の 遺 伝 子 発 現 データを MAS5, GCRMA で 数 値 化 し 散 布 図 を 作 成 することで 数 値 の 再 現 性 への 影 響 を 確 認 した また 学 習 用 データを 用 いて 実 際 に 予 測 モデルを 構 築 し 精 度 の 評 価 を 行 った 数 値 の 再 現 性 への 影 響 を 比 較 した 結 果 では プロ トコル 間 およびプロトコル 内 のいずれにおいても MAS5 と 比 較 して GCRMA のほうが 実 験 間 の 再 現 性 が 高 い 傾 向 が 確 認 された(Fig. 2) Protocol A Protocol A MAS5 Protocol B Protocol B Protocol A GCRMA Protocol A Protocol B Protocol B Fig. 2 Scattered plot of Log2ratio 遺 伝 子 発 現 プロファイルはラット 初 代 培 養 肝 細 胞 から 取 得 それぞれisoniazidで 処 理 を 行 った 24 時 間 後 の 初 代 培 養 肝 細 胞 から 2 種 類 の 実 験 プロトコル(A,B)によりデータを 取 得 また 毒 性 予 測 モデルの 精 度 を 比 較 した 結 果 では GCRMA により 構 築 されたモデルにおいて Sensitivity, Specificity がそれぞれ 66.67% %と MAS5 の 16.67% %と 比 べてより 高 い 予 測 精 度 が 得 られ た (Fig. 3) 100% 80% 60% 40% 20% 0% MAS5 GCRMA Sensitivity Specificity Fig. 3 Accuracy of MAS5 and GCRMA based models MAS5およびGCRMAにより 数 値 化 されたデータから 構 築 したモデルの 精 度 次 に GeneChip のプローブの 絞 り 込 み 手 法 による 影 響 を 調 べるため Affymetrix 社 の 提 供 するプロー ブに 関 する 情 報 と Ensembl を 用 いた 独 自 の 評 価 法 を 用 いてプローブを 分 類 し 比 較 を 行 った Affymetrix 社 ではプローブのターゲットへのマッチングにより Annotation Grade を 5 段 階 に 分 類 している(Affymetrix 社 ; 本 研 究 で 使 用 した Rat Genome Array について 調 べたところ タ ーゲットへのマッチングが 確 認 されている Grade A のプローブは 16,327 あり その 他 の 14,772 プローブ は 部 分 的 にマッチングしているものや EST クラスタ ーにのみマッチングしているプローブであった 次 に 我 々は GeneChip の オリゴプローブ 配 列 を Ensembl Transcript (Release54) 配 列 上 へマッピングし た(Ensembl; その 結 果 ターゲットにマップされたプローブが 8,259 クロスハイブリダイズしていたプローブが 3120 タ ーゲット 以 外 にマップされたプローブが 1,907 逆 向 きにマップされたプローブが 1,172 全 くマップされ なかったプローブが 16,579 確 認 された(Table 2) Annotation Grade で Grade A とされているプローブの うち 単 一 のターゲットにマップされたプローブは 46.6%(7,609)で その 他 53.4%(8,718)はクロスハイブ リダイゼーションや 逆 向 きに 設 計 遺 伝 子 以 外 の 配 列 を 認 識 しているプローブを 含 む 可 能 性 が 示 唆 され た(Table 2) 一 方 Ensembl 上 で 単 一 ターゲットにマ ップされたプローブのうち 92.1%は Grade A のプロ ーブで その 他 はわずか 7.9%であった Table 2 Probe annotation and mapping results Ensembl Mapping // Annotation Grade GradeA Grade B,C,E,R Total Mapped (single target) 7, ,259 Mapped (multiple targets) Mapped (cross hybridization)* 2, ,120 Mapped (target not covered)*+ 1, ,907 Mapped Reverse 163 1,009 1,172 Unmapped 4,108 12,471 16,579 Total 16,327 14,772 31,099 *include mapped on Unknown +include partially coverd(mutiple targets) これらの 結 果 に 基 づき あらかじめ 解 析 に 用 いる プローブセットの 設 定 を 変 えて(GradeA, Mapped ( 単 一 ターゲット)およびそれらの 組 み 合 わせ) 毒 性 予 測 モデルの 構 築 を 行 い 精 度 の 比 較 を 行 った( 数 値 化 処 理 はすべて GCRMA で 実 行 ) 構 築 されたモデルの 予 測 精 度 は Mapped および Grade A Mapped で 最 も 高 く いずれも Sensitivity 83.33%, Specificity %と 高 い 値 を 示 した(Fig. 4) 100% 80% 60% 40% 20% Sensitivity Specificity 0% All GradeA Mapped GradeA Mapped Fig. 4 Accuracy of different Probe set based models プローブセットを 変 更 して 構 築 したモデルの 精 度 Mappedは Ensembl 上 の 単 一 ターゲットにマップされた8,259プローブ Grade A Mapped はMappedのうち Grade Aであった7,609プロー ブを 利 用 した 265

279 3.2. 毒 性 予 測 データ 前 処 理 検 討 の 結 果 最 も 予 測 精 度 の 高 かっ た GCRMA で 数 値 化 を 行 い Mapped プローブを 用 いて 構 築 されたモデルを 利 用 して 評 価 用 データの 毒 性 予 測 を 行 った その 結 果 全 151 化 合 物 中 15 化 合 物 について 少 なくともいずれかの 条 件 で 毒 性 有 りと 予 測 された(Table 3) Table 3 Predicted toxic compounds and experimental conditions Compound // Time point & Dose 4 day 8 day 15 day 29 day L M H L M H L M H L M H acetamidofluorene T T T T T T T* allyl alcohol T* cisplatin T colchicine T ethambutol T T lomustine T T* lornoxicam T T NA meloxicam T NA methapyrilene T T T* monocrotaline T T T T NA naphthyl isothiocyanate T T T* naproxen T T NA nitrosodiethylamine T T T T NA phalloidin NA NA T NA NA T NA NA NA NA NA NA thioacetamide T T T T* L:Low dose, M:Middle dose, H:High dose, T:Toxic(predicted), NA:Not Avalable Shaded:Proliferation, bile duct was observed *Training sample 評 価 用 データで Proliferation, Bile duct の 病 理 所 見 が 確 認 された 8 化 合 物 中 学 習 データに 用 いた 6 化 合 物 では allyl alcohol を 除 き いずれも 学 習 に 用 いた 条 件 以 外 のサ ンプルでも 毒 性 有 りと 予 測 され 学 習 データに 用 いてい ない nitrosodiethylamine, phalloidin も 毒 性 有 りと 予 測 された また thioacetamide については 所 見 が 確 認 される 以 前 のサ ンプルでも 毒 性 有 りと 予 測 された 一 方 所 見 が 確 認 されていない 143 化 合 物 では,7 化 合 物 を 除 いて 大 半 がいずれの 条 件 でも 毒 性 無 しと 予 測 された 4 考 察 本 研 究 ではまず GeneChip データの 数 値 化 手 法 に よる 毒 性 予 測 モデルの 精 度 への 影 響 を 検 討 した そ の 結 果 MAS5 と 比 較 して GCRMA を 用 いたときに 高 い 精 度 の 予 測 モデルが 構 築 できた GCRMA によ る 数 値 化 では MAS5 と 比 較 して 低 発 現 でのばら つきが 抑 えられることが 確 認 されていることから (Data not shown) コントロールとの Ratio データに 変 換 すると 発 現 の 小 さなプローブのばらつきの 影 響 が 少 なく MAS5 と 比 較 して Fig. 2 のように 実 験 間 での 再 現 性 が 高 くなり その 結 果 毒 性 予 測 精 度 に 差 が 出 たと 考 えられる またトキシコゲノミクスに よる 毒 性 予 測 では 化 合 物 により 溶 媒 ( 化 合 物 を 溶 解 する 溶 液 )が 異 なるなど(Table 1) データ 採 取 の 条 件 を 統 一 することが 難 しく Ratio のようなコント ロールとの 比 較 値 を 用 いることが 重 要 となるため 数 値 化 手 法 として MAS5 よりも GCRMA を 用 いるこ とが 適 切 と 考 えられた 次 に プローブをあらかじめ 選 別 することによる 毒 性 予 測 モデルの 精 度 への 影 響 を 検 討 した 結 果 全 プローブを 用 いるよりも Ensembl 上 で 単 一 ターゲ ットにマッピングされたプローブを 用 いたときに 高 い 精 度 の 予 測 モデルが 構 築 できた 一 方 Affymetrix 社 の Annotation Grade で Grade A とされているプロー ブを 用 いたときは 全 プローブを 用 いたときと 予 測 精 度 に 差 がなかった Table 2 で 示 したように Grade A のプローブであっても クロスハイブリダ イゼーションや 逆 向 き 遺 伝 子 以 外 の 配 列 にマップ されたプローブを 含 んでおり 毒 性 予 測 に 関 連 のな いノイズを 含 んでいた 可 能 性 が 考 えられた 最 後 に 構 築 したモデルを 用 いて 評 価 用 データの 毒 性 予 測 を 行 った 結 果 所 見 が 確 認 された 化 合 物 に ついては allyl alcohol を 除 いて 学 習 用 に 用 いた 条 件 以 外 でも 毒 性 有 りと 予 測 された Allyl alcohol につい ては 学 習 用 に 用 いた 条 件 (29 日 高 用 量 )では 所 見 が 確 認 されていなかったため 正 しく 学 習 できなか った 可 能 性 が 考 えられた 実 験 条 件 ごとに 見 ると 29 日 高 用 量 のデータを 用 いてモデルを 構 築 したに もかかわらず それよりも 早 いタイムポイントや 低 用 量 の 条 件 でも 毒 性 有 りと 予 測 されたものも 多 く 見 られた また thioacetamide については 所 見 が 現 れる 以 前 のサンプルでも 毒 性 有 りと 予 測 されるなど 病 理 所 見 が 確 認 されるより 早 い 段 階 で 毒 性 を 予 測 できる 可 能 性 が 示 唆 された 一 方 で 所 見 が 確 認 されていない 化 合 物 で 毒 性 有 りと 予 測 されているサンプルや 所 見 が 確 認 されている 条 件 で 毒 性 無 しと 予 測 されている サンプルもあった 前 者 については Proliferation, Bile duct の 病 理 所 見 に 先 立 つ 遺 伝 子 発 現 の 変 動 を 捕 らえている 可 能 性 もあるが 前 述 したような 学 習 デ ータの 特 性 により 正 しく 学 習 できていない 可 能 性 もある したがって 構 築 された 予 測 モデルの 生 物 学 的 意 義 については 精 査 していく 必 要 があると 考 え られた 本 研 究 では 生 物 学 情 報 への 機 械 学 習 解 析 の 応 用 として トキシコゲノミクスデータの 解 析 検 討 を 行 った 肝 臓 における 胆 管 増 生 を 毒 性 のターゲットと して SVM により 予 測 モデルを 構 築 した 結 果 精 度 の 高 い 予 測 モデルの 構 築 に 成 功 した(Sensitivity 83.3%, Specificity 100%) 本 研 究 の 結 果 から トキシコゲノ ミクスデータに 機 械 学 習 を 応 用 することで 化 合 物 の 毒 性 を 予 測 できる 可 能 性 が 示 唆 された また 複 数 の データ 処 理 の 方 法 を 比 較 し 適 当 な 手 法 の 選 択 およ び 生 物 学 的 背 景 を 考 慮 した 検 討 を 行 うことによって 精 度 の 高 い 予 測 モデルが 構 築 できた このことから 最 適 な 手 法 を 選 択 使 用 することが 精 度 の 向 上 に 重 要 な 因 子 であることが 確 認 された 手 法 の 最 適 化 を 行 うことで より 高 精 度 な 予 測 モデルが 構 築 できる と 思 われることから 今 後 も 検 討 を 進 めていく 参 考 文 献 [1] Nie AY, McMillian M, Parker JB, Leone A, Bryant S, Yieh L, Bittner A, Nelson J, Carmen A, Wan J, Lord PG. Predictive toxicogenomics approaches reveal 266

280 underlying molecular mechanisms of nongenotoxic carcinogenicity., Mol Carcinog Dec;45(12): [2] Fielden MR, Brennan R, Gollub J. A gene expression biomarker provides early prediction and mechanistic assessment of hepatic tumor induction by nongenotoxic chemicals., Toxicol Sci Sep;99(1): [3] Ellinger-Ziegelbauer H, Gmuender H, Bandenburg A, Ahr HJ. Prediction of a carcinogenic potential of rat hepatocarcinogens using toxicogenomics analysis of short-term in vivo studies., Mutat Res Jan 1;637(1-2): [4] Steiner G, Suter L, Boess F, Gasser R, de Vera MC, Albertini S, Ruepp S. Discriminating different classes of toxicants by transcript profiling., Environ Health Perspect Aug;112(12): [5] Zidek N, Hellmann J, Kramer PJ, Hewitt PG. Acute hepatotoxicity: a predictive model based on focused illumina microarrays., Toxicol Sci Sep;99(1): [6] Hirode M, Ono A, Miyagishima T, Nagao T, Ohno Y, Urushidani T. Gene expression profiling in rat liver treated with compounds inducing phospholipidosis., Toxicol Appl Pharmacol Jun 15;229(3): [7] Ganter B, Tugendreich S, Pearson CI, Ayanoglu E, Baumhueter S, Bostian KA, Brady L, Browne LJ, Calvin JT, Day GJ, Breckenridge N, Dunlea S, Eynon BP, Furness LM, Ferng J, Fielden MR, Fujimoto SY, Gong L, Hu C, Idury R, Judo MS, Kolaja KL, Lee MD, McSorley C, Minor JM, Nair RV, Natsoulis G, Nguyen P, Nicholson SM, Pham H, Roter AH, Sun D, Tan S, Thode S, Tolley AM, Vladimirova A, Yang J, Zhou Z, Jarnagin K., Development of a large-scale chemogenomics database to improve drug candidate selection and to understand mechanisms of chemical toxicity and action., J Biotechnol Sep 29;119(3): [8] Ganter B, Snyder RD, Halbert DN, Lee MD., Toxicogenomics in drug discovery and development: mechanistic analysis of compound/class-dependent effects using the DrugMatrix database., Pharmacogenomics Oct;7(7): [9] Hubbell E, Liu WM, Mei R. Robust estimators for expression analysis., Bioinformatics Dec;18(12): [10] Lim WK, Wang K, Lefebvre C, Califano A. Comparative analysis of microarray normalization procedures: effects on reverse engineering gene networks., Bioinformatics Jul 1;23(13):i [11] Zhijin Wu, Rafael A. Irizarry, Robert Gentleman, Francisco Martinez Murillo, Forrest Spencer, A Model Based Background Adjustment for Oligonucleotide Expression Arrays., Johns Hopkins University, Dept. of Biostatistics Working Papers. Working Paper 1., 2004 May. [12] Guyon IM, Weston J, Barnhill S, Vapnik VN. Gene selection for cancer classification using support vector machines., Mach Learn : [13] Baldi P, Brunak S, Chauvin Y, Andersen CA, Nielsen H., Assessing the accuracy of prediction algorithms for classification: an overview., Bioinformatics May;16(5): 謝 辞 本 研 究 は 厚 生 労 働 科 学 研 究 費 補 助 金 H14-トキシコ-001 およ び H19-トキシコ-001 による 267

281 2009 Technical Report on Information-Based Induction Sciences 2009 (IBIS2009) Ellipsoidal Support Vector Machines Michinari Momma Abstract: This paper proposes the ellipsoidal SVM (e-svm) that uses an ellipsoid center, in the version space, to approximate the Bayes point. Since SVM approximates it by a sphere center, e-svm provides an extension to SVM for better approximation of the Bayes point. Although the idea has been mentioned before [11], no work has been done for formulating and kernelizing the method. Starting from the maximum volume ellipsoid problem, we successfully formulate and kernelize it by employing relaxations. The resulting e-svm optimization framework has much similarity to SVM; it is naturally extendable to other loss functions and other problems. A variant of the sequential minimal optimization is provided for efficient implementation. The empirical results are shown to be consistent with the Bayes point machines, in terms of classification performance, and difference from other related methods is highlighted by using high dimensional datasets. Keywords: Bayes point machines, Support vector machines 1 Introduction The most common interpretation of the support vector machines (SVMs) [19, 12] is that it separates positive and negative examples by maximizing the margin that is the distance between supporting hyperplanes of both examples. Another interpretation comes from a concept called the version space. The version space is a space of consistent hypotheses, or models with no error. SVM maximizes the inscribing hypersphere to find the center that is the SVM weight vector w. Given the version space, the sphere center completely characterizes the SVM model. The Bayes point is a point through which all hyperplanes bisect the version space by half, and is shown to have better generalization ability theoretically and empirically[6, 11]. Attempts to approximately find the Bayes point have been done since the early studies of the version space and the Bayes point. SVM can be considered as an example. The Bayes point machines (BPM) [6] uses a kernel billiard algorithm to find the center of mass NEC, , tel , m-momma@cd.jp.nec.com, NEC Common Platform Software Research Laboratories, 1753 Shimonumabe, Nakahara-Ku, Kawasaki, Kanagawa in the version space. The analytic center machines (ACM) [18] approximate the Bayes point by analytic points of linear constraints. The idea of using an ellipsoid rather than a sphere has been mentioned in [11], although it was neither formulated nor implemented because of its projected high computational cost O(n 3.5 ). Then a billiard algorithm including BPM has been developed to alleviate the computational challenge. However, as we have seen in the history of SVM, seemingly expensive problem can be made efficient by exploiting special structures in the problem. Sequential minimal optimization (SMO) or decomposition methods are notable examples of such algorithms[9, 3]. Furthermore, recent development of large scale linear SVMs [13, 8] impressively improves the scalability of the quadratic optimization into practially linear order. Learning from the experience, we are encouraged to develop and study the method of ellipsoidal approximation to BPM, which we refer to as the ellipsoidal SVM (e-svm). e-svm is formulated just like SVMs. Advantages in formulating in such a way include possible adaptation of theoretical characterization, optimization methods developed for SVM. Furthermore, extensions to chang-

282 ing loss function or application to other kind of problems should be possible. These advantages would not be obtained if we stick to BPM that has to rely on sampling techniques that scale poorly on a large scale dataset; In BPM, even the soft boundary formulation is nontrivial and the kernel regularization is used after all. The e-svm formulation is based on the maximum volume inscribed ellipsoid (MVIE) problem. The original MVIE problem consists of second order cone (SOC) constraints and a semidefinite constraint. Although the problem is convex and an interior point type method can be applied for polynomial time convergence, the natural kernelization, i.e, the inner-product based mapping, is left as a challenge. For example, the second order cone program in [16] resorts to the direct kernel method where the original data matrix is replaced by the kernel matrix. e-svm uses relaxation of SOC constraints to make the dual kernelizable. This technique has not been used before to the best of our knowledge. As with the MVIE problem, e-svm has the logdeterminant term in the objective. Similar problems can be seen in [14, 5, 4, 16]. e-svm problem may be interpreted as a combination of the regular SVM and the minimum volume covering ellipsoid (MVCE) problem. Thus, when optimized separately, e-svm becomes very similar to ellipsoidal kernel machine (EKM) [14]. If applied to one class problem, e-svm would become similar to [5]. In other words, e-svm can subsume other related methods and possesses bigger optimization problem. By changing the loss function to a strict convex Bregman function, the Bregman s method would be applicable to solving e-svm [4]. One direct interpretation of e-svm is that it learns the Mahalanobis metric in margin. That is, e-svm maximizes margin by adjusting the metric. In terms of adjusting margin, the relative margin machines (RMM) [15] address impact of data scaling on the performance in SVM. Since SVMs do not take into consideration of spread of data, a bad scaling can hurt the performance. Although e-svm has more degree of freedom to adjust the margin, these two methods learn quite different models as we will see in Section 4. As the first step to solving the challenging e-svm problem efficiently, we adopt the sequential minimal optimization (SMO). The modified SMO algorithm indeed shares many convenient features with that for SVM, such as the closed-form solution for the minimal problem, Karush-Kuhn-Tucker (KKT) condition violation check, etc. Although there should exist faster algorithm to solve depending on the type of problems, we decide to start from the simpler SMO algorithm and study how e-svm compares against BPM, SVM and other related methods. Section 2 formulates the e-svm optimization problem. Section 3 describes the SMO algorithm adapted for the e-svm problem. Section 4 compares related methods by a simple example. Section 5 gives experimental results. Section 6 concludes the paper. Notation: Throughout the paper, we assume that m data points x i in n-dimensional space and the corresponding (target) label y i { 1, 1} are given. The bold small letters represent vectors and the capital letters represent matrices. The vector/matrix transpose is T. The kernel matrix is given by K with K ij as its element. tra denotes the trace of a matrix A. s.t. in optimization problems means subject to. I is an index set of m data points: I {1,...,m}. A 2 denotes the matrix 2-norm and x 2 the L2-norm of a vector x. 2 Ellipsoidal support vector machine formulations In this section, the e-svm optimization problem is formulated starting from that of SVM in the version space, since it is a simpler counterpart of e-svm. The version space is a space of error zero models. For linear models, it is the error-zero subspace of weight vectors w. The data points are considered as hyperplanes and the classification constraints are the feasible region that is a polyhedron. The problem of finding a maximum hypersphere inside the polyhedron can be formulated as follows: max ρ,w,b ρ s.t. y i ( x T i w + b ) x i 2 ρ, w 2 1, i I which corresponds to maximization of the minimum distance between the center and the hyperplanes, in the absence of the bias b. By allowing errors in the above problem, we can get a soft-margin version of the 269

283 above problem. min ρ,w,b,ζ mρ + 1/ν m i=1 ζ i s.t. y i ( x T i w + b ) + t 2 i ζ i t 2 i ρ, w 2 1, i I (1) where t i is defined to be x i 2 and ν > 0 is a given constant. Note in the special case with t i = 1, Problem 1 becomes identical to the ν-svm formulation. To better approximate the center of models, an ellipsoid, instead of a hypersphere, will be used to inscribe the polyhedron. The MVIE problem is a wellknown log-determinant optimization problem, see e.g. [2]. A representation of an ellipsoid centered at w is given by E = {Eu + w u 2 1, E 0}. Thus the constraints for SVM (1) are modified as follows: y i ( x T i (Eu + w) + b ) +t 2 i ζ i t 2 i ρ, u, u 2 1 (2) Since Equation 2 holds for any u, it suffices to use the lower bound of lhs in order to remove u: y i ( x T i (Eu + w) + b ) + t 2 i ζ i y i ( x T i w + b ) Ex i 2 + t 2 iζ i t 2 i ρ (3) where yiexi Ex i 2 = arg min u, u =1 (y i x T i Eu) is used. Furthermore, in order to obtain the largest ellipsoid inscribing a polyhedron, the volume of the ellipsoid should be maximized, which corresponds to maximizing the determinant of E ( E ), as the volume of an ellipsoid is proportional to the determinant. In an optimization problem, log det is easier to handle and thus adopted here as well. The resulting optimization problem is given as follows: min E,ρ,ζ,w,b s.t. λ(r log E + (1 r)tre) mρ + 1 ν ζi y i ( x T i w + b ) Ex i 2 t 2 i ρ t2 i ζ i w 2 1, ζ i 0, i I, E 0, (4) where λ > 0 is a trade-off parameter and r is a constant whose value takes 0 < r 1. The additional term tre is introduced to gain numerical stability as suggested in [5]. Note the role of ρ and E as maximizing margin is similar and redundant; the determinant maximization term can subsume the linear maximization of ρ 1. Hence, ρ is dropped from the problem hereafter, 1 Our preliminary study confirmed that ρ becomes zero in most cases allowing us to remove λ: min E,ζ,w,b r log E + (1 r)tre + 1 ν ζi s.t. y i ( x T i w + b ) + t 2 i ζ i Ex i 2 w 2 1, ζ i 0, i I, E 0. (5) This MVIE problem can be solved by using existing techniques, including interior point methods or cutting plane based approaches. Here we relax the SOC constraint in Problem 5 in order to ease the high computational complexity. This change, as we shall see, plays a significant role in making the kernelized formulation possible. As the first step, assume the matrix E is written as E = E 0 + B, where E 0 is the current solution and B is a deviation from it. By the Taylor expansion, the SOC constraint is written as Ex i 2 = κ i + 1 κ i x T i E 0Bx i + O( B 2 2 ) where κ i is given by κ i = E 0 x i 2. Using the convexity of SOC, we get the following inequality. Ex i 2 κ i + (1/κ i )x T i E 0 Bx i. (6) Now the SOC constraints are replaced by linear constraints that are much easier to handle. In the special case with E 0 = ci, c +0, the problem becomes simple and may be used as the initial problem. min B,ξ,w,b s.t. r log B + (1 r)trb + i y i ( x T i w + b ) + ξ i x T i Bx i C i ξ i w 2 1, ξ 0, i I, B 0 (7) where we define ξ i = κ 2 i ζ i and C i = 1 t 2 i ν. This formulates the ellipsoidal support vector machines primal problem. Note the Taylor approximation gets less accurate when B 2 becomes larger, which is the cost for making the formulation feasible for kernelization done in Section 2.2. Problem 7 has some interesting similarity with other methods. By putting B = Σ 1, it can be seen as a variant of MVCE problem in which the radius in the original problem is modified to a prediction dependent constraint. Hence it can be viewed as a supervised version of [14, 5]; unlike EKM, e-svm solves the classification problem at the same time. Shivaswamy et al. s formulation for handling missing and uncertain data [16] looks similar to Problem 4, where the metric in margin is given by the uncertainty in the data point. 270

284 In e-svm, margin is given by the B-norm, which is r log B 1 in the objective. By the matrix determinant lemma, the following equality can be shown to optimized simultaneously. hold; B 1 = I + A1/2 XX T A 1/2 (1 r) 1 r r 2.1 Dual formulation I, where X is the data matrix X = [x 1...x m ] T and A is a diagonal It can be readily shown that Problem 7 is a convex matrix whose elements are give by A i,i = α i. Note optimization problem with no duality gap. Hence the that the last factor is a constant so it can be ignored. complementarity can be used to solve the primal and By employing the kernel defined feature mapping the dual problems, just like SVMs. The Lagrangian is x φ(x), or XX T K, we have given as follows: L = r log B + (1 r) trb + I + 1 (1 r) A1/2 XX T A 1/2 1 I + C i ξ (1 r) A1/2 KA 1/2 i i ( ( α i yi x T i w + b ) x T ) = I + 1 i Bx i ξ (1 r) A1/2 ZZ T A 1/2 i i +γ ( w ) = π T ξ tr(bd), I + 1 (1 r) ZT AZ where α, γ, π and D are the Lagrange multipliers for the classification constraints, norm constraint on w, nonnegativity on ξ and positive semidefiniteness on B, respectively. The optimality condition gives the following relations 2 : ( B 1 = 1 (1 r)i + α i x i x T i, D = 0 r i w = 1 α i y i x i, y i α i = 0, C i α i π i = 0. 2γ i i Thus using the above equations the dual problem is written as follows: max α,γ s.t. r log B 1 1 α i α j y i y j x T i x j γ 4γ i,j ( ) B 1 = 1 (1 r)i + α i x i x T i r i y i α i = 0, 0 α i C i, γ > 0 (8) i A pleasant surprise is that B 1 is always positive definite since α i 0, which is a great advantage, allowing us to remove the constraint B 1 0 in (8). 2.2 Kernel formulation In this subsection, we show how Problem 8 is kernelized. For notational convenience, we use the matrix notation as well as the vector notation wherever appropriate. Note Problem 8 is very similar to the SVM problems, with the only difference being the additional 2 The log B term forces B to be full-rank. Thus D = 0 holds by complementarity. ) where K is decomposed as K = ZZ T From the 2nd line to the 3rd line the Sylvester s determinant theorem, a generalization of the Matrix determinant lemma, is used. Note also the dimensionality of I changes before and after the theorem is applied. In general, Z can be any matrix such that K = ZZ T including Z = K 1/2 so that a rank reduction method or sparsification method such as the incomplete Cholesky factorization may be used. After removing the constant terms, the kernel e-svm optimization problem is given by max r log I + 1 T α i z i z i (1 r) i 1 y i y j α i α j K ij γ 4γ i,j s.t. y i α i = 0, 0 α i C i, γ 0, (9) i with z i being the transpose of the i-th row of the matrix Z: Z = [z 1... z m ] T. 3 Sequential minimal optimization Although Problem 9 can be solved by an optimization package, a customized solver should be developed to take advantage of its similarity to the familiar SVM formulation; ideally an SVM solver can be modified to handle e-svm. For this purpose, we develop a variant of SMO for e-svm. The differences from the standard implementation of SMO include w 2 being normalized to one, step 271

285 size optimization formula, and KKT conditions. The weight normalization concerns optimization with respect to γ and can be done via the iterative projection. Step size optimization and active set selection using the KKT condition are done very similar to those for SVM. This section focuses on describing essential differences as a guide to implementation. 3.1 Optimality conditions SMO chooses an active set, a pair of data points, to optimize at any iteration. The selection of a pair critically affects the convergence speed. We adopt the selection heuristic described in [9]: choose ones that violate the KKT condition most. This subsection derives the KKT condition and thus gives the criterion for choosing the active set. First, consider the dual of (9). The Lagrangian is given by 1 r L = r log I + 1 T α i z i z i r r i + 1 y i y j α i α j K ij + γ η 4γ i i ij δ i α i + µ i (α i C i ). Solving the optimality conditions, we have y i α i T (F i γ)y i δ i + µ i z i Bzi = 0 (10) γ = y i y j α i α j K ij /2. (11) ij with F i = 1 2γ Kij y j α j and B = ( 1 r r I + i α iz i z ) T 1 i. Hence, by the complementarity, we have the following KKT conditions: For α i = 0, δ i > 0, µ i = 0 (H i β)y i 0 For 0 < α i < C i, δ i, µ i = 0, (H i β) y i = 0 For α i = C i, δ i = 0, µ i > 0 (H i β) y i 0 with H i = F i y i z i T Bzi. Note the first term F i corresponds to that in [9] and the second term is newly introduced for the e-svm problem. This means that replacing F i by H i suffices to establish a version of the SMO algorithm for e-svm and can be easily integrated into an existing SVM solver. adopted to the existing SMO algorithm. Another important piece in SMO algorithm is to find the optimal step size. The incremental step for α i can be expressed as α new = α old + s (e i y i y j e j ), (13) which satisfies the constraint i y iα new i = 0 given α old is a feasible solution. e i is a vector of zeros except for the i-th element being unity. Consider the following objective function, U(s), after removing any constant terms with respect to s: U(s) = r log det B 1 i,j 1 4γ αnew i α new j y i y j K ij γ. (14) The first term is modified using the update formula: B 1 = Bold 1 + s ( zi z T T i y i y j z j z ) j [ r ] = I + s T z i B old [z r T i y j z j ] y i z j B old s r = ω s ii r y jω ij s r y iω ij 1 s r y const (15) iy j ω jj where ω ij is defined to be ω ij = z i T Bold zi and the matrix determinant lemma is used for deriving the 2nd line. The resulting matrix is merely a 2 2 matrix determinant and easily expandable. Likewise, we can rewrite the second term in (14) as follows: i,j α new i α new j y i y j K ij = i,j α old i α old j y i y j K ij 4βsy i (F i F j ) s 2 (K ii 2K ij + K jj ). Hence by putting all the pieces together, we have the following optimality condition on s. 1 U(s) log det B = r 1 α i α j y i y j K ij = 0 s s s 4γ ra 1 a 3 + (2ra 2 a 1 a 3 a 4 )s (a 2 a 3 + a 1 a 4 )s 2 a 2 a 4 s 3 = 0 with a 1 = r(ω ii y i y j ω jj ), a 2 = y i y j (ω 2 ij ω iiω jj ), a 3 = y i (G i G j ), a 4 = Kii+Kii 2Kij 2γ. This is merely a cubic equation and can be solved analytically. ij 3.2 Step size computation As explained, the KKT condition for e-svm is easily 3.3 Computing B At each iteration, access to B is needed to calculate ω s. Specifically, the diagonal elements ω ii are required 272

286 for the KKT violaiton check and ω ij as well as ω ii and ω jj for the step size computation concerning an update of α i and α j. Since we solve the dual α, as well as γ in SMO, B 1 is easily obtained, but getting B, in a naive way, would require an inverse matrix operation that is never done in practice. A way to efficiently compute B is to employ the rank-one update of matrix inversion and factorize the matrix in the following way: B = B0 + i σ iv i v T i. By using the Woodbury formula, B is updated at each SMO step involving update of α i and α j : Bnew = B old + σ i v i v T i + σ j v j v T j, where v i = B old z i, ω ii = ( ) z T i v i, σ i = s r+sω ii. v j = Bold + σi v i v T i z j, σ j = sy iy j r sy iy jz jt v j. Note this decomposition formula on B enables us to do an incremental update of ω: ω new kl = ω old kl + σ i z i T z k z i T z l + σ j z j T z k z j T z l where ωkl old T = z Bold k zl. This update formula is particularly useful when exploiting efficiency; it is evident that computing ω kl with B close to full-rank or dense is overkill as SMO requires many updates. ω ij s that are cached will be updated via the vector-vector multiplications. For ω ij s that are not in the cache may require the full matrix-vector multiplication, if B is computed and kept in memory. Otherwise, the conjugate gradient would need to be employed to calculate ω ij from B 1, see [7], for example. A recommendation is that the diagonal elememts ω ii are kept in memory as all of them are used in any case for the KKT condition violation check. 4 Related methods Recently, several methods are proposed for improving generalization ability of SVMs. Ellipsoidal kernel machines (EKM) are ellipsoidal gap-tolerant classifiers that have lower bound of the VC dimension than that of SVMs. The EKM algorithm first finds the minimum volume ellipsoid enclosing data points. Then the normal SVM is applied to the transformed space where data points are placed in a hypersphere. Relative margin machines (RMM) addresses the issue of scale invariance in SVMs. Being motivated by a simple example where enlarging one feature dimension can drastically change the solution of an SVM, RMM regularizes the amplitude of prediction value, or projection of data point onto the weight vector w so as to reduce the effect of influence from badly scaled features. 5 Experimental study 5.1 Comparison against BPM It is important to examine the quality of e-svm solutions to understand how the approximation and relaxation used in e-svm affect the performance. In this paper, we directly compare the classification performance using the standard real dataset as a delegate to an assessment from an optimization perspective, which is left for a longer version of the paper. Also, BPM and e-svm are compared against SVM to understand how much performance lift can be realized. In this regard, a wide range of datasets in the UCI machine learning repository [1] are used. Both SVM and e-svm are implemented in pure MATLAB, using the SMO. Note ν-svm formulation is adopted in this study, since e-svm is based on ν-svm. BPM s implementation follow [6] and is implemented in C with an R interface. For the experimental setting, 100 randomizations are done and the average error rates in percent is reported in Table 1 and 2. In order to evaluate significance of statistics, the paired t-tests are conducted for comparing BPM with SVM, and e-svm with SVM. SVM. Bold numbers denote the test results being significant. Both hard and soft margin/boundary cases are examined. The training and testing splits are identical to [10] except for sonar, iono, wisconsin-breastcancer (WISC-BC). The splits for sonar are set identical to [6] and those for iono are created randomly in advance and identical splits are applied for all methods. For e- SVM and SVM, the tolerance of KKT violation is set to For BPM the tolerance parameter for convergence check is set to Model parameters are selected following [10]: using five-fold cross validation on the first five set and take the median of the best parameters. The best model parameters are fixed to conduct the 100 repetitions. Parameters including C and r for e-svm, ν for SVM and λ in BPM are selected by the cross validation, in which parameter values are used to cover a wide range of the search space. For hard margin e-svm, r is set to The radial basis function kernel is used for this experiment. 273

287 Table 1: Error rates for hard boundary/margin classifiers Data set SVM BPM e-svm thyroid 4.96 (.24) 4.24 (.22) 4.42 (.25) heart (.40) (.33) (.32) diabetes (.21) (.22) (.24) wave (.12) (.08) (.07) banana (.14) (.10) (.08) wisc-bc 4.22 (.13) 2.56 (.10) 3.28 (.12) bupa (.39) 34.5 (.38) (.35) german (.22) (.24) (.27) brest (.51) (.48) (.51) sonar (.38) (.36) (.38) iono 7.94 (.25) (.25) 5.92 (.21) The kernel width parameters are set to identical to those reported in [6], except for iono, for the rest of the datasets, those reported for SVMs in [10] are used. The overall performance for BPM and e-svm is very similar. This suggests that the approximations made to formulate e-svm do not affect the classification performance for the datasets examined. In comparison with SVM, the hard boundary/margin classifiers significantly outperform those of SVM. For soft boundary/margin cases, however, the advantage is reduced. Although performance of BPM and e-svm is slightly better than that of SVM, the difference is small, which is a consistent observation with [6]. Scalability The computational cost is illustrated to see how e- SVM scales in comparison with BPM, using the adult dataset [1]. The entire dataset is split into training, testing and validation. Model selection is done using the validation set of size 1000 and fixed for the rest of the experiments. The size of the training set is increased from 100 up to Test performance is observed to check if the there is no significant difference between the methods. Obviously, e-svm runs much faster than BPM as the data size grows. Using the log-log fit, e-svm scales as m 2.1 whereas BPM takes as much as m 2.7. Note SVM takes only 30 secs for the problem of size 4,000 (not shown); although e-svm significantly beats BPM, there is much room to improve the efficiency when compared with SVM. 5.2 High dimensional datasets The datasets used in Subsection 5.1 are in relatively low dimensions. With the ability to adjust the margin metric, it is more interesting to see the performance Table 2: Error rates for soft boundary/margin classifiers Data set SVM BPM e-svm thyroid 5.05 (.22) 4.32 (.20) 4.44 (.22) heart (.30) (.29) (.32) diabetes (.17) (.18) (.19) wave 9.88 (.04) (.05) (.04) banana (.07) (.04) (.05) wisc-bc 2.70 (.10) 2.32 (.10) 2.77 (.09) bupa (.30) (.30) (.32) german (.20) (.22) (.22) breast (.43) (.46) (.48) sonar (.38) (.36) (.36) iono 5.94 (.18) (.23) 5.16 (.18) time (sec) BPM ESVM(SMO) size Figure 1: Scalability for BPM and e-svm with a high dimensional dataset. We compare the e- SVM with other methods mentioned in Section 4. The 20-newsgroup (20NG) and mnist [1] are used for this purpose. EKM is implemented in MATLAB Optimization Toolbox. RMM is implemented in SDPT3 [17]. For 20NG, the number of words is limited to the top 5,000 frequent words and each data point is normalized to length one after the tf-idf weighting. There are 11,250 data points in the training and 7,486 in the testing set. mnist contains 60,000 observations in the training and 10,000 in the testing set in 784 dimensions. The one-versus-one strategy is applied to solving the multiclass problem. We set the number of training data points to 50, 100 and 200 for each pair of classes in training (100, 200 & 400 in training samples for both classes). The rest of the training data is reserved as the validation set that is used for model selection. The linear kernel is used for all the methods. The results are shown in Table 3. For 20NG, e-svm outperforms other methods. Interestingly, both EKM and RMM show slightly worse performance than SVM. This indicates the transformation in EKM and regularization for scaling resistance in RMM seem to have adversary effect. In turn, for mnist, RMM outperforms other 274

288 Table 3: Error rates for SVM, EKM, RMM and e-svm SVM EKM RMM e-svm ng ng ng mnist mnist mnist methods, which confirms RMM works well when scale invariance is desired. 6 Conclusion In this paper, the ellipsoidal support vector machine was proposed. The formulation is based on that of the familiar SVM and the sequential minimal optimization was successfully adapted to solve the e-svm optimization problem. The framework is flexible for possible modification of loss functions or application to other problems. Also, by the minimum volume ellipsoid interpretation, it can be used to learn the metric guided through the maximum margin framework. None of these advantages is available in BPM and thus novel in e-svm. Furthermore, e-svm showed comparable performance with BPM, indicating the approximations in e-svm do not affect the performance over wide variety of datasets. The SMO algorithm was shown to provide acceptable scalability and was much more efficient than BPM. e-svm thus can be applicable to real world applications up to several thousands of data points. Future work includes developing a more efficient algorithm that handles caching better and an algorithm for outof-memory computation. References [1] C. L. Blake and C. J. Merz. UCI Repository of machine learning databases, mlearn/ MLRepository.html. [2] Stephen Boyd and Lieven Vandenberghe. Convex Optimization. Cambridge University Press, New York, NY, USA, [3] Pai-Hsuen Chen, Chih-Jen Lin, and Bernhard Schölkopf. A tutorial on ν-support vector machines: Research articles. Appl. Stoch. Model. Bus. Ind., 21(2): , [4] Jason V. Davis, Brian Kulis, Prateek Jain, Suvrit Sra, and Inderjit S. Dhillon. Information-theoretic metric learning. In ICML 07: Proceedings of the 24th international conference on Machine learning, pages , New York, NY, USA, ACM. [5] A.N. Dolia, T. De Bie, C.J. Harris, J. Shawe-Taylor, and D.M. Titterington. The minimum volume covering ellipsoid estimation in kernel-defined feature spaces. In Proceedings of the 17th European Conference on Machine Learning (ECML 2006), Berlin, September [6] Ralf Herbrich, Thore Graepel, and Colin Campbell. Bayes point machines. Journal of Machine Learning Research, 1: , [7] Steven C. H. Hoi, Rong Jin, and Michael R. Lyu. Learning nonparametric kernel matrices from pairwise constraints. In ICML, pages , [8] C. Hsieh, K. Chang, C. Lin, S. Keerthi, and S. Sundararajan. A dual coordinate descent method for large-scale linear SVM. pages , [9] S. S. Keerthi, S. K. Shevade, C. Bhattacharyya, and K. R. K. Murthy. Improvements to Platt s SMO algorithm for SVM classifier design. Neural Comput., 13(3): , [10] Gunnar Rätsch. Benchmark repository, projects/bench/benchmarks.htm. [11] Pál Ruján. Playing billiards in version space. Neural Comput., 9(1):99 122, [12] B. Schölkopf and A. J. Smola. Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. MIT Press, [13] Shai Shalev-Shwartz, Yoram Singer, and Nathan Srebro. Pegasos: Primal Estimated sub-gradient SOlver for SVM. In ICML 07: Proceedings of the 24th international conference on Machine learning, pages , New York, NY, USA, ACM. [14] P. Shivaswamy and T. Jebara. Ellipsoidal kernel machines. Artificial Intelligence and Statistics, [15] P. Shivaswamy and T. Jebara. Relative margin machines. In Neural Information Processing Systems, [16] Pannagadatta K. Shivaswamy, Chiranjib Bhattacharyya, and Alexander J. Smola. Second order cone programming approaches for handling missing and uncertain data. J. Mach. Learn. Res., 7: , [17] K. C. Toh, M. J. Todd, and R. Tutuncu. SDPT3 a Matlab software package for semidefinite programming. Optimization Methods and Software, 11: , [18] Theodore B. Trafalis and Alexander M. Malyscheff. An analytic center machine. Mach. Learn., 46(1-3): , [19] V. N. Vapnik. The Nature of Statistical Learning Theory. Springer, New York,

289 2009 Technical Report on Information-Based Induction Sciences 2009 (IBIS2009) Statistical Performance Analysis in Probabilistic Image Processing by Belief Propagation Shun Kataoka Muneki Yasuda Kazuyuki Tanaka Abstract: We propose new schemes for evaluating statistical performances of image restoration based on Bayesian statistics. The schemes are constructed by means of belief propagation, which is a powerful approximate methods in probabilistic inference. They are reduced to problems involving the solution of simultaneous integral equations for distributions of messages in belief propagation. Our schemes are proposed for two cases. One of them is for the case in which the original image is given explicitly. The other one is for the case in which the prior probability for the original image is given. We show some numerical results of the statistical performance analysis for the probabilistic restoration of binary images. Keywords: Bayesian statistics, Markov random fields, belief propagation, probabilistic information processing, statistical mechanical informatics, Bayesian network 1 (Probabilistic Image Processing) (Bayesian Statistics),.,,,,., (Belief Propagation: BP),, [1, 2, 3].,.,., , tel , Graduate School of Information Sciences, Tohoku University,6-3-09,Aramaki-Aza-Aoba, Aobaku, Sendai [4, 5].,,. (Low Density Parity Check: LDPC), (Code Division Multiple Access: CDMA) (Replica Method) [6, 7, 8].,. [6, 9, 10, 11].,,

290 ,,. 2, ,. i, V = {1, 2,, V }., {i, j} E. i f i, i f i = +1, f i = 1. f = (f 1, f 2,, f).. ( ) P f J = 1 exp Z f J (f i f j ) 2 2 (1) {i,j} E J, Z f. i f i, i g i. f g = (g 1, g 2,, g V ) T ( P g f, ) σ ( ) { } V 1 = exp 1 2πσ 2 2σ 2 (g i f i ) 2 i V (2). σ 2. (1) (2) g f. h = (h 1, h 2,, h V ), J σ 2 α β, g h ( ) P h g, α, β = 1 exp α h i h j + β g i h i Z h {i,j} E i V (3) ( ).Z h P h g, α, β. 2.2 (3) i h i (Maximum Posterior Marginal: MPM). i h i P i (h i g, α, β) = ( ) P h g, α, β h\hi (4). 2, h i h i h i g, α, β ( ) h i = arg max P i (h i g, α, β) = sgn h i g, α, β h i (5) g h i h 2.3 MPM, f h. D h ( f, h ) 1 ( ) 1 δfi,h V i i V = V. f i sgn( h i g, α, β ) (6) i V MPM D h ( f, h ). D h ( f, h ) f g [D h ] f,g d g D h ( f, ( h )P g ) ( ) f, σ P f J f = d g f i sgn( h i g, α, β ) 2 V i V f ( P g f, ) ( ) σ P f J (7) 277

291 (1),. (7)., D h ( f, h ) g [D h ] g d g D h ( f, ( h )P g f, ) σ = d g f i sgn( h i g, α, β )P 2 V i V ( g f, ) σ (8) f. (8)( [D) h ] g (7) [D h ] fg P f J f. 3, (7) (8)., [12] (3) (4) { λ ij, λ ji {i, j} E }. P i (h i g, α, β) 1 exp βg Z i i + λ ij h i (9) j i Z i, i = { j {i, j} E } i. { λ ij, λ ji {i, j} E }. λ ij = tanh 1 tanh(α) tanh βg j + k j\{i} λ jk (10) 1 (Bethe Approximation) (6) (9) D h ( f, h ) V f i sgn βg i + λ ij (11) j i i V. (10) (11), f g h. 3.2 (7) (8)... λ ij P 1 ij (λ ij),., (1) J (2) σ, α, β λ ij P 1 ij (λ ij) (10). Pij(λ 1 ij ) = dg j P j (g j ) k j\{i} dλ jk P 1 jk(λ jk ) { δ [λ ij tanh 1 tanh(α) tanh βg j + k j\{i} λ jk }] (12) δ[ ] Dirac, P i (g i ). P i (g i ) dg j P j V \{i} f ( g f, ) ( ) σ P f J (13) P i (g i ), P i (g i ),. (12) (7) [D h ] f,g 278

292 . [D h ] f,g dg i P i (g i ) 2 V i V f i j i dλ ij Pij(λ 1 ij )f i sgn βg i + λ ij (14) j i f P 2 ij (λ ij). Pij(λ 2 ij ) = dg j k j\{i} dλ jk P 2 jk(λ jk ) { 1 exp (g j f j ) 2 } 2πσ 2 2σ 2 { δ [λ ij tanh 1 tanh(α) tanh βg j + k j\{i} λ jk }] (15) (8) [D h ] g. [D h ] g dg i dλ ij P 2 V i V 2πσ 2 ij(λ 2 ij ) j i f i sgn βg i + { λ ij (gi f i ) 2 } exp 2σ 2 (16) j i 4 3 D h (14), (16) α, β. (2) σ 1. (12)(15) 4.1, (1) J, [D h ] f,g. [D h ] f,g (12), P 1 ij (λ ij) (14). α, β [D h ] f,g 1. (1) J β = 1 [D h ] f,g. 1, 2 [D h ] f,g (α, β) = (0.465, 1) α = J, β = σ 2, [6, 9] 2. 1 {J, σ} {α, β},. ( ).. [D h ] f,g α β : J = (1) σ 2 = 1 [D h ] fg (α, β). [D h ] f,g α : J = (1) σ 2 = 1 β = 1 [D h ] fg α-. 279

293 4.2 3( ) 1, [D h ] g. [D h ] g (15), P 2 ij (λ ij) (16). α, β [D h ] g 4. 4 β = 1 5. [D h ] g α = 0.56 α = 0.56 [D h ] g (α, β) = (0.56, 1). 2 [D h ] g [D h ] f,g f. f, [D h ] f,g [D h ] g. 4,.. [D h ] g : α : 3 σ 2 = 1 [D h ] g (α, β). β [D h ] g α : 3 σ 2 = 1 [D h ] g β = 1 α. 5.,. 2,, (Expectation Maximization: EM) [5, 10, 11]. (No , No ) COE Center of Education and Research for Information Electronics Systems. [1] A. S. Willsky: Multiresolution Markov models for signal and image processing, Proceedings of the IEEE, vol.90, no.8, pp (2002). [2] K. Tanaka, Statistical-mechanical approach to image processing, Journal of Physics A: Mathematical and General, vol.35, no.37, pp.r81-r150 (2002). 280

294 [3],, (2006) [4] K. Tanaka and J. Inoue, Maximum likelihood hyperparameter estimation for solvable Markov random field model in image restoration, IEICE Transactions on Information and Systems, vol.e85- D, no.3, pp (2002). [5] K. Tanaka and D. M. Titterington, Statistical trajectory of approximate EM algorithm for probabilistic image processing, Journal of Physics A: Mathematical and Theoretical, vol.40, no.37, pp (2007). [6] H. Nishimori, Statistical Physics of Spin Glass and Information Processing, Oxford University Press (2001). [7] Y. Kabashima and D. Saad, Statistical mechanics of low density parity check codes, Journal of Physics: Mathematical and General, vol.37, no.6, pp.r1-r43 (2004). [8] T. Tanaka, A statistical-mechanics approach to large-system analysis of CDMA multiuser detectors IEEE Transactions on Information Theory, vol.48, no.11, pp (2002). [9] H. Nishimori and K. Y. M. Wong, Statistical mechanics of image restoration and error-correcting codes, Physical Review E, vol.60, no.1, pp (1999). [10] J. Inoue and K. Tanaka, Dynamics of the maximum marginal likelihood hyperparameter estimation in image restoration: Gradient descent versus expectation and maximization algorithm, Physical Review E, vol.65, no.1, Article No (2002). [11] J. Inoue and K. Tanaka, Mean field theory of EM algorithm for Bayesian gray scale image restoration, Journal of Physics A: Mathematical and General, vol.36, no.43, pp (2003). [12] T. Morita, Spin-glass and ferromagnetic phases of the random-bond Ising model on the Bethe lattice, Physica A, vol.125, no.2/3, pp (1984). 281

295 2009 Technical Report on Information-Based Induction Sciences 2009 (IBIS2009) Chow-Liu A generalized version of Chow-Liu algorithm for data-mining. Yuu Ishida Joe Suzuki 1 (Ω, F, µ) F X : Ω R X F X ( f X ) X, Y I(X, Y ) := I(X, Y ) := x X(Ω) y Y (Ω) P XY (x, y) log P XY (x, y) P X (x)p Y (y) f XY (x, y) log f XY (x, y) f X (x)f Y (y) dxdy X, Y Kullback-Leibler Chow-Liu ( ) N X (1),, X (N) Markov ( ) I(X (i), X (j) ), , yawn.ikaryaku@gmail.com Graduate school of Science School of Science,Osaka University, Machikaneyamacho1-1.Toyonakashi.Osaka , , suzuki@math.sci.osaka-u.ac.jp Graduate school of Science School of Science,Osaka University, Machikaneyamacho1-1.Toyonakashi.Osaka Kullback-Leibler (Chow-Liu, 1968) n {(x i,1,, x i,n )} n i=1, x i,j X (j), j = 1,, N empirical Chow-Liu Chow-Liu N (Ω, F, µ) N Kullback-Leibler () MDL (2 ) (Suzuki, 1993)N N U E E := {{X, Y } X, Y U, X Y } U, E (U, E) G = (U, E) U G G E G G G G G G 282

296 2.2 Kullback-Leibler B R Borel X : Ω R D B = {ω Ω X(ω) D} F (F ) X (Ω, F, µ) 2 F ν(a) = 0 = µ(a) = 0 µ ν µ << ν A F µ << ν dµ dν µ(a) = A fdν := f (Radon-Nikodym ) D(µ ν) := log( dµ dν )dµ µ ν Kullback-Leibler(K-L) D(µ ν) 0 D(µ ν) = 0 µ = ν K-L µ ν X (X(Ω) < ) P (x) := µ X ({x}), x X(Ω), Q(x) := ν X ({x}), x X(Ω) Q(x) = 0 = P (x) = 0 D(P Q) = x X(Ω) P (x) log P (x) Q(x) f, g g(x)dx = 0 = f(x)dx = 0 D(f g) = f(x) log f(x) g(x) dx (Ω, F, µ) 2 X, Y X, Y X, Y σ- F X, F Y, F XY µ X, µ Y, µ XY µ X µ Y F XY I(X, Y ) := D(µ XY µ X µ Y ) X, Y X, Y (X(Ω), Y (Ω) < ) P X (x) := µ X ({x}), x X(Ω), P Y (y) := µ Y ({y}), y Y (Ω) P XY (x, y) := µ XY ({x}, {y}), x X(Ω), y Y (Ω) I(X, Y ) = x X(Ω) y Y (Ω) P XY (x, y) log P XY (x, y) P X (x)p Y (y) f X, f Y, f XY I(X, Y ) = 2.3 Chow-Liu f XY (x, y) log f XY (x, y) f X (x)f Y (y) dxdy n ( ) P X (1),...X (n)(x(1),..., x (N) ), x (1) X (1) (Ω),...,x (N) X (N) (Ω) ˆP X (1),...,X (N)(x(1),..., x (N) ) := P X (a j ) X (a λ [j] )(x (aj), x (a λ ) [j] ) j N (Dendoroid ) 1 λ[j] j 1,j 1,λ[1] = 0, P X (j) X (0)( x(0) ) = P x (j)( ) (a 1,..., a N ) (1,..., N) 1 U := {X (i) i = 1,..., N}, E E := {{X (i), X (j) } i j} G = (U, E) G E = {{X (a j), X (a λ[j]) } j = 1,..., N} 1 λ[j] j 1,j 1,λ[1] = 0, (1,..., N) (a 1,..., a N ) 1 (Chow-Liu) [1] D := (P X (1),...,X (N) ˆP X (1),...,X (N)) P x (1)...X (N)(x(1),..., x (N) ) x (1) X (1) (Ω),...,x (N) X (N) (Ω) P log x (1)...X (N)(x(1),..., x (N) ) j N P X (a j ) X (a λ [j] )(x (aj), x (a λ ) [j] ) (a 1,..., a N ) λ[k]2 k N I(i, j) := x (i) X (i) (Ω),...,x (j) X (j) (Ω) log P X (i) X (j)(x(i), x (j) ) P X (i)p X (j) P X (i) X (j)((x(i), x (j) ) U := {X (i) i = 1,..., N}, w(x (i), X (j) ) := I(X i, X j ) 283

297 : D = (P X (1),,X (N) ˆP X (1),...,X (N)) P X (1)...X (N)(x(1),..., x (N) ) x (1) X (1) (Ω),...,x (N) X (N) (Ω) P log X (1)...X (N)(x(1),..., x (N) ) j N P X (a j ) X (a λ [j] )(x (aj), x (a λ ) [j] ) = I(X (aj), X (a λ ) [j] ) + H(X (1),, X (N) ) a λ[j] a j N H(X (i) ) i=1 H(X (1),, X (N) ) := X (1),,X (N) P X (1),,X (N)(x(1),, x (N) ) log P X (1),,X (N)(x(1),, x (N) ) H(X (i) ) := x (i) P X (i)(x (i) ) log P X (i)(x (i) ) D(P ˆP ) a λ[j] a j I(X (a λ j ), X (a λ [j] ) ) 1 {X (a j), X (a λ[j]) } n x n = {x 1, x 2,...x n }, x i X(Ω) = N j=1 X(j) (Ω),i = 1,..., n I(i, j) I n (i, j) := c i,j [x, y] log c i,j[x, y] c i [x]c j [y] 0 x X (i),y X (j) c i [x],c i,j [x, y] x 1,..., x n X (i) = x X (i) (Ω) X (i) = x X (i) (Ω)X (j) = y X (j) (Ω) D(P X (1),...,X (n) ˆP X (1),...,X(n)) H[π](x n ) := c 1,...,N (x (1),..., x (N) ) x (i) X (i) (Ω) log c aλ[j] (x (a λ[j]) ) c aj,a λ[j] (x, (a j) x (a λ[j]) ) c 1,...,N (x (1),..., x (N) ) X (1) = x (1) X (1) (Ω),..., X (N) = x (N) X (N) (Ω) n x n H[π](x n ) T 1. E {} 2. E = {} (a) I n (i, j) {X (i), X (j) } E E E {X (i), X (j) } (b) (U, E {X (i), X (j) }) E E {X (i), X (j) } (c) T (U, E) 1 I n (i, j) 1. I n (1, 2) X (1), X (2) 2. I n (1, 3) X (1), X (3) 3. I n (2, 3) X (2), X (3) X (2), X (3) 4. I n (1, 4) X (1), X (4) 5. 1: (i,j) i j I n (i, j) X (1) X (3) X (2) X (4) X (1) X (3) X (2) X (4) 1 ( Chow-Liu ) 284 X (1) X (3) X (2) X (4) X (1) X (3) X (2) X (4)

298 2.4 Chow-Liu n x i X(Ω)i=1,2,,n π H[π](x n ) K[π] L[π](x n ) := H[π](x n ) + K[π] 2 log n L[π](x n ) H[π](x n ) π x n K[π] 2 log n π [3, pp ] J n (i, j) := I n (i, j) ((i) 1)( (j) 1) 2 log n ( (i) := X (i) (Ω) ) H[π](x n ) 2 U := {X (i) i = 1,..., N} E := {{X (i), X (j) } i j} G = (U, E) G E = {{X (a j), X (a λ[j]) } j = 1,..., N}0 λ[j] j 1 j = 1,..., Nλ[1] = 0 (1,..., N) (a 1,..., a N ) 3. (a) J n (i, j) {X (i), X (j) } E E {X (i), X (j) } (b) (U, E {X (i), X (j) }) J n (i, j) 0 E E {X (i), X (j) } (c) F (U, E) 2 I n (i, j) (1) = 5,α (2) = 2,α (3) = 3, α (4) = 4 1. J n (1, 2) = 8 X (1), X (2) 2. J n (2, 3) = 6 X (2), X (3) 3. J n (1, 3) = 2 X (1), X (3) 4. I n (2, 4) = 1 X (2), X (4) 5. J n < 0 2: i j I n (i, j) (i) (j) J n (i, j) X (1) X (3) X (1) X (3) 1 λ[j] j 1 0 λ[j] j 1 X (2) X (4) X (2) X (4) 2 () (suzuki,1993)[2] : n x n : L[π](x n ) F X (1) X (2) X (3) X (4) X (1) X (3) X (2) X (4) 1. E {} 2. E = {} n x n L[π](x n ) F 285

299 : X (a j) X a λ j Kπ= N (a j 1)(a λj 1) j=0 L[π](x n ) N I n (a j, a λj ) j=0 = N (a j 1)(a λj 1) log n j=0 N {I n (a j, a λj ) 1 2 (a j 1)(a λj 1) log n} j=0 2 (X (a j), X a λ j) J n (i, j) < 0 i,j X (i), X (j) (X (i), X (j) ) n j=1 J n(a j, a λj ) X (j) (Ω) 2 X (j) I n (i, j) J n (i, j) J n (i, j) {X (a j), X (a λ[j]) } 3 Chow-Liu 3.1 X (1),, X (N) Chow-Liu Kullback-Leibler µ j i : X (i) X (j) µ i,j : X (i), X (j) I(X (i), X (j) ) := D(µ ν) := dµ i,j log d2 µ i,j dµ i dµ j dµ log( dµ dν ) : j i i j i 0 i Radon-Nikodym dµ 1,,N d i j µ j i = dn µ 1,,N i j dµ j i = dn µ 1,,N N k=1 dµ k = dn µ 1,,N N k=1 dµ k N k=1 dµ k i j dµ j i d 2 µ i,j [ ] 1 dµ i dµ j i j,i 0 Kullback-Leibler D(µ 1,,N i j µ j i ) = + i j,i 0 I(X (i), X (j) ) dµ 1,,N log{ dn µ 1,,N N k=1 dµ } k I(X (i), X (j) ) Kullback-Leibler 3.2 X (i) N(0, σ 2 ) f X (i)(x (i) ) := 1 e x(i)2 2σ ii 2πσii X (i), X (j) f X (i) X (j)(x(i), x (j) 1 ) := e 2π Σ 1 2 Σ = ρ ij = I(i, j) = ( σ ii σ ji σ ij σ jj σij σiiσ jj 1 2 (x(i) ) p X (i) X (j)(x(i), x (j) ) x (j) )Σ 1 x(i) x (j) log p X (i) X (j)(x(i), x (j) ) p X (i)(x (i) )p X (j)(x (j) ) dx(i) dx (j) σii σ jj = log Σ 1 2 = log (1 ρ ij 2 )

300 I(i, j) ρ ij X (i) ρ ij Chow-Liu J n (i, j) = I n (i, j) 1 2 log n = log (1 ρ ij 2 ) log n 3 () : n x n : L[π](x n ) F 1. E {} 2. E = {} 3. (a) J n (i, j) {X (i), X (j) } E E {X (i), X (j) } (b) (U, E {X (i), X (j) }) J n (i, j) 0 E E {X (i), X (j) } (c) F (U, E) ρ 3.3 Chow- Liu f(x Σ): µ = 0, Σ R N N ˆΣ(x n ): x n = (x 1,, x n ) Σ H := log f(x ˆΣ(x n ))dx k: Σ MDL L := H + k log n 2 J n (i, j) := I n (i, j) 1 2 log n X (i), i = 1,..., N I n (i, j) J n (i, j) X i β i N i=1 β i i j β iβ j J n (i, j) := I n (i, j) β iβ j 2 log n 3.4 Chow-Liu Conjecture X (i) X (j) J n (i, j) := I n (i, j) ((j) 1) 2 log n ( (j) := X (j) (Ω) ) L[π](x n ) 4 Chow- Liu Kullback-Leibler(K-L) D(P Q) I(i, j) Chow-Liu 287

301 [1] C.K.CHOW,C.N.LIU: Approximating Discrete Probability Distributions with Dependence Trees.(1968) [2] J.Suzuki, A construction of bayesian networks from databases on an MDL principle,proc 9th UAI,Morgan,Kaufmann,pp ,1993. [3] : (2009) 288

302 2009 Technical Report on Information-Based Induction Sciences 2009 (IBIS2009) Kullback-Leibler Importance Estimation Procedure Restricted Boltzmann Machine Learning Algorithm in Restricted Boltzmann Machines using Kullback-Leibler Importance Estimation Procedure Tetsuharu Sakurai Muneki Yasuda Kazuyuki Tanaka Abstract: Deep Belief Networks (DBN) are generative neural network models with many layers of hidden units which were recently introduced along with a greedy layer-wise learning algorithm by Hinton et al. The main building block of a DBN is a bipartite undirected graphical model called restricted Boltzmann machine. In the present paper, we propose a new and less greedy learning algorithm for restricted Boltzmann machines within DBNs using Kullback-Leibler Importance Estimation Procedure. We also show its validity by comparing our proposed algorithm with the exactly calculated KL(P D PV D) learning algorithm using numerical experiments based on artificial data. Keywords: deep belief network, variational bound, restricted Boltzmann machine, learning algorithm, Kullback-Leibler Importance Estimation Procedure 1 Deep Belief Network (DBN) Hinton [1]Greedy [2] DBN Restricted Boltzmann Machine (RBM)[3] RBM RBM 2 2 DBN RBM [1, 4] DBN RBM RBM, , { tsakurai,muneki,kazu }@smapip.is.tohoku.ac.jp, Graduate School of Information Sciences, Tohoku University. NP-hard Contrastive divergence [5, 6] RBM Roux and Bengio DBN Variational Bound RBM [7] 3 DBN Greedy RBM DBN [7] RBM Sugiyama [8] Kullback- Leibler Importance Estimation Procedure (KLIEP) Roux and Bengio[7] RBM 289

303 2 Deep Belief Network Greedy 3 KLIEP RBM Deep Belief Network DBN Greedy RBM 2.1 Restricted Boltzmann Machine 1: RBM V = 4, H = 3 RBM 2 {0, 1} v = {0, 1} V h = {0, 1} H RBM V = {1,, V } H = {1,, H } RBM 1 ( ) P RBM (v, h Θ) = Z RBM (Θ) exp E(v, h Θ) Z RBM (Θ) E(v, h Θ) = i V a i v i j H b j h j i V (1) w ij v i h j j H (2) a = {a i i V }b = {b j j H} w = {w ij i V, j H} Θ = {a, b, w} v h P V (v Θ) h P H (h Θ) v P RBM (v, h Θ) = G V (v, b, w) Z RBM (Θ) P RBM (v, h Θ) = G H(h, a, w) Z RBM (Θ) ( ) exp a i v i i V (3) exp b j h j (4) j H G V (v, b, w) { ( 1 + exp b j + )} w ij v i (5) j H i V G H (h, a, w) 1 + exp a i + w ij h j i V j H (6) P H V (h v, b, w) P H V (h v, b, w) = exp {(b j + } i V w ijv i )h j 1 + exp (b j + ) i V w ijv i j H (7) P V H (v h, a, w) P V H (v h, a, w) = exp {(a i + ) } j H w ijh j v i i V 1 + exp (a i + ) j H w ijh j (8) v h M {d µ {0, 1} V µ = 1,, M} P D (v) P D (v) 1 M M δ(v i, d µ i ) (9) µ=1 i V δ(x, y) RBM Kullback- Leibler (KL) KL(P D P V ) = v P D (v) ln P D (v) P V (v Θ) (10) Θ = arg min Θ KL(P D P V ) (11) (11) O(e V + H ) CD 290

304 2.2 Deep Belief NetworkGreedy 2: L DBN 2 RBM 2 DBN 2 2 RBM Hinton DBN Greedy L DBN l h (l) {0, 1} ν(l) ν(l) l l = 0 h (0) = v h (l) P tr (h (l) h (l+1), W (l+1) ) {( = exp θ (l) i + j Ω l+1 w (l+1) ij ( i Ω l 1 + exp θ (l) i + j Ω l+1 w (l+1) ij ) h (l+1) j h (l) i h (l+1) j ) } (12) θ (l) h (l) w (l+1) h (l) h (l+1) Ω l l W (l) = {θ (l 1), w (l) } DBN Greedy STEP 1. l 0 STEP 2. h (l) h (l+1) P RBM (h (l), h (l+1) θ (l), θ (l+1), w (l+1) ) RBM {θ (l), θ (l+1), w (l+1) } h (l) h (l+1) STEP 3. W (l+1) W (l+1) h (l) h (l+1) (7) STEP 4. l l + 1 if(l L 2) STEP 2. if(l = L 1) Greedy Wake-Sleep [9] Hinton Greedy Variational Bound DBN [1]Roux and Bengio Variational Bound (11) RBM [7] Roux and Bengio 2.1 P D (v) RBM P V (v Θ) KL P D (v) P D V (v Θ) h,v 0 P V H (v h, a, w)p H V (h v 0, b, w)p D (v 0 ) (13) KL RBM Θ = arg min Θ KL(P D P D V ) (14) PV D (v Θ) P H V (h v, b, w) P V H (v h, a, w) (14) DBN Variational Bound L = 3 DBN Greedy RBM DBN [7] 3 Restricted Boltzmann Machine KLIEP Roux and Bengio[7] RBM 291

305 3.1 Roux and Bengio Roux and Bengio (14) PV D (v Θ) KL KL(P D P D V ) = v P D (v) ln P D (v) P D V (v Θ) (15) Θ P D (v) (9) Θ = {a, b, w} KL(P D P D V ) a i + 1 M = 1 M M µ=1 d µ i M f i (h, a i, w)w (h, Θ)P H V (h d µ, b, w) µ=1 h KL(P D P D V ) b j = 1 M M ( ) h j g j (d µ, b j, w) µ=1 h W (h, Θ)P H V (h d µ, b, w) (16) (17) { KL(P D PV D) = 1 M h j W i (h, Θ) w ij M µ=1 h } ( ) + d µ i h j h j f i (h, a i, w) d µ i g j(d µ, b j, w) W (h, Θ) P H V (h d µ, b, w) (18) f i (h, a i, w) g j (v, b j, w) exp (a i + ) j H w ijh j f i (h, a i, w) 1 + exp (a i + ) j H w ijh j (19) g j (v, b j, w) exp ( b j + i V w ) ijv i 1 + exp ( b j + i V w ) ijv i (20) W (h, Θ)W i (h, Θ) W (h, Θ) v = 1 M W i (h, Θ) v = 1 M P D (v)p V H (v h, a, w) P D V (v Θ) M P V H (d µ h, a, w) µ=1 P D V (dµ Θ) P D (v)p V H (v h, a, w) v i PV D (v Θ) M µ=1 d µ i P V H (d µ h, a, w) P D V (dµ Θ) (21) (22) (16)(18) (21)(22) W (h, Θ)W i (h, Θ) Kullback-Leibler Importance Estimation Procedure (21)(22) W (h, Θ)W i (h, Θ) P D (v)/pv D (v Θ) Sugiyama KLIEP[8] (16)(18) 2 P D (v)/p D V (v Θ) KLIEP W (h, Θ)W i (h, Θ) P D (v)/pv D (v Θ) ( ) P D (v)/pv D (v Θ) exp c i v i i V (23) P D (v) ( ) 1 Z(c, Θ) exp c i v i PV D (v Θ) i V Q V (v c, Θ) (24) Z(c, Θ) ( ) exp c i v i PV D (v Θ) (25) v i V KLIEP KL KL(P D Q V ) c c KLIEP c M M = f i (h, a i + c i, w)p H V (h d µ, b, w) µ=1 d µ i µ=1 h (26) (24) (21) (22) W (h, Θ) 1 G H (h, a + c, w) Z(c, Θ) G H (h, a, w) (27) 292

306 W i (h, Θ) f i(h, a i + c i, w) G H (h, a + c, w) Z(c, Θ) G H (h, a, w) (28) (27)(28) (16)(18) KL(P D PV D) a i 1 M d µ i M + 1 M f i (h, a i, w) MZ(c, Θ) µ=1 µ=1 G H(h, a + c, w) P H V (h d µ, b, w) (29) G H (h, a, w) KL(P D PV D) 1 M ( h j b j MZ(c, Θ) µ=1 h ) g j (d µ GH (h, a + c, w), b j, w) P H V (h d µ, b, w) G H (h, a, w) (30) KL(P D PV D) w ij 1 M ( d µ i MZ(c, Θ) h j + h j f i (h, a i + c i, w) µ=1 h ) h j f i (h, a i, w) d µ i g j(d µ GH (h, a + c, w), b j, w) G H (h, a, w) P H V (h d µ, b, w) (31) (26) 1 M M µ=1 d µ i = 1 MZ(c, Θ) h M f i (h, a i + c i, w) µ=1 h G H(h, a + c, w) P H V (h d µ, b, w) G H (h, a, w) (32) (29) (26) c KL(P D PV D) a i 1 M ( f i (h, a i + c i, w) MZ(c, Θ) µ=1 h ) GH (h, a + c, w) f i (h, a i, w) P H V (h d µ, b, w) G H (h, a, w) (33) KLIEP (29)(31) (30) (31)(33) Z(c, Θ) 1 c KL(P D P D V ) a ib j w ij a i b j w ij KLIEP a i M ( ) f i (h, a i + c i, w) f i (h, a i, w) µ=1 h G H(h, a + c, w) P H V (h d µ, b, w) (34) G H (h, a, w) M ( ) b j h j g j (d µ GH (h, a + c, w), b j, w) G H (h, a, w) µ=1 h P H V (h d µ, b, w) (35) M ( w ij d µ i h j + h j f i (h, a i + c i, w) µ=1 h ) h j f i (h, a i, w) d µ i g j(d µ, b j, w) G H(h, a + c, w) P H V (h d µ, b, w) (36) G H (h, a, w) c (26) (34)(36) h O(e H ) h P H V (h d µ, b, w) P H V (h d µ, b, w) 2.1 P H V (h d µ, b, w) K c (34)(36) O(KM V H ) STEP 1. Θ STEP 2. (26) c STEP 3. STEP 2. c (34)(36) STEP 4. Θ STEP 5. STEP

307 4 Roux and Bengio (RB) (14) (16)(18) RBM RB KLIEP RB-K RB-K (26) c (29) (31) RB-KZ (26) c (34)(36) RB-KERB-KE c P H V (h d µ, b, w) P H V (h d µ, b, w) RB-KS RB RB-K RB P RB (v) RB-K P RB K (v) KL KL(P RB P RB K ) 1 V P RB (v) ln v P RB(v) P RB K (v) (37) V = 4 H = 4 () () RBM RBM Θ = {a, b, w} 0 (0.2) 2 N (0, 0.2) {0, 1} V 2 M = 100 {d µ {0, 1} V µ = 1,, M} RB RBM 500 RB-K RBM KL RB-K 0.05 RB-KS K = 5 3 RB-KZRB-KERB-KS RB RB-KZ RB- KERB-KS KL(P P ) RB RB K 10 3 RB KZ RB KE RB KS update step 3: KL(P RB P RB K ) 100 (29) (31) RB-KZ (34)(36) RB- KERB-KE RB-KS Z(c, Θ) RB-K RB 2 RB 5 RBM Roux and Bengio (14) RBM KLIEP Roux and Bengio (14) M K. (No and No ) COE 294

308 Center of Education and Research for Information Electronics Systems. [1] G. E. Hinton, S. Osindero, and Y. W. Teh: A fast learning algorithm for deep belief nets. Neural Computation, Vol.18, No.7, pp , [2] G. E. Hinton and R. R. Salakhutdinov: Reducing the dimensionality of data with neural networks. Science, 313, pp , [3] G. E. Hinton: Products of experts. In Proceedings of the Ninth International Conference on Artificial Neural Networks (ICANN), Vol.1, pp.1-6, [4] Y. Bengio, P. Lamblin, D. Popovici, and H. Larochelle: Greedy layer-wise training of deep networks. IAdvances in Neural Information Processing Systems 19, pp , [5] G. E. Hinton: Training products of experts by minimizing contrastive divergence. Neural Computation, Vol.14, No.8, pp , [6] M. A. Carreira-Perpinan and G. E. Hinton: On contrastive divergence learning. In Artificial Intelligence and Statistics, [7] N. Le Roux and Y. Bengio: Representational Power of Restricted Boltzmann Machines and Deep Belief Networks. Neural Computation, Vol.20, No.6, pp , [8] M. Sugiyama, S. Nakajima, H. Kashima, P. von Bünau, and M. Kawanabe: Direct Importance Estimation with Model Selection and Its Application to Covariate Shift Adaptation. In J. C. Platt, D. Koller, Y. Singer, and S. Roweis (Eds.), Advances in Neural Information Processing Systems 20, pp , [9] R. Neal and P. Dayan: Factor Analysis Using Delta-Rule Wake-Sleep Learning. Neural Computation, Vol.9, pp ,

309 2009 Technical Report on Information-Based Induction Sciences 2009 (IBIS2009) - - Hidden Structures Detection in Nonstationary Binary Time Series Data - Application to Neuroscientific Data - Ken Takiyama Masato Okada Abstract: We propose the algorithm which can estimate event rate, timimngs of change point and the number of states in a binary time series data. Event rate can be estimated with high accuracy even if the rate has nonstationary temporal correlation and mean value. Our algorithm consists of nongaussian switching state space model, variational bayes, and local variational. We demonstrate the algorithm applying to neuroscientific data. Our algorithm can estimate nonstationary event rate, change point and the number of neural states using only one observation data, which is a neuroscientific requirement in some cases. Synthetic data analysis reveals our algorithm can discriminate neural states based on both mean event rates and temporal correlations though many previous algorithm can detect change points based only on mean event rate. Our algotrithm can detect change points in area MT neural data and estimate the number of neural states based on temporal correlation. These results indicate our algotihm is applicable to wide range of nonstationary binary time series data, to neural data specifically. Keywords: nongaussian switching state space model, variational bayes, local variational, pruning, binary time series data, firing rate estimation 1, [1, 2].,. (State Space Model(SSM)). SSM,,., SSM 1,, tel , takiyama@mns.k.u-tokyo.ac.jp Graduate School of Frontier Sciences, The University of Tokyo Kashiwanoha 5-1-5, Kashiwa-shi,Chiba, Japan, okada@mns.u-tokyo.ac.jp, Graduate School of Frontier Sciences, The University of Tokyo, Brain Science Institute, RIKEN. SSM,, (Switching State Space Model (SSSM) ) [3],,, [3, 4, 5]. SSSM, 1.,.,. 0, 1 2. [6], [7], [8] 2,. Brown 296

310 1 2 N x 1 N x 2 N x 3 N x M N x 1 1 x 2 1 x 3 1 x M (msec) z z z z M η 1 η 2 η 3 η M 1: (a):, 0, 1. (b):.,. (c):. Kolmogorov-Smilnov [7] [9]., 2.,, 2. SSSM,, [10, 11], [12, 13]. [14], Abeles, [15].,, [15, 16, 18, 19].,,. ( 1(a)).,,., [24]., ( 1(b)).,,, ( ) [20, 21]., [22, 23], 1, [24].,,.,..,, EM [25]., (SSSM) SSSM N,. N. m n z n m = 1, z n m = 0 z = {z 1 1,..., z N 1,..., z 1 M,..., zn M }. M. n x n = {x n 1,..., x n M } (n = 1,..., N), s,. p(s, x, z) = p(s x, z)p(z)p(x 1 )...p(x N ) (1) SSSM 1(c). 2.2, p(z 1 π) = N n=1 (πn ) zn 1 (2) p(z m+1 z m, a) = N n=1 N k=1 (ank ) zn m zk m+1 (3) 297

311 p(x n β n, µ n ) = β n Λ (2π) N exp( β n 2 (xn µ n ) T Λ(x n µ n )) (4). π n, a nk, n, n k ( N n πn = 1, N k ank = 1). β n, µ n n,. Λ, p(x n ) m exp( βn 2 (x m x m 1 µ m ) 2 ) Λ. π, a, (2), (3) p(π γ n ) = C(γ n ) N n=1 (πn ) γn 1 (5) p(a γ nk ) = N n=1 [C(γnk ) N k=1 (ank ) γnk 1 ] (6). C(γ n ) = Γ( N n=1 γn ) Γ(γ 1 )...Γ(γ N ), C(γnk ) = Γ( N k=1 γnk ) Γ(γ n1 )...Γ(γ nn ), C(γ n ), C(γ nk ) p(π γ n ), p(a γ nk ). γ n, γ nk n, n k. 2.3 T K ( 1), 1. k +1, 1 η k (k = 1,..., K). k λ k, p(s λ) = K k (λ k ) 1+η k 2 (1 λ k ) 1 η k 2 (7) [26, 27].,,. 1 λ k [0, 1)., exp(2x k ) = λ k 1 λ k (x k (, )). x k.,. 1, K,.,. T M C, r = C.,. p(s x, z) = N,M n,m [exp(ˆη mx n m C log 2 cosh x n m)] zn m (8)., ˆη m = 2 r u=1 η (m 1)r+u C. 3 SSSM x, z a, π. SSSM p( s) [3],., w, θ F[q] = dwdθq(w)q(θ) log q(w)q(θ) p(s, w, θ) = U[q] S[q] (9) q()., U[q] = dwdθq(w)q(θ) log p(s, w, θ), H[q] = dwdθq(w)q(θ) log q(w)q(θ),,.., (4), (8). [12, 13], SSSM. ξm n (8) p ξ (s x, z) M,N m,n [exp( Ln m 2 (x n m η n m) 2 )] zn m (10) (4)., η m n = ˆη m /L n m, L n m = C ξ tanh(ξm). n (10) m n F ξ [q] F ξ [q] F[q], EM [25] ξ q[] [28]., [29]. T F[q; T ] = U[q] T S[q] (11), T 1,. x n, z q(x n W n ) = (2π) N exp( 1 2T (xn ˆµ n ) T W n (x n ˆµ n )) q(z) N exp( ˆπn n=1 M 1 N n T )z 1 N m=1 n=1 k=1 N n=1 m=1 exp(ânk m M exp(ˆb n m n T )z m (12) n T )z m zk m+1 (13) 298

312 ., W n = CL n + β n Λ (14) ˆµ n = (W n ) 1 (w n + β n Λµ n ) (15) ˆπ n = log π n (16) ˆbn m = ˆη m x n m C 2 l(ξn m)( (x n m) 2 (ξ n m) 2 ) C log 2 cosh ξ n m (17) â nk = log a nk (18), L n zm n tanh ξn m ξ (m, m) m n, w n (1, m) zm ˆη n m.. π, a N q(π) (π n ) z n 1 +γ1 1+T T 1 q(a) N n=1 n=1 k=1 (19) M 1 N (a nk m=1 zn m zk m+1 +γnk 1+T ) T 1 (20). z, Forward-Backward [30]. T i+1 = 1 2 T i (T 1 = 100) [3]. i. 4 EM ξ, µ, β EM., Q Q(θ θ t ) = log p(η, x, z θ) (21), θ (t)., θ = {ξ, µ, β}, t. EM, SSSM [3]. (21) Q (9) U[q; θ, θ t ], (21) (9). Q(θ θ (t) ) θ = 0 ξ n m = (x n m) 2 (22) µ n = x n (23) β n = M Tr[Λ((W n ) 1 + ( x n µ n )( x n µ n ) T )] (24). EM, [8] (κ = 2.4), ( 2(a)). [31],. [32, 33].,, ( 2(a)).,,,. T = 4.0, = 0.001, C = 0.04, γ n = 1(n = 1,..., 5), γ nk = 100(n = k), γ nk = 2.5(n k). 2. 2(a),. 2(a), m n zm n, zm n 1. 2(d),. 1, 2, 3,,,. 4, 5, 3., [11]. 2(e). 3,. 2(b) 7 3, zm n. zm n,,,, (Kernel Smoothing(KS))[34], (Kernel Band Optimization(KBO))[33], (Adaptive Kernel Smoothing(KSA))[35], (Bayesian Adaptive 299

313 (Hz) (Hz) (msec) < z > 1 (Hz) (Hz) MT K-S (msec) (msec) (msec) < z > (msec) < z > 1 0 2:,. (a):. (msec), (Hz)., zm n.,. (b): 3, 7. (c): MT. Kolmogorov-Smilnov (K-S ),,. (d): (a).. (e): (a),. (f): (c).. Regression Spilne(BARS))[36], (Bayesian Binning)[37]. KS f(x, y, σ) = 1 2πσ exp( 1 2σ 2 (x y)2 ) (25), σ σ = 30(msec)(KS30), σ = 50(msec)(KS50), σ = 100(msec)(KS100).,., KSA, KBO KSA. BARS, BB kass/bars/bars.html, (Variational Bayes Switching State Space Model(VB-SSSM))., Cunningham et al.[24]. (Mean Absolute Error (MAE) ) MAE = 1 T λ t T ˆλ t (26) t=1., λ t, ˆλ t. 2(a) 7,. 7 MAE, 3(a). 3(a),. 3(b), (c), [15, 16, 17, 300

314 KS30 KS50 KS100 LOC KSA BARS BB VB-SSSM (Hz) 0 mphmm 0 < z > 1 < z > (msec) (Hz) 30 mphmm (msec) 3: (a): 2(a).. 7 ±.. (b):, (multivariate Poisson - Hidden Markov Model (mphmm)). (c):, mphmm. 18, 19](, mphmm ),., mphmm 10. 3(b),. 3(c).,. 5.3 (Medial Temporal Area(MT )), [39]. MT. Neural Signal Archive [40], Britten (1992) [41]. 2(c) ( 6.4%) MT (nsa j001 T2). 2(c) - (Kolmogorov-Smirnov plot (K-S plot))[9],. K-S 95%,,. 2(c), 2(f), 2, 4,,., 1, 2. T = 2.0, = 0.001, C = 0.02, γ n = 1(n = 1,..., 5), γ nk = 100(n = : 28,. k), γ nk = 2.5(n k). Neural Signal Archive MT, nsa2004.1, j % ,. 28, 2 20, β n µ n. 1 2, 1, 2 β, µ 4. (c), (d) µ t,, µ n t = 1 Tn T n t=1 µn t, dµ n = 1 Tn T n t=1 (µn t µ n t ) 2., T n n,, t ,,,,., SSSM, ( 2(a), 2(c)). MT 1, 2 301

315 (sec) β 4 x μ> 4: (a): 1 2. (b): 1(), 2() β. (c): 1(), 2() µ t. (d): 1(), 2() µ t. <dμ>, [42, 43, 44]. [43], [44]. 2, 2(f) 2, 4. 1, 2.,,. 2(c) MT,,., [42].,., [42]., [43]., 2 MT.., MT.,, ,., Jones (2007)[17],,., 1, 2. 4(a),.,. MT,. 3(b), (c),,. 4(c), (d), 4(b),. MT,., MT ()., [45, 46]. (), DOWN, UP 2. UP, UP DOWN. Chen et al(2009), UP DOWN [46]. Chen [47],.,,. [15, 16, 19, 17, 18], MT 302

316 . 3(a), 3(b), 3(c),, 2(b), 2(e),. Abeles [15], Gat [16] articulate sulcus, mphmm. 6,,..,,.,.,,.,. [15, 16, 19, 17, 18, 46],.,,. 1. Shimazaki (2007)[48],..., Watanabe [38].. [1] V. Guralnik and J. Srivastava, ACM-SIGKDD, (1999) 33. [2] J. Takeuchi and K. Yamanishi, IEEE Trans. Know. Data Eng., 18(4) (2006) 482. [3] Z. Gharamani and G. E. Hinton. Neural Compt. 12, (2000) 831. [4] Hamilton, J. D. Econometrica 57, (1989) 357. [5] V. Pavlovic et al. Advances in NIPS 13, (2001) 981. [6] Y. Hung et al. J. American Stat. Association 103(483), (2008) [7] Y. Ogata. J. American Stat. Association 83(401), (1988) 9. [8] R. Barbieri. J. Neurosci. Methods 105, (2001) 25. [9] E. N. Brown et al. Neural Compt. 14 (2001) 325. [10] H. Attias Proc. 15th Conf. on UAI (1999) [11] Beal M. J. Ph.D thesis University College London (2003) [12] T. S. Jaakola and M. I. Jordan. Stat. and Compt. 10, (2000) 25. [13] K. Watanabe, and M. Okada, Proc. of ICONIP (2008). [14] A. Corduneanu and C. M. Bishop. (2001) Morgan Kaufmann [15] M. Abeles et al. PNAS. 92, (1995) [16] I. Gat et al. Network. 8, (1997) 297. [17] L. M. Jones et al. PNAS. 104, (2007) [18] C. Kemere et al. J. Neurophysiol.. 100, (2008) [19] N. Achtman et al. J. Neural Eng. 4 (2007) 336. [20] M. Churchland et al. J. Neurosci. 26(14) (2006) [21] B. Yu et al. J. Neurophysiol. 102, (2009) 614. [22] J. P. Donoghue. Nat. Neurosci. Supple. 5, (2002) [23] L. R. Hochberg et al. Nature 442, (2006) 164. [24] J. Cunningham et al. Neural Netw. 19, (2009) in press. [25] A. P. Dempster et al. J. Roy. Statist. Soc. B 39 (1977) 1. [26] E. N. Brown et al. In Computational Neuroscience: Comprehensive Approach (2003) [27] W. Truccolo et al. J Neurophysiol. 93 (2005) [28] C. M. Bishop and M. Svensen Proc. on UAI (2003) 57. [29] K. Katahira et al. J. Phys. 95 (2008) [30] L. E. Baum et al. Annals of Math. Stat. 41 (1970) 164. [31] S. N. Baker and R. N. Lemon. J. Neurophysiol. 84(4) (2000) [32] A. C. Smith and E. N. Brown, Neural Computation 15 (2003) 965. [33] H. Shimazaki and S. Shinomoto, Neural Coding 2007 Montevideo, Uruguay. [34] M. Nawrot et al. J. Neurosci. Methods 94 (1999) 81. [35] B. Richmond et al. J. Neurophysiol. 64(2) (1990). [36] I. DiMatteo et al. Biometrika 88 (2001) [37] Endres et al. Advances in NIPS 20 (2008) [38] K. Watanabe et al. IEICE trans. Info. and Syst., (in press). [39] J. H. Maunsell and D. C. Essen, J. Neurophysiol 49 (1983) [40] K. H. Britten et al. The Neural Signal Archive nsa [41] K. H. Britten et al. J. Neurosci. 12(12) (1992) [42] W. Bair and C. Koch, Neural Compt. 8 (1996) [43] S. G. Lisberger and J. A. Movshon j. Neurosci. 19(6) (1999) [44] L. C. Obsorne et al. j. Neurosci. 24(13) (2004) [45] D. Ji and W. A. Wilson, Nat. Neurosci. 10(1) (2007) 100. [46] Z. Chen et al. Neural Compt. 21 (2009) [47] P. J. Green, Biometrika 82(4) (1995) 711. [48] H. Shimazaki and S. Shinomoto, Neural Comput. 19 (2007)

317 2009 Technical Report on Information-Based Induction Sciences 2009 (IBIS2009) SVM Cross Subspace Learning for Subspace Support Vector Machine Abstract: Naoya INOUE Yukihiko YAMASHITA The support vector machine (SVM) classifies an input pattern by using a hyperplane. The SVM demonstrated high generalization ability, and is being widely researched now. We propose a new SVM-based classifier called the subspace SVM (SSVM). The SSVM has a restriction that the normal vector of its hyperplane is included in a subspace. In order to provide the subspace, we split a set of samples into two sets. One set is used to compose the normal vector, and the other set is used to train parameters in the normal vector and the threshold, similarly to the cross validation method. We call this method the cross subspace learning for the SSVM. We conducted experiments with 13 datasets in order to show the advantage of the SSVM. Keywords: Pattern recognition, Support vector machine, Cross subspace learning, Crossvalidation Vapnik [4, 5] Support Vector Machine (SVM) [1] 1960 Vapnik Optimal Hyperplane Classifier 0 (OHC) OHC SVM 2 [6] Minsky [7] [2]SVM SSVM SVM (SSVM) SVM SSVM SVM 2 SSVM SVM SSVM SVM 2 SSVM, , tel , {n708i,yamasita}@ide.titech.ac.jp, Tokyo Institute of Technology, Ookayama, Meguro-ku, Tokyo 304 SSVM

318 1: SVM y i ( w, x i +θ) 1 i w 2 SVM ξ l 0 (l = 1, 2,..., L) y l ( w, x l + θ) 1 ξ l. (2) ξ l = 0 ξ l > 0 ξ l > 1 (2) ξ l C ξ l SVM l = 1, 2,..., L y l ( w, x l + θ) 1 + ξ l 0, (3) ξ l 0, (4) 1 L 2 w 2 + C ξ l (5) l=1 Lagrange L 0 α l C, α l y l = 0. (6) 2 l=1 13 α l (l = 1, 2,..., L) L 2 Support Vector Machine L D = α l 1 L L α l α k y l y k x l, x k (7) 2 SVM Y = {+1, 1} +1 1 Ω +1 Ω 1. Φ w θ OHC 2 x z SVM k(x, z) = Φ(x), Φ(z). (8) d(x) = w, x + θ. (1) SV x d(x) x SV = {x l 0 < α L C}. (9) Ω +1 d(x) x Ω 1 1 {x d(x) = 0} R N d(x) = α l y l k(x l, x) + θ. (10) M 305 l x l SV l=1 k=1

319 3 SVM 3.1 SVMSSVM w SVM SSVM ξ l 0, α l 0, µ l 0. (20) {z n } N n=1 [ ( N ) ] {β n } N n=1, α l y l β n z n, x l + θ 1 + ξ l = 0, (21) w = N β n z n. (11) n=1 w 2 = N m=1 n=1 N β m β n z m, z n. (12) {(x l, y l )} L l=1 SSVM l = 1, 2,, L ( N ) y l β n z n, x l + θ 1 + ξ l 0, ξ l 0 (13) n=1 1 2 N m=1 n=1 N β m β n z m, z n + C L ξ l (14) l=1 α l, µ l (l = 1, 2,..., L) L S P = 1 2 N m=1 n=1 +C N β m β n z m, z n L ξ l l=1 L µ l ξ l l=1 ] L N α l [y l ( β n z n, x l + θ) 1 (15) l=1 n=1 KKT l = 1, 2,..., L n = 1, 2,..., N L S P β n = N β m z n, z m m=1 L α l y l z n, x l = 0, (16) l=1 306 L S P θ L S P L = α k y k = 0, (17) k=1 = C α l µ l = 0, (18) ξ l ( N ) y l β k z k, x l + θ 1 + ξ l 0, (19) k=1 n=1 µ l ξ l = 0. (22) A A T α = (α 1, α 2,..., α L ) T, (23) β = (β 1, β 2,..., β L ) T, (24) y = (y 1, y 2,..., y L ) T, (25) 1 = (1, 1,..., 1) T. (26) (N, N)- Z (Z) mn = z m, z n. (27) Z Z Z + εi ε > 0 I (N, L)-matrix D (D) nl = y l z n, x l. (28) (16) (17) Zβ Dα = 0, (29) y T α = y, α = 0, (30) SSVM SVM α 2 l = 1, 2,..., L 0 α l C, (31) y T α = 0, (32) α L S D = 1 T α 1 2 αt D T Z 1 Dα (33)

320 2 α β = Z 1 Dα. (34) w SV S = {x l 0 < α l < C, 1 l L}. θ = 1 SV S x l SV S [ y l ] N β n z n, x l n=1 (35) Experiment 1 Experiment 2 Experiment 3 Experiment 4 For subspace For parameter Total number of Training Data θ. 2: (K = 4 (1,3)) N d(x) = β n k(z n, x) + θ. (36) (27) (28) z n x l 2 n=1 d U,V (x). 3.2 d(x) (7) SVM d(x) = d S1,T S 1 (x)+d S2,T S 2 (x)+ +d SK,T S K (x). α k SVM 2 K = 4 SSVM T S 1,T S 2,..., T S k U S 1,S 2,..., S k V d(x) K 1 d(x) = d T S1,S 1 (x)+d T S2,S 2 (x)+ +d T SK,S K (x). K 1 (39) K 3 K = 4 K K. K 1 K-fold [4, 5] SVM 1 SVM SSVM 1 U V, L T = {x i } L i=1 S V 1,S 2,..., S K 4 K = 4 K T = S 1 S 2 S 3... S K. (37) U V U S 1,S 2,..., S k T S 1,T S 2,..., T S k V K. 4 U V 2 13 SSVM UV (38)

321 Experiment 1 Experiment 2 Experiment 3 Experiment 4 For subspace For parameter Total number of Training Data : (K = 4 (3,1)) 11 SSVM Experiment 1 Experiment 2 Experiment 3 Experiment 4 For subspace For parameter Total number of Training Data 4: (K = 4 (1,4)) splice SSVM SVM U V (k, l) l = K 1 SVM SSVM l = 1, 2,... K C 5-fold SSVM [10] SVM SSVM SSVM 1 SVM SVM [3] [8, 9] 2, 13 SVM 9 SVM banana ringnorm thyroid twonorm t- 5% K = 3K = 4 (1, 2)(1, 3) K = 3 K = 4 K = 2 K = K = 3 K = 4 K = SSVM 2 K 2 K = germen image titanic SVM SSVM SSVM SVM 2 3 SSVM U V K 5 308

322 Dataset # of training # of test # of dimension DN name patterns patterns realizations 1 banana breast cancer diabetis flare solar german heart image ringnorm splice thyroid titanic twonorm waveform : [1] C. Cortes and V. Vapnik Support-vector networks, Machine Learning, vol.20, no.3, pp , Sep [2] V. Vapnik, Statistical Learning Theory, Wiley Interscience, NewYork, [3],, [10] G. Rätsch and T. Onoda and K.-R. Müller, Soft Margins for AdaBoost, Technical Report NC-TR-1998-, 2004 Workshop on Information-Based Induction Sciences (IBIS2004), 021, Department of Computer Science, Royal Holloway, University of London, Aug. Tokyo, Nov [4] M. Stone, Cross-Validatory Choice and Assessment of Statistical Predictions, Journal of the Royal Statistical Society. Series B (Methodological), vol.36, no.2, pp , [5] S. Geisser, The predictive sample reuse method with applications, Journal of the American Statistical Association vol.70, pp , Jun [6] D. C. Manning and P. Raghavan and H. Schuetze, Introduction to Information Retrieval, Journal of the American Statistical Association chapter 15.2, Cambridge University Press, [7] M. Minsky and S. Papert, Perceptrons, MIT Press, Cambridge, [8] G. Rätsch and T. Onoda and K. R. Müller, Soft Margins for AdaBoost, Machine Learning, vol.42, no.3, pp , Mar [9] S. Mika and G. Rätsch and J. Weston and B. Schölkopf and K. R. Müller, Fisher discriminant analysis with kernels, Neural Networks for Signal Processing IX pp.41-48, Jun

323 DN SVM SSVM, K=2 (1,1) SSVM K = 3 (1,2) SSVM K = 3 (2,1) ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± 0.47 (a) SVM SSVM (K = 2, 3) DN SSVM K = 4 (1,3) SSVM K = 4 (2,2) SSVM K = 4 (3,1) ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± 0.45 (b) SSVM (K = 4) 2: ( ) 310

324 DN SVM SSVM, K=2 (1,2) SSVM K = 3 (1,3) SSVM K = 3 (2,3) ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± 0.41 (a) SVM SSVM (K = 2, 3) DN SSVM K = 4 (1,4) SSVM K = 4 (2,4) SSVM K = 4 (3,4) ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± 0.39 (b) SSVM (K = 4) 3: ( ) 311

325 2009 Technical Report on Information-Based Induction Sciences 2009 (IBIS2009) Linear Time Model Selection for Mixture of Heterogeneous Components via Expectation Minimization of Information Criteria Ryohei Fujimaki Satoshi Morinaga Michinari Momma Kenji Aoki Takayuki Nakata Abstract: Our main contribution is to propose a novel model selection methodology, expectation minimization of information criterion (EMIC). EMIC makes a significant impact on the combinatorial scalability issue pertaining to the model selection for mixture models having types of components. A goal of such problems is to optimize types of components as well as the number of components. One key idea in EMIC is to iterate calculations of the posterior of latent variables and minimization of expected value of information criterion of both observed data and latent variables. This enables EMIC to compute the optimal model in linear time with respect to both the number of components and the number of available types of components despite the fact that the number of model candidates exponentially increases with the numbers. We prove that EMIC is compliant with some information criteria and enjoys their statistical benefits. Keywords: Mixture of Heterogeneous Components, Expectation Minimization of Information Criteria 1 ( ) [6] () [14] NEC, , r-fujimaki@bx.jp.nec.com, URL NEC Common Platform Software Research Laboratoies, 1753, Shimonumabe, Nakahara-ku, Kawasaki-shi, Kanagawa (A) (B) 2 (A) 312

326 (B) (A) ( (MDL) [10] (AIC) [1] (BIC) [13] ) [9] (B) [6] [8] (Expectation Minimization of Information Criterion; EMIC) EMIC EM [5] 1 EM M ( (B) ) 3.4 MDL/BIC, AIC EMIC MDL/BIC AIC EMIC (A) UCI [2] (A) (B) 1 2 P ( ; ) () ˆ 2.1 V j = {P (X; ϕ Vj ) ϕ Vj Φ j } ϕ V j = (ϕ V j 1,..., ϕv j J Vj ) V j Φ j P (X; ϕ Vj ) j J Vj S = {V j j = 1,..., S } S { P (X; θ) = C c=1 } π c P (X; ϕ Sc c ), (1) 2 H = {H i i = 1,..., H } 3 H (1) C S c S (c = 1,..., C) c H π = (π 1,..., π C )ϕ = (ϕ S 1 1,..., ϕs C C )θ = {π, ϕ} 2.2 x N = x 1,..., x N H H () 4 IC(x N ; H) = N log P (x n ; ˆθ H ) + l(ˆθ H ), (2) n=1 5 l(ˆθ H ) MDL/BIC 6 AIC MDL/BIC(x N ; H) = N n=1 log P (x n ; ˆθ H )+ J H 2 log N, 2 [3] [7] 3 H H 4 [15] () [11] 5 6 MDL crude MDL O(log N) O(1) refined MDL Rissanen [12] refined MDL crude MDL 313

327 N AIC(x N ; H) = 2 log P (x n ; ˆθ H ) + 2J H, n=1 EM [5] ˆθ H H X Z = (Z 1,..., Z C ) Z c X c 1 0 x n z N = z 1,..., z N (X, Z) (x N, z N ) P (X, Z ϕ) P (X Z, ϕ) P (Z π) P (X Z, ϕ) Z P (X Z c = 1, ϕ) = P (X; ϕ Sc c ) IC(x N, z N ; H) IC(x N, z N ; H) = n=1 c=1 N n=1 c=1 N C C z nc log ˆπ c + C z nc log P (x n ; c=1 ˆϕ Sc c ) l(ˆϕ Sc c ) + l(ˆπ) (3) P (Z x, θ, H) N n=1 c=1 C ( Zc. π c P (x; ϕ S c c )) (5) IC(x N, z N ; H) E Z [IC(x N, z N ; H)] 3.4 E Z [ ] P (Z x, θ, H) E Z [IC(x N, z N ; H)] E Z [IC(x N, z N ; H)] = N n=1 c=1 N C n=1 c=1 C [ C E Z [z nc ] log ˆπ c + E Z E Z [z nc ] log P (x n ; ˆϕ S c c ) c=1 l(ˆϕ S ] c c ) + l(ˆπ) (6) (6) EM 3.2 EMIC C EM S c ϕ S c c C (t) EM t C Sc c=1 l(ˆϕ c ) + l(ˆπ) MDL/BIC AIC ( C J Sc c=1 2 log N ) n=1 z nc C c=1 J S c + (C 1) + C 1 2 log N for MDL/BIC for AIC (4) (2) (3) IC(x N, z N ; H) EM E E P (Z x, θ, H) M E Z [z nc ] E Z [ C c=1 l(ˆϕ S c c ) + l(ˆπ)] E Z [z nc ] EM E (t) Z [z nc] = π c (t 1) C c=1 π(t 1) c P (x n ; ϕ S(t 1) c c ) P (x n ; ϕ S(t 1) c c ). (7) z n O(N) E Z [ C Sc c=1 l(ˆϕ c )+l(ˆπ)] MDL/BIC AIC MDL/BIC E (t) Z [log( N n=1 z nc)] z 1c,..., z Nc 2 N 314 1

328 (B) 3.3 N AIC (4) M M E Z [IC(x N, z N ; H)] EMIC { argmin S c,ϕ S c c N n=1 Z [z nc] log P (x n ; ϕ S c c ) + l( ˆϕ } S c c ), E (t) (8) S c ϕ S c c N ˆπ c (t) n=1 = E(t) Z [z nc], (9) N EMIC EM S c ϕ S c c EM IC(x N ; H (t 1) ) IC(x N ; H (t) ) EM ˆθ (t) EMIC ˆθ (t) ˆθ (t) Algorithm 1 EMIC C EM IC IC(x N ; H) Algorithm 1 C max Ghahramani [6] Algorithm 1 Expectation Minimization of Information Criterion 1: Input: x N, S and C max 2: Initialization : H NULL and IC 3: for C = 1,..., C max do 4: t 1 and initialize H (t) and ˆθ (t). 5: repeat 6: t t : Evaluate E Z [z nc ] and E Z [ C c=1 8: for c = 1,..., C do 9: Calculate S c (t), 10: end for Sc l(ˆϕ c )+l(ˆπ)]. ˆϕ (t) S c, ˆπ (t) c by (8) and (9). 11: Update: H (t) {C, S (t) 1,..., S(t) ˆθ (t) {ˆπ (t), ˆϕ (t) }. C } 12: Evaluate IC(x N ; H (t) ) and convergence. 13: until IC(x N ; H (t) ) converges 14: if IC(x N ; H (t) ) < IC then and 15: IC IC(x N ; H (t) ), H H (t) and θ ˆθ (t). 16: end if 17: end for 18: Output: the optimal model H and parameter θ 3.3 MDL/BIC MDL/BIC E E (t) Z [log( N n=1 z nc)] O(2 N ) N E (t) Z [log( N n=1 z nc)] µ nc = E Z [z nc ] µ c = N n=1 µ nc E (t) Z [log( N n=1 z nc)] N n=1 z nc µ c M E Z [log ( N ) M [ ( N ) m z nc ] b m E Z z nc ], (10) n=1 m=0 n=1 b m M (10) z nc z k ic = z ic (k 1) E Z [z n1 cz n2 c] = µ n1 cµ n2 c (n 1 n 2 ) m E Z [( N n=1 z nc) m ] O(N) m = 2 N N E Z [( z nc ) 2 ] = µ 2 c + µ c µ 2 ic (11) n=1 i=1 315

329 3.4 EMIC MDL AIC BIC MDL MDL MDL 1 EMIC MDL N 3.2 EM MDL(x N ; H) 1 MDL J H /2 log N θ H [4] θ H 7 () θ MDL(x N ; H) = log P (x N, ˆθ)MDL(x N, z N ; H) = log P (x N, z N, ˆθ) [ MDL(x N ; H (t) ) = E (t) Z log P (xn, z N, ˆθ (t) ) P (z N x N, ˆθ (t) ) (12) E (t) Z [MDL(xN, z N ; H (t) )] Jensen [4] MDL(x N ; H (t+1) ) MDL(x N ; H (t) ) (13) 1 MDL N N 1 EMIC MDL/BIC AIC 2 EMIC l(ˆθ H ) = E Z [ C c=1 l(ˆϕ S c c ) + l(ˆπ)] 3.2 EM IC(x N ; H) 2 EMIC E (t+1) Z [IC(x N, z N ; θ (t+1) ] E (t+1) Z [IC(x N, z N ; θ (t) ], (14) 7 ]. Jensen E (t+1) Z [log P (z N x N ; θ (t+1) ] E (t+1) Z [log P (z N x N ; θ (t) ] (15) 8 (14) l(ˆθ H ) = E Z [ C c=1 l(ˆϕ S c c ) + l(ˆπ)] IC(x N ; H (t+1) ) IC(x N ; H (t) ) (16) 2 AIC l(ˆθ H ) = E Z [ C Sc c=1 l(ˆϕ c )+ l(ˆπ)] EMIC IC(x N ; H) 1 2 MDL/BICAIC EMIC (H (t), ˆθ (t) ) IC(x N ; H) EMIC 4 2 EMIC GAUSS 2 POLY 10 ( 10-fold ) 10 H C 4.1 EMIC MDL MDL EMIC (EMIC MDL ) 1 8 E (t+1) Z [ ] = P Z P (Z x, (t) ) MDL/BICAIC AIC MDL/BIC 316

330 FULL MDL [9] PAT MDL EM EM EMIC MDL PAT MDL FULL MDL (EMIC MDL PAT MDL ) ) MDL(x N ; H)2) H = H (R H ) 3) C = C (R C )4) CPU 1) 2) 3) GAUSS [ 5, 5] [0, 1] GAUSS C = 5 C max POLY [1, 4] POLY C max = 5 D max N = 500 EMIC EM GAUSS C max PAT MDL FULL MDL CPU EMIC CPU () C max EMIC : POLY 8 1 :y = 10, 2 :y = 10, 3 :y = 2x, 4 :y = 2x, 5 :y = 2x 2 3, 6 :y = 2x 2 +3, 7 :y = 0.5x 3 1.5x 2 2x+4, 8 :y = 0.5x x 2 + 2x 4. R H R C EMIC EM EMIC EMIC () 3POLY CPU D max PAT MDL FULL MDL EMIC ( ) 12 EMIC GAUSS IC(x N ; H)R H R C EMIC MDL EMIC EM POLY 4 t EM F i i D i F i t = 1 () 12 0 D max

331 2: GAUSS EMIC MDL PAT MDL FULL MDL 4: EMIC POLY t = UCI 3: POLY EMIC MDL PAT MDL FULL MDL EMIC t = 3t = 10t = 20 (D 1 D 4 ) ( F 1 ) ( F 4 t = 10) N n=1 E Z[z nc ] (E ) (M ) UCI [2] ecolihousingiris wineyeastvowel context 13 CPU EMIC MDL AIC 3 3 GAUSS 3 C max = 5 1 CPU MDL AIC EMIC EMIC