i E

Size: px

Start display at page:

Download "i E"

れいなながおか
7 years ago
Views:

2 i E

3 ii i

2.2..................... 10 2.2.3........................ 10 2.3............................ 10 2.3.1..................... 11 2.3.2.................. 14 2.

4 iii

3.3.................... 25 3.4........................ 26 3.5..................... 26 3.5.1..................... 26 3.5.2............... 27 3.6.................................. 28 4 29 4.1................................. 29 4.1.1............................ 30 4.

5 iv A: B: A: B: C: N X

3.1................................. 53 5.3.2................................. 56 5.3.3................................. 57 6 61 6.1........................ 61 6.1.1 N................... 61 6.1.2 X.

6 7 68 v A B

7 vi ([1] ) A/B ([2] ) Optimizely ( ) Bricolage A A A A A B B C A A B x 1 : x 2 : x 3 : x 4 :

8 A....... 52 5.9 B.......................... 54 5.10.................... 56 5.11....................... 58 5.12 x 1 :................................ 60 5.

8 5.16 x 1 x 2 : vii A.1 x = (0, 0, 0, 0) A.2 x = (0, 0, 0, 1) A.3 x = (1, 0, 0, 0) A.4 x = (1, 2, 2, 1) A.5 x = (0, 1, 0, 0) A.6 x = (0, 0, 1, 0)

......................... 71 A.4 x = (1, 2, 2, 1).......................... 71 A.5 x = (0, 1, 0, 0).

9 viii A B B

2 A....................... 51 5.3 B........................ 55 5.4 B.

10 E [3][4] Eric Ries * 1 [1] *1 Lessons Learned: The lean startup

11 2 アイデア学ぶ構築するデータ製品計測する ([1] ) *2 (LPO: Landing Page Optimization) A/B *2

12 3 な違いがユーザの行動を左右することが知られているたとえば 2008 年アメリカ合衆国大統領選挙ではバラックオバマ陣営がウェブページ最適化を活用して多額の献金を集めることに成功した [2] 当時バラックオバマ氏の公式ウェブサイトには支援者からの献金を受け付ける機能が備わっていたが献金を行うにはメールアドレスを登録する必要があり訪問者のメールアドレス登録率が低いことがボトルネックになっていたそこでバラックオバマ陣営はトップページで A/B テストを行い訪問者のメールアドレス登録率を向上させるデザインを探索することにした実験では 6 種類の写真と 4 種類のボタンのバリエーションから 24 種類のパターンが生成され約 30 万人もの訪問者に対してテストが行われた図 1.2 参照その結果最もパフォーマンスの高いパターンはオリジナルのパターンに対して 40.6% も登録率が高いことがわかったこのパターンを採用したところバラックオバマ陣営は 2,880,000 のメールアドレスと 6000 万ドルの献金を追加で獲得することに成功した Media Variation Original Image Family Image Change Image Barack s Video Sam s Video Springfield Video Button Variation 図 1.2 バラックオバマ氏公式ウェブサイトのトップページの A/B テストで用いられた写真とボタンのバリエーション ([2] から引用)

種類のパターンが生成され約 30 万人もの訪問者に対してテストが行われた図 1.2 参照その結果最もパフォーマンスの高いパターンはオリジナルのパターンに対して 40.

13 Optimizely *3 A/B Google Google Website Optimizer *4 Visual Website Optimizer *5 planbcd * *3 Optimizely *4 Google Website Optimizer Google Analytics *5 Visual Website Optimizer *6 planbcd

com/ *4 Google Website Optimizer http://www.google.

15 [5] A/A A/A 2 5% A/A 2 5% 5 [6]

16 7 [7] URL Optimizely ( 2.1 ) 2.1 Optimizely ( )

17 8 *1 planbcd Webpage Segmentation [8][9] DOM 2 *1

18 9 DOM HTML *2 DOM *3 HTML <table> <ul> [10] [9] [11] HTML HTML5 *4 CSS *5 DOM [12] W3C *6 HTML CSS *7 VIPS [13] VIPS DOM Zone Tree Model [14] HTML CSS *2 HyperText Markup Language: *3 Document Object Model: HTML XML *4 HTML 5 HTML CSS *5 Cascading Style Sheets: *6 World Wide Web Consortium: *7 Conformance: requirements and recommendations deprecated ( )

Model: HTML XML *4 HTML 5 HTML CSS *5 Cascading Style Sheets: *6 World Wide Web Consortium: *7

19 E [15] [16] Web Page Segmention [17][18] HTML [19] [12]( 2.2 ) 2.3

20 Example-Based Retargeting for Web Design mar Jerry O. Talton Salman Ahmad Scott R. Klemmer Stanford University HCI Group Department of Computer Science 11 s of design examples unparalleled er, leveraging existing designs to n difficult. This paper introduces r transferring design and content lage employs a novel, structuredearns to create coherent mappings on human-generated exemplars. e then used to automatically transge into the style and layout of anage can learn to accurately repros, and that it provides a general, hnique for retargeting content beb pages. argeting, structured prediction. ords es and presentation: Hypertext/ n Factors Figure 1. Bricolage computes coherent mappings between Web pages by matching visually and semantically similar page elements. Bricolage The produced mapping can then be used to guide the transfer of content from one page into the design and layout of the other. 2.2 Most design reuse today is accomplished with templates rely on examples for inspiration [13]. Templates use standardized page semantics to render content into predesigned layouts. This strength is also a cilitate better design work [22]. e space of possible solutions and weakness: templates homogenize page structure, limit customization and creativity, and yield cookie-cutter designs. ssibilities [2, 3]. Furthermore, reents from prior ideas can be more Ideally, tools should offer both the ease of templates and the hem from scratch [12, 21, 14]. diversity of the entire Web. What if any Web page could be a design template? us of design examples unparal- 2008, Google had indexed more This paper introduces the Bricolage algorithm for transferring design and content between Web pages. The term URLs [1]. However, we hyource is underutilized for design bricolage refers to the creation of a work from a diverse range of things that happen to be available. Bricolage s assist with browsing examples ign elements, adapting the gestalt remains a time-intensive, A/B matches visually and semantically similar elements in pages manual to create coherent mappings between them. These mappings A/B can then be used to automatically transfer the content from one page into the style and layout of the other (Figure 1). [5] A/B A ard copies of all or part of this work for Bricolage uses structured prediction [7] to learn how to nted without fee provided B that copies are transfer 50% content between pages. It trains on a corpus of or commercial advantage and that copies human-generated mappings, collected using a Web-based on on the first page. To copy otherwise, or crowdsourcing interface, A the Bricolage B 2 Collector. The Collector was seeded with 50 popular Web pages that were redistribute to lists, requires prior specific decomposed into a visual hierarchy by a novel, constraint- couver, British Columbia /11/05...$10.00 A/B A/B

This paper introduces r transferring design and content lage employs a novel, structuredearns to create coherent mappings on human-generated exemplars.

21 12 (Multivariate Testing) [20] (Full Factorial Test Design) (Fractional Factorial Test Design) Sparsity-Of-Effects Principle 3 2 [21][22] [21] [23] epsilon-greedy Softmax UCB1

22 13 epsilon-greedy epsilon-greedy 0 < ε < 1 ε 1 ε Softmax Softmax X x X y x x p(x) 2.1 p(x) = exp(y x /τ) x X exp(y x/τ) (2.1) τ τ x p(x) 1/ X y x 1 UCB1 UCB1 UCB1 y x x X t x x UCB u x u x = y x + 2 log( x X t x) t x UCB x [23] Amazon.com *8 *8 Amazon

23 14 [5][24] (HC: Hill Climbing) [25] (Multi-start Hill Climbing) (SA: Simulated Annealing) ( )

24 15 ( ) [25] (TS: Tabu Search) [25] E [26]

25 [27][28] [29] [30] [31] *9 *9

26 17 [32] [5] javascript * 10 javascript HTML Google Website Optimizer, Optimizely javascript javascript [5] *10

27 18 [5] [6][7] [5][24][33] [5][6]

28 [23]

29 x = (x 1,, x m ) m x i V i = {v i1,, v ili } x i l i x x 1 x 2 x 3 3 x 3 V 3 = {,,, } f(x) x x = arg max f(x) (3.1) x X

30 21 x f(x) x y p(y x) f(x) y E[y x] x = arg max E[y x] (3.2) x X N n x n y n Y = {(x 1, y 1 ),, (x n, y n )} 3.3 Y N (3.3) x X x X y N Y x x = arg max E[y x] s.t. x X Y N (3.4) x X 3.2

31 22 x X x f(x) p(y x) y N Proximate Optimality Principle [34] (Algorithm 1 ) Algorithm 1 Choose x X. repeat X Neighbors(x) x Move(x, X ) until return x x X X [35]

32 [35] x x i V i X x i x i V i X x x f(x) x E[y x] 3.3 x y 3.3.1

33 24 (p ) p α α α [36] t t Y x = {(x t, y t ) Y x t = x} Y Y x 95% Y x f(x) µ Lx = E[y x] 1.96 s x Yx f(x) E[y x] s Yx = µ Ux (3.5) x yi Yx (y i E[y x]) 2 s x = Y x µ Lx f(x) µ Ux f(x) x X [µ Lx, µ Ux ]

34 25 x l 3.6 µ Uxl < max x X\{x l } µ Lx (3.6) x w 3.7 µ Lxw > max x X\{x w } µ Ux (3.7) n 2 n α 1 β µ µ σ = µ µ σ α = β = 0.80 n = 16 2 = 16σ2 (µ µ) 2 [37]

35 y = a 0 + l 1 1 i=1 a 1i x i l m 1 i=1 a mi x i m x i x [7] E 3.1 t

36 A/B A/B x 0 x i x X x A/B x j 3.4 x X x E[y x] A/B

37 28 3.6

39 n 0 n 1 n 2

40 31 n 0 x i l i 4.1 n 0 = m l i (4.1) i=1 n 1 n 1 α β k k = m i=1 l i n t n 0, n 1, n 2 N N n 0 N n 0 n 1 N N n x X Y N n 1 Y x

41 x 4.2 x y 4.2 {(x 1, y 1 ), (x 2, y 2 ), } (4.2) 3.4

42 33 x m N x X N α 1 β α 1 β α = 0.05 β = 0.20 [38] Cohen [39] n 0, n 1, n 2 (Algorithm ) N < n 0 (Algorithm 2 6 )

43 34 Algorithm 2 Require: α as the significant level. Require: 1 β as the power. Require: as the effect size. Require: N > 0 as the number of visitors. Require: X as the set of solutions. 1: Set Y as an empty set for observed data. 2: Set n 0 m i=1 l i 3: Set n 1 n ANOV A (α, β,, X) 4: Set n 2 n T T EST (α, β, ) 5: if N < n 0 then 6: Give up optimization. 7: x, N, Y Initialization(N, X, Y, n 1 ) 8: while N > 0 do 9: x, N, Y Move(x, N, Y, n 2 ) 10: return x as optimal pattern. N n 0 N X n 1 x (Algorithm 2 7 ) N > n 2 x (Algorithm 2 9 ) N N = 0 x (Algorithm 2 10 ) A: A x X Y

44 35 x (Algorithm 3 ) Algorithm 3 A: Choose x X randomly. y Observe(x) Add (x, y) to Y. x arg max E[y x] x X return x B: B n 2 (Algorithm 4 ) Algorithm 4 B: : function Initialization(N, X, Y, n 1 ) Choose x X randomly. return x, N, Y : function Move(x, N, Y, n 2 ) X {x } Neighbors(x ) for n = 1 to n 2 do for x X do y Observe(x) Add (x, y) to Y. N N 1 x arg max x X E[y x] return x, N, Y

45 A: A B Algorithm 5 Algorithm 5 A: : function Initialization(N, X, Y, n 1 ) Set x as empty vector. for n = 1 to n 1 do Choose x X randomly. y Observe(x) Add (x, y) to Y. N N 1 for i = 1 to m do v arg max v V i E[y x i = v] x i v return x, N, Y A n 1 x X Y x i E[y x i = v] v V i x B: A ( 3.7 )

46 37 n 1 A (Algorithm 6 ) Algorithm 6 B1: : function Initialization(N, X, Y, n 1 ) Set x as empty vector. for n = 1 to n 1 do Choose x X randomly. y Observe(x) Add (x, y) to Y. N N 1 for i = 1 to m do v arg max v V i E[y x i = v] if E[y x i = v ] is winner then x i v if every element of x is filled then return x, N, Y for i = 1 to m do v arg max v V i E[y x i = v] x i v return x, N, Y (Algorithm 7 )

47 38 Algorithm 7 B2: : function Move(x, N, Y, n 2 ) X {x } Neighbors(x ) for n = 1 to n 2 do for x X do y Observe(x) Add (x, y) to Y. N N 1 if E[y x] is winner then x x return x, N, Y x arg max E[y x] x X return x, N, Y C: B ( 3.6 ) X (Algorithm 8 ) X (Algorithm 9 ) 4.4 3

48 39 Algorithm 8 C1: : function Initialization(N, X, Y, n 1 ) Set x as vector. for n = 1 to n 1 do Choose x X randomly. y Observe(x) Add (x, y) to Y. N N 1 for i = 1 to m do if E[y x i = v] is loser then X {x x i v} if E[y x i = v] is winner then x i v if every element of x is filled then return x, N, Y for i = 1 to m do v arg max v V i E[y x i = v] x i v return x, N, Y n 0 n 1 n 2 3 n 1 X 4.1

49 40 Algorithm 9 C2: : function Move(x, N, Y, n 2 ) X {x } Neighbors(x ) for n = 1 to n 2 do for x X do y Observe(x) Add (x, y) to Y. N N 1 if E[y x] is loser then Pop out x from X. if E[y x] is winner then x x return x, N, Y x arg max E[y x] x X return x, N, Y 4.1 A B A B + + C

50 f(x) x 1 x 2 ϵ f(x) y N n n

51 f(x) ϵ N 0 ϵ = 0 n 0 < N < n 1 1 ϵ > 0 n 0 < N < n 1 2 ϵ > 0 n 1 N 3 ϵ > 0 n 0 < N < n 1 4 ϵ > 0 n 1 N m = 5 x i l i = 3 M = l m = 3 5 (= 243) X = {x 1, x 2,, x M } = {(0, 0, 0, 0, 0), (0, 0, 0, 0, 1),, (2, 2, 2, 2, 2)} N x α = 0.05, 1 β = 0.8, = 0.2 n 0 = 5 3 = 15, n 1 = 500, n 2 = 400 R pwr *1 x x 100 *1 CRAN - Package pwr

52 ( ϵ = 0 n 0 < N < n 1 ) 0 A B A 5.1 t 1.00 Average accuracy of each algorithm 0.75 Accuracy 0.50 Algorithm Baseline A Baseline B Proposal A Sample Size: n A A( ) X x B( ) n 2 N

53 44 A 1 ( ϵ > 0 n 0 < N < n 1 ) 1 A B A ϵ A ϵ Average accuracy of each algorithm 0.75 Accuracy 0.50 Algorithm Baseline A Baseline B Proposal A Sample Size: n A B 0 0

54 45 A 2 ( ϵ > 0 n 1 N) 2 A B A N t > 500 A 1.00 Average accuracy of each algorithm 0.75 Accuracy 0.50 Algorithm Baseline A Baseline B Proposal A Sample Size: n A

55 46 3 ( ϵ > 0 n 0 < N < n 1 ) 3 A B A N A N A A Average accuracy of each algorithm 0.3 Accuracy 0.2 Algorithm Baseline A Baseline B Proposal A Sample Size: n A 4 ( ϵ > 0 n 1 N) 4 A A B N A B

56 47 N = n Average accuracy of each algorithm 0.75 Accuracy 0.50 Algorithm Baseline A Proposal A Proposal B Sample Size n A B A A B 4 C 5.6 C

57 Average accuracy of each algorithm 0.75 Accuracy 0.50 Algorithm Baseline A Proposal B Proposal C Sample Size n B C 5.1.3

58 *2 A A A A A ( 5.7 ) A A A *2

59 50 図 5.7 実サイト A の相関図ページたという文脈を共有していることになり訪問者間の行動特性の差は小さいと考えられるまた相関図ページ一枚がメインコンテンツであるため相関図ページを訪問して広告をクリックするか否かという非常にシンプルな構図でウェブページのパフォーマンスを捉えることができる実サイト A では積極的にウェブページ最適化を開発に取り入れており訪問者にデザインや機能が一部異なる相関図ページを表示することによって施策の価値を評価しているそのため実サイト A ではそれぞれの施策が広告のクリック率に与える効果が蓄積されており施策の有無によるクリック率の違いをシミュレートするための環境が整っている以上の理由から対象サイトとして実サイト A を用いることが適切と判断しシミュレーションを行うことにした問題設定ウェブサイトの目的を顧みて相関図ページにおける広告クリック率を最大化する評価指標として設定する 2013 年 5 月 14 日から 18 日にかけて行なった実験で相関図の広告の配置

60 51 18% χ 2 p p A χ 2 p A 14, % B 9, % C 5, % ,883 4, % x q(x) = x x x 3 f(x) x q(x) 1 1 q(x) 0 : N = : 100 : α = 0.05 : 1 β = 0.8 : = 0.05 : n 1 = 5500 : n 2 =

61 A 1.00 Average accuracy of each algorithm 0.75 Accuracy 0.50 Method Baseline A Baseline B Proposal A Proposal B Proposal C Time 5.8 A B

62 53 n 1 = 5500 B C A Imagerous* *3 B B ( 5.9 ) B *3 Imagerous*

63 B B SNS B B B B B

64 55 B 5.3 A 5.3 B x 1 0px, 5px x 2 0px, 5px, 10px x 3 100px, 200px, 300px x 4, 0 x = (0, 2, 1, 0) :0px :10px :200px :

65 B( ) A α = β = 0.8 = 0.3 : n 1 = 120 : n 2 = ID! javascript!!!!!! ID! HTML!!! ID!!!! HTML!! 5.10 B HTML javascript javascript

66 57 ID ID B HTML DOM style javascript javascript HTML DOM javascript ID B javascript 5 javascript B B x E[y x ] Y x B (1,0,0,0) A (0,0,0,0) B x = (1, 0, 0, 0) :5px

67 58 :0px :100px : x E[y x ] = B x 178 A x = (0, 0, 0, 0) :0px :0px :100px : E[y x ] = B t E[y x ] =

68 B 0 (0,0,0,1) N/A N/A B 282 (1,0,0,1) B 552 (1,0,0,0) A 0 N/A N/A N/A A 259 (1,0,0,0) A 525 (0,0,0,0) B n 2 A n < n 0 x = (1, 0, 0, 0) x = (1, 0, 0, 0) x = (0, 0, 0, 0) % x i v = 0 A x 1 x x 1 v = 1, 2 x 2 v = 1 x 1 v = 0 x 2 v = 2 x 1 x 2 x = (0, 0, 0, 0) 5.5 A B B x = (1, 0, 0, 0)

69 x 1 : 5.13 x 2 : 5.14 x 3 : 5.15 x 4 : 5.16 x 1 x 2 :

70 N N N n 0 m O(c m ) n O(m) n 0

71 X X N [25] x d x e x d

72 x 1 x 2 V 1 = V 2 = {, } 6.1 x 1 x 2

73 A B 2 A 0 1 B SNS E [40][41]

74 x X y N Y x x = arg max E[y x] s.t. x X Y N x X x x x Y N x x y x:, y: x: y:

75 YouTube *1 *2 Brightcove *3 Treepodia *4 E A/B *5 A/B/C D CAD 3D [42] CAD *1 YouTube *2 *3 Brightcove *4 Treepodia *5 Ecommerce Video A/B testing - Test Your Intuitions! : Treepodia /07/ecommerce-video-ab-testing-test-your-intuitions/

76 67 3D CAD 3D Ponoko *6 Shapeways *7 *6 Ponoko *7 Shapeways

77 D 3D

78 69

79 70 付録 A 実サイト B 仮説パターン例図 A.1 仮説パターン x = (0, 0, 0, 0) 図 A.2 仮説パターン x = (0, 0, 0, 1)

80 71 図 A.3 仮説パターン x = (1, 0, 0, 0) 図 A.4 仮説パターン x = (1, 2, 2, 1) 図 A.5 仮説パターン x = (0, 1, 0, 0) 図 A.6 仮説パターン x = (0, 0, 1, 0)

82 73 [1] Eric Ries. The Lean Startup: How today s entrepreneurs use continuous innovation to create radically successful businesses. Random House Digital, Inc., [2] Dan Siroker. How obama raised $60 million by running a simple experiment [3] Thomas Eisenmann, Eric Ries, and Sarah Dillard. Hypothesis-driven entrepreneurship: The lean startup. Harvard Business School Entrepreneurial Management Case, No , [4] Carmen Nobel. Teaching a lean startup strategy, [5] Ron Kohavi, Roger Longbotham, Dan Sommerfield, and Randal M Henne. Controlled experiments on the web: survey and practical guide. Data Mining and Knowledge Discovery, Vol. 18, No. 1, pp , [6] Thomas Crook, Brian Frasca, Ron Kohavi, and Roger Longbotham. Seven pitfalls to avoid when running controlled experiments on the web. In Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining, pp ACM, [7] Saharon Rosset and Slava Borodovsky. A/b testing using the negative binomial distribution in an internet search application [8] Zaiqing Nie, Ji-Rong Wen, and Wei-Ying Ma. Webpage understanding: beyond pagelevel search. ACM SIGMOD Record, Vol. 37, No. 4, pp , [9] Christian Kohlschütter and Wolfgang Nejdl. A densitometric approach to web page segmentation. In Proceedings of the 17th ACM conference on Information and knowledge management, pp ACM, [10] Shian-Hua Lin and Jan-Ming Ho. Discovering informative content blocks from web

83 74 documents. In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, pp ACM, [11] Parag Mulendra Joshi and Sam Liu. Web document text and images extraction using dom analysis and natural language processing. In Proceedings of the 9th ACM symposium on Document engineering, pp ACM, [12] Ranjitha Kumar, Jerry O Talton, Salman Ahmad, and Scott R Klemmer. Bricolage: example-based retargeting for web design. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp ACM, [13] Deng Cai, Shipeng Yu, Ji-Rong Wen, and Wei-Ying Ma. Vips: a visionbased page segmentation algorithm [14] Jie Zou, Daniel Le, and George R Thoma. Combining dom tree and geometric layout analysis for online medical journal article segmentation. In Digital Libraries, JCDL 06. Proceedings of the 6th ACM/IEEE-CS Joint Conference on, pp IEEE, [15] Dave Gehrke and Efraim Turban. Determinants of successful website design: relative importance and recommendations for effectiveness. In System Sciences, HICSS- 32. Proceedings of the 32nd Annual Hawaii International Conference on, pp. 8 pp. IEEE, [16] O Byung Kwon, Choong-Ryuhn Kim, and Eun Jong Lee. Impact of website information design factors on consumer ratings of web-based auction sites. Behaviour & information technology, Vol. 21, No. 6, pp , [17] Imran Sarwar Bajwa, Imran Siddique, and M Choudhary. Web layout mining (wlm): A new paradigm for intelligent web layout design. In Proceedings of the ITI 4th International Conference on Information and Communications Technology (ICICT 06). IEEE, Los Alamitos, CA, [18] Allison Thompson. Automated generation of website content and layout. Michigan Celebration of Women in Computing, p. 43, [19] Srinivasan H Sengamedu and Rupesh R Mehta. Web page layout optimization using section importance. In Proceedings of the 17th international conference on World Wide Web. ACM, [20] T. Ash, M. Ginty, and R. Page. Landing Page Optimization: The Definitive Guide

84 75 to Testing and Tuning for Conversions. ITPro collection. Wiley, [21].., [22],... A,, No. 12, pp , [23] John White. Bandit Algorithms for Website Optimization. O Reilly, [24] Round M Kohavi R. Front line internet analytics at amazon.com stanford.edu/~ronnyk/emetricsamazon.pdf, [25],,,,,.., [26] Arben Asllani and Alireza Lari. Using genetic algorithm for dynamic and multiple criteria web-site optimizations. European journal of operational research, Vol. 176, No. 3, pp , [27] Belarmino Adenso-Diaz and Manuel Laguna. Fine-tuning of algorithms using fractional experimental designs and local search. Operations Research, Vol. 54, No. 1, pp , [28] Steven P Coy, Bruce L Golden, George C Runger, and Edward A Wasil. Using experimental design to find effective parameter settings for heuristics. Journal of Heuristics, Vol. 7, No. 1, pp , [29] Verena Heidrich-Meisner and Christian Igel. Hoeffding and bernstein races for selecting policies in evolutionary direct policy search. In Proceedings of the 26th Annual International Conference on Machine Learning, pp ACM, [30] Oded Maron and Andrew W Moore. Hoeffding races: Accelerating model selection search for classification and function approximation. Robotics Institute, p. 263, [31] Mauro Birattari, Thomas Stützle, Luis Paquete, and Klaus Varrentrapp. A racing algorithm for configuring metaheuristics. In GECCO, Vol. 2, pp Citeseer, [32] E. Hansen. System and method for optimizing website visitor actions, November US Patent App. 11/140,288. [33] Ron Kohavi, Randal M Henne, and Dan Sommerfield. Practical guide to controlled experiments on the web: listen to your customers not to the hippo. In Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data

85 76 mining, pp ACM, [34],. 207 proximate optimality principle tabu search., Vol. 2002, No. 5, pp , [35],.., Vol. 47, No. 6, pp , [36]. Excel 2., [37] Gerald van Belle and Steven P Millard. Struts: Statistical rules of thumb. Departments of Environmental Health and Biostatistics, University of Washington, [38], [39] Jacob Cohen. A power primer. Psychological bulletin, Vol. 112, No. 1, p. 155, [40]. :., [41] Steven J Spear. Learning to lead at toyota. harvard business review, Vol. 82, No. 5, pp , [42] C. Anderson. Makers: The New Industrial Revolution. Crown Publishing Group, 2012.

189 2015 1 80

189 2015 1 A Design and Implementation of the Digital Annotation Basis on an Image Resource for a Touch Operation TSUDA Mitsuhiro 79 189 2015 1 80 81 189 2015 1 82 83 189 2015 1 84 85 189 2015 1 86 87