実践・競馬データサイエンス.key

Size: px

Start display at page:

Download "実践・競馬データサイエンス.key"

すずりみうら
5 years ago
Views:

1 Practical Data Science for Horse Racing AlphaImpact NUKUI JP

2 Profile (NUKUI Speciality : Machine Learning Jobs: Fringe81 Co.,Ltd. Ad Tech HR Tech (Professional tipster) Experience of horse racing: 12 years ( ) Favorite horse: (Heart's Cry) 2

3 AlphaImpact " Developing horse racing AI (2016/06~) " Members NUKUI Shun : Machine Learning, Horse racing domain knowledge OMOTO Tsukasa : Machine Learning, System Architect, One of committers of LightGBM HARA Tomonori : Horse racing hacker " Activities HP: Sell predictions on netkeiba.com (2017/03~ 3

4 Agenda (Data of horse racing) (Design of objectives) (Feature Engineering) (Training of prediction models) (Evaluation of prediction models) 4

5 Agenda (Data of horse racing) (Design of objectives) (Feature engineering) (Training prediction models) (Evaluation of prediction models) 5

6 What is Horse Racing An exciting sport that horses with jockeys compete A gambling to predict its results 6

7 Why Horse Racing x Data Science? New data is added weekly or everyday We can watch the process of output from live streaming Variety of data Exciting (important!!) 7

8 Hypothesis Practice Verification hypothesis 1 week cycle verification practice 8

9 Agenda (Data of horse racing) (Design of objectives) (Feature engineering) (Training prediction models) (Evaluation of prediction models) 9

10 Problem Setting Horse 1 Horse 2 Horse N score 1 score 2 score N data of N horses in race score of performance bets Focus on 10

11 Selection of Objectives What to solve is more critical than creating features or models (basic objectives) (order) (finishing time) 1 (delta time with winner) (prize) 11

12 Engineering of Objectives (normalization in race) (identify unplaced horses) Good objectives require the domain knowledge and understanding of the problem 12

13 Agenda (Data of horse racing) (Design of objectives) (Feature engineering) (Training prediction models) (Evaluation of prediction models) 13

14 (Horse Table) race info attributes of horse Odds Jockey race history 14

15 Flow of Data Processing Processing features from collected data directly lowers the maintainability feature1 feature2 featuren collected data feature vectors 15

16 Flow of Data Processing Horse table data simplify the interfaces of data processing HorseTable Horse1 Race feature1 feature2 featuren Odds collected data horse table data feature vectors 16

17 (Past Race History) X (Use past X histories as feature) X ) 2 3 X (Summarize past X histories) X X (Summarize histories in the past X monthes) 17

18 Categorical Data Categorical data in horse racing Categorical data able to be used as numeric Can use both of categorical and numeric as a feature 18

19 Encoding of Categorical Data One-Hot-Encoding one hot vector Target Encoding mean, count, sum, ) (hit ratio) (return ratio) 19

20 Automated Achievement Features We made more than 1500 achievement features horse jockey trainer owner sire ( ) track type course length course length weather field condition pace count win hit ratio place hit ratio win return ratio place return ratio subject race condition statistics 20

21 Smoothing Ratio Features " When the categories is small, set the aggregate value closer to the overall average " α Select optimal α for each category count=2 hit ratio=0.5 count=100 hit ratio=0.4 hit ratio=0.2 low confidence A B average where α=0.1 hit ratio=0.254 hit ratio=

22 (Seasonal Features) Cyclic features can be represented by trigonometric functions 4 sinθ 7 cosθ

23 Agenda (Data of horse racing) (Design of objectives) (Feature engineering) (Training prediction models) (Evaluation of prediction models) 23

24 LightGBM The fast, light, accurate implementation of gradient boosting Category variables can be treated as categorical variables Missing values can be treated as missing values AlphaImpact OMOTO LightGBM committer We have a committer of LightGBM 24

25 (Model Training) hyperopt Race1 Race2 Race Race update params LightGBM average LightGBM valid score score feedback RaceN Race LightGBM split into k-folds by the race (not by the horse) cross validation (the valid set is used for early stopping) 25

26 hyperopt Tree-structured Parzen Estimator TPE Grid Search Random Search More efficient than grid search or random search 26

27 Efficient Search with GCE hyperopt LightGBM hyperopt Since LightGBM has a lot of hyper parameters, it takes much time to search even with hyperopt Google Cloud Engine (GCE) The price of the GCE preemptible instance is reasonable, but the instance might be shut down at any time GCE 70% 24 hyperopt Trials epoch pkl If you save the intermediate state as Trials object for each epoch, you can resume searching on the way 27

28 Tips of LightGBM Tuning Dummying categorical variables is often more accurate early stopping Early stopping is necessary to avoid overfitting random_state Accuracy varies considerably with different random_state 28

29 Feature Analysis with LightGBM Check feature importance Check the contribution of features for predictions cf. SHAP (SHapley Additive explanations) 29

30 The Contribution of Features for Predictions Group thousands of features, and calculate the total contribution of each group 30

31 Agenda (Data of horse racing) (Design of objectives) (Feature engineering) (Training prediction models) (Evaluation of prediction models) 31

32 Evaluation Metrics ndcg Apply ndcg, a well-used metric for ranking problem 1 High relevant score should be positioned at high rank Should define the variety of relevant scores 32

33 The Relevant Score of ndcg " (inverse of order) 1/1, 1/2,, 1/N consider the whole ranking " (prize) 15000, 6000, 3800, 2300, 1500, 0, 0,..., 0 consider only top 5 (prize@3) 15000, 6000, 3800, 0, 0, 0, 0,..., 0 " (place payoff) emphasize the dark horses " (win betting share) how close to popularity 33

34 Comparison of Models with ndcg (A) (B) all turf races (A) (B) (A) is more accurate than (B) (= ) (B) (B) is closer to win betting share (A) (B) (A) is more profitable than (B) 34

35 Evaluation of Top-N Box Betting all turf races in 2017 Box : hit ratio, return ratio, std of return ratio : win: place: quinella place: quinella: exacta: trio: 3 trifecta: 3 35

36 Evaluation of Top-N Box Betting As a result of the effort of objective and feature engineering... all turf races in 2017 Return ratio of win betting is 123%!! 36

37 (Qualitative Evaluation) See the predictions for a few of representative races Check if the predictions make sense Overconfidence will lead over fitting UP Seeing the actual predictions lifts motivation 37

38 Summary Horse racing is supreme as a theme of machine learning Objective variables should be engineered to fit the problem setting Feature engineering is required for improving the accuracy LightGBM LightGBM is very effective for horse racing prediction Model performance is evaluated by both quantitative and qualitative 38

39 Thank you 39

Introduction Purpose This training course describes the configuration and session features of the High-performance Embedded Workshop (HEW), a key tool

Introduction Purpose This training course describes the configuration and session features of the High-performance Embedded Workshop (HEW), a key tool for developing software for embedded systems that