Hadoopの全て

Size: px

Start display at page:

Download "Hadoopの全て"

しょうこなかじゅく
6 years ago
Views:

1 Kazuki Ohta ( ) Preferred Infrastructure <kzk@preferred.jp> 1

net/ @kzk_mover l l l Hadoop Hadoop-Gfarm with

2 l l l l l ( ) Preferred Infrastructure, CTO Sedue Hadoop Hadoop l l l l l l Hadoop Hadoop-Gfarm with I/O Project: IOFSL (I/O Forwarding and Scalability Layer) 2

3 Agenda Hadoop? MapReduce MapReduce MapReduce 3

4 4

5 ? l l l 1: l POS A ( : ) l 2 l 2: l l 5

6 l l l 1: l TB PB l 1 l => l 2: l l l 6

7 Hadoop? 7

8 Hadoop? l Google OSS l Apache l Google File System l MapReduce l l Java Yahoo! 4000 l MapReduce 58

9 Hadoop Distributed File System? l l ( :, DC) l 9

10 MapReduce? l HDFS l Map, Reduce l HDFS HDFS 10

11 Cloudera ( : Yahoo Research) Doug Cutting Lucene Doug Yahoo! Facebook, Cloudera 11

12 Hadoop l Web l l l l Twitter Blog l POS l l l l l l l l l l l... l Yahoo!, Facebook, ebay, Amazon l NTT 12

13 Google & The Google File System Sanjay Ghemawat, Howard Gobioff, and Shu-Tak Leong, SOSP 2003 MapReduce: Simplified Data Processing on Large Clusters Jeffrey Dean and Sanjay Ghemawat, SOSP 2004 Parallel Architectures and Compilation Techniques (PACT) 2006, KeyNote program/mapreduce-pact06-keynote.pdf 13

14 MapReduce, HPC,,, etc. MAPREDUCE 10 The First International Workshop on MapReduce and its Applications HPDC MAPRED 2010 The First Internatinal Workshop on Theory and Practice of MapReduce CloudCom mapreduce2010.html 14

15 MapReduce 用 / / mapreduce- hadoop- algorithms- in- academic- papers- may update/ : Analyzing Human Genomes with hadoop human- genomes- with- hadoop/ 15

16 用 1. Modeling true risk 2. Customer churn analysis 3. Recommendation engine 4. Ad targeting 5. PoS transaction analysis 6. Analyzing network data to predict failure 7. Threat analysis 8. Trade surveillance 9. Search quality 10. Data sandbox Ten Hadoopable Problems cloudera- 10- hadoopable- problems- webinar

17 Yahoo! 用 % 用 60% 用 RSS 3 26 日 - > 20 C++ Python > 2-3 日 17

18 Hadoop Cloudera Hadoop Enterprise Yahoo!, Inc. Hadoop Pig, Oozie Facebook Hadoop Hive 18 5

19 Hadoop 19 8

20 Hadoop Hadoop Wiki: Hadoop, hbase on Codezine Hadoop Hadoop 20

21 Apache Hadoop wins TeraSort Benchmark 100T Yahoo! minutes, 3452 nodes 40 nodes per rack 2 quad core Xeon 8 GB Memory 4 Sata Disks 1Gbps Ethernet, 8Gbps uplinks per rack 570MB/sec 21

22 22

23 Hadoop l l l Hadoop Hive l SQL l SELECT user, COUNT(1) FROM log_tbl WHERE user.sex == male GROUP BY user HBase l HDFS l HiveQL 23

24 MapReduce? 24

25 Web Web 200 * 20KB = 400 TB Disk 50MB/sec (SATA) G

26 G ok

27 27

28 MPI (Message Passing Interface) (Send, Recv, All-to-All) 28

29 MPI 29

30 MPI

31 MapReduce / 1 MapReduce 31

32 MapReduce WordCount Grep Sort Log Analysis Web Graph Generation Inverted Index Construction Machine Learning NaiveBayes, K-means, Expectation Maximization, etc. 32

33 MapReduce 用 / / mapreduce- hadoop- algorithms- in- academic- papers- may update/ 33

34 Google 34 22

35 MapReduce 35

36 MapReduce 36

37 MapReduce <key, value>* Map map: <key, value> <key, value >* Shuffle shuffle: <key, reducers> destination reducer Reduce reduce: <key, <value > * > <key, value >* <key, value >* 37

38 : map(string key, string value) { foreach word in value: emit(word, 1); } reduce(string key, vector<int> values) { int result = 0; for (int i = 0; I < values.size(); i++) result += values[i]; emit(key, result); } 38

39 MapReduce <k, v>* <k, v >* Data <k, v>* Map <k, <v >*>* <k, v >* <k, v>* <k, v >* Reduce Data Data <k, v>* Map Shuffle <k, v>* <k, v >* Reduce Data Data <k, v>* Map <k, <v >*>* <k, v >* 39

40 : doc1 : foo foo foo bar bar buzz Data Map Reduce Data Data Map Shuffle Reduce Data Data Map 40

41 : doc1 : foo foo foo bar bar buz Data Map doc1: foo doc1: foo Reduce Data Data Map Shuffle doc1: foo doc1: bar Reduce Data Data Map doc1: bar doc1: buz 41

42 : doc1 : foo foo foo bar bar buz doc1: foo doc1: foo Data Data doc1: foo doc1: bar Map Map Shuffle Reduce Data Data doc1: bar doc1: buz Map Reduce Data 42

43 : doc1 : foo foo foo bar bar buz doc1: foo doc1: foo foo: 1 foo: 1 Data Map doc1: foo doc1: bar bar: 1 foo: 1 Reduce Data Data Map doc1: bar doc1: buz bar: 1 buz: 1 Reduce Data Data Map 43

44 : doc1 : foo foo foo bar bar buz Data foo: 1 foo: 1 Map bar: <1, 1> buz: <1> bar: 1 foo: 1 Reduce Data Data Map foo: <1, 1, 1> bar: 1 buz: 1 Reduce Data Data Map 44

45 : doc1 : foo foo foo bar bar buz Data Map bar: <1, 1> buz: <1> Reduce Data Data Map foo: <1, 1, 1> Reduce Data Data Map 45

46 : doc1 : foo foo foo bar bar buz Data Map bar: <1, 1> buz: <1> bar: 2 buz: 1 Reduce Data Data Map foo: <1, 1, 1> foo: 3 Reduce Data Data Map 46

47 : doc1 : foo foo foo bar bar buz Data Map bar: 2 buz: 1 Reduce Data Data Map foo: 3 Reduce Data Data Map 47

48 MapReduce Map Reduce Map, Reduce Map, Reduce Moving Computation is Cheaper Than Moving Data 48

49 100TB : Yahoo TeraSort 49

50 Hadoop MapReduce Master/Slave JobTracker Master Job Task Task TaskTracker Job: MapReduce Task: MapTask, ReduceTask Task Task TaskTracker TaskTracker Slave JobTracker Task 50

51 MapReduce Architecture TaskTracker JobTracker 51

52 Hadoop Master-Slave 52

53 MapReduce 53

54 Hive SQL MapReduce Java hive> CREATE TABLE shakespeare (freq INT, word STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY \t STORED AS TEXTFILE; hive> LOAD DATA INPATH shakespeare_freq INTO TABLE shakespeare; hive> SELECT * FROM shakespeare LIMIT 10; hive> SELECT * FROM shakespeare WHERE freq > 100 SORT BY freq ASC LIMIT 10; 54

55 Pig MapReduce DSL (Domain Specific Language) Pig Latin A = LOAD myfile AS (x, y, z); B = FILTER A by x> 0; C = GROUP B BY x; D = FOREACH A GENERATE x, COUNT(B); STORE D INTO output ; pig.jar: parses checks optimizes plans execution submits jar to Hadoop monitors job progress Execution Plan Map: Filter Reduce: Count 55

56 Enjoy Hadoop J Thank you! 56

57 57

58 58

59 Hadoop 59

60 Hadoop Hadoop 1 hadoop fit 60 71

61 RDBMS vs Hadoop RDBMS Hadoop Transactions/ second 1000 s n/a Concurrent Queries 100 s 10 s Update Patterns Read / Write Append Only Join Complexity 100 s of tables Arbitrary keys Schema Complexity Structured Structured or Unstructured Total Data Volume 100 s of TBs 10 s of PBs Per Job Data Volume 10 s of TBs 10 s of PBs Processing Freedom SQL MapReduce, Streaming, Pig, Hive, etc. Hardware Profile High-end servers Commodity / Utility Hardware 61 72

62 Hadoop 62

63 Hadoop 63

64 Hadoop Hadoop Distributed File System (HDFS) GFS MapReduce Hadoop MapReduce MapReduce, 64

65 HDFS Master/Slave Master NameNode Master ( ) DataNode Slave ( ) 65

67 HDFS Client 67

68 (1) ( DC) 68 55

69 (2) Hadoop 1 2 DC(Rack) 3 DC(Rack) 4 (1) (3) (2) 69 56

70 NameNode NameNode CPU Google Google BigTable GoogleFileSystem? 70

71 Hadoop MapReduce Master/Slave JobTracker Master Job Task Task TaskTracker Job: MapReduce Task: MapTask, ReduceTask Task Task TaskTracker TaskTracker Slave JobTracker Task 71

72 MapReduce Architecture TaskTracker JobTracker 72 59

73 MapReduce MapReduce Shuffle Mapper <-> Reducer Shuffle (Map ) Map Map Map Shuffle 73 Reduce Reduce

74 Hadoop 74

$Hive SQL MapReduce Java hive> CREATE TABLE shakespeare (freq INT, word STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY \t STORED AS TEXTFILE; hive> LOAD DATA$

75 Hive SQL MapReduce Java hive> CREATE TABLE shakespeare (freq INT, word STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY \t STORED AS TEXTFILE; hive> LOAD DATA INPATH shakespeare_freq INTO TABLE shakespeare; hive> SELECT * FROM shakespeare LIMIT 10; hive> SELECT * FROM shakespeare WHERE freq > 100 SORT BY freq ASC LIMIT 10; 75 61

76 : Hive Join MapReduce join hive> INSERT OVERWRITE TABLE merged SELECT s.word, s.freq, k.freq FROM shakespeare s JOIN shakespeare2 k ON (s.word = k.word) WHERE s.freq >= 1 AND k.freq >= 1; hive> SELECT * FROM merged LIMIT 20; 76 62

77 Hive Hive join 77

78 Pig MapReduce DSL (Domain Specific Language) Pig Latin A = LOAD myfile AS (x, y, z); B = FILTER A by x> 0; C = GROUP B BY x; D = FOREACH A GENERATE x, COUNT(B); STORE D INTO output ; pig.jar: parses checks optimizes plans execution submits jar to Hadoop monitors job progress Execution Plan Map: Filter Reduce: Count 78

79 : Pig Deta-flow set, associative array, tuple : # named_events = FOREACH events_by_time GENERATE $1 as event, $2 as hour, $3 as minute; # 12 noon_events = FILTER named_events BY hour = 12 ; # unique distinct_events = DISTINCT noon_events; 79 65

80 hbase: Google BigTable interactive TB PB key lookup transaction RDBMS 80 66

81 hbase: key Web lookup 81 67

82 hbase: MapReduce row MapReduce MapReduce sort/search/indexing hbase sequential scan MapReduce 82

83 More and more projects... fuse-hdfs: HDFS fuse Zookeeper: Sqoop: RDBMS HDFS Avro: Serialization + RPC Scribe, Flume: Mahout: MapReduce Oozie:... 83

84 Hadoop 84

85 (CDH) Cloudera Distribution for Hadoop Apache Project Cloudera bugfix & security fix ( ) yum/apt yum install hadoop-0.20-* Hive/Pig/Hbase/Flume 85

86 HDFS # ls alias dfsls='~/hadoop/bin/hadoop dfs -ls # ls -r alias dfslsr='~/hadoop/bin/hadoop dfs -lsr # rm alias dfsrm='~/hadoop/bin/hadoop dfs -rm # rm -r alias dfsrmr='~/hadoop/bin/hadoop dfs -rmr' # cat alias dfscat='~/hadoop/bin/hadoop dfs -cat # mkdir alias dfsmkdir='~/hadoop/bin/hadoop dfs -mkdir 86

87 HDFS HDFS alias dfsput='~/hadoop/bin/hadoop dfs -put dfsput <local-path> <hdfs-path> HDFS alias dfsget='~/hadoop/bin/hadoop dfs -get dfsget <hdfs-path> <local-path> 87 77

88 HadoopStreaming MapReduce sh C++ Ruby Python MapReduce Hadoop Streaming wrapper <key, value> Amazon, Facebook Streaming 88

89 ./bin/hadoop jar contrib/hadoop streaming.jar -input inputdir [HDFS ] -output outputdir [HDFS ] -mapper map [map ] -reduce reduce [reduce ] -inputformat [TextInputFormat SequenceFileAsTextInputFormat] -outputformat [TextOutputFormat] 89 79

90 : Ruby $./bin/hadoop jar contrib/hadoop streaming.jar -input wcinput -output wcoutput -mapper /home/hadoop/kzk/map.rb -reducer /home/hadoop/kzk/reduce.rb -inputformat TextInputFormat -outputformat TextOutputFormat map.rb #!/usr/bin/env ruby while!stdin.eof? line = STDIN.readline.strip ws = line.split ws.each { w puts "#{w}\t1 } end reduce.rb #!/usr/bin/env ruby h = {} while!stdin.eof? line = STDIN.readline.strip word = line.split("\t")[0] unless h.has_key? word h[word] = 1 else h[word] += 1 end end h.each { w, c puts "#{w}\t# {c} } 90

91 MapReduce (1) Join 91

92 Join? : : EMP: 42,, loc(13) LOC: 13, EMP: 42,, loc(13), 92

93 MapReduce join Map-Side Join mapper join Reduce-Side Join reducer join String getlocation(int locid) { // or } void map(k, v) { int locid = parse_locid(v); String location = getlocation(locid); } map-side join 93 41

94 Reduce-Side Join: Union class Record { enum Type { emp, loc } Type type; // EMP String empname; int empid; }; // LOC String locationname; int locid; 94

95 Reduce-Side Join: Mapper mapper reducer mapper type key locid void map(k, v) { // employee Record r = parse(v); r.type = Type.emp; emit (r.locid, r); } void map(k, v) { // location } Record r = parse(v); r.type = Type.loc; emit (r.locid, r); 95 43

96 Reduce-Side Join: Reducer values Type.loc employees void reduce(k, values) { Record thislocation; List<Record> employees; for (Record v in values) { if (v.type == Typ.loc) } thislocation = v; else employees.add(v); } for (Record e in employees) { } e.locationname = thislocation.locationname; emit(e); 96

97 MapReduce (2) 97

98 : MapReduce BFS 1 1 DistanceTo(StartNode) == 0 DistanceTo(n) == 1 n StartNode DistanceTo(m) = 1 + min(distanceto(n), n S) S m MapReduce Mapper, Reducer? 98

100 : 2, 4 2: 1, 3, 4 3: 1 4: 1, 3 100

101 Parallel BFS by MapReduce Map Key: n Value: D (start ), S (n ) S m (m, D + 1) Reduce : m : MapReduce1 1 hop MapReduce 101

102 Hadoop OSS / MapReduce JOIN, Shuffle :-) 102

103 Enjoy Playing Around Hadoop J Thank you! 103

Microsoft PowerPoint _Hadoop.pptx

Microsoft PowerPoint _Hadoop.pptx Hadoop で行う大規模データ処理 kzk Hadoop とは? Google の基盤ソフトウェアのクローン Google File System Yahoo Research の Doug Cutting 氏が開発元々は Nutch Crawler のサブプロジェクト Doug の子供の持っているぬいぐるみの名前 Java で記述 Amazon S3 との親和性