ビッグデータアナリティクス - 第3回: 分散処理とApache Spark

Size: px

Start display at page:

Download "ビッグデータアナリティクス - 第3回: 分散処理とApache Spark"

たつぞうあんさい
6 years ago
Views:

1 3 : Apache Spark / 32

2 ZB ZB 1ZB = = 1,000,000,000,000 GB Word Excel XML CSV JSON text / 32

3 CPU SPECfp Pentium G ,946 Xeon Gold , Xeon Platinum , PRIMERGY RX GB 19 PRIMERGY RX GB 45 PRIMERGY RX GB 180 1TB SATA 200MB/s 2500 SAS 300MB/s 5.1 SSD 600MB/s / 32

4 CPU / 32

5 R Excel Excel / 32

6 Mapreduce MapReduce Hadoop/Spark Map: Reduce: Map Hadoop/Spark1.x Spark 2.x / 32

7 Apache Spark / 32

8 twitter (JSON) /project/bigdata-lab/bda/tweet_ json 150GB (10/15 ) twitter 1% spark % time cat /project/bigdata-lab/bda/tweet_ json wc -l cat tweet_ json 0.13s user 46.46s system 4% cpu 18:29.32 total wc -l 19.21s user 29.22s system 4% cpu 18:29.32 total I/O / 32

9 JSON CSV XML JavaScript {"id": , "text":" \n ","place":null,"lang":"ja"} { :, :, :, :,...} { :, :{ :{ :, :,...}, :,...}} / 32

10 {...,"lang":"ja",...} % lang : ja % cat /project/bigdata-lab/bda/tweet_ json grep \"lang\":\"ja\" w ) / 32

11 Spark MLlib GraphX Spark Scala, Java, Python, R Hadoop HDFS Hive Text CSV JSON / 32

12 Hadoop Hadoop Spark bda1node0x (x ) bda1node01 04 ssh bda1node05 $ ssh bda1node05 Last login: Mon Oct 16 16:00: from mm-dhcp naist.jp :12:14 JST [ysuzuki@bda1node05 ~]$ mandara $ cat /project/bigdata-lab/bda/tweet_ json head [Tue Oct 03 20:40:40 JST 2017]Establishing connection. [Tue Oct 03 20:40:42 JST 2017]Connection established. [Tue Oct 03 20:40:42 JST 2017]Receiving status stream. {"in_reply_to_status_id_str":null,"in_reply_to_status_id":null, "created_at":"tue Oct 03 11:40: ","in_reply_to _user_id_str":null,"source":"<a href=\" / 32

13 HDFS Hadoop Spark Linux HDFS HDFS % hadoop fs -put /project/bigdata-lab/bda/tweet_ json HDFS % hadoop fs -ls -h Found 4 items drwx ysuzuki is-staff :00.Trash drwxr-xr-x - ysuzuki is-staff :59.sparkStaging drwx ysuzuki is-staff :32.staging -rw-r--r-- 3 ysuzuki is-staff G :42 tweet_ json / 32

14 Spark Spark Scala Java Python R Python spark % pyspark2 --master yarn % pyspark2 % spark2-shell --master yarn scala % pyspark2 --driver-memory 16g --executor-memory 16g --master yarn Welcome to / / / / \ \/ _ \/ _ / / _/ / /. /\_,_/_/ /_/\_\ version cloudera2 /_/ Using Python version (r266:84292, Aug :36:59) SparkSession available as spark. >>> / 32

15 DataFrame Spark (DataFrame) (select,filter,groupby) (join) (read, show) / 32

16 >>> df = spark.read.json("/user/ysuzuki/tweet_ json") df tweet_ json >>> df.count() / 32

17 >>> df.filter("lang= ja ").count() ja en es groupby / 32

18 >>> df.groupby("lang").count().sort("count", ascending=false).show() lang count en ja es ar und ko pt th only showing top 20 rows / 32

19 1) 2) Spark / 32

20 count.py count.py from pyspark.sql import SparkSession spark = SparkSession \.builder \.appname("app example") \.config("master", "yarn") \.getorcreate() df = spark.read.json("tweet_ json") print df.count() count.py % spark2-submit count.py tee output.txt % cat output.txt / 32

21 >>> from pyspark.sql.functions import explode,split >>> df.filter("lang = en ").select("text").distinct().select(explode(split( text, ))).groupby("col").count().sort("count",ascending=false).show() col count RT the to a I be me only showing top 20 rows / 32

22 >>> df.groupby("text").count().show() text count Big economic call... reall count >>> df.groupby("text").count().sort("count",ascending=false).show() text count / 32

23 >>> df.groupby("text").count().sort("count",ascending=false).show(20,false) text #L 10/15 1 #L (^^) 3 10/13 10:59 # 4284 I appreciate Eminem / 32

24 >>> df.groupby("user.name").count().sort("count",ascending=false).show(20,false) name count ; ID / 32

25 ID >>> a = df.groupby("user.id").count().sort("count",ascending=false) >>> a.show() id count / 32

26 ID >>> b = df.select("user.id","user.name").distinct() >>> b.show() id name [6 ] Vibes Liseth Valencia R / 32

27 a b >>> c = a.join(b, a.id == b.id, inner ).select("name","count") >>> c.show() name count McDonalds Japan Test Account (NESCAF ) / 32

28 text >>> df.where(col( text ).like("% %")).select( text ).show() text / 32

29 CSV >>> c.write.csv("output_csv") >>> exit $ hadoop fs -get output_csv output_csv Excel utf-8 Excel / 32

30 / 32

31 / 32

32 Apache Spark 100GB / 32

WebOS aplat WebOS WebOS 3 XML Yahoo!Pipes Popfry UNIX grep awk XML GUI WebOS GUI GUI 4 CUI

7 XML Week Web WebOS WebShell WebOS WebOS GUI WebOS WebOS 2 WebOS aplat WebOS WebOS 3 XML Yahoo!Pipes Popfry UNIX grep awk XML GUI WebOS GUI GUI 4 CUI CUI JavaScript I/O CommandClass WebShell webshell