Cloud connect the world as a Glue

Size: px

Start display at page:

Download "Cloud connect the world as a Glue"

ゆきしろみず
7 years ago
Views:

1 Cloud connect the world as a Glue AWS Dev Day 2017 Track 2 Masahiro

2 Me Masahiro Nagano Mercari, Inc Principal Engineer Site Reliability Engineering (SRE) Team BASE, Inc Technical Advisor

3 SRE Team

4 SRE Site Reliability Engineering Google Ben Treynor Google / Software Engineering/Team = Google SRE

5 Google SRE (SWE) 50% 50% SRE SWE SLA

7 Mercari SRE 2015/11 SRE 6

8 Mercari SRE Operations OnCall ( ) Automation Software Eng.

9 Agenda / 3 /

10 Mercari 国内最大級のフリマアプリ 3分で簡単に出品安心安全な決済

11 Mercari KPI 6500 DL(JP+US) GMV( ) 100

12 (JP) JP 4000 (2016/11)

13 1,200 1 (peek )

14 24 50% 24

15 Global Service JP UK US 2016/08 US AppStore 3位 2017/03/15 リリース

16 Global Development Team San Francisco London Tokyo San Francisco/London

17 Global Development Team Tokyo JP region San Fransisco London /

18 Global Development 7 18:00 10:00 Tokyo 9 San Francisco 3 1:00 London

19 Global Development (1) Pull Reqeuest Slack Video Conference ( )

20 Global Development (2) ios/android Region fork branch

21 SRE 6 1 US 1 US Sync MTG 9 (PDT 17:00) Video Conference UK MTG OnCall 9 US

22 Mercari Architecture

23 Infrastructure UK US JP 石狩DC 専用サーバ Cloud Hybrid & Multi Cloud Cloud

24 Infrastructure history (1) 2013/07 JP VPS 1 Web DB 2

25 Infrastructure history (2) 2014/09 US AWS (Oregon) JP AWS AWS US US

26 Infrastructure history (3) (2015/02 kazeburo ) 2015/11 SRE AWS 2017/03 UK GCP

27 Architecture JP DNS-RR nginx nginx nginx +α Reverse Proxy = nginx Application = Apache+mod_php App App App App App App Database = MySQL Cache = memcached Search = Solr MySQL MySQL memcached memcached Diagonal Scale Database iomemory NVMe cloud util cloud util

28 US EC2 DNS-RR EC2 nginx nginx nginx App App App App App App EC2 EC2 EC2 EC2 EC2 EC2 EC2 Architecture JP EC2/GCE ( ) US cloud load balancer nginx nginx nginx App App App App App App UK GCE GCE GCE GCE GCE GCE GCE GCE GCE MySQL EC2 MySQL EC2 RDS EC2 memcached EC2 memcached DB RDS UK Cloud Load Balancer MySQL GCE MySQL GCE GCE memcached GCE memcached util EC2 util EC2 util GCE util GCE

29 Architecture Ansible Playbook JP US App Store 3 EC2 IaaS

30 Mercari Architecture 3 Region JP/US/UK Architecture AWS JP JP/US/UK

31 Mercari Global Infrastructure

32 Global Infrastructure Mercari JP/US/UK

33 Global Infrastructure DNS: Amazon Route53 CDN: Akamai, CloudFront 各Region サーバが中心 JP US UK 共通アーキテクチャクラウドが中心信頼性の高いAWSのサービスが挟み込む決済/物流/Domestic Service 決済/物流/Domestic Service Common Micro Services Analysis: Google BigQuery Storage: Amazon S3 決済/物流/Domestic Service

34 Amazon Route53 DNS Roadworker github.com/codenize-tools/roadworker Routefile Github #Routefile hosted_zone "mercari.jp." do rrset "api.mercari.jp.", "CNAME" do ttl 30 resource_records( "endpoint-api.mercari.jp" ) end end PR Pull Request merge CI Github Travis-CI Route53

35 Amazon Route53 + HealthCheck DNS-RR DNS DNS-RR DNS-RR Route53 Health Check ( )

36 Route53 + Health Check with Roadworker #Routefile [ 153.x.y.150, "153.x.y.151"].each do ip rrset "endpoint-ha.mercari.jp.", "A" do ttl 30 weight 1 set_identifier endpoint-ha- + ip.gsub(/\./,'-') health_check " :request_interval => 30, :failure_threshold => 3 resource_records( "#{ip}" ) end end Health Check DNS-RR

37 ( ) DNS unbound BIND Consul/DNS resolv.conf *.local unbound *.consul DNS DNS DNS DNS unbound *.local BIND *.consul consul DNS interface unbound unbound unbound App App App unbound unbound unbound App App App

38 ( ) DNS CNAME DNS CNAME CNAME db-cstool-master IN CNAME cstool-db.xxxxx.us-west-2.rds.amazonaws.com. EC2

39 Amazon S3 IAM

40 :! App App App App App App /day / AWS SDK for PHP PUT

41 : batch + aws-cli App App App Log App App App Log > 1TB/day fluent S3 aws-cli fluent-plugin-s3

42 : MySQL Master MySQL BackupSlave xtrabackup + aws-cli MySQL Master MySQL BackupSlave > 1.2TB( )/day MySQL xtrabackup( 1 mysqldump) backup slave backup aws-cli

43 Amazon S3 as a Hub nginx nginx nginx App App App App App App consul Microservices ML API ML API import + IAM SaaS / MySQL import/export goofys SFTP ACL Partner S3 Hub

44 Amazon ML

46 50msec ( )

47 18ms AWS GCP ms GCP 6ms 140ms / /

48 HTTPS TCP Handshaking RTT 26msec HTTPS 200msec RTT 100msec 600msec ) mercari API (90percentile) 100msec

49 , US / ( ) SaaS

50 CDN Cloudfront, Akamai, Fastly CDN TLS Handshaking CDN Origin CDN

man Intelligence Assignment/ Human Intelligence on Tasks (HIT) Task Tasks (HIT) l Turk Assignment/ Task Workers Amazon Mechanical Turk Requester Human Intelligence Workers Tasks (HIT) Workers Amazon

Amazon Task Tasks (HIT) Task Mechanical Turk Human Intelligence Amazon Tasks (HIT) Mechanical Turk Non-Service Specific User Client Client Users Mobile Client AM Add-on MobileUser Client Client

51 man Intelligence Assignment/ Human Intelligence on Tasks (HIT) Task Tasks (HIT) l Turk Assignment/ Task Workers Amazon Mechanical Turk Requester Human Intelligence Workers Tasks (HIT) Workers Amazon Assignment/ Mechanical Turk Task Assignment/ Task HumanWorkers Intelligence Tasks (HIT) Workers Requester Amazon Mechanical Turk Assignment/ Requester Workers Requester Human Intelligence Assignment/ Amazon Task Tasks (HIT) Task Mechanical Turk Human Intelligence Amazon Tasks (HIT) Mechanical Turk Non-Service Specific User Client Client Users Mobile Client AM Add-on MobileUser Client Client Multimedia Multimedia Users Mobile Client Corporate data center Internet anagement IAM Add-on et AWS Management nsole Console Requester Assignment/ Task Workers Internet Example: IAM Add-on IAM Add-on Example: IAM Add-on Reques Non-Service Specific Non-Service Specific Non-Service Specific Amazon Mechanical Turk Users Workers CDNの利用: mercari Web Requester Non-Service Specific vice Specific Human Intelligence Tasks (HIT) Assignment/ Amazon Requester Task Mechanical Turk AWS Management Example: Console IAM Add-on Users Client Corporate Client User Multimedia data center Traditional Users Client Mobile Client Traditional Multimedia Corporate server data center server Multimedia Corporate Mobile Client User data center Corporate Traditional Non-Service Specific data center server Traditional Multimedia Corporate Users Client server data center User Users Assignment/ Task Traditional Mobile Client server Client Workers Multimedia Mobile Client Corporate data center Multimedia CDN Traditional server AWS Management Console Internet IAM Add-on JP Mobile Client Human Intelligence Tasks (HIT) IAM Add-on AWS Management Example: Console IAM Add-on Example: IAM Add-on IAM Add-on User Users Example: AWS Management Internet IAM Add-on IAM Add-on Console Internet AWS Management Console Client Example: Add-on IAMIAM Add-on US UK mercari.com/ mercari.com/uk/ 2011 Amazon Web Services LLC or its affiliates. All rights reserved. Mobile Client Mul Example: IAM Add-on Internet 2011 Amazon Web Services LLC or its affiliates. All rights reserved. AWS Management IAM Add-on Example: 2011 Amazon Web Services LLC or its affiliates All rights Amazon reserved. Web Services LLC or its affiliates. All rights reserved Amazon Web Services LLC or its affiliates. All rights reserved Amazon Web Services LLC or its affiliates. All rights reserved. Console IAM Add-on 2011 Amazon Web Services LLC or its affiliates. All rights reserved. azon Web Services LLC or its affiliates. All rights reserved. mercari.com/jp/ 2011 Amazon Web Services LLC or its affiliates. All rights re 石狩DC

52 HTTPS KeepAlive PHP Application KeepAlive TCP KeepAlive => Connection Pooling Proxy Server

53 chocon Go Proxy Server OSS github.com/kazeburo/chocon

54 chocon Private Network Client http chocon http or https keepalive Web % curl -H Host: example.com.ccnproxy-https proxy DNS URL *.ccnproxy-https IN CNAME chocon.local. % curl

55 Before chocon $./httpstat.sh /dev/null HTTP/ OK Server: nginx/ Date: Thu, 01 Jun :43:49 GMT Content-Type: application/json; charset=utf-8 Content-Length: 22 Expires: Thu, 01 Jun :43:49 GMT Cache-Control: max-age=3600,public Body stored in: /tmp/httpstat-body DNS Lookup TCP Connection SSL Handshake Server Processing Content Transfer [ 2ms 24ms 197ms 25ms 0ms ] namelookup:2ms connect:26ms pretransfer:223ms starttransfer:248ms total:248ms

56 After chocon $./httpstat.sh /dev/null HTTP/ OK Cache-Control: max-age=3600,public Content-Length: 22 Content-Type: application/json; charset=utf-8 Date: Thu, 01 Jun :43:49 GMT Expires: Thu, 01 Jun :43:49 GMT Server: nginx/ X-Chocon-Req: bsczjrcmz9wbrn8tyhz3wv Body stored in: /tmp/httpstat-body DNS Lookup TCP Connection Server Processing Content Transfer [ 1ms 1ms 19ms 0ms ] namelookup:1ms connect:2ms starttransfer:21ms total:21ms

57 Why chocon? middleware forward proxy HTTPS HTTPS end to end MITM Proxy Go HTTP/2

58 chocon in JP App App http or https App App keepalive Microservices chocon SaaS App App Cloud API endpoint App App DC(Cloud) Cloud(DC) 90msec 19msec RTT AWS SDK endpoint

59 chocon & Pacific Ocean App App App App chocon HTTPS, HTTP/2 Keepalive US Cloud Cloudfront/CDN 100msec Region US

61 JP/US/UK 3 Region Amazon Route53, Amazon S3

62 We re Hiring! SRE

Microsoft Azure Microsoft Corporation Global Blackbelt Sales Japan OSS TSP Rio Fujita

Microsoft Azure Microsoft Corporation Global Blackbelt Sales Japan OSS TSP Rio Fujita Agenda Microsoft Azure Microsoft Azure Microsoft OSS 2 Microsoft Azure Promotion 3 https://azure.microsoft.com/ja-jp/free/