Cloud connect the world as a Glue AWS Dev Day 2017 Track 2 Masahiro Nagano @kazeburo
Me Masahiro Nagano / @kazeburo Mercari, Inc Principal Engineer Site Reliability Engineering (SRE) Team BASE, Inc Technical Advisor
SRE Team
SRE Site Reliability Engineering Google Ben Treynor Google / Software Engineering/Team = Google SRE
Google SRE (SWE) 50% 50% SRE SWE SLA http://landing.google.com/sre/book.html
Mercari SRE 2015/11 SRE 6
Mercari SRE Operations OnCall ( ) Automation Software Eng.
Agenda / 3 /
Mercari 国内最大級のフリマアプリ 3分で簡単に出品 安心安全な決済
Mercari KPI 6500 DL(JP+US) 1 100 GMV( ) 100
(JP) JP 4000 (2016/11)
1,200 1 (peek )
24 50% 24
Global Service JP UK US 2016/08 US AppStore 3位 2017/03/15 リリース
Global Development Team San Francisco London Tokyo San Francisco/London
Global Development Team Tokyo JP region San Fransisco London /
Global Development 7 18:00 10:00 Tokyo 9 San Francisco 3 1:00 London
Global Development (1) Pull Reqeuest Slack Video Conference ( )
Global Development (2) ios/android Region fork branch
SRE 6 1 US 1 US Sync MTG 9 (PDT 17:00) Video Conference UK MTG OnCall 9 US
Mercari Architecture
Infrastructure UK US JP 石狩DC 専用サーバ Cloud Hybrid & Multi Cloud Cloud
Infrastructure history (1) 2013/07 JP VPS 1 Web DB 2
Infrastructure history (2) 2014/09 US AWS (Oregon) JP AWS AWS US US
Infrastructure history (3) (2015/02 kazeburo ) 2015/11 SRE AWS 2017/03 UK GCP
Architecture JP DNS-RR nginx nginx nginx +α Reverse Proxy = nginx Application = Apache+mod_php App App App App App App Database = MySQL Cache = memcached Search = Solr MySQL MySQL memcached memcached Diagonal Scale Database iomemory NVMe cloud util cloud util
US EC2 DNS-RR EC2 nginx nginx nginx App App App App App App EC2 EC2 EC2 EC2 EC2 EC2 EC2 Architecture JP EC2/GCE ( ) US cloud load balancer nginx nginx nginx App App App App App App UK GCE GCE GCE GCE GCE GCE GCE GCE GCE MySQL EC2 MySQL EC2 RDS EC2 memcached EC2 memcached DB RDS UK Cloud Load Balancer MySQL GCE MySQL GCE GCE memcached GCE memcached util EC2 util EC2 util GCE util GCE
Architecture Ansible Playbook JP US App Store 3 EC2 IaaS
Mercari Architecture 3 Region JP/US/UK Architecture AWS JP JP/US/UK
Mercari Global Infrastructure
Global Infrastructure Mercari JP/US/UK
Global Infrastructure DNS: Amazon Route53 CDN: Akamai, CloudFront 各Region サーバが中心 JP US UK 共通アーキテクチャ クラウドが中心 信頼性の高いAWSの サービスが挟み込む 決済/物流/Domestic Service 決済/物流/Domestic Service Common Micro Services Analysis: Google BigQuery Storage: Amazon S3 決済/物流/Domestic Service
Amazon Route53 DNS Roadworker github.com/codenize-tools/roadworker Routefile Github #Routefile hosted_zone "mercari.jp." do rrset "api.mercari.jp.", "CNAME" do ttl 30 resource_records( "endpoint-api.mercari.jp" ) end end PR Pull Request merge CI Github Travis-CI Route53
Amazon Route53 + HealthCheck DNS-RR DNS DNS-RR DNS-RR Route53 Health Check ( )
Route53 + Health Check with Roadworker #Routefile [ 153.x.y.150, "153.x.y.151"].each do ip rrset "endpoint-ha.mercari.jp.", "A" do ttl 30 weight 1 set_identifier endpoint-ha- + ip.gsub(/\./,'-') health_check "http://#{ip}/hc", :request_interval => 30, :failure_threshold => 3 resource_records( "#{ip}" ) end end Health Check DNS-RR
( ) DNS unbound BIND Consul/DNS resolv.conf *.local unbound *.consul DNS DNS DNS DNS unbound *.local BIND *.consul consul DNS interface unbound unbound unbound App App App unbound unbound unbound App App App
( ) DNS CNAME DNS CNAME CNAME db-cstool-master IN CNAME cstool-db.xxxxx.us-west-2.rds.amazonaws.com. EC2
Amazon S3 IAM
:! App App App App App App /day / AWS SDK for PHP PUT
: batch + aws-cli App App App Log App App App Log > 1TB/day fluent S3 aws-cli fluent-plugin-s3
: MySQL Master MySQL BackupSlave xtrabackup + aws-cli MySQL Master MySQL BackupSlave > 1.2TB( )/day MySQL xtrabackup( 1 mysqldump) backup slave backup aws-cli
Amazon S3 as a Hub nginx nginx nginx App App App App App App consul Microservices ML API ML API import + IAM SaaS / MySQL import/export goofys SFTP ACL Partner S3 Hub
Amazon ML
50msec ( )
18ms AWS GCP 70-100ms GCP 6ms 140ms / /
HTTPS TCP Handshaking RTT 26msec HTTPS 200msec RTT 100msec 600msec ) mercari API (90percentile) 100msec
, US / ( ) SaaS
CDN Cloudfront, Akamai, Fastly CDN TLS Handshaking CDN Origin www.mercari.com CDN
man Intelligence Assignment/ Human Intelligence on Tasks (HIT) Task Tasks (HIT) l Turk Assignment/ Task Workers Amazon Mechanical Turk Requester Human Intelligence Workers Tasks (HIT) Workers Amazon Assignment/ Mechanical Turk Task Assignment/ Task HumanWorkers Intelligence Tasks (HIT) Workers Requester Amazon Mechanical Turk Assignment/ Requester Workers Requester Human Intelligence Assignment/ Amazon Task Tasks (HIT) Task Mechanical Turk Human Intelligence Amazon Tasks (HIT) Mechanical Turk Non-Service Specific User Client Client Users Mobile Client AM Add-on MobileUser Client Client Multimedia Multimedia Users Mobile Client Corporate data center Internet anagement IAM Add-on et AWS Management nsole Console Requester Assignment/ Task Workers Internet Example: IAM Add-on IAM Add-on Example: IAM Add-on Reques Non-Service Specific Non-Service Specific Non-Service Specific Amazon Mechanical Turk Users Workers CDNの利用: mercari Web Requester Non-Service Specific vice Specific Human Intelligence Tasks (HIT) Assignment/ Amazon Requester Task Mechanical Turk AWS Management Example: Console IAM Add-on Users Client Corporate Client User Multimedia data center Traditional Users Client Mobile Client Traditional Multimedia Corporate server data center server Multimedia Corporate Mobile Client User data center Corporate Traditional Non-Service Specific data center server Traditional Multimedia Corporate Users Client server data center User Users Assignment/ Task Traditional Mobile Client server Client Workers Multimedia Mobile Client Corporate data center Multimedia CDN Traditional server AWS Management Console Internet IAM Add-on JP Mobile Client Human Intelligence Tasks (HIT) IAM Add-on AWS Management Example: Console IAM Add-on Example: IAM Add-on IAM Add-on User Users Example: AWS Management Internet IAM Add-on IAM Add-on Console Internet AWS Management Console Client Example: Add-on IAMIAM Add-on US UK mercari.com/ mercari.com/uk/ 2011 Amazon Web Services LLC or its affiliates. All rights reserved. Mobile Client Mul Example: IAM Add-on Internet 2011 Amazon Web Services LLC or its affiliates. All rights reserved. AWS Management IAM Add-on Example: 2011 Amazon Web Services LLC or its affiliates. 2011 All rights Amazon reserved. Web Services LLC or its affiliates. All rights reserved. 2011 Amazon Web Services LLC or its affiliates. All rights reserved. 2011 Amazon Web Services LLC or its affiliates. All rights reserved. Console IAM Add-on 2011 Amazon Web Services LLC or its affiliates. All rights reserved. azon Web Services LLC or its affiliates. All rights reserved. mercari.com/jp/ 2011 Amazon Web Services LLC or its affiliates. All rights re 石狩DC
HTTPS KeepAlive PHP Application KeepAlive TCP KeepAlive => Connection Pooling Proxy Server
chocon Go Proxy Server OSS github.com/kazeburo/chocon
chocon Private Network Client http chocon http or https keepalive Web % curl -H Host: example.com.ccnproxy-https http://10.0.0.1/v1/foo https://example.com/ proxy DNS URL *.ccnproxy-https IN CNAME chocon.local. % curl http://example.com.ccnproxy-https/v1/foo
Before chocon $./httpstat.sh /dev/null https://microservice.example.com/hc HTTP/1.1 200 OK Server: nginx/1.11.5 Date: Thu, 01 Jun 2017 00:43:49 GMT Content-Type: application/json; charset=utf-8 Content-Length: 22 Expires: Thu, 01 Jun 2017 01:43:49 GMT Cache-Control: max-age=3600,public Body stored in: /tmp/httpstat-body.263264511496278239 DNS Lookup TCP Connection SSL Handshake Server Processing Content Transfer [ 2ms 24ms 197ms 25ms 0ms ] namelookup:2ms connect:26ms pretransfer:223ms starttransfer:248ms total:248ms
After chocon $./httpstat.sh /dev/null https://microservice.example.com.ccnproxy-https/hc HTTP/1.1 200 OK Cache-Control: max-age=3600,public Content-Length: 22 Content-Type: application/json; charset=utf-8 Date: Thu, 01 Jun 2017 00:43:49 GMT Expires: Thu, 01 Jun 2017 01:43:49 GMT Server: nginx/1.11.5 X-Chocon-Req: bsczjrcmz9wbrn8tyhz3wv Body stored in: /tmp/httpstat-body.390174181496278775 DNS Lookup TCP Connection Server Processing Content Transfer [ 1ms 1ms 19ms 0ms ] namelookup:1ms connect:2ms starttransfer:21ms total:21ms
Why chocon? middleware forward proxy HTTPS HTTPS end to end MITM Proxy Go HTTP/2
chocon in JP App App http or https App App keepalive Microservices chocon SaaS App App Cloud API endpoint App App DC(Cloud) Cloud(DC) 90msec 19msec RTT AWS SDK endpoint
chocon & Pacific Ocean App App App App chocon HTTPS, HTTP/2 Keepalive US Cloud Cloudfront/CDN 100msec Region US
JP/US/UK 3 Region Amazon Route53, Amazon S3
We re Hiring! SRE www.mercari.com/jp/jobs/