IIJ Technical WEEK Cloudbusting Machine（CBM）

Size: px

Start display at page:

Download "IIJ Technical WEEK Cloudbusting Machine（CBM）"

はすなきちや
6 years ago
Views:

1 Cloudbusting Machine CBM IIJ

2 Project Gryfon PaaS

3 Cloudbusting Machine CBM Project Gryfon PaaS Key-Value Store KVS C GQL PHP-C MySQL Cassandra MongoDB zipbench

4 Key-Value Store API

5 Key-Value Store KVS Key-Value Store NoSQL Key-Value Store Key-Value Store Google BigTable Amazon Dynamo

6 Key-Value Store RDB SQL vs NoSQL RDB KVS vs RDB KVS ACID vs Eventual consistency CAP RDB ACID ACID AtomicityConsistencyIsolationDurability KVS Eventual consistency ConsistencyAvailabilityPartitionTolerance RDB C+A KVS C+P A+P

7 Key-Value Store KVS C+P BigTableHBaseHypertableMongoDB P2PA+P DynamoVoldemortCassandra Yahoo! Cloud Serving Benchmark CouchDBMongoDB BigTableHbaseHypertableCassandra Voldemort

8 Key-Value Store SQL SQL SQL MongoDB SQL RDBNoSQLRDBNoSQL

9 CQI KVS CBM KVS SQLKVS PHP KVS/RDB KVSRDB KVS RDB KVS KVS

10 Google AppEngine appserver XMPP URL

11 Google AppEngineGAE GAE GAE SQL API Python: Java: JDO/JPA Go: CQI C++/PHP GAE GAE/P Google AppEngine for Python SDK GQE TyphoonAE GAE SDK GAE/P

12 GAE/P Python Execution Environment Application Backend Servers Indexing Megastore Model Interface Google RPC Transactio n Datastore API BigTable Table GFS Store

13 GAE/P Datastore API Datastore API: Google AppEngine db Datastore API GQLdb GAE/P SDK Python google/appengine/api google/appengine/datastore api/datastore.py API datastore/datastore_query.py datastore/datastore_rpc.py api/datastore_types.py datastore/datastore_index.py datastore/entity_pbpy

14 GAE/P Datastore API api/datastore.py Query Class Run Get Count Get Put Delete datastore/datastore_query.py Query Class YAML datastore/datastore_index.py make_rpc_call datastore/datastore_rpc.py Get Put Delete

15 GAE/P class MainPage(webapp.RequestHandler): def get(self): self.response.out.write('<html><body>') greetings = db.gqlquery("select * FROM Greeting ORDER BY date DESC LIMIT 10") for greeting in greetings: if greeting.author: self.response.out.write('<b>%s</b> wrote:' % greeting.author.nickname()) else: self.response.out.write('an anonymous person wrote:') self.response.out.write('<blockquote>%s</blockquote>' % cgi.escape(greeting.content)) # Write the submission form and the footer of the page self.response.out.write(""" <form action="/sign" method="post"> <div><textarea name="content" rows="3" cols="60"></textarea></div> <div><input type="submit" value="sign Guestbook"></div> </form> </body> </html>""")

16 PHP Execute Environment Storage A CQI API Storage A Driver User Defined Object Datastore Bridge Storage B User Defined Object Query Engine Storage B Driver User Defined Object Storage C GAE DataStore API Storage C Driver

17 L PHP Application Java Application YYY Application ZZZ Application PHP Interface Java Interface YYY Interface ZZZ Interface PHP-C Binding Java-C Binding YY-C Binding ZZZ-C Binding Query Engine QE GQL Parser DataStore Bridge S KVS Driver MongoDB KVS Driver XXX KVS Driver Cassandra KVS Driver YYY MongoDB XXX Cassandra YYY

18 Query Engine GQL Datastore Bridge API Storage Driver GAE/P Datastore API KVS Query Engine GQLSELECT SELECTConditionOrdering ANCESTOR Datastore Bridge Storage Driver Storage Driver KVS Cassandra/MongoDB Hypertable C

19 API CAPI ${PREFIX}/include/cqi.h CQI CQI_t *cqi_new(char *driver, char *user, char *passwd, char *host, int port); void cqi_free(cqi_t *cqi); GQL query_t *cqi_analyze(cqi_t *cqi, char *gqlstr); int cqi_attach(cqi_t *cqi, char *aux); int cqi_detach(cqi_t *cqi); int cqi_put(cqi_t *cqi, entity_t *ep); int cqi_delete(cqi_t *cqi, entity_t *ep); entity_t *cqi_fetch(cqi_t *cqi, query_t *qp, int *count); int cqi_count(cqi_t *cqi, query_t *qp);

20 C int main(int argc, char *argv[]) { CQI_t *cqi; query_t *qp; entity_t *ep; char *gqlstr = "SELECT * FROM xzip"; int i, n; cqi = cqi_new( Cassandra", // driver "root", // user "root", // passwd " ", // host 9160); // port cqi_attach(cqi, "xzipcode.xzip"); // connect datastore qp = cqi_analyze(cqi, gqlstr); cqi_query_show(qp); n = cqi_count(cqi, qp); printf("\n# %d entries\n", n); if (n > 0) { ep = cqi_fetch(cqi, qp, &n); if (ep!= NULL) { for (i = 0; i < n; i++) { printf("\n# %d\n", i); cqi_entity_show(&ep[i]); } } cqi_entity_free(ep); } } cqi_query_free(qp); cqi_detach(cqi); cqi_free(cqi); exit(0); // disconnect datastore

21 PHP php if(!extension_loaded('cqi_php')) { dl('cqi_php.'. PHP_SHLIB_SUFFIX); } $query_string = 'SELECT * FROM xzip'; $queryarray = array(); $gql = array('main_sentence' => $query_string, 'param_count' => count($queryarray), 'param' => $queryarray); $result = cqi_analyze($gql); $qarray = $result['query']; $query = $qarray[0]; $driver = array('driver' => 'cassandra', 'user' => 'root', 'passwd' => 'root', 'host' => ' ', 'port' => 9160, 'aux' => 'xzipcode.xzip'); $result = cqi_count($driver, $query); $result = cqi_fetch($driver, $query);?>

22 ORM KVSMongoDB RDB KVS KVSCassandra KVS KVS RDB KVS KVS

23 KVS KVS SQL CQL: Cassandra HQL: Hypertable SQL RDB SQL KVS Oracle Cassandra Netfilix: Replacing Datacenter Oracle with global apache cassandra on AWS

24 KVS DataStax Cassandra Oracle NoSQL KVS CQI SQLKVS PHP PDOPHP Data Object

25 PHP Data Object PDO PHPRDB MySQLSQLite CPEAR:DB PDO Core PDO Driver CQIPDO PHPPHP PDO PDOStatement

26 PDOCQI PDOSQL CQI SQL SQL KVS KVS SQL KVSCQI API PDOAPICQIAPI

27 PDOCQI CQI PDOCQI CBM 0.2 L PHP Application PDO Application PDO Application PHP Interface PHP-C Binding Query Engine PDO Core QE GQL Parser PDO CQI Driver SQL Parser PDO Cassandra Driver SQL Parser PDO MongoDB Driver SQL Parser S KVS Driver MongoDB DataStore Bridge KVS Driver Cassandra KVS Driver Cassandra KVS Driver MongoDB MongoDB Cassandra Cassandra MongoDB

28 PDOCQI Cassadra MongoDB PDO Driver PDO Core Cassandra/MongoDB PDO CQI Driver SQL Parser SQL SQLite SQLtokenize/parse CREATE TABLE/INSERT/SELECT/DELETE/DROP TABLE KVS KVS Driver KVS KVS KVS Driver PDO

29 PDOCQICassandra Driver Apache Cassandra 1.0 Thrift Driver SQL CQLCassandra Query Language Secondary Index WHERE CREATE TABLE INSERT varchar NOT NULL/PRIMARY KEY UNIQUE KEY Cassandra PDO

30 PDOCQICassandra Driver DELETE WHERE DROP TABLE SELECT WHERE ORDER BY GROUP BY HAVING JOIN KEYKEY *KEY

31 PDOCQIMongoDB Driver MongoDB 2.0 mongo-c-driver SQL Mongo QueryBSON CREATE TABLE INSERT varchar NOT NULL/PRIMARY KEYUNIQUE MongoDBid mongo-c-driver API PDO

32 PDOCQIMongoDB Driver DELETE/DROP TABLE SELECT WHERE ORDER BY GROUP BY HAVING JOIN SELECT

33 Cloudbusting MachineCBM0.2 PDOCQI Cassandra/ MongoDB CQI CQI Hypertable SQL ORDER BY Zend Engine JOINSQL KVS PHP KVSPaxos

34 In Progress KVS BigTable Store SQL SQL KVSSQL SQLQL SQL KVSRDB KVS RDB KVS KVS

…l…b…g…‘†[…N…v…“…O…›…~…ﬁ…OﬁÁŸ_

…l…b…g…‘†[…N…v…“…O…›…~…ﬁ…OﬁÁŸ_ 13 : Web : RDB (MySQL ) DB (memcached ) 1: MySQL ( ) 2: : /, 3: : Google, 1 / 23 testmysql.rb: mysql ruby testmem.rb: memcached ruby 2 / 23 ? Web / 3 ( ) Web s ( ) MySQL PostgreSQL SQLite MariaDB (MySQL