lucene-gosen Solr1

ElasticSearch 2013/08/29 @johtani

Twitter@johtani lucene-gosen Solr1 http://blog.johtani.info

N-gram ElasticSearch

ElasticSearch

Full text search Wikipedia

RDB RDB RDB Term/Token

1 2

1 2 1 2

1 2 1 2 1 1 2 1 id 1 2 1 2 2 2

1 1 2 1 1 2 1 2 2 2

AND 1 1 2 1 1 2 1 2 2 2

AND 1 1 2 1 1 2 1 2 2 2

AND 1 1 2 1 ANDid 1 2 1 2 2 AND 2

AND 1 1 2 1 ANDid 1 2 1 2 2 AND 2

N-gram 2 N-gram N1

N-gram N2

ElasticSearch

ElasticSearch Amazon Web Services

ElasticSearch RESTful OSSApache Software License Version 2.0 Apache Lucene0.90.3Lucene 4.4.0

ElasticSearch

インデックス構成 論理 インデックス/タイプ マッピングタイプ /ドキュメント という構成 Solrにはタイプという概念がない doc1 doc3 doc2 type novel type magazine index books index dvds ElasticSearch 13年8月30日金曜日

RDB RDB ES RDB

ElasticSearchURL curl -XPUT http://localhost:9200/books/novel/doc1 -d {...} curl -XGET http://localhost:9200/books/_search?q=... curl -XGET http://localhost:9200/books/novel/_search?q=... ID curl -XGET http://localhost:9200/books/novel/doc1

OK curl -XGET http://localhost:9200/books,dvds/_search?q=... curl -XGET http://localhost:9200/_search?q=... curl -XGET http://localhost:9200/_all/magazine/_search?q=...

インデックス構成 物理 クラスタとノード インデックスとシャード shard の関係 books primary shard1 books replica shard1 dvds replica shard1 dvds primary shard1 dvds primary shard2 dvds replica shard2 books primary shard2 books replica shard2 ノード ElasticSearchプロセス１ ノード ElasticSearchプロセス２ クラスタ elasticsearch 13年8月30日金曜日

ncluster.name n 51/1 Solrshard splitting2013/04

Zen Discovery or http://www.elasticsearch.org/guide/reference/modules/discovery/zen/ ZooKeeper

TTL Geo Percorator

REST API REST API curl -XGET 'http://localhost:9200/_cluster/health?pretty=true' curl -XPUT 'http://localhost:9200/twitter/' curl -XDELETE 'http://localhost:9200/twitter/'

JSON numeric_detectiontrue/false dynamic_date_formatsjoda-time

24long hoge string ignore_malformedtrue 24long12.3float 12

Solr mappingsdynamic_templates { "person" : { "dynamic_templates" : [ { "template_1" : { "match" : "multi*", "mapping" : { "type" : "multi_field", "fields" : { "{name}" : {"type": "{dynamic_type}", "index" : "analyzed"}, "org" : {"type": "{dynamic_type}", "index" : "not_analyzed"} } } } },...

templateorder 0ha_5 curl -XPUT localhost:9200/_template/main_template -d ' { "template" : "*", "order" : "1", "settings" : { "index.number_of_replicas" : 0 } curl -XPUT localhost:9200/_template/ha_template -d ' } { "template" : "ha_*", "order" : "10", "settings" : { "index.number_of_replicas" : 5 } }

Solr T { "variation" : [ { { "query" : { "nested" : { "color" : "blue", "size" : 4 "path" : "variation", "query" : { }, { "bool" : { "must" : [ "name" : "green", "size" : 6 { "term" : {"variation.name" : "green"} }, { "term" : {"variation.size" : "6"} } ] } } ] } } } } }

mapping ID $ curl -XPUT localhost:9200/shop/cloth/1 -d '{ "name" : "something" }' $ curl -XPUT localhost:9200/shop/variation/1?parent=1 -d '{ "color" : "red", "size" : "XL" }' $ curl -XPUT localhost:9200/shop/variation/2?parent=1 -d '{ "color" : "green", "size" : "S" }' { } "has_child" : { "type" : "variation", "query" : { "bool" : { "must" : [ { "term": {"size" : "XL"} }, { "term": {"color" : "red"} } ] } } }

_sourcejson Solr _source curl -XPOST 'localhost:9200/test/type1/1/_update' -d '{ "script" : "ctx._source.counter += count", "params" : { "count" : 4 } }'

TTL TTLTime To Live 1 Expire60s mapping { } "tweet" : { } "_ttl" : { "enabled" : true, "default" : "1d" }

index slow log/search slow logsolr #index.search.slowlog.threshold.query.warn: 10s #index.search.slowlog.threshold.query.info: 5s #index.search.slowlog.threshold.query.debug: 2s #index.search.slowlog.threshold.query.trace: 500ms #index.search.slowlog.threshold.fetch.warn: 1s #index.search.slowlog.threshold.fetch.info: 800ms #index.search.slowlog.threshold.fetch.debug: 500ms #index.search.slowlog.threshold.fetch.trace: 200ms

ElasticSearch RiverElasticSearchRabbitMQ AnalysisKuromoji Site plugin http://www.elasticsearch.org/guide/reference/modules/ plugins/

Percorator percorator curl -XPUT localhost:9200/_percolator/test/kuku -d '{ "query" : { "term" : { "field1" : "value1" } } } percorator curl -XGET localhost:9200/test/type1/_percolate -d '{ "doc" : { "field1" : "value1" } }' And the matches are part of the response: {"ok":true, "matches":["kuku"]}

Geo geohashgeo_shape geohash

API_bulk http://www.elasticsearch.org/guide/reference/api/bulk/ UDP { "index" : { "_index" : "test", "_type" : "type1", "_id" : "1" } } { "field1" : "value1" } { "delete" : { "_index" : "test", "_type" : "type1", "_id" : "2" } } { "create" : { "_index" : "test", "_type" : "type1", "_id" : "3" } } { "field1" : "value3" } { "update" : {"_id" : "1", "_type" : "type1", "_index" : "index1"} } { "doc" : {"field2" : "value2"} }

Kibana3ElasticSearch(logstash OR fluentd) ElasticSearch

http://www.elasticsearch.org http://www.elasticsearch.org/guide/ http://elasticsearch.com Solr vs ElasticSearch https://github.com/minghai/elasticsearch_vs_solr Sematext http://blog.sematext.com

ElasticSearch ServerPackt Publishing2013/02 http://elasticsearchserverbook.com Mastering ElasticSearchPackt Publishing2013/12 http://www.packtpub.com/elasticsearch-server-for-fast-scalable-flexiblesearch-solution/book http://www.packtpub.com/mastering-elasticsearch-querying-and-datahandling/book Elasticsearch in ActionManning Publications2014/MEAP http://www.manning.com/hinman/

LuceneES ElasticSearch Lucene 0.90.3 4.4 0.90.2 4.3.1 0.90.1 4.3 0.90.0 4.2.1 0.90.0rc2 4.2.1 0.90.0rc1 4.2 0.20.6 3.6.2 0.20.5 3.6.2 0.90.0 beta1 4.1