Table of Contents
Elasticsearch is a distributed, RESTful search and analytics engine capable of solving a growing number of use cases. As the heart of the Elastic Stack, it centrally stores your data so you can discover the expected and uncover the unexpected. | ||
| -- Elasticsearch Overview | ||
JanusGraph supports Elasticsearch as an index backend. Here are some of the Elasticsearch features supported by JanusGraph:
- Full-Text: Supports all
Textpredicates to search for text properties that matches a given word, prefix or regular expression. - Geo: Supports all
Geopredicates to search for geo properties that are intersecting, within, disjoint to or contained in a given query geometry. Supports points, circles, boxes, lines and polygons for indexing. Supports circles, boxes and polygons for querying point properties and all shapes for querying non-point properties. Note that JTS is required when using line and polygon shapes (see Geoshape documentation for more information). - Numeric Range: Supports all numeric comparisons in
Compare. - Flexible Configuration: Supports remote operation and open-ended settings customization.
- TTL: Supports automatically expiring indexed elements.
- Collections: Supports indexing SET and LIST cardinality properties.
- Temporal: Nanosecond granularity temporal indexing.
- Custom Analyzer: Choose to use a custom analyzer
Please see Appendix B, Version Compatibility for details on what versions of ES will work with JanusGraph.
![[Important]](images/icons/important.png){kind=icon} | Important |
|---|---|
Beginning with Elasticsearch 5.0 JanusGraph uses sandboxed Painless scripts for inline updates, which are enabled by default in Elasticsearch 5.x. Using JanusGraph with Elasticsearch 2.x requires enabling Groovy inline scripting by setting |
JanusGraph supports connections to a running Elasticsearch cluster. JanusGraph provides two options for running local Elasticsearch instances for getting started quickly. JanusGraph server (see Section 7.1, “Getting Started”) automatically starts a local Elasticsearch instance. Alternatively JanusGraph releases include a full Elasticsearch distribution to allow users to manually start a local Elasticsearch instance (see this page for more information).
$ elasticsearch/bin/elasticsearch
![[Note]](images/icons/note.png){kind=icon} | Note |
|---|---|
For security reasons Elasticsearch must be run under a non-root account |
JanusGraph supports HTTP client connections to a running Elasticsearch cluster. Please see Appendix B, Version Compatibility for details on what versions of ES will work with the different client types in JanusGraph.
| Note |
|---|---|
JanusGraph’s index options start with the string " |
![[Tip]](images/icons/tip.png){kind=icon} | Tip |
|---|---|
It’s recommended that index names contain only alphanumeric lowercase characters and hyphens, and that they start with a lowercase letter. |
The Elasticsearch client is specified as follows:
index.search.backend=elasticsearchWhen connecting to Elasticsearch a single or list of hostnames for the Elasticsearch instances must be provided. These are supplied via JanusGraph’s index.[X].hostname key.
index.search.backend=elasticsearch index.search.hostname=10.0.0.10:9200
Each host or host:port pair specified here will be added to the HTTP client’s round-robin list of request targets. Here’s a minimal configuration that will round-robin over 10.0.0.10 on the default Elasticsearch HTTP port (9200) and 10.0.0.20 on port 7777:
index.search.backend=elasticsearch index.search.hostname=10.0.0.10, 10.0.0.20:7777
JanusGraph only uses default values for index-name and health-request-timeout. See Chapter 13, Configuration Reference for descriptions of these options and their accepted values.
index.[X].elasticsearch.index-nameindex.[X].elasticsearch.health-request-timeout
The REST client accepts the index.[X].bulk-refresh option. This option controls when changes are made visible to search. See ?refresh documentation for more information.
If using Elasticsearch 5.0 or higher, a different ingest pipelines can be set for each mixed index. Ingest pipeleine can be use to pre-process documents before indexing. A pipeline is composed by a series of processors. Each processor transforms the document in some way. For example date processor can extract a date from a text to a date field. So you can query this date with JanusGraph without it being physically in the primary storage.
index.[X].elasticsearch.ingest-pipeline.[mixedIndexName] = pipeline_id
See ingest documentation for more information about ingest pipelines and processors documentation for more information about ingest processors.
Elasticsearch does not perform authentication or authorization. A client that can connect to ES is trusted by ES. When Elasticsearch runs on an unsecured or public network, particularly the Internet, it should be deployed with some type of external security. This is generally done with a combination of firewalling and tunneling of Elasticsearch’s ports. Elasticsearch has two client-facing ports to consider:
- The HTTP REST API, usually on port 9200
- The native "transport" protocol, usually on port 9300
A client uses either one protocol/port or the other, but not both simultaneously. Securing the HTTP protocol port is generally done with a combination of firewalling and a reverse proxy with SSL encryption and HTTP authentication. There are a couple of ways to approach security on the native "transport" protocol port:
- Tunnel ES’s native "transport" protocol
- This approach can be implemented with SSL/TLS tunneling (for instance via stunnel), a VPN, or SSH port forwarding. SSL/TLS tunnels require non-trivial setup and monitoring: one or both ends of the tunnel need a certificate, and the stunnel processes need to be configured and running continuously. The setup for most secure VPNs is likewise non-trivial. Some Elasticsearch service providers handle server-side tunnel management and provide a custom Elasticsearch
transport.typeto simplify the client setup. - Add a firewall rule that allows only trusted clients to connect on Elasticsearch’s native protocol port
- This is typically done at the host firewall level. Easy to configure, but very weak security by itself.
JanusGraph supports customization of the index settings it uses when creating its Elasticsearch index. It allows setting arbitrary key-value pairs on the settings object in the Elasticsearch create index request issued by JanusGraph. Here is a non-exhaustive sample of Elasticsearch index settings that can be customized using this mechanism:
index.number_of_replicasindex.number_of_shardsindex.refresh_interval
Settings customized through this mechanism are only applied when JanusGraph attempts to create its index in Elasticsearch. If JanusGraph finds that its index already exists, then it does not attempt to recreate it, and these settings have no effect.
JanusGraph iterates over all properties prefixed with index.[X].elasticsearch.create.ext., where [X] is an index name such as search. It strips the prefix from each property key. The remainder of the stripped key will be interpreted as an Elasticsearch index creation setting. The value associated with the key is not modified. The stripped key and unmodified value are passed as part of the settings object in the Elasticsearch create index request that JanusGraph issues when bootstrapping on ES. This allows embedding arbitrary index creation settings settings in JanusGraph’s properties. Here’s an example configuration fragment that customizes three Elasticsearch index settings using the create.ext config mechanism:
index.search.backend=elasticsearch index.search.elasticsearch.create.ext.number_of_shards=15 index.search.elasticsearch.create.ext.number_of_replicas=3 index.search.elasticsearch.create.ext.shard.check_on_startup=true
The configuration fragment listed above takes advantage of Elasticsearch’s assumption, implemented server-side, that unqualified create index setting keys have an index. prefix. It’s also possible to spell out the index prefix explicitly. Here’s a JanusGraph config file functionally equivalent to the one listed above, except that the index. prefix before the index creation settings is explicit:
index.search.backend=elasticsearch index.search.elasticsearch.create.ext.index.number_of_shards=15 index.search.elasticsearch.create.ext.index.number_of_replicas=3 index.search.elasticsearch.create.ext.index.shard.check_on_startup=false
| Tip |
|---|---|
The |
Check that the Elasticsearch cluster nodes are reachable on the HTTP protocol port from the JanusGraph nodes. Check the node listen port by examining the Elasticsearch node configuration logs or using a general diagnostic utility like netstat. Check the JanusGraph configuration.
For bulk loading or other write-intense applications, consider increasing Elasticsearch’s refresh interval. Refer to this discussion on how to increase the refresh interval and its impact on write performance. Note, that a higher refresh interval means that it takes a longer time for graph mutations to be available in the index.
For additional suggestions on how to increase write performance in Elasticsearch with detailed instructions, please read this blog post.
- Please refer to the Elasticsearch homepage and available documentation for more information on Elasticsearch and how to setup an Elasticsearch cluster.