Elasticsearch graph visualization tutorial

Elasticsearch is part of the open source portfolio known as the Elastic Stack. It’s a fast and scalable search engine built on the Apache Lucene software library. Lucene is a high-performance technology for searching and indexing data, but it can be very complex to use. Elasticsearch makes the power of Lucene more accessible by pre-selecting sensible defaults and providing a more intuitive REST API.

The power and simplicity of Elasticsearch makes it popular with organizations that need to search very large volumes of data, including Facebook, Wikimedia and Stack Exchange. It supports near real-time data searching on a petabyte scale, using a system of sharding and routing to scale outwards from the beginning.

Visualizing graphs in your Elasticsearch index with KeyLines is a great way to make complex connections easy to understand. A well-designed visualization means users can find and interpret patterns and outliers, explore connections in an intuitive way and answer questions more quickly using data insight.

In this tutorial, we walk through the steps of connecting KeyLines to Elasticsearch. This guide assumes a basic understanding of JavaScript and explains how to load data into Elasticsearch with Kibana, before using this store as the backend for your KeyLines graph visualization application. We’ll be using two technologies from the Elastic Stack:

• Elasticsearch – the core search technology
• Kibana – a tool to visualize your Elasticsearch data and navigate the Elastic Stack

Visualization architecture

KeyLines is a JavaScript file that’s deployed to a web server. A chart element (powered by either HTML5 Canvas or WebGL) is then embedded into a web page for the user to access. As the user interacts with the chart, events – like click, double-click, drag – are raised. You can then customize the response to these events, creating a custom user experience.

KeyLines’ integration architecture is very simple. Elasticsearch provides a REST API and works with the JSON data structure:

In this scenario, KeyLines runs in the web browser. Every time a user clicks, hovers or interacts with the graph data on a chart, it raises an event. When the user requests data, the browser application sends an AJAX request to the Elasticsearch REST API. Elasticsearch returns the data as a JSON object that KeyLines styles and displays on the chart.

Step 1: Installation

This tutorial works with KeyLines 5.10 and earlier. If you’re using a later version, you can access a Long Term Availability (LTA) release from the SDK’s Download page.

In the KeyLines SDK, find the Integrate with Elastic demo and download the source files. They’ll be extracted to a folder called elasticsearch.

Next, download the latest versions of Elasticsearch and Kibana:

• Elasticsearch


• Kibana

Unzip these two folders and store them locally.

Step 2: Run Elasticsearch & Kibana

In your terminal, navigate to the parent folder of the elasticsearch download (for example, `cd ~/Downloads/elasticsearch-7.13.2/`) and run:

Windows: `bin\elasticsearch.bat`

Linux/Mac: `bin/elasticsearch`

Once the process is complete, curl or browse to localhost:9200. You should see information about the Elasticsearch instance you just started:

{
  "name" : "My-MacBook-Pro.local",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "mldTY1-GTMag6QyUmuoVsQ",
  "version" : {
    "number" : "7.13.2",
    "build_flavor" : "default",
    "build_type"   : "tar",
    "build_hash"   : "4d960a0733be83dd2543ca018aa4ddc42e956800",  
    "build_date"   : "2021-08-10T21:01:55.251515791Z",
    "build_snapshot" : false,
    "lucene_version" : "8.8.2",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

Now, navigate to the parent folder of the Kibana download (for example, `cd ~/Downloads/elasticsearch-7.13.2/`) and execute the following:

Windows: `bin\kibana.bat`

Linux/Mac: `bin/kibana`

Step 3: Import the dataset

Once the terminal processes are complete, navigate to localhost:5601 to see the Kibana dashboard.

The files we downloaded from KeyLines in step 1 included a pre-prepared subset of the data saved in JSON format. We’ll use this to populate the database for the demo.

To do this, once Kibana is running, search for ‘Machine Learning’ and select Import data.

Select the movies.json and actors.json files downloaded from KeyLines.

Once the files are uploaded, at the bottom left of the page, click Import. For simplicity, we’ll set the index name for the movies.json file to ‘movies’ and the actors.json file to ‘actors’.

Step 4: Test with a query

To navigate to the Kibana console, search for ‘Console’ (Dev tools/Console).

Now we can run a query to check that the data is populated:

`GET /movies/_search?q=RoboCop`

At the top right hand corner of the console, click Play. You should see data retrieved from Elasticsearch in the right-hand pane.

Step 5: Install demo dependencies

This demo uses Express server and the official Elasticsearch driver for node.js.

To install them, from inside the elasticsearch folder downloaded from KeyLines, run `yarn install` or `npm install`.

Step 6: Run the web server

To run the standalone web server downloaded from KeyLines, run `yarn run start` or `npm run start`.

Browse to http://localhost:8080 to open the demo application and start exploring the data: