Start a KeyLines trial

To follow this tutorial, you’ll need a KeyLines evaluation account.

Request a KeyLines trial

Elasticsearch is part of the open source portfolio known as the Elastic Stack. It’s a fast and scalable search engine built on the Apache Lucene software library. Lucene is a high-performance technology for searching and indexing data, but it can be very complex to use. Elasticsearch makes the power of Lucene more accessible by pre-selecting sensible defaults and providing a more intuitive REST API.

The power and simplicity of Elasticsearch makes it popular with organizations that need to search very large volumes of data, including Facebook, Wikimedia and Stack Exchange. It supports near real-time data searching on a petabyte scale, using a system of sharding and routing to scale outwards from the beginning.

In this tutorial, we walk through the steps of connecting KeyLines to Elasticsearch. This guide assumes a basic understanding of Javascript and explains how to use Elasticsearch as the backend for your KeyLines graph visualization application. We’ll use two technologies from the Elastic Stack:

  • Elasticsearch – the core search technology
  • Logstash – a tool for streaming, munging and loading data into Elasticsearch

Visualization architecture

KeyLines is a JavaScript file that’s deployed to a web server. A chart element (powered by either HTML5 Canvas or WebGL) is then embedded into a web page for the user to access. As the user interacts with the chart, events – like click, double-click, drag – are raised. You can then customize the response to these events, creating a custom user experience.

KeyLines’ integration architecture is very simple. Elasticsearch provides a REST API and works with the JSON data structure:

The KeyLines - Elasticsearch visualization architecture
The KeyLines – Elasticsearch visualization architecture

In this scenario, KeyLines runs in the web browser. Every time a user clicks, hovers or interacts with the graph data on a chart, it raises an event. When the user requests data, the browser application sends an AJAX request to the Elasticsearch REST API. Elasticsearch returns the data as a JSON object that KeyLines styles and displays on the chart.

Step 1: Download prerequisites

To build a KeyLines-Elasticsearch integration, download the following:

Step 2: Set up your file structure

You’ll find the files you need in the Elasticsearch.zip file. For the KeyLines/Elasticsearch JavaScript app, use the following structure:

  • Database.js contains the main functions to initialize KeyLines and to add visual interactions.
  • Elasticsearch.js contains the functions required to send queries to, and interact with, the server.
  • Keylines.js contains components to build the visualization part of our application. Make sure to include the assets folder from the KeyLines download.
  • Index.htm contains the KeyLines chart and some customization code to describe the general UI.

Step 3: Load data into Elasticsearch

If your Elasticsearch instance is populated already, you should skip this step.

For this example, we used a dataset of movies and actors taken from IMDB.com and saved as JSON files.

  1. Denormalize the data to make graph queries faster and more efficient. We’ve done this for you already. The denormalized data is in the JSON files you downloaded with the Elasticsearch.zip file. For more information, see Denormalizing Your Data.
  2. Load the data into your Elasticsearch instance.
  3. Write configuration files to run Logstash and import our dataset. They look like this:
  4. input {  
      file {
        type => "movies"
        path => "/path/to/json/movies.json"
        start_position => "beginning"
        ignore_older => 0
      }
      file {
        type => "actors"
        path => "/path/to/json/actors.json"
        start_position => "beginning"
        ignore_older => 0
      }
    }
    filter {
      json {
        source => "message"
      }
    }
    output {
      if [type] == "movies" {
        elasticsearch {
            action => "index"
            hosts => ["localhost:9200"]
            index => "movies"
            document_type => "movie"
        }
      }
      if [type] == "actors" {
        elasticsearch {
            action => "index"
            hosts => ["localhost:9200"]
            index => "actors"
            document_type => "actor"
        }
      }
      stdout {}
    }
    
  5. To import your data into Elasticsearch with Logstash, run the following command:
    path/to/logstash/bin/logstash -v -f logstash-json.conf

    To query Elasticsearch directly from the web browser, edit the elasticsearch.yaml config file you downloaded when you installed Elasticsearch. Add the following:

        http.cors.enabled: true
        http.cors.allow-origin: "*"

    Note: For security reasons, we recommend that you don’t configure Elasticsearch in this way in a production environment.

  6. Restart Elasticsearch.

Step 4: Embed KeyLines in your webpage

It’s easy to create a KeyLines chart. First, make sure to import the main keylines.js somewhere in the <head> of index.html:

<script type="text/javascript" src="js/keylines.js"/>

Then, once the page is created, call KeyLines.create() to initialize a chart instance. This code assumes that there’s an element with id ‘kl’ somewhere in the page:

var options = {
  // Configure your chart here - see the API reference for details
};
KeyLines.create({ id: 'kl', options: options }, function(chart) {
    // chart is an instance of a KeyLines chart
});

It won’t look too exciting until you’ve loaded some data, but you should now have a live chart:

KeyLines chart
KeyLines chart

Now you can add styling and interaction controls that suit the way you want to work with charts. For inspiration about what functionality is available, browse the Demos page of the SDK site.

Step 5: Fetch data from Elasticsearch

Here’s an example of how to find movies with “The Matrix” in the ‘title’ field.
If your Elasticsearch server is running locally, you can use URL to make a HTTP request, like this:

curl 'localhost:9200/movies/movie/_search' -d ' 
{
  "query": {
    "query_string": {
      "query":"The Matrix",
      "fields": ["title"]
    }
  }
}'

In your HTML page, you can make an Ajax request for the same data like this:

var data = {
  query: {
    query_string: {
      query: 'The Matrix',
      fields: ['title']
     }
  }
};
fetch('http://localhost:9200/movies/movie/_search', {
   method: 'POST',
   body: data
})
.then(function (res) {
  res.json().then(function(json) {
    // json contains query results in elastic search format
  });
});

In response to this query, you’ll receive a JSON object describing The Matrix movie. Now we need to parse this into a format KeyLines can use.

Step 6: Parse our result in KeyLines format

The Elasticsearch response contains the information we need to create the results on a KeyLines chart, so parsing your JSON is a relatively simple process.

Here’s the structure of the search results:

{
   _shards:{},
   hits:{
       hits:[],
       max_score:1,
       total:4
   },
   timed_out:false,
   Took:3
}

The search results are in hits.hits[]. Each element looks like this:

{
  _id: "AVVOMTZEtt2NBAP44u0z",
  _index: "movies",
  _score: 1,
  _source: {},
  _type: "movie"
}

The data we want is listed under the source property, and looks something like this:

{
  "id": "1",
  "title": "The Matrix",
  "tagline": "Welcome to the Real World.",
  "summary": "Thomas A. Anderson is a man living two lives...",
  "image": "http://image.tmdb.org/t/p/w185/gynBNzwyaHKtXqlEKKLioNkjKgN.jpg",
  "duration": "136",
  "rated": "R",
  "actors": [{
    "name": "Laurence Fishburne",
    "role": "Morpheus"
   } /* .... */ ]
}

You’ll need to convert this data into nodes and links before we can load it into KeyLines and visualize it. Add this transformer function to the page:

function makeKeyLinesItems(json) {
  var items = [];
  if (json && json.hits) {
    json.hits.hits.forEach(function (hitItem) {
      // get the type first
      var searchType = hitItem._type;
      // get the source of the object
      var item = hitItem._source;
      var searchNode = seachType === 'movie' ? item.title : item.name;
      items.push(makeNode(searchNode, searchType));
      var linkType = searchType === 'movie' ? 'actor' : 'movie'
      var links = item[linkType + 's'];
      items.forEach(function (link) {
        var linkedNode = link.name;
        items.push(makeNode(linkedNode, linkType));
        var actor = searchType === 'actor' ? searchNode : linkedNode;
        var movie = searchType === 'movie' ? searchNode : linkedNode;
        items.push(makeLink(actor, movie, link.role));
      });
    });
  }
  return items;
}

The makeLink and makeNode functions are reproduced below. You may want to customize them with your own style information (see the API reference in the SDK site for more details):

function makeNode(name, type) {
  return {
    id: name,
    type: 'node',
    t: name,
    d: { type: type }
  };
}
function makeLink(actor, movie) {
  return {
    id: actor + '-' + movie,
    type: 'link',
    id1: actor,
    id2: movie
  };
}

Finally, parse in the JSON data from the fetch callback to our transformer:

res.json().then(function(json) {
   var items = makeKeyLinesItems(JSON);
   // We’ll pass items into the chart later
});

Step 7: Visualize the data in KeyLines

Once we have KeyLines-formatted JSON, we can load it into the chart and run a standard layout to position nodes nicely:

chart.load({ type: 'LinkChart', items: items }, function () {
    chart.layout('standard');   
});
The KeyLines - Elasticsearch demo
The KeyLines SDK site includes a KeyLines-Elasticsearch sample demo, with downloadable source code. Request a KeyLines trial account to get started.

Step 8: Run more sophisticated searches

This simple example is intended to help get you started. Now that your infrastructure is working, you can perform more sophisticated searches.

For example, to search multiple fields and not just the title field, in Step 5, change the query to:

{
   "query": {
    "query_string": {
      "query":"The Matrix",
      "fields": ["title", "name"]
    }
  }
}

To widen your search even further, use wildcards. To search the database for words that start with The, append an asterisk * to the search term. For example:

{
   "query": {
    "query_string": {
      "query":"The*",
      "fields": ["title", "name"]
    }
  }
}

Next steps: Extending the UI

As you worked through the steps, you’ll have noticed some additional KeyLines controls such as running automatic layouts. This is just one way you can extend the functionality of KeyLines to help users explore and understand their data.

KeyLines offers a wide range of ways to customize your final application – far too many to outline here. Instead we recommend taking a detailed look through the KeyLines SDK documentation, especially the API reference and sample demos.

From the blog

How to visualize graphs with Elasticsearch

Over the last few weeks, we have been looking at Elasticsearch – the open source enterprise search technology. Blog post...

Visualizing the Elasticsearch Graph API with KeyLines

KeyLines is an impressively powerful graph visualization technology. Its winning combination of scalability, performance and functionality has seen it deployed...

Subscribe to our newsletter

Get occasional data visualization updates, stories and best practice tips by email