Hooking KeyLines up to the Neo4j Bolt Protocol

6th March, 2017 Estimated reading time 7–12 minutes

Neo4j 3.0 has many new features, and we’re particularly interested in the Bolt Protocol.

Bolt promises to be a faster and more secure alternative to the ‘traditional’ REST protocol for interacting with Neo4j databases. For KeyLines customers using Neo4j, this sounds like a great way to crank up performance and build even better visualization applications.

In this post we’ll take a closer look at the Neo4j Bolt protocol. If you’re a more experienced Neo4j user, you can also follow the code snippets and step-by-step instructions to visualize your Neo4j data in KeyLines faster than ever.

What is Bolt?

The Neo4j team describes Bolt as a “connection-oriented protocol to access the graph”. It’s basically another way for Neo4j to communicate with other technologies. It encodes plain text Cypher queries to a binary format, then sends them directly to the database via Websockets or a standard TCP connection.

The Bolt approach has three advantages over using a REST API:

  • Speed – binary encoding means less data gets sent over the wire, and access is faster.
  • Performance – using Websockets, high volumes of queries are executed more quickly, giving higher throughput and lower latency.
  • Security – Bolt has TLS security built and enabled by default in to protect the data connection.
    The Bolt team has created official drivers in multiple languages – including JavaScript, which can run on the client or server-side.

Let’s try using a KeyLines/Neo4j Bolt combo to visualize graph data.

A quick disclaimer: Currently, the KeyLines demos on our SDK site connect to the Neo4j server using a REST API connection. We’re providing this Bolt example code to show how to set this up yourself. We don’t intend to update this blog post in the future, so we can’t guarantee the setup will work for all future versions of Neo4j Bolt.

the Neo4j Bolt Protocol

Step 1: Setting up the environment

For the purposes of this blog post, we’ll use the official Neo4j JavaScript driver 1.1, and Neo4j v3.x, loaded with dummy data from the movie database.

Use these commands to start a new project:

npm init --yes
npm i neo4j-driver -S

Next, download your copy of KeyLines and unzip it into the same directory. If you’re new to KeyLines, you can register for a free trial.

Step 2: Connect KeyLines to Neo4j

Next we’ll use the Bolt protocol to connect KeyLines and Neo4j. There are several ways to use the JavaScript bolt library, but we’ll focus on the simplest one: running in NodeJS or directly in the browser (client-side).

  • To setup and authenticate the Bolt connection for a NodeJS setup, require (‘neo4j-driver’) in Node.
  • For the browser setup, include the browser version of the Neo4j driver, which is in node_modules.

Step 3: Set up the driver

The next step is to set up the driver with some credentials.

First, we probably want to reassign the Neo4j variable at the top of the JavaScript file (whether client-side or server side). We’ll access v1 on the namespace:

neo4j = neo4j.v1;

Then we’ll set up the driver with our credentials:

var driver = neo4j.driver("bolt://localhost", neo4j.auth.basic("neo4j", "neo4j"));

Now we can create sessions to run queries:

function queryNeo (query) {
	var session = driver.session();
	return session.run(query)
		.catch(e => console.log(e))
		.then(results => {
			session.close();
// do something with results here
			return results;
		});
}

queryNeo(‘MATCH (movie:Movie{title: "The Lord of the Rings: The Fellowship of the Ring"})<-[rel]-(actor:Actor) RETURN *’);

Neo4j recommend closing the sessions between requests, unless you want to run a batch of queries in parallel, to reduce simultaneous connections overloading the server.

Step 4: Process the Bolt response

One disadvantage of the Bolt protocol for graph visualizers is that the JSON response you get is flatter that the JSON we’re used to from the REST protocol.

With the REST API, we can request that the results are returned in a graph-like structure, which is easy to convert into a KeyLines format. This isn’t the case with the Bolt protocol, because we get a response like this:

{
  "records": [
    {
      "keys": [
        "actor",
        "movie",
        "rel"
      ],
      "length": 3,
      "_fields": [
        {
          "identity": {
            "low": 819,
            "high": 0
          },
          "labels": [
            "Person",
            "Actor"
          ],
          "properties": {
            "name": "Andy Serkis",
            "lastModified": "1299996916000",
            "id": "1333",
            "biography": "",
            "version": {
              "low": 134,
              "high": 0
            },
            "profileImageUrl": "http://cf1.imgobject.com/profiles/02e/4ceeb6645e73d654f900002e/andy-serkis-profile.jpg"
          }
        },
        {
          "identity": {
            "low": 802,
            "high": 0
          },
          "labels": [
            "Movie"
          ],
          "properties": {
            "studio": "New Line Cinema",
            "releaseDate": "1008716400000",
            "imdbId": "tt0120737",
            "runtime": {
              "low": 178,
              "high": 0
            },
            "description": "Young hobbit Frodo Baggins, after inheriting a mysterious ring from his uncle Bilbo, must leave his home in order to keep it from falling into the hands of its evil creator. Along the way, a fellowship is formed to protect the ringbearer and make sure that the ring arrives at its final destination: Mt. Doom, the only place where it can be destroyed.",
            "language": "en",
            "title": "The Lord of the Rings: The Fellowship of the Ring",
            "version": {
              "low": 410,
              "high": 0
            },
            "trailer": "http://www.youtube.com/watch?v=Pki6jbSbXIY",
            "imageUrl": "http://cf1.imgobject.com/posters/8b8/4cc48d685e73d677850018b8/the-lord-of-the-rings-the-fellowship-of-the-ring-mid.jpg",
            "genre": "Action",
            "tagline": "One ring to rule them all",
            "lastModified": "1299912557000",
            "id": "120",
            "homepage": "http://www.lordoftherings.net/"
          }
        },
        {
          "identity": {
            "low": 1005,
            "high": 0
          },
          "start": {
            "low": 819,
            "high": 0
          },
          "end": {
            "low": 802,
            "high": 0
          },
          "type": "ACTS_IN",
          "properties": {
            "name": "Gollum"
          }
        }
      ],
      "_fieldLookup": {
        "actor": 0,
        "movie": 1,
        "rel": 2
      }
    },
...

But all is not lost. We can convert this into a JSON format KeyLines can use.

A side note on the number format

Numbers in the Bolt response are encoded in a different format in the response. For example ‘1’ in a Neo4j database may be returned as {high: 0, low: 1}.

This is because Neo4j supports 64-bit integers, but JavaScript engines only safely support 32-bit integers.

Most integers you encounter in a web application will be 32-bit, for instance timestamps, but you will need to do a bit more processing to convert the numbers back into their original, usable representation.

Most of the integer functions are not accessible to the Neo4j developer, but the integer representations are based on the long.js library. We can either include that library to process the numbers or write basic functionality to convert.

Step 5: Convert the response into KeyLines JSON

There are three stages to convert the Bolt format into KeyLines JSON:

  1. Loop through each field to look for KeyLines nodes with the right IDs that haven’t been stored already, then convert them.
  2. Convert the ‘start’ and ‘end’ properties of relationships, and match them to an element in the fields array (nodes).
  3. Convert other numeric properties if possible, e.g. time stamps for the Time Bar.

To convert the number formats, we loop through every property to see if it’s a Neo4j Integer object:

neo4j.isInt(object);

Next, if we’re sure that this number is 32-bit, we use this code to convert the object into a standard integer:

function convertInt (neoInt) {
  return neo4j.integer.inSafeRange(neoInt) ? neo4j.integer.toNumber(neoInt) : neoInt;
}

If you want further functionality (e.g. testing if it is safe to convert an integer or not before you do the conversion), consider using long.js.

We’ve created a JavaScript file called klbolt.js that you can download. To convert your Neo4j response into a KeyLines-friendly format, we’ve created a JavaScript file you can use. Download 'klbolt.js.zip'.

The file works in a number of scenarios - including in a NodeJS setup or just in the browser. To get klbolt.js running in a NodeJS environment, add this code to your server.js file:

// in a server.js file:

var klbolt = require('klbolt.js');
// official neo4j bolt js driver
var neo4j = require('neo4j-driver').v1;

var driver = neo4j.driver('bolt://localhost', neo4j.auth.basic(user, pass));
    driver.onError = function (error) {
      logger.error("Error starting neo4j bolt driver. (server: " + boltPath + ')', error);
    };

  function runNeoBoltQuery (query, params) {
    var session = driver.session();
    return session.run(query, params)
      .catch(function (error) {
        logger.error('Error running neo4j query', error);
        session.close();
      })
      .then(function (results) {
        session.close();
        try {
          var parser = new klbolt.Parser();
          var data = parser.parse(results);
        } catch (e) {
          throw e;
        }
        return data
      });
  }

// And an express handler:
app.post('/neo4j', function (req, res) {
  runNeoBoltQuery(req.body.query, req.body.params)
    .then(res.send);
});

Ready to give it a try?

If you’re using KeyLines with Neo4j, give the Bolt protocol a try - and remember to let us know if you see any performance improvements.

Start a KeyLines trial

| | |

Try KeyLines