Web based graph visualization with D3 and KeyLines

6th July, 2016

This is the first in a series of blog posts written for D3 developers interested in network visualization

Read part 2 and part 3


My passion for network visualization first became apparent during my PhD in cultural anthropology. I was investigating how graph tools can be used to empower community members – a process which included building an interactive graph visualization with D3. I found the library to be powerful, but often confusing and counter-intuitive.

I was very excited to join Cambridge Intelligence – a company well known for its excellent KeyLines network visualization toolkit.

In this post I will – as objectively as possible! – compare my first experiences with D3 and KeyLines, explaining what a developer should know before migrating from one to the other.

Some quick caveats

This is not a full feature comparison, and I am not a professional JavaScript developer. I belong to the category of advanced users, people who deal with JavaScript (and other languages) libraries and frameworks alongside databases and desktop applications in a daily workflow.

Introducing D3 and the KeyLines network visualization toolkit

The first thing to understand is that D3 and KeyLines tackle different use cases and are built on different technologies.
Although there is much overlap in how they are used, they are built for different functions:

  • D3 is an open source, general purpose library for web-based data visualization. Graphical elements are rendered as svg shapes by default. Some people use D3 with networks, more with geospatial data, and most for other sort of charts.
  • KeyLines is a commercial library specialized in web-based visualization of connected data. It is focused on graphs (node-link structures), geospatial graphs, and dynamic networks (graphs changing through time). It uses HTML5 canvas or WebGL renderers.

Most organizations become interested in network visualization when they need to expose their graph data to business users. By providing users with a tool to explore and understand patterns in their vast stores of connected data, they can more easily exploit its value.

Many visualization projects start with an open source JavaScript toolkit such as D3.js. Later on, when developers are struggling to introduce more advanced network features (for analytics, performance, user interaction, or developer friendliness reasons) or to improve readability of the visualization, they migrate to an optimized graph visualization toolkit such as KeyLines.

Let’s follow their process.

The graph dataset

For this comparison I will use a dataset generated with freedatagenerator.com, containing fake company details.

The data itself is a JSON file containing two arrays of objects, one for nodes and one for the links between nodes. This “nodes table” and “links table” structure is very common and looks like this:

{
    "nodes": [
    {
        "Id": 1,
        "Name": "Company Name",
        "City": "City Name",
        "Address": "Address line",
        "Latitude": 50.0,
        "Longitude": -80.0,
        "Event1": "2013-04-02",
        "Event2": "2011-09-17"
    }, {
        "Id": 2,
        "Name": "...",
        "City": "...",
        "Address": "...",
        "Latitude": 30.0,
        "Longitude": -90.0,
        "Event1": "2014-01-10",
        "Event2": "2015-08-04"
    }
  ],
    "links": [
    {
        "Id": 1,
        "Transaction": "...",
        "Event1": "2013-11-13",
        "Event2": "2012-09-18",
        "Source": 1,
        "Target": 2,
        "Event3": "2014-08-30",
        "Value1": 246042,
        "Value2": 70571,
        "Value3": 670796
    }
  ]
}

Let’s start our visualization.

Step 1 – Setup your data

Setup your data with D3

The first step consists of loading the data into the web page, parsing it into the required format, and running a layout.

By default D3 connects links to source and target nodes using the zero-based nodes array index. That means a link from node 0 to node 1 will actually connect the first and the second nodes in the nodes array, not nodes with id0 and id1 stored in the dataset, which is what one would probably expect. In order to use our own data ids, we need to create a lookup dictionary object and then map D3 properties link.source and link.target to the corresponding node objects in the dictionary.

function ParseJSONData(data) {

    var nodesDict = data.nodes.reduce(function(dict, node) {
        dict[node.Id] = node;
        return dict;
    }, {});

    data.links.forEach(function(link) {
        link.source = nodesDict[link.Source];
        link.target = nodesDict[link.Target];
    });
...

This was my first stumbling block with D3 – it took a while to get my head around this approach and I could only understand it with the help of a developer colleague.

Set up your data with KeyLines

KeyLines has a documented object structure. A mapping function allows us to specify which piece of data corresponds to which KeyLines object and property, with nodes connected through their ids.

function ParseJSONData(obj, chart) {
    
    var chartData = {
        type: 'LinkChart',
        items: []
    };

    var nodes = obj.nodes.map(function(node) {
        return { 
            id: node.Id,
            type: 'node'
        };
    });
    var links = obj.links.map(function(link) {
        return { 
            id: "l" + link.Id, 
            type: 'link', 
            id1: link.Source, 
            id2: link.Target 
        };
    });

    chartData.items = nodes.concat(links);
…

As KeyLines has been designed specifically for network visualization, it has more suitable and convenient data format requirements. A D3 developer will find it easier to understand and work with.

Step 2 – Create the graph

With a JSON loading function and a parsing function ready, we have our data set up, and we can create the visualization.

Create the graph with D3

In D3 we select an existing DOM element in the web page (a div for instance) which will be the wrapper for the visualization, and we append the main svg element to it:

window.onload = function() {
    ...
    var svg = d3.select("#graph-div").append("svg")
        .attr("width", width)
        .attr("height", height);
    ...

Every other graphical object in the visualization will be appended to the svg element. We will create a line for each data link and a circle for every data node using D3’s enter selection. Since our selectAll function returns an empty selection (there is no DOM object corresponding to the specified selection criteria), the enter sub-selection will contain a placeholder for each data element:

var link = svg.selectAll(".link")
    .data(data.links)
    .enter().append("svg:line")
    .attr("class", "link");

var node = svg.selectAll(".node")
    .data(data.nodes)
    .enter().append("svg:circle")
    .attr("class", "node")
    .attr("r", 5);

Note that we give a class name to each element type, allowing us to manage visual styles with CSS:

.node {
    stroke: #fff;
    stroke - width: 2 px;
}

.link {
    stroke: #999;
    stroke-width: 1.1px;
    stroke-opacity: .7;
}

Alternatively, styles can be assigned inside JavaScript code, using the dot syntax, e.g.:

.style("stroke", "#999");

Create the graph with KeyLines

In KeyLines we specify the DOM element which will be replaced with the canvas element where the graph will be drawn.

window.onload = function() {
    ...
    KeyLines.create({ id: 'kl' }, chartLoaded);
};

The only style option we will specify is node color, but KeyLines has plenty of other styling options. In order to change the defaults we need to add the corresponding line to the data mapping function. The following specifies fill and stroke color for nodes:

chartData.items.push({
    ...
    c: "#000",
    b: "#fff"
});

Then we just need to load the data using chart.load function.

As you can see, KeyLines doesn’t need the added layer of complexity introduced by D3 with selections, because the data formatted by the parsing function is all that’s needed to create the chart.

Step 3 – Run a force-directed layout

Once a developer has his data in a chart, the first thing they will usually do is run a force-directed layout. This will detangle the nodes and links, producing a clearer view of the data.

Run a force-directed layout with D3

If we run the D3 page with the SVG elements without a layout, we would see all the nodes stacked at position 0,0. We need a layout function! We can create a force object, which will control the graph layout, and assign to it the same data we used for the shapes:

var force = d3.layout.force()
    .size([width, height]);

force
    .nodes(data.nodes)
    .links(data.links)
    .start();

Next we can use the force tick event to change the positions of the shapes according to the positions calculated by the force layout:

force.on("tick", tick);

function tick() {
    link.attr("x1", function(d) { return d.source.x; })
        .attr("y1", function(d) { return d.source.y; })
        .attr("x2", function(d) { return d.target.x; })
        .attr("y2", function(d) { return d.target.y; });

    node.attr("cx", function(d) { return d.x; })
        .attr("cy", function(d) { return d.y; });
};

The first result needs to be tweaked a little, because it’s too tightly-knit, as the picture shows:

D3 force directed layout

We can specify some parameters of the layout forces. For instance, augmenting node repulsion with .charge(-250) and link length with .linkDistance(80) will reduce link tightness as required, resulting in a more readable graph:

D3 force directed layout

Run a force-directed layout with KeyLines

When loading the data, we can call the layout function and use KeyLines standard (force-directed) layout among the 7 layouts available.

chart.load(chartData, doLayout);

    function doLayout() {
    chart.layout('standard');
}
KeyLines standard layout

You should find the KeyLines’ default layout parameters a little more useable. For instance the graph is fitted to screen and node size is automatically assigned. Nevertheless, we can also easily tweak them to increase readability. The tightness option allows us to control how tight nodes are positioned.

chart.layout('standard', {tightness: 2});

Setting the tightness level to 2 will do the job for this sample network:

KeyLines standard layout

You can compare the two results by watching their behavior.

Step 4: Graph navigation

Usually the last step of the initial data exploration is to add some chart interaction controls for the user. This is where KeyLines really shines.

Graph navigation in KeyLines

The following is possible, out of the box:

Zoom
The user can use the mouse wheel to intuitively zoom in and out the view.

Pan
If in hand mode, the user can pan the graph by dragging the background.

Navigation controls
Placed in the top left of the view, navigation controls allows the user to visually pan and zoom the network, quickly revert to the home view (fitted to screen), toggle the “hand” mode.

Overview window
The user can open the overview window in the bottom right, and use it to pan around the graph, especially useful together with a deeper zoom level to explore the neighborhood without losing orientation.

Select
By default KeyLines allows many selection behaviors on nodes and links: left click selection, selection highlight, shift + click for multiple selection, selection box.

Drag
The user can click and drag nodes and links. Dragging a node changes its position, dragging a link bends its line.

Multiple links
KeyLines automatically applies an offset and bending to multiple links.

These are features that everyone is likely to want and eventually build for their network visualization (I always wanted them all!), and that’s precisely why they are switched on by default in KeyLines.

Nevertheless, we can also deactivate them, tweak them, and even replace the defaults with custom behaviors taking advantage of the full list of KeyLines event bindings.

Keyboard shortcuts

Some useful keyboard shortcuts are enabled by default. These include Ctrl + A => select all visible elements; Del => remove selected items; Arrow keys => precision move selected items.

Graph navigation in D3

In order to replicate in D3 what KeyLines gives us out of the box, we need multiple steps.
Some of these are easy, some more difficult.

Drag
For instance we can recreate the same node drag functionality by calling D3 drag behavior on nodes:

var node = svg.selectAll(".node")
    ...
    .call(drag);

var drag = d3.behavior.drag()
    .on("dragstart", dragStart)
    .on("drag", dragNode);

function dragStart() {
    force.stop();
}

function dragNode(d) {
    d.x += d3.event.dx;
    d.y += d3.event.dy;
    tick();
}

Dragging links requires a different, and more complex, function.

Pan, zoom, select, highlight and other advanced functionalities

Each of these interactions will need to be implemented separately, using various SVG transforms and JavaScript functions. Some are simple, and some are far beyond the skill levels of this particular author.

In my view, this is where KeyLines really shows its worth.

Conclusion – General purpose vs graph optimized visualization

In this first post I showed the very basic network graph drawing functionality of D3 and KeyLines libraries, side by side, to highlight the main differences.

Both from the developer and user perspectives, KeyLines is easier to use and optimized for this sort of task. Data bindings, event bindings, styling options and layout functions, all assume the visualization context is one of connected data in a graph and therefore can focus on the peculiarities of this visualization domain.

Stay tuned for my next post, where I’ll try to take graph visualization to the next level with more advanced functionality.

Try KeyLines for yourself

If you want to try KeyLines for yourself and see how easy and fast it is to visualize your connected data, you can request a free trial, using the form at the bottom of this page.

As always, don’t forget to tell us what you think. Send us your questions, opinions and suggestions – we would love to hear from you!

| | |

Subscribe to our newsletter

Get occasional data visualization updates, stories and best practice tips by email