Skip to content

Software Development Blogs: Programming, Software Testing, Agile Project Management

Methods & Tools

Subscribe to Methods & Tools
if you are not afraid to read more than one page to be a smarter software developer, software tester or project manager!

Mark Needham
Syndicate content
Thoughts on Software Development
Updated: 1 day 5 hours ago

Jython/Neo4j: java.lang.ExceptionInInitializerError: java.lang.ExceptionInInitializerError

Wed, 02/05/2014 - 13:21

I’ve been playing around with calling Neo4j’s Java API from Python via Jython and immediately ran into the following exception when trying to create an embedded instance:

$ jython -Dpython.path /path/to/neo4j.jar
Jython 2.5.3 (2.5:c56500f08d34+, Aug 13 2012, 14:48:36)
[Java HotSpot(TM) 64-Bit Server VM (Oracle Corporation)] on java1.7.0_45
Type "help", "copyright", "credits" or "license" for more information.
 
>>> import org.neo4j.graphdb.factory
>>> org.neo4j.graphdb.factory.GraphDatabaseFactory().newEmbeddedDatabase("/tmp/foo")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
	at org.neo4j.graphdb.factory.GraphDatabaseFactory$1.newDatabase(GraphDatabaseFactory.java:83)
	at org.neo4j.graphdb.factory.GraphDatabaseBuilder.newGraphDatabase(GraphDatabaseBuilder.java:198)
	at org.neo4j.graphdb.factory.GraphDatabaseFactory.newEmbeddedDatabase(GraphDatabaseFactory.java:69)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
 
java.lang.ExceptionInInitializerError: java.lang.ExceptionInInitializerError

I eventually ended up at GraphDatabaseSettings so I tried to import that:

>>> import org.neo4j.graphdb.factory.GraphDatabaseSettings
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
java.lang.ExceptionInInitializerError
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:270)
	at org.python.core.Py.loadAndInitClass(Py.java:909)
	at org.python.core.Py.findClassInternal(Py.java:844)
	at org.python.core.Py.findClassEx(Py.java:895)
	at org.python.core.packagecache.SysPackageManager.findClass(SysPackageManager.java:133)
	at org.python.core.packagecache.PackageManager.findClass(PackageManager.java:28)
	at org.python.core.packagecache.SysPackageManager.findClass(SysPackageManager.java:122)
	at org.python.core.PyJavaPackage.__findattr_ex__(PyJavaPackage.java:137)
	at org.python.core.PyObject.__findattr__(PyObject.java:863)
	at org.python.core.PyObject.impAttr(PyObject.java:1001)
	at org.python.core.imp.import_next(imp.java:720)
	at org.python.core.imp.import_logic(imp.java:782)
	at org.python.core.imp.import_module_level(imp.java:842)
	at org.python.core.imp.importName(imp.java:917)
	at org.python.core.ImportFunction.__call__(__builtin__.java:1220)
	at org.python.core.PyObject.__call__(PyObject.java:357)
	at org.python.core.__builtin__.__import__(__builtin__.java:1173)
	at org.python.core.imp.importOne(imp.java:936)
	at org.python.pycode._pyx5.f$0(<stdin>:1)
	at org.python.pycode._pyx5.call_function(<stdin>)
	at org.python.core.PyTableCode.call(PyTableCode.java:165)
	at org.python.core.PyCode.call(PyCode.java:18)
	at org.python.core.Py.runCode(Py.java:1275)
	at org.python.core.Py.exec(Py.java:1319)
	at org.python.util.PythonInterpreter.exec(PythonInterpreter.java:215)
	at org.python.util.InteractiveInterpreter.runcode(InteractiveInterpreter.java:89)
	at org.python.util.InteractiveInterpreter.runsource(InteractiveInterpreter.java:70)
	at org.python.util.InteractiveInterpreter.runsource(InteractiveInterpreter.java:46)
	at org.python.util.InteractiveConsole.push(InteractiveConsole.java:110)
	at org.python.util.InteractiveConsole.interact(InteractiveConsole.java:90)
	at org.python.util.jython.run(jython.java:317)
	at org.python.util.jython.main(jython.java:129)
Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
	at org.neo4j.graphdb.factory.GraphDatabaseSettings.<clinit>(GraphDatabaseSettings.java:69)
	... 33 more
 
java.lang.ExceptionInInitializerError: java.lang.ExceptionInInitializerError

This bit of code is used to work out the default cache type. It does by first reading the available cache types from a file on the class path and then picking the first one on the list.

I could see that the file it was trying to read was in the JAR and I thought passing the JAR to ‘python.path’ would put it on the classpath. Alistair suggested checking that assumption by checking what was actually on the classpath:

>>> import java.lang
>>> java.lang.System.getProperty("java.class.path")
u'/usr/local/Cellar/jython/2.5.3/libexec/jython.jar:'

As you can see our JAR is missing from the list which explains the problem with reading the file. If we launch the jython REPL with our JAR passed explicitly to the Java classpath then it works:

$ jython -J-cp /path/to/neo4j.jar
Jython 2.5.3 (2.5:c56500f08d34+, Aug 13 2012, 14:48:36)
[Java HotSpot(TM) 64-Bit Server VM (Oracle Corporation)] on java1.7.0_45
Type "help", "copyright", "credits" or "license" for more information.
 
>>> import java.lang
>>> java.lang.System.getProperty("java.class.path")
u'/usr/local/Cellar/jython/2.5.3/libexec/jython.jar:/path/to/neo4j.jar:'

We can now create an embedded database with no problems:

>>> import org.neo4j.graphdb.factory
>>> org.neo4j.graphdb.factory.GraphDatabaseFactory().newEmbeddedDatabase("/tmp/foo")
EmbeddedGraphDatabase [/tmp/foo]
Categories: Programming

Java: Handling a RuntimeException in a Runnable

Sat, 02/01/2014 - 00:59

At the end of last year I was playing around with running scheduled tasks to monitor a Neo4j cluster and one of the problems I ran into was that the monitoring would sometimes exit.

I eventually realised that this was because a RuntimeException was being thrown inside the Runnable method and I wasn’t handling it. The following code demonstrates the problem:

import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.*;
 
public class RunnableBlog {
    public static void main(String[] args) throws ExecutionException, InterruptedException {
        ScheduledExecutorService executor = Executors.newSingleThreadScheduledExecutor();
 
        executor.scheduleAtFixedRate(new Runnable() {
            @Override
            public void run() {
                    System.out.println(Thread.currentThread().getName() + " -> " + System.currentTimeMillis());
                    throw new RuntimeException("game over");
            }
        }, 0, 1000, TimeUnit.MILLISECONDS).get();
 
 
        System.out.println("exit");
        executor.shutdown();
    }
}

If we run that code we’ll see the RuntimeException but the executor won’t exit because the thread died without informing it:

Exception in thread "main" pool-1-thread-1 -> 1391212558074
java.util.concurrent.ExecutionException: java.lang.RuntimeException: game over
	at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
	at java.util.concurrent.FutureTask.get(FutureTask.java:111)
	at RunnableBlog.main(RunnableBlog.java:11)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:601)
	at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)
Caused by: java.lang.RuntimeException: game over
	at RunnableBlog$1.run(RunnableBlog.java:16)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
	at java.lang.Thread.run(Thread.java:722)

At the time I ended up adding a try catch block and printing the exception like so:

public class RunnableBlog {
    public static void main(String[] args) throws ExecutionException, InterruptedException {
        ScheduledExecutorService executor = Executors.newSingleThreadScheduledExecutor();
 
        executor.scheduleAtFixedRate(new Runnable() {
            @Override
            public void run() {
                try {
                    System.out.println(Thread.currentThread().getName() + " -> " + System.currentTimeMillis());
                    throw new RuntimeException("game over");
                } catch (RuntimeException e) {
                    e.printStackTrace();
                }
            }
        }, 0, 1000, TimeUnit.MILLISECONDS).get();
 
        System.out.println("exit");
        executor.shutdown();
    }
}

This allows the exception to be recognised and as far as I can tell means that the thread executing the Runnable doesn’t die.

java.lang.RuntimeException: game over
pool-1-thread-1 -> 1391212651955
	at RunnableBlog$1.run(RunnableBlog.java:16)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
	at java.lang.Thread.run(Thread.java:722)
pool-1-thread-1 -> 1391212652956
java.lang.RuntimeException: game over
	at RunnableBlog$1.run(RunnableBlog.java:16)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
	at java.lang.Thread.run(Thread.java:722)
pool-1-thread-1 -> 1391212653955
java.lang.RuntimeException: game over
	at RunnableBlog$1.run(RunnableBlog.java:16)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
	at java.lang.Thread.run(Thread.java:722)

This worked well and allowed me to keep monitoring the cluster.

However, I recently started reading ‘Java Concurrency in Practice‘ (only 6 years after I bought it!) and realised that this might not be the proper way of handling the RuntimeException.

public class RunnableBlog {
    public static void main(String[] args) throws ExecutionException, InterruptedException {
        ScheduledExecutorService executor = Executors.newSingleThreadScheduledExecutor();
 
        executor.scheduleAtFixedRate(new Runnable() {
            @Override
            public void run() {
                try {
                    System.out.println(Thread.currentThread().getName() + " -> " + System.currentTimeMillis());
                    throw new RuntimeException("game over");
                } catch (RuntimeException e) {
                    Thread t = Thread.currentThread();
                    t.getUncaughtExceptionHandler().uncaughtException(t, e);
                }
            }
        }, 0, 1000, TimeUnit.MILLISECONDS).get();
 
        System.out.println("exit");
        executor.shutdown();
    }
}

I don’t see much difference between the two approaches so it’d be great if someone could explain to me why this approach is better than my previous one of catching the exception and printing the stack trace.

Categories: Programming

Neo4j 2.0.0: Optimising a football query

Fri, 01/31/2014 - 23:41

A couple of months ago I wrote a blog post explaining how I’d applied Wes Freeman’s Cypher optimisation patterns to a query – since then Neo4j 2.0.0 has been released and I’ve extended the model so I thought I’d try again.

The updated model looks like this:

2014 01 31 22 25 20

The query is similar to before – I want to calculate the top away goal scorers in the 2012-2013 season. I started off with this:

MATCH (game)<-[:contains_match]-(season:Season),
      (team)<-[:away_team]-(game),
      (stats)-[:in]->(game),
      (team)<-[:for]-(stats)<-[:played]-(player)
WHERE season.name = "2012-2013"
RETURN player.name, 
       COLLECT(DISTINCT team.name), 
       SUM(stats.goals) as goals
ORDER BY goals DESC
LIMIT 10

When I executed this query using my Python query tuning tool and the average time was 3.31 seconds.

I separated the MATCH statements into smaller individual statements just to see what would happen:

MATCH (game)<-[:contains_match]-(season:Season)
MATCH (team)<-[:away_team]-(game)
MATCH (stats)-[:in]->(game)
MATCH (team)<-[:for]-(stats)<-[:played]-(player)
WHERE season.name = "2012-2013"
RETURN player.name, 
       COLLECT(DISTINCT team.name), 
       SUM(stats.goals) as goals
ORDER BY goals DESC
LIMIT 10

That reduced the time to 178 milliseconds which is quite a nice improvement for so little effort. As I understand it, this is down to the traversal matcher handling smaller patterns more effectively than it handles longer patterns.

The next step was to move the WHERE clause up so that it filtered out rows right at the beginning of the query rather than letting them hang around for another 3 MATCH statements:

MATCH (game)<-[:contains_match]-(season:Season)
WHERE season.name = "2012-2013"
MATCH (team)<-[:away_team]-(game)
MATCH (stats)-[:in]->(game)
MATCH (team)<-[:for]-(stats)<-[:played]-(player)
RETURN player.name, 
       COLLECT(DISTINCT team.name), 
       SUM(stats.goals) as goals
ORDER BY goals DESC
LIMIT 10

That took the time down to 131 milliseconds. At this stage I also tried putting the ‘MATCH (team)<-[:away_team]-(game)' line first to see what would happen.

MATCH (team)<-[:away_team]-(game)
MATCH (game)<-[:contains_match]-(season:Season)
WHERE season.name = "2012-2013"
MATCH (stats)-[:in]->(game)
MATCH (team)<-[:for]-(stats)<-[:played]-(player)
RETURN player.name, 
       COLLECT(DISTINCT team.name), 
       SUM(stats.goals) as goals
ORDER BY goals DESC
LIMIT 10

I expected it to be a bit slower since we were now keeping around more games than necessary again and as expected the time rose to 172 milliseconds – slightly quicker than our first attempt at tweaking.

I reverted that change and realised that there would be a lot of ‘stats’ nodes which didn’t have any goals associated with them and would therefore have a ‘goals’ property value of 0. I tried filtering those nodes out before the ‘SUM’ part of the query:

MATCH (game)<-[:contains_match]-(season:Season)
WHERE season.name = "2012-2013"
MATCH (team)<-[:away_team]-(game)
MATCH (stats)-[:in]->(game)
MATCH (team)<-[:for]-(stats)<-[:played]-(player)
WHERE stats.goals > 0
RETURN player.name,
COLLECT(DISTINCT team.name),
SUM(stats.goals) as goals
ORDER BY goals DESC
LIMIT 10

This proved to be a very good optimisation – the time reduced to 47 milliseconds, an improvement of almost 3x on the previous optimisation and 63x quicker than the original query.

The main optimisation pattern used here was reducing the number of rows being passed through the query.

Ideally you don’t want to be passing unnecessary rows through each stage of the query – rows can be filtered out either by using more specific MATCH clauses or in this case by a WHERE clause.

Wes and I presented a webinar on Cypher query optimisation last week which so if you want to learn more about tuning queries that might be worth a watch.

Categories: Programming

Neo4j 2.0.0: Cypher – Index Hints and Neo.ClientError.Schema.NoSuchIndex

Fri, 01/31/2014 - 08:14

One of the features added into the more recent versions of Neo4j’s cypher query language is the ability to tell Cypher which index you’d like to use in your queries.

We’ll use the football dataset, so let’s start by creating an index on the ‘name’ property of nodes labelled ‘Player’:

CREATE INDEX ON :Player(name)

Let’s say we want to write a query to find ‘Wayne Rooney’ while explicitly using this index. We might start with the following query:

MATCH p
USING INDEX p:Player(name)
WHERE p.name = "Wayne Rooney"
RETURN p

If we run that we’ll see this error:

Cannot use index hint in this context. The label and property comparison must be specified on a non-optional node
Label: `Player`
Property name: `name`
Neo.ClientError.Schema.NoSuchIndex

We need to specify the label on the node in the ‘MATCH’ part of the query for our hint to work e.g.

MATCH (p:Player)
USING INDEX p:Player(name)
WHERE p.name = "Wayne Rooney"
RETURN p

Now we might decide that we want to find ‘Wayne Rooney’ or ‘Robin Van Persie’ so we change our query slightly:

MATCH (p:Player)
USING INDEX p:Player(name)
WHERE p.name = "Wayne Rooney" OR p.name = "Robin Van Persie"
RETURN p

But this one fails too!

Cannot use index hint in this context. The label and property comparison must be specified on a non-optional node
Label: `Player`
Property name: `name`
Neo.ClientError.Schema.NoSuchIndex

The problem here is that when you use ‘OR’ it’s currently not using an index but rather a label scan so the hint to use the index doesn’t make sense to Cypher.

The final way I’ve seen people get confused using index hints is when matching node properties inline e.g.

MATCH (p:Player {name: "Wayne Rooney"})
USING INDEX p:Player(name)
RETURN p

If we run that we’ll be back in the land of exceptions:

Cannot use index hint in this context. The label and property comparison must be specified on a non-optional node
Label: `Player`
Property name: `name`
Neo.ClientError.Schema.NoSuchIndex

We can work around that by pulling out the inline matching of the ‘name’ property”:

MATCH (p:Player)
USING INDEX p:Player(name)
WHERE p.name = "Wayne Rooney"
RETURN p
Categories: Programming

Java: Work out the serialVersionUID of a class

Fri, 01/31/2014 - 07:51

Earlier in the week I wanted to work out the serialVersionUID of a serializable class so that I could override its toString method without breaking everything.

I came across Frank Kim’s blog post which suggested using the serialver tool which comes with the JDK.

I created a little Maven project to test this tool out on a very simple class:

import java.io.Serializable;
 
public class SerialiseMe implements Serializable
{
 
}

If we compile that class into a JAR and then run the serialver tool we see the following output:

$ serialver -classpath target/serialiser-0.0.1-SNAPSHOT.jar SerialiseMe
SerialiseMe:    static final long serialVersionUID = -6060222249255158490L;

I wanted to quickly confirm that I could serialise and deserialise this class using this value so I wrote the following bit of code to serialise the class (when it didn’t have a serial version UID):

public class Serialiser
{
    public static void main( String[] args ) throws IOException, ClassNotFoundException
    {
        ByteArrayOutputStream bout = new ByteArrayOutputStream(  );
        ObjectOutputStream oout = new ObjectOutputStream( bout );
 
        Object value = new SerialiseMe();
 
        oout.writeObject( value );
        oout.close();
        byte[] bytes = bout.toByteArray();
 
        FileOutputStream fileOuputStream = new FileOutputStream("/tmp/foo.txt");
        fileOuputStream.write(bytes);
        fileOuputStream.close();
    }
}

After I’d done that, I wrote the following bit of code to deserialise the file:

public class Deserialiser
{
    public static void main( String[] args ) throws IOException, ClassNotFoundException
    {
        FileInputStream fileInputStream = new FileInputStream( new File( "/tmp/foo.txt" ) );
        byte[] bytes = IOUtils.toByteArray( fileInputStream );
 
        ByteArrayInputStream in = new ByteArrayInputStream( bytes, 0, bytes.length );
        ObjectInputStream oin = new ObjectInputStream( in );
        Object object = oin.readObject();
    }
}

I plugged the serial version UID into the class and was able to deserialise it correctly. I tried changing one of the digits just to check it would blow up and indeed it did:

import java.io.Serializable;
 
public class SerialiseMe implements Serializable
{
    static final long serialVersionUID = -6060222249255158491L;
}
Exception in thread "main" java.io.InvalidClassException: SerialiseMe; local class incompatible: stream classdesc serialVersionUID = -6060222249255158490, local class serialVersionUID = -6060222249255158491
	at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:604)
	at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1620)
	at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1515)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1769)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1348)
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
	at Deserialiser.main(Deserialiser.java:18)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:601)
	at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)

serialver #ftw!

Categories: Programming

Neo4j: org.eclipse.jetty.io.EofException – Caused by: java.io.IOException: Broken pipe

Mon, 01/27/2014 - 12:32

From scouring the Neo4j google group and Stack Overflow I’ve noticed that a few people have been hitting the following exception when executing queries against Neo4j server:

SEVERE: The response of the WebApplicationException cannot be utilized as the response is already committed. Re-throwing to the HTTP container
javax.ws.rs.WebApplicationException: javax.ws.rs.WebApplicationException: org.eclipse.jetty.io.EofException
	at org.neo4j.server.rest.repr.OutputFormat$1.write(OutputFormat.java:174)
	at com.sun.jersey.core.impl.provider.entity.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:71)
	at com.sun.jersey.core.impl.provider.entity.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:57)
	at com.sun.jersey.spi.container.ContainerResponse.write(ContainerResponse.java:306)
	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1437)
	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
	at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:698)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1506)
	at org.neo4j.server.rest.security.SecurityFilter.doFilter(SecurityFilter.java:112)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1477)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:503)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:211)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1096)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:432)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:175)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1030)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:136)
	at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
	at org.eclipse.jetty.server.Server.handle(Server.java:445)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:268)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:229)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.run(AbstractConnection.java:358)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:601)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:532)
	at java.lang.Thread.run(Thread.java:744)
Caused by: javax.ws.rs.WebApplicationException: org.eclipse.jetty.io.EofException
	at org.neo4j.server.rest.repr.formats.StreamingJsonFormat$StreamingListWriter.writeValue(StreamingJsonFormat.java:316)
	at org.neo4j.server.rest.repr.ListWriter.writeValue(ListWriter.java:75)
	at org.neo4j.server.rest.repr.ValueRepresentation.addTo(ValueRepresentation.java:49)
	at org.neo4j.server.rest.repr.ListRepresentation.serialize(ListRepresentation.java:65)
	at org.neo4j.server.rest.repr.Serializer.serialize(Serializer.java:75)
	at org.neo4j.server.rest.repr.MappingSerializer.putList(MappingSerializer.java:61)
	at org.neo4j.server.rest.repr.ListRepresentation.putTo(ListRepresentation.java:85)
	at org.neo4j.server.rest.repr.ObjectRepresentation$PropertyGetter.putTo(ObjectRepresentation.java:133)
	at org.neo4j.server.rest.repr.ObjectRepresentation.serialize(ObjectRepresentation.java:144)
	at org.neo4j.server.rest.repr.MappingRepresentation.serialize(MappingRepresentation.java:41)
	at org.neo4j.server.rest.repr.OutputFormat$1.write(OutputFormat.java:160)
	... 30 more
Caused by: org.eclipse.jetty.io.EofException
	at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:186)
	at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:335)
	at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:125)
	at org.eclipse.jetty.server.HttpConnection$ContentCallback.process(HttpConnection.java:784)
	at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:79)
	at org.eclipse.jetty.server.HttpConnection.send(HttpConnection.java:356)
	at org.eclipse.jetty.server.HttpChannel.sendResponse(HttpChannel.java:631)
	at org.eclipse.jetty.server.HttpChannel.write(HttpChannel.java:661)
	at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:198)
	at com.sun.jersey.spi.container.servlet.WebComponent$Writer.write(WebComponent.java:307)
	at com.sun.jersey.spi.container.ContainerResponse$CommittingOutputStream.write(ContainerResponse.java:134)
	at org.codehaus.jackson.impl.Utf8Generator._flushBuffer(Utf8Generator.java:1754)
	at org.codehaus.jackson.impl.Utf8Generator.writeNumber(Utf8Generator.java:886)
	at org.codehaus.jackson.map.ser.StdSerializers$LongSerializer.serialize(StdSerializers.java:170)
	at org.codehaus.jackson.map.ser.StdSerializers$LongSerializer.serialize(StdSerializers.java:158)
	at org.codehaus.jackson.map.ser.StdSerializerProvider._serializeValue(StdSerializerProvider.java:610)
	at org.codehaus.jackson.map.ser.StdSerializerProvider.serializeValue(StdSerializerProvider.java:256)
	at org.codehaus.jackson.map.ObjectMapper.writeValue(ObjectMapper.java:1613)
	at org.codehaus.jackson.impl.JsonGeneratorBase.writeObject(JsonGeneratorBase.java:314)
	at org.neo4j.server.rest.repr.formats.StreamingJsonFormat$StreamingListWriter.writeValue(StreamingJsonFormat.java:312)
	... 40 more
Caused by: java.io.IOException: Broken pipe
	at sun.nio.ch.FileDispatcherImpl.writev0(Native Method)
	at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:51)
	at sun.nio.ch.IOUtil.write(IOUtil.java:148)
	at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:524)
	at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:167)
	... 59 more

I’d not come across it myself but the error message suggests that the client has closed the connection while the server is still trying to write out a response.

This suggests that you need to write a query which runs for a long time. I created the following data set to try and force this to happen:

public class DenseNodes
{
    private static final DynamicRelationshipType FOO = DynamicRelationshipType.withName( "FOO" );
    private static final Label BAR = DynamicLabel.label( "Bar" );
 
    public static void main( String[] args ) throws IOException
    {
        String path = "/tmp/dense";
 
        FileUtils.deleteRecursively( new File( path ) );
 
        GraphDatabaseService db = new GraphDatabaseFactory().newEmbeddedDatabase( path );
 
        for ( int i = 0; i < 10; i++ )
        {
            try ( Transaction tx = db.beginTx() )
            {
                Node node = db.createNode( BAR );
 
                for ( int j = 0; j < 100_000; j++ )
                {
                    node.createRelationshipTo( db.createNode(), FOO );
                }
 
                tx.success();
            }
        }
    }
}

This script creates 10 nodes with 100,000 relationships and as a by-product we have 1,000,000 nodes with 1 relationship.

We can execute the following query using a Jersey client to take us over the timeout and force the EOFException:

public class DenseMe
{
    public static void main( String[] args ) throws IOException
    {
        String query = "MATCH (a:Bar)-[:`FOO`]->(b) RETURN a,b";
 
        long start = System.currentTimeMillis();
        executeViaHTTP( query );
        long end = System.currentTimeMillis();
 
        System.out.println(end - start);
    }
 
    private static void executeViaHTTP( String query ) throws IOException
    {
        ObjectNode entity = JsonNodeFactory.instance.objectNode();
        entity.put("query", query );
 
        ClientResponse response = client().resource( "http://localhost:7474/db/data/cypher" )
                .accept( MediaType.APPLICATION_JSON_TYPE )
                .type(MediaType.APPLICATION_JSON_TYPE)
                .post( ClientResponse.class, entity );
 
        InputStream stream = response.getEntityInputStream();
 
        BufferedReader reader = new BufferedReader( new InputStreamReader( stream ) );
        char[] buffer = new char[1024];
        int bytesRead;
        while ( (bytesRead = reader.read( buffer )) != -1 )
        {
            for ( int i = 0; i < bytesRead; i++ )
            {
                System.out.print( buffer[i] );
            }
        }
    }
 
    private static Client client()
    {
        DefaultClientConfig defaultClientConfig = new DefaultClientConfig();
        defaultClientConfig.getClasses().add( JacksonJsonProvider.class );
        Client client = Client.create( defaultClientConfig );
        return client;
    }
}

There are two ways that we can attempt to work around this problem:

  1. Increase the timeout on the Jersey Client
  2. Stream the response to the client so we don’t build up such a huge response on the server

We’ll start with the former, which requires the following tweak to the client function:

private static Client client()
{
    DefaultClientConfig defaultClientConfig = new DefaultClientConfig();
    defaultClientConfig.getClasses().add( JacksonJsonProvider.class );
    Client client = Client.create( defaultClientConfig );
    client.setConnectTimeout(1_000_000);
    client.setReadTimeout( 1_000_000 );
    return client;
}

If we run that we’ll end up with the following response using a 4GB heap:

{
  "message" : "Java heap space",
  "exception" : "OutOfMemoryError",
  "fullname" : "java.lang.OutOfMemoryError",
  "stacktrace" : [ ]
}

I was tailing the GC logs while running the query and the majority of time was being spent in Full GC while the query was running:

2014-01-27T10:27:26.101+0000: 239848.812: Total time for which application threads were stopped: 5.5309550 seconds
2014-01-27T10:27:26.101+0000: 239848.812: [Full GC2014-01-27T10:27:26.101+0000: 239848.812: [CMS: 3512768K->3512768K(3512768K), 5.4359920 secs] 4126207K->4126207K(4126208K), [CMS Perm : 41512K->41512K(69300K)], 5.4360820 secs] [Times: user=5.43 sys=0.00, real=5.43 secs]
2014-01-27T10:27:31.537+0000: 239854.249: [Full GC2014-01-27T10:27:31.537+0000: 239854.249: [CMS: 3512768K->3512768K(3512768K), 5.4878690 secs] 4126207K->4126207K(4126208K), [CMS Perm : 41512K->41512K(69300K)], 5.4879470 secs] [Times: user=5.49 sys=0.01, real=5.49 secs]
2014-01-27T10:27:37.025+0000: 239859.737: Total time for which application threads were stopped: 10.9243140 seconds
2014-01-27T10:27:37.025+0000: 239859.737: [Full GC2014-01-27T10:27:37.025+0000: 239859.737: [CMS: 3512768K->3512768K(3512768K), 5.4437040 secs] 4126207K->4126207K(4126208K), [CMS Perm : 41512K->41512K(69300K)], 5.4437790 secs] [Times: user=5.44 sys=0.01, real=5.44 secs]
2014-01-27T10:27:42.469+0000: 239865.181: [Full GC2014-01-27T10:27:42.469+0000: 239865.181: [CMS: 3512768K->3512768K(3512768K), 5.4283480 secs] 4126207K->4126207K(4126208K), [CMS Perm : 41512K->41512K(69300K)], 5.4284400 secs] [Times: user=5.43 sys=0.00, real=5.43 secs]
2014-01-27T10:27:47.898+0000: 239870.609: Total time for which application threads were stopped: 10.8724950 seconds
2014-01-27T10:27:47.898+0000: 239870.609: [Full GC2014-01-27T10:27:47.898+0000: 239870.609: [CMS: 3512768K->3512768K(3512768K), 5.4385630 secs] 4126208K->4126207K(4126208K), [CMS Perm : 41512K->41512K(69300K)], 5.4386540 secs] [Times: user=5.43 sys=0.01, real=5.44 secs]
2014-01-27T10:27:53.337+0000: 239876.048: Total time for which application threads were stopped: 5.4389110 seconds

The second option is to stream back the response by adding the following header to our request:

ClientResponse response = client().resource( "http://localhost:7474/db/data/cypher" )
    .accept( MediaType.APPLICATION_JSON_TYPE )
    .type(MediaType.APPLICATION_JSON_TYPE)
    .header( "X-Stream", "true" )
    .post( ClientResponse.class, entity );

If we do that then we’ll get back the first rows of the query immediately although although we’ll have to be careful with what we do with the response or we could see an OutOfMemory exception on the client instead.

We might also want to think whether we actually need to return that many rows in the first place. A lot of the time a subset is more than enough.

Categories: Programming

Neo4j HA: org.neo4j.graphdb.TransactionFailureException: Timeout waiting for database to allow new transactions. Blocking components (1): []

Mon, 01/27/2014 - 10:42

As I mentioned in my previous post, I’ve been spending quite a bit of time working with Neo4j HA and recently came across the following exception in data/graph.db/messages.log:

org.neo4j.graphdb.TransactionFailureException: Timeout waiting for database to allow new transactions. Blocking components (1): []
	at org.neo4j.kernel.ha.HighlyAvailableGraphDatabase.beginTx(HighlyAvailableGraphDatabase.java:199)
	at org.neo4j.kernel.TransactionBuilderImpl.begin(TransactionBuilderImpl.java:43)
	at org.neo4j.kernel.InternalAbstractGraphDatabase.beginTx(InternalAbstractGraphDatabase.java:949)
	at org.neo4j.server.rest.transactional.TransactionalRequestDispatcher.dispatch(TransactionalRequestDispatcher.java:52)
	at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
	at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
	at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
	at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:698)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1506)
	at org.neo4j.server.rest.security.SecurityFilter.doFilter(SecurityFilter.java:112)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1477)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:503)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:211)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1096)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:432)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:175)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1030)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:136)
	at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
	at org.eclipse.jetty.server.Server.handle(Server.java:445)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:268)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:229)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.run(AbstractConnection.java:358)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:601)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:532)
	at java.lang.Thread.run(Thread.java:744)

I traced the error message back to AvailabilityGuard#describeWhoIsBlocking and discovered that this error message is thrown when we don’t want the cluster to accept transactions.

Usually the blocking component (the thing that’s stopping transactions from being accepted) is described but in this case the error message indicates that the cluster hasn’t been formed.

I was able to reproduce the error message by trying to commit a transaction from a slave in a cluster which was misconfigured so that the slaves were unable to interact with the master.

To find the root cause for this error message we need to scroll up the log a bit and look for the exception that happened immediately before this one. In this case we see the following:

2014-01-24 11:48:53.889+0000 WARN  [o.n.k.h.c.HighAvailabilityModeSwitcher]: Consistency checker failed
org.neo4j.com.ComException: MasterClient20 could not connect to /10.137.62.4:6001
	at org.neo4j.com.Client$1.create(Client.java:141) ~[neo4j-com-2.0.0.jar:2.0.0]
	at org.neo4j.com.Client$1.create(Client.java:123) ~[neo4j-com-2.0.0.jar:2.0.0]
	at org.neo4j.com.ResourcePool.acquire(ResourcePool.java:175) ~[neo4j-com-2.0.0.jar:2.0.0]
	at org.neo4j.com.Client.getChannel(Client.java:336) ~[neo4j-com-2.0.0.jar:2.0.0]
	at org.neo4j.com.Client.sendRequest(Client.java:216) ~[neo4j-com-2.0.0.jar:2.0.0]
	at org.neo4j.kernel.ha.MasterClient20.getMasterIdForCommittedTx(MasterClient20.java:359) ~[neo4j-ha-2.0.0.jar:2.0.0]
	at org.neo4j.kernel.ha.cluster.HighAvailabilityModeSwitcher.checkDataConsistencyWithMaster(HighAvailabilityModeSwitcher.java:679) [neo4j-ha-2.0.0.jar:2.0.0]
	at org.neo4j.kernel.ha.cluster.HighAvailabilityModeSwitcher.checkDataConsistency(HighAvailabilityModeSwitcher.java:498) [neo4j-ha-2.0.0.jar:2.0.0]
	at org.neo4j.kernel.ha.cluster.HighAvailabilityModeSwitcher.access$900(HighAvailabilityModeSwitcher.java:110) [neo4j-ha-2.0.0.jar:2.0.0]
	at org.neo4j.kernel.ha.cluster.HighAvailabilityModeSwitcher$2.run(HighAvailabilityModeSwitcher.java:393) [neo4j-ha-2.0.0.jar:2.0.0]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_51]
	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_51]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) [na:1.7.0_51]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) [na:1.7.0_51]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_51]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_51]
	at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51]

A quick configuration change was able to get me going again but you could also see this error message if firewall rules were misconfigured so that’s something else to look out for.

Categories: Programming

Neo4j HA: Election could not pick a winner

Fri, 01/24/2014 - 11:30

Recently I’ve been spending a reasonable chunk of my time helping people get up and running with their Neo4j High Availability cluster and there’s sometimes confusion around how it should be configured.

A Neo4j cluster typically consists of a master and two slaves and you’d usually have it configured so that any machine can be the master.

However, there is a configuration parameter ‘ha.slave_only’ which can be set to ‘true’ to ensure that a machine will never be elected as master when an election takes place.

We might configure our machine with that setting if it’s acting as a reporting instance but we need to make sure that 2 members don’t have that setting otherwise we won’t have any failover in the cluster.

For example, if we set two of the machines in the cluster to be slave only and then stop the master we’ll see output similar to the following in data/graph.db/messages.log:

2014-01-23 11:17:24.510+0000 INFO  [o.n.c.p.a.m.MultiPaxosContext$ElectionContextImpl]: Doing elections for role coordinator
2014-01-23 11:17:24.510+0000 DEBUG [o.n.c.p.e.ElectionState$2]: ElectionState: election-[performRoleElections]->election from:cluster://10.239.8.251:5001 conversation-id:3/13#
2014-01-23 11:17:24.513+0000 DEBUG [o.n.c.p.e.ElectionState$2]: ElectionState: election-[vote:coordinator]->election from:cluster://10.151.24.237:5001 conversation-id:3/13#
2014-01-23 11:17:24.515+0000 DEBUG [o.n.c.p.e.ElectionState$2]: ElectionState: election-[voted]->election from:cluster://10.138.29.197:5001 conversation-id:3/13#
2014-01-23 11:17:24.516+0000 DEBUG [o.n.c.p.e.ElectionState$2]: ElectionState: election-[voted]->election from:cluster://10.151.24.237:5001 conversation-id:3/13#
2014-01-23 11:17:24.519+0000 DEBUG [o.n.c.p.a.m.MultiPaxosContext$ElectionContextImpl$2]: Elections ended up with list []
2014-01-23 11:17:24.519+0000 WARN  [o.n.c.p.e.ElectionState]: Election could not pick a winner

This message initially looks confusing but what it’s telling us is that the cluster was unable to elect a new master, in this case because there were no machines that could be elected as master.

So if you see that message in your logs, check your config to make sure that there are actually machines to choose from.

Categories: Programming

Neo4j Backup: Store copy and consistency check

Wed, 01/22/2014 - 18:36

One of the lesser known things about the Neo4j online backup tool, which I wrote about last week, is that conceptually there are two parts to it:

  1. Copying the store files to a location of your choice
  2. Verifying that those store files are consistent.

By default both of these run when you run the ‘neo4j-backup’ script but sometimes it’s useful to be able to run them separately.

If we want to just run the copying the store files part of the process we can tell the backup tool to skip the consistency check by using the ‘verify‘ flag:

$ pwd
/Users/markneedham/Downloads/neo4j-enterprise-2.0.0
$ ./bin/neo4j-backup -from single://127.0.0.1 -to /tmp/foo -verify false
Performing full backup from 'single://127.0.0.1'
Files copied
................        done
Done

If we ran that without the ‘verify’ flag we’d see the output of the consistency checker as well:

$ ./bin/neo4j-backup -from single://127.0.0.1 -to /tmp/foo
Performing full backup from 'single://127.0.0.1'
Files copied
................        done
Full consistency check
....................  10%
....................  20%
....................  30%
....................  40%
....................  50%
....................  60%
....................  70%
....................  80%
....................  90%
.................... 100%
Done

If we already have a backup and only want to run the consistency checker we can run the following command:

$ java -cp 'lib/*:system/lib/*' org.neo4j.consistency.ConsistencyCheckTool /tmp/foo
Full consistency check
....................  10%
....................  20%
....................  30%
....................  40%
....................  50%
....................  60%
....................  70%
....................  80%
....................  90%
.................... 100%

The consistency tool itself takes a ‘config‘ flag which gives you some control over what things you want to consistency check.

The various options are defined in org.neo4j.consistency.ConsistencyCheckSettings.

For example, if we want to change the file that the consistency check report is written to we could add the following property to our config file:

$ tail -n 1 conf/neo4j.properties
consistency_check_report_file=/tmp/foo.txt

And then run the consistency tool like so:

$ java -cp 'lib/*:system/lib/*' org.neo4j.consistency.ConsistencyCheckTool -config conf/neo4j.properties /tmp/foo

If there are any inconsistencies they’ll now be written to that file rather than to a file in the store directory.

You can also pass that ‘config’ flag to the backup tool and it will make use of it when it runs the consistency check. e.g.

$ ./bin/neo4j-backup -from single://127.0.0.1 -to /tmp/foo -verify false -config conf/neo4j.properties

Most of the time you don’t need to worry too much about either of these commands but I always forget what the various options are so I thought I’d better write it up while it’s fresh in my mind.

Categories: Programming

Neo4j Backup: java.lang.ClassCastException: org.jboss.netty.buffer.BigEndianHeapChannelBuffer cannot be cast to org.neo4j.cluster.com.message.Message

Sun, 01/19/2014 - 20:29

When using Neo4j’s online backup facility there are two ways of triggering it, either by using the ‘single://‘ or ‘ha://‘ syntax and these behave slightly differently.

If you’re using the ‘single://’ syntax and don’t specify a port then it will connect to ’6362′ by default:

./neo4j-backup -from single://192.168.1.34 -to /mnt/backup/neo4j-backup

If you’ve changed the backup port via the ‘online_backup_server’ property in conf/neo4j.properties you’ll need to set the port explicitly:

online_backup_server=192.168.1.34:6363
./neo4j-backup -from single://192.168.1.34:6363 -to /mnt/backup/neo4j-backup

If you’re using the ‘ha://’ syntax then the backup client joins the HA cluster, works out which machine is the master and then creates a backup from that machine.

In order for the backup client to join the cluster it connects to port ’5001′ by default:

./neo4j-backup -from ha://192.168.1.34 -to /mnt/backup/neo4j-backup

If you’ve changed the ‘ha.cluster_server’ property then you’ll need to set the port explicitly:

ha.cluster_server=192.168.1.34:5002
./neo4j-backup -from ha://192.168.1.34:5002 -to /mnt/backup/neo4j-backup

A mistake that I made when first using this utility was to use the ‘ha://’ syntax with the backup port. e.g.

./neo4j-backup -from ha://192.168.1.34:6362 -to /mnt/backup/neo4j-backup

If you do this you’ll end up with the following exception:

2014-01-19 19:24:30.842+0000 ERROR [o.n.c.c.NetworkSender]: Receive exception:
java.lang.ClassCastException: org.jboss.netty.buffer.BigEndianHeapChannelBuffer cannot be cast to org.neo4j.cluster.com.message.Message
	at org.neo4j.cluster.com.NetworkSender$NetworkMessageSender.messageReceived(NetworkSender.java:409) ~[neo4j-cluster-2.0.0.jar:2.0.0]
	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) ~[netty-3.6.3.Final.jar:na]
	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) ~[netty-3.6.3.Final.jar:na]
	at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) ~[netty-3.6.3.Final.jar:na]
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:107) ~[netty-3.6.3.Final.jar:na]
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312) ~[netty-3.6.3.Final.jar:na]
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88) ~[netty-3.6.3.Final.jar:na]
	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) ~[netty-3.6.3.Final.jar:na]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_45]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_45]
	at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]

Let me know in the comments if any of this doesn’t make sense. There are lots of other examples to follow on neo4j-backup’s man page in the manual as well.

Categories: Programming

Learning about bitmaps

Sun, 01/12/2014 - 18:44

A few weeks ago Alistair and I were working on the code used to model the labels that a node has attached to it in a Neo4j database.

The way this works is that chunks of 32 nodes ids are represented as a 32 bit bitmap for each label where a 1 for a bit means that a node has the label and a 0 means that it doesn’t.

For example, let’s say we have node ids 0-31 where 0 is the highest bit and 31 is the lowest bit. If only node 0 has the label then that’d be represented as the following value:

java> int bitmap = 1 << 31;
int bitmap = -2147483648

If we imagine the 32 bits positioned next to each other it would look like this:

2014 01 12 15 45 16
java> 0X80000000;
Integer res16 = -2147483648

The next thing we want to do is work out whether a node has a label applied or not. We can do this by using a bitwise AND.

For example to check whether the highest bit is set we would write the following code:

java> bitmap & (1 << 31);
Integer res10 = -2147483648

That is set as we would imagine. Now let’s check a a few bits that we know aren’t set:

java> bitmap & (1 << 0);
Integer res11 = 0
 
java> bitmap & (1 << 1);
Integer res12 = 0
 
java> bitmap & (1 << 30);
Integer res13 = 0

Another operation we might want to do is set another bit on our existing bitmap for which we can use a bitwise inclusive OR.

A bitwise inclusive OR means that a bit will be set if either value has the bit set or if both have it set.

Let’s set the second highest bit. and visualise that calculation:

2014 01 12 15 45 16

If we evaluate that we’d expect the two highest bits to be set:

java> bitmap |= (1 << 30);
Integer res14 = -1073741824

Now if we visualise the bitmap we’ll see that is indeed the case:

2014 01 12 17 16 21
java> 0XC0000000;
Integer res15 = -1073741824

The next operation we want to do is to unset a bit that we’re already set for which we can use a bitwise exclusive OR.

An exclusive OR means that a bit will only remain set if there’s a combination of (0 and 1) or (1 and 0) in the calculation. If there are two 1′s or 2 0′s then it’ll be unset.

Let’s unset the 2nd highest bit so that we’re left with just the top bit being set.

If we visualise that we have the following calculation:

2014 01 12 17 33 21

And if we evaluate that we’re back to our original bitmap:

java> bitmap ^= (1 << 30);
Integer res2 = -2147483648

I used the Java REPL to evaluate the code samples in this post and this article explains bitshift operators very clearly.

The Neo4j version of the bitmap described in this post is in the BitmapFormat class on github.

Categories: Programming