RelationshipGroupStore mapped_memory setting for BatchInserter?

Vote count:

0

I'm inserting relationships into Neo4j via the BatchInserter (as part of the initial data insertion). The process quickly slows down to 1,000,000 per ~1hr. The rest of the data (12,390,251 relationships, 6 million nodes, 74 million properties) goes in in less than an hour.

I've identified the culprit as the RelationshipGroupStore is increasing by only a few bytes at a time. While it starts off at 2megs increases in the file size, it eventually slows down to only a few bytes at a time.

I'm curious if this is related to the memory_mappnig, but there doesn't seem to be an option for the RelationshipGroupStore for the BatchInserter (probably the rest of the Kernel as well).

I've tried putting the following into the configuration but it did not seem to have an effect

neostore.relationshipgroupstore.db.mapped_memory=2G 

Any ideas? Should I not do this in BatchInserter (50 mln Relationships for my current dataset, many more for others). I also believe this was much faster with older versions of Neo4j. As you can imagine 50 million relationships over 6 million nodes creates lots of highly connected nodes, which Neo4j 2.1.X have handled very well when it comes to query speed. I'm just trying to speed up insertion now.

Neo4j 2.1.5 Java: Various Versions, 1.7, 1.8 OS: Ubuntu, Mac OS X, CentOS

Neo4j Configuration:

neostore.propertystore.db.index.mapped_memory=1M
neostore.propertystore.db.strings.mapped_memory=1G
neostore.propertystore.db.index.keys.mapped_memory=1M
store_dir=data.db
dump_configuration=true
use_memory_mapped_buffers=true
neostore.propertystore.db.arrays.mapped_memory=512M
neostore.propertystore.db.mapped_memory=2G
neostore.relationshipstore.db.mapped_memory=2G
neostore.nodestore.db.mapped_memory=1G

JVM Settings

"-Xms10G"
"-Xms10G"
"-XX:+UseConcMarkSweepGC"
"-XX:+UseBiasedLocking"
"-XX:+AggressiveOpts"
"-XX:+UseCompressedOops"
"-XX:+UseFastAccessorMethods"
"-XX:+DoEscapeAnalysis"
"-Xss4096k"
"-d64"
"-server"

Any help is appreciated. Thanks!

asked 1 min ago

Joseph G

3

RelationshipGroupStore mapped_memory setting for BatchInserter?