Development work last week was focused on trying to reduce memory usage. The work is still on-going, but there is an update now in the develop branch that significantly reduces the memory usage by changing the b-tree operations to act directly on the underlying page data buffers instead of converting them into C# objects internally. This also has the side-effect of increasing the speed of import. Import of 1M triples is down from 240 seconds to 69 seconds; import of 10M triples down from over 3000 seconds to under 900 (and memory usage during that import is down from ~4.5GB to ~2.5GB). However there is still a progressive memory leak during import so for very large files it is better to split the file into several smaller files (of the order of 25M triples if you have 8GB RAM, you can probably get away with larger chunks if you have more memory free on your machine). Tracking down this leak will be the next focus of development work – ideally we want memory usage to be stable and to be capped at a little over the amount assigned to the page cache so as to allow BrightstarDB to import even on memory-constrained devices.
In other news, it is nice to be receiving a number of inquiries about licensing – it shows that people are considering using BrightstarDB. Just to be completely clear – the MIT license applies to all usage of BrightstarDB, both commercial and non-commercial. We decided not to use complicated dual-licensing. If you like BrightstarDB so much you feel you want to pay for it ;-), please get in touch with NetworkedPlanet and ask about support contracts, training and services – these are the channels by which we hope to raise the revenue to keep up development on BrightstarDB.