I’m going to try to keep the blog updated on a regular basis with details of what we are currently working on in BrightstarDB.
In the last week or two I’ve made some progress on two fronts. Firstly the develop branch of BrightstarDB is now more or less done for a 1.3 release. 1.3 is quite an important milestone for us because it will be the first “official” open-source release of BrightstarDB. The main focus of this release has been to remove some dependencies on third-party tools for documentation and obfuscation and to simplify and update the packaging.
Documentation Differences: I’ve moved the main prose documentation over from the commercial Help & Manual tool to the open-source Sphinx toolkit (I wrote about running Sphinx on Windows on my personal blog), and I’ve set up an account on readthedocs.org. The main advantage of this is that we can have release builds of the documentation and a “latest” build that tracks the current state of the documentation in GitHub. The prose docs are now no longer part of the installer package, instead the installer simply adds a shortcut to the online documentation. This also means that we have the ability to patch / fix documentation after a release goes out more easily.
Packaging Differences: The commercial release had 4 different packages representing different licensing levels – this can now be simplified down to a single installer for BrightstarDB which installs the full suite including the mobile libraries and the SD-Share server. The other main difference is that we are no longer performing and obfuscation or IL merging on the assemblies. The main affect this has is that the third-party dependency assemblies are now included in the libs directory rather than being merged into BrightstarDB.dll. On the downside it does mean that you need to copy three or four assemblies rather than one, but on the up-side it means that you aren’t necessarily tied to the versions we ship if a critical bug-fix comes along. I’ve also changed our NuGet package definitions so that the dependencies are retrieved from NuGet automatically.
Release Schedule: The 1.3 release is more or less ready to go. We are just waiting on word from the dotNetRDF guys on getting a new release that incorporates a patch I provided for a SPARQL query bug. I want to release with that fix as the bug breaks a number of our LINQ unit tests. Hopefully that dotNetRDF release should be forthcoming soon!
The other area of progress has been around controlling BrightstarDB’s memory usage during import. Graham and I have managed to track down the main culprit for memory usage getting out of control when importing larger files and I’ve started working on a fix for this. The initial work looks promising, I’ve managed to get the total memory allocation during an import down from over 6.5GB to just over 2GB (this is total #bytes allocated over the course of the import, actual peak memory usage is lower than this). In doing this work I’ve also come across part of the code that can be drastically simplified to remove an unnecessary layer of caching and indirection. As this is in the core B-Tree code my expectation is that this will help not only control memory usage during import but also go some way to helping control memory usage during query / read operations too.