3 ms·

If you're still reading this: I'd want to see experience on your team with database internals. 25 years experience making a CMS makes you good at web apps, but

by gwbas1c 7mo ago

If you're still reading this: I'd want to see experience on your team with database internals. 25 years experience making a CMS makes you good at web apps, but not the kind of things that make a good database. It's a significantly lower-level skillset than CMS development. > Regarding scale, it will currently not scale to "Big Data" territory, that was never the intention. That's not the scalability that I'm referring to. A "binary append-only log file with an in-memory index system" won't scale to the needs of a typical web app; because at some point either the lifetime of the app will make the file grow too large, or you'll run out of RAM for your indexes. This is well outside of "big data" territory. (IE, where I work the production SQL database is tiny, but we've been in business so long, and have had enough state change, that "a binary append-only log file with an in-memory index system" would have long failed.)

rettepelo 7mo ago
Still reading :-) Sure, this is our first attempt at building a database engine from scratch, and we have not done this before, but we enjoy learning as we go. In the end it is the product that matter, and we are not done, but so far it is meeting our own expectations. Having said that I would like to mention we also build ORM systems too (One was demonstrated by Microsoft in a MIX conference many years ago, OData), and as I mentioned the key aim here is to build a system that makes it easy to build Web Applications. The ambition is not a state of the art typical database engine. Ours is different in that it has a built in object level revision system, culture versions with fallback logic, file conversion ( media files ), persisted task queue, AI integration, external backups, native ORM, record and field level access control etc. Postgres and others have plugins where similar things can be achieved, but it is not available as one cohesive package. ( BTW, Relatude.DB has MIT license ) Regarding the binary append only file format. It has its pros and cons, and we certainly choose it is easier to implement reliably. However, there are some implementation details I like to point out. First, it is not appending indefinitely, it is periodically rebuilding a second file in the background, that only contains the latest state. ( A bit like the VACUUM function of other dbs ) It switches silently to this file in the background to keep the disk use down. How often this happens is something you can configure, and depends of how much you value a full transaction history vs disk use. ( It also have support for streaming to a second "infinite" transaction log on a cheap and slow storage like blobstore ). Every write to the transaction log is a complete object serialization, ( like most document databases ). In memory the system keeps a record of where in the log file the latest object is serialized and if the object is not in the cache it reads from the log file. The register of object locations are regularly serialized to state files on disc for a faster startup. An advantage with this is that inserts are written only once to disc in a linear fashion, making it very fast for inserts. A disadvantage is the memory needed to keep track of where every object is located in the log file, but it is only a long variable per object and it is not that bad. Regarding the indexes it is using a plugin system and you have several options. It is configurable on a per system and per field level. Indexes can either be memory based, ( hashmaps, btrees, trie etc ). Or they can be disc bases to save memory at the cost of some performance loss. Currently we have plugins for SQLite and Lucene, and will probably offer other options later, or you can write your own. Using only disc based indexes reduces memory use to a minimum but at the cost of performance. The ORM is central to the system, classes and mappers are dynamically compiled at startup to maximize performance. Having it all integrated gives us certain advantages with simplicity and performance optimization compared to combining different systems into a common API. Data integrity is also easier when it is one system. For example you can upload a file, change a relation, subtracting a int value to a field, all in a single transaction that either validates and is confirmed written to disc or fails cleanly. Sorry, this became a rather long post... it is a complicated topic where details matter and a lot more could be said. In any case, I agree db engines are both fun and challenging.

by gwbas1c 7mo ago

rettepelo 7mo ago

Still reading :-) Sure, this is our first attempt at building a database engine from scratch, and we have not done this before, but we enjoy learning as we go. In the end it is the product that matter, and we are not done, but so far it is meeting our own expectations. Having said that I would like to mention we also build ORM systems too (One was demonstrated by Microsoft in a MIX conference many years ago, OData), and as I mentioned the key aim here is to build a system that makes it easy to build Web Applications. The ambition is not a state of the art typical database engine. Ours is different in that it has a built in object level revision system, culture versions with fallback logic, file conversion ( media files ), persisted task queue, AI integration, external backups, native ORM, record and field level access control etc. Postgres and others have plugins where similar things can be achieved, but it is not available as one cohesive package. ( BTW, Relatude.DB has MIT license ) Regarding the binary append only file format. It has its pros and cons, and we certainly choose it is easier to implement reliably. However, there are some implementation details I like to point out. First, it is not appending indefinitely, it is periodically rebuilding a second file in the background, that only contains the latest state. ( A bit like the VACUUM function of other dbs ) It switches silently to this file in the background to keep the disk use down. How often this happens is something you can configure, and depends of how much you value a full transaction history vs disk use. ( It also have support for streaming to a second "infinite" transaction log on a cheap and slow storage like blobstore ). Every write to the transaction log is a complete object serialization, ( like most document databases ). In memory the system keeps a record of where in the log file the latest object is serialized and if the object is not in the cache it reads from the log file. The register of object locations are regularly serialized to state files on disc for a faster startup. An advantage with this is that inserts are written only once to disc in a linear fashion, making it very fast for inserts. A disadvantage is the memory needed to keep track of where every object is located in the log file, but it is only a long variable per object and it is not that bad. Regarding the indexes it is using a plugin system and you have several options. It is configurable on a per system and per field level. Indexes can either be memory based, ( hashmaps, btrees, trie etc ). Or they can be disc bases to save memory at the cost of some performance loss. Currently we have plugins for SQLite and Lucene, and will probably offer other options later, or you can write your own. Using only disc based indexes reduces memory use to a minimum but at the cost of performance. The ORM is central to the system, classes and mappers are dynamically compiled at startup to maximize performance. Having it all integrated gives us certain advantages with simplicity and performance optimization compared to combining different systems into a common API. Data integrity is also easier when it is one system. For example you can upload a file, change a relation, subtracting a int value to a field, all in a single transaction that either validates and is confirmed written to disc or fails cleanly. Sorry, this became a rather long post... it is a complicated topic where details matter and a lot more could be said. In any case, I agree db engines are both fun and challenging.