3 ms·

To be clear, it would depend on some (equally small) libraries, so it's not like 500 lines of no dependency C or something

_a_a_a_ 3y ago
I think what you're suggesting it is possible, to get halfway decent database using components, but I think you need to take a very high level view of these components. For a start, a parser. You almost certainly want a parser generator because that can be reused in other places. You need stuff to handle low level disc writes with logging, again, as a pluggable component (or maybe not, if you plan to shovel everything through your database [to clarify, that your DB effectively becomes the usual end to your filesystem, and everyone uses that, which isn't a terrible idea] in which case it's only got to be written once). You need an optimiser for the queries, even 'simple' ones. For this you might need a prolog-like facility, which again would be useful in other places. [to clarify, a prolog-like component system with some constraint+costs solver component built on top] You'll need a lot more than that as well. You will have to take a very high level view of things, ruthlessly componentised. I understand and endorse what you're trying to do, but a low level infrastructure with a bunch of libraries isn't quite going to cut it. Just my opinion anyway, take it as you will!
- vitiral 3y ago
  Thanks, I appreciate the thoughts. It's for the civboot.org project, so it has VERY constrained requirements. Basically it is just to build things like a VCS and log collector on. As I outlined in the post I'm only concerned with two requirements: appending mutation quickly and indexing static databases. If I add an in-memory index for recent writes that haven't yet been indexed then it pretty much meets all requirements. I don't plan on supporting SQL or anything like that -- it's a single-file database that Lua can (indexed) read and write to. Others can also trivially read the index as well, but concurrent writes are probably not going to be possible. Transactions are easy enough I think.
  - _a_a_a_ 3y ago
    Hi, then a database is not much more than an indexed file system (sort of). I'll check out your link in a bit more detail, thanks. I wasn't expecting it to be multiple user, I assumed single use only which seriously simplifies some aspects (lock the world, do stuff, unlock). To support transactions you'll need a write ahead log and recovery to preserve data through crashes/power loss – that's not trivial. Still, good luck. Regarding your processor, I like the idea of a very simple process are easy to produce. I haven't looked at what you're suggesting but perhaps instructions on making a full-sized one out of discrete components might be worthwhile? There are a few projects online (I haven't been able to find them but I seem to remember a 6502 made by hand the size of a mattress). While this would only run at a few kilohertz, it's better than no computer at all and useful for bootstrapping to something smaller, and possible if you've got no VLSI or anything similar. An even simpler processor might be possible. Sorry if you've already considered this, it was just a thought.
    - vitiral 3y ago
      Ya, basically just a file you dump things at in incrementing indexes and a index file to quickly look up keys. Your other comments are the kinds of thoughts I approach this with. I'm hoping for the final (simple fab) CPU @100MHz and at least a few MiB of RAM (I think about 1990's tech). I did the pre-work in thinking about the CPU and hardware to convince myself of what features were needed. Eventually I'll get to building the CPU itself. I first want the OS it will target :)

_a_a_a_ 3y ago

I think what you're suggesting it is possible, to get halfway decent database using components, but I think you need to take a very high level view of these components. For a start, a parser. You almost certainly want a parser generator because that can be reused in other places. You need stuff to handle low level disc writes with logging, again, as a pluggable component (or maybe not, if you plan to shovel everything through your database [to clarify, that your DB effectively becomes the usual end to your filesystem, and everyone uses that, which isn't a terrible idea] in which case it's only got to be written once). You need an optimiser for the queries, even 'simple' ones. For this you might need a prolog-like facility, which again would be useful in other places. [to clarify, a prolog-like component system with some constraint+costs solver component built on top] You'll need a lot more than that as well. You will have to take a very high level view of things, ruthlessly componentised. I understand and endorse what you're trying to do, but a low level infrastructure with a bunch of libraries isn't quite going to cut it. Just my opinion anyway, take it as you will!

vitiral 3y ago
Thanks, I appreciate the thoughts. It's for the civboot.org project, so it has VERY constrained requirements. Basically it is just to build things like a VCS and log collector on. As I outlined in the post I'm only concerned with two requirements: appending mutation quickly and indexing static databases. If I add an in-memory index for recent writes that haven't yet been indexed then it pretty much meets all requirements. I don't plan on supporting SQL or anything like that -- it's a single-file database that Lua can (indexed) read and write to. Others can also trivially read the index as well, but concurrent writes are probably not going to be possible. Transactions are easy enough I think.
- _a_a_a_ 3y ago
  Hi, then a database is not much more than an indexed file system (sort of). I'll check out your link in a bit more detail, thanks. I wasn't expecting it to be multiple user, I assumed single use only which seriously simplifies some aspects (lock the world, do stuff, unlock). To support transactions you'll need a write ahead log and recovery to preserve data through crashes/power loss – that's not trivial. Still, good luck. Regarding your processor, I like the idea of a very simple process are easy to produce. I haven't looked at what you're suggesting but perhaps instructions on making a full-sized one out of discrete components might be worthwhile? There are a few projects online (I haven't been able to find them but I seem to remember a 6502 made by hand the size of a mattress). While this would only run at a few kilohertz, it's better than no computer at all and useful for bootstrapping to something smaller, and possible if you've got no VLSI or anything similar. An even simpler processor might be possible. Sorry if you've already considered this, it was just a thought.
  - vitiral 3y ago
    Ya, basically just a file you dump things at in incrementing indexes and a index file to quickly look up keys. Your other comments are the kinds of thoughts I approach this with. I'm hoping for the final (simple fab) CPU @100MHz and at least a few MiB of RAM (I think about 1990's tech). I did the pre-work in thinking about the CPU and hardware to convince myself of what features were needed. Eventually I'll get to building the CPU itself. I first want the OS it will target :)