5 ms·

Thank you for sharing My perspective is that writing this kind of system in a language such as Python is actually a great thing because for myself Python is mo

Thank you for sharing My perspective is that writing this kind of system in a language such as Python is actually a great thing because for myself Python is more widely readable and approachable compared to C++ or C which is what databases are often programmed in. If someone wants to be serious they can port it to a low level language . As it stands it's educational and useful for studying. I wrote a distributed pseudo multi model SQL/graph Cypher/Document and dyanmodb style database in Python https://GitHub.com/samsquire/hash-db https://GitHub.com/samsquire/hash-db for the same goal of learning how database engines could work in a distributed way.

devman0 3y ago
This is why you see a community of pure Java RDBMSs (Hypersonic, H2, Derby, etc), if you don't need the big iron scale it's easier to ship/use the database or even embed it in memory if needed.
- fuzztester 3y ago
  Yes. It's an interesting area, irrespective of implementation language. I had tried out one or two Java-based RDBMSes back in the day, via programs written in Java, for fun. I think one was HSQLDB. https://en.m.wikipedia.org/wiki/HSQLDB https://en.m.wikipedia.org/wiki/HSQLDB There was also another interesting one called PointBase, which was developed by Bruce Scott, an Oracle founder, and others. https://en.m.wikipedia.org/wiki/PointBase https://en.m.wikipedia.org/wiki/PointBase
  - gabereiser 3y ago
    H2 is pretty great and fully supports JSONB and the like [1]. Full java-only RDBMS. [1]: https://www.h2database.com/html/main.html https://www.h2database.com/html/main.html
    - p2detar 3y ago
      H2 is pretty solid. I used to ship a product with it as a default database. Customers could optionally use MySQL, MSSQL or Oracle, but for small businesses H2 was more than enough.
    - fuzztester 3y ago
      Thanks, sounds worth checking out.
- lopatin 3y ago
  H2 has quite heavy use in production too. For example Apache Ignite uses it as the core of it's distributed SQL engine. Ignite takes a distributed SQL query and splits it up into single node queries, which are executed on each node by H2, and then Ignite merges the results back.
spanspan 3y ago
Cool stuff. I had similar intuitions- Python allow me to focus on the high-level concepts. Albeit, there were times where I wished I had gone with a statically-typed + compiled language.
p4bl0 3y ago
I totally agree with that. I loved the ugit [1] build Git from scratch in Python series for that. [1] https://www.leshenko.net/p/ugit/ https://www.leshenko.net/p/ugit/
- tmh88j 3y ago
  Thanks for sharing. This looks like another fun project to work through over a weekend.
crabbone 3y ago
Idk. Python is just as bad as C / C++, with the downside that you cannot do much of the interesting stuff one would need to if they wanted to learn how to make databases, if they use Python. Both C and Python are very "approachable" if you ignore the bad language design, inconsistencies and other "gotchas" and only take the "easy" parts, disregarding edge cases. However, with C you could at least have a fighting chance to learn how to do things right, but with Python you'll never even know what the real thing is like.
- Eddygandr 3y ago
  > but with Python you'll never even know what the real thing is like What do you mean? Surely the concepts are similar regardless of using Python or C?
  - dumpsterdiver 3y ago
    The concepts and syntax are similar, but Python will always be "riding upon a horse", whereas C can be the horse.
  - crabbone 3y ago
    Not even close. The real problems you will have to solve when working with a database anyone would want to use are, for example: * Memory layout of the buffers used to write / read / cache this data. In Python, you don't even have a concept of memory alignment / layout -- it's all happening somewhere in the interpreter. * How to best service multiple requests concurrently. Again, Python offers nothing here, and nothing to experiment with. But this is a huge part of working with databases. The whole two-stage commit, transaction, consistency guarantees -- it's the central point of databases, but Python gives you no tools to even try anything like that. * Working with various underlying storage... most of it has no Python interface (but does have C interface). Eg. if you want to understand how to optimize storing data by using some filesystem -- those filesystems do often expose similar interfaces that go beyond VFS, but they won't be immediately available to Python. * Working with vectorization of queries -- again, Python doesn't have a concept of ISA, doesn't have any way to instruct the code to utilize any particular CPU instructions... but this is where a lot of work is done by people who work on real databases. Not being able to get any meaningful access to query optimization, planning becomes pointless / useless -- what are your concerns going to be when you write a query planner if you still have no idea how it's going to be executed? * Similar stuff goes for networking -- whenever you need to solve something that goes beyond the absolute trivial you will at best rely on Python bindings to some library that actually does that rather than on Python code proper. In other words, if your goal was to learn how to do it, you will not achieve that goal because the actual work will happen elsewhere. So... I don't know... there is no way you can really learn how to make databases in Python. You can probably learn something, depending on what's your baseline, but you will not be ready to make a real thing, if all you have is Python. It's a difference between toy doctor set and learning to be a surgeon...

devman0 3y ago

This is why you see a community of pure Java RDBMSs (Hypersonic, H2, Derby, etc), if you don't need the big iron scale it's easier to ship/use the database or even embed it in memory if needed.

fuzztester 3y ago
Yes. It's an interesting area, irrespective of implementation language. I had tried out one or two Java-based RDBMSes back in the day, via programs written in Java, for fun. I think one was HSQLDB. https://en.m.wikipedia.org/wiki/HSQLDB https://en.m.wikipedia.org/wiki/HSQLDB There was also another interesting one called PointBase, which was developed by Bruce Scott, an Oracle founder, and others. https://en.m.wikipedia.org/wiki/PointBase https://en.m.wikipedia.org/wiki/PointBase
- gabereiser 3y ago
  H2 is pretty great and fully supports JSONB and the like [1]. Full java-only RDBMS. [1]: https://www.h2database.com/html/main.html https://www.h2database.com/html/main.html
  - p2detar 3y ago
    H2 is pretty solid. I used to ship a product with it as a default database. Customers could optionally use MySQL, MSSQL or Oracle, but for small businesses H2 was more than enough.
  - fuzztester 3y ago
    Thanks, sounds worth checking out.
lopatin 3y ago
H2 has quite heavy use in production too. For example Apache Ignite uses it as the core of it's distributed SQL engine. Ignite takes a distributed SQL query and splits it up into single node queries, which are executed on each node by H2, and then Ignite merges the results back.

spanspan 3y ago

Cool stuff. I had similar intuitions- Python allow me to focus on the high-level concepts. Albeit, there were times where I wished I had gone with a statically-typed + compiled language.

p4bl0 3y ago

I totally agree with that. I loved the ugit [1] build Git from scratch in Python series for that. [1] https://www.leshenko.net/p/ugit/ https://www.leshenko.net/p/ugit/

tmh88j 3y ago
Thanks for sharing. This looks like another fun project to work through over a weekend.

crabbone 3y ago

Idk. Python is just as bad as C / C++, with the downside that you cannot do much of the interesting stuff one would need to if they wanted to learn how to make databases, if they use Python. Both C and Python are very "approachable" if you ignore the bad language design, inconsistencies and other "gotchas" and only take the "easy" parts, disregarding edge cases. However, with C you could at least have a fighting chance to learn how to do things right, but with Python you'll never even know what the real thing is like.

Eddygandr 3y ago
> but with Python you'll never even know what the real thing is like What do you mean? Surely the concepts are similar regardless of using Python or C?
- dumpsterdiver 3y ago
  The concepts and syntax are similar, but Python will always be "riding upon a horse", whereas C can be the horse.
- crabbone 3y ago
  Not even close. The real problems you will have to solve when working with a database anyone would want to use are, for example: * Memory layout of the buffers used to write / read / cache this data. In Python, you don't even have a concept of memory alignment / layout -- it's all happening somewhere in the interpreter. * How to best service multiple requests concurrently. Again, Python offers nothing here, and nothing to experiment with. But this is a huge part of working with databases. The whole two-stage commit, transaction, consistency guarantees -- it's the central point of databases, but Python gives you no tools to even try anything like that. * Working with various underlying storage... most of it has no Python interface (but does have C interface). Eg. if you want to understand how to optimize storing data by using some filesystem -- those filesystems do often expose similar interfaces that go beyond VFS, but they won't be immediately available to Python. * Working with vectorization of queries -- again, Python doesn't have a concept of ISA, doesn't have any way to instruct the code to utilize any particular CPU instructions... but this is where a lot of work is done by people who work on real databases. Not being able to get any meaningful access to query optimization, planning becomes pointless / useless -- what are your concerns going to be when you write a query planner if you still have no idea how it's going to be executed? * Similar stuff goes for networking -- whenever you need to solve something that goes beyond the absolute trivial you will at best rely on Python bindings to some library that actually does that rather than on Python code proper. In other words, if your goal was to learn how to do it, you will not achieve that goal because the actual work will happen elsewhere. So... I don't know... there is no way you can really learn how to make databases in Python. You can probably learn something, depending on what's your baseline, but you will not be ready to make a real thing, if all you have is Python. It's a difference between toy doctor set and learning to be a surgeon...