5 ms·

> I wonder if I’ll ever be able to get this done. I can’t help but be extremely pessimistic. This is an insanely challenging project. Looking at the author’s o

by yashap 6y ago

> I wonder if I’ll ever be able to get this done. I can’t help but be extremely pessimistic. This is an insanely challenging project. Looking at the author’s other side projects: https://euandre.org/about.html https://euandre.org/about.html I’d guess this project is about 10,000x-100,000x as challenging as the Songbooks one. And the author has no obvious experience writing DBs, that I can see? This project seems at least as ambitious as CockroachDB. Development started on CockroachDB 6.5 years ago, was founded by a group of Google alums with relevant distributed systems experience (working on GFS), and it since became well funded and has had a sizeable team working on it full time for years. This project would be almost impossible to make significant progress against without a strong, well funded team working on it for years, but the author speaks about it as a side project.

marta_morena_25 6y ago
You are assuming that one needs to build a new database to achieve the requirements. That is unlikely. This sounds more like a glue-code project, where you combine different existing solutions, like Git, local SQL and probably a SQL server to form a whole, pre-made solution. What the author wants is implemented essentially by many many teams. Each time from scratch. All we need is someone baking this into a re-usable framework. That is STILL ambitious for a side project, but far from impossible.
StavrosK 6y ago
He doesn't have to start from scratch, he can implement syncing/atoms/whatever else on top of SQLite.
- yashap 6y ago
  I don't think so. For one, he wants it to run in the browser, and SQLite doesn't run in the browser. In his "embedded" section, he makes it pretty clear that he doesn't want to build on SQLite: > SQLite is a great example of that: it is a very powerful relational database that runs almost anywhere. What I miss from it that SQLite doesn’t provide is the ability to run it on the browser > Mentat was an interesting project, but its reliance on SQLite makes it inherit most of the downsides (and benefits too) of SQLite itself. Also, I'm sceptical you could make SQLite efficient for an immutable DB, IF you want to support full SQL. A DB like Datomic has to do some pretty crazy stuff to be remotely efficient, and it doesn't even try to support full SQL - a great read on Datomic internals: https://tonsky.me/blog/unofficial-guide-to-datomic-internals https://tonsky.me/blog/unofficial-guide-to-datomic-internals Mentat, the DB he links to, is much less ambitious than this proposal, and had a couple years of work by a number of devs before eventually being shuttered/abandoned. It's about 35K lines of Rust, despite being built on top of SQLite, isn't remotely done, doesn't run in the browser, etc. Even if you did build this proposed DB on top of another DB, to make it efficient, and have all of the features discussed, I strongly suspect that it would be an absolutely massive amount of work.
kieckerjan 6y ago
What does "10,000x-100,000x as challenging" mean? If the songbook cost him one week, this will cost him at least 200 years? I love a good hyperbole, but this is just disparaging.
- inopinatus 6y ago
  That could also be written as 40-50dB more challenging.
- brulard 6y ago
  A comma is being used as decimal separator in some locales...
- bufferoverflow 6y ago
  Most databases that we use are written by hundreds to thousands of people and take years to become reliable and mostly bug-free. It can easily take much more than 200 programmer-years.
  - kieckerjan 6y ago
    By that yardstick nobody should ever attempt to write an OS or programming language or game engine from scratch. I am glad there are plenty of individuals who try.
    - Quarrelsome 6y ago
      ye but be realistic. If you're just one dude that has no experience in creating operating systems then you shouldn't seriously try to write one from scratch. For giggles, sure go into the project with the idea that it will fail, that's not an issue. However if you're serious then what the poster is saying is that its over a lifetime's worth of work.
      - majewsky 6y ago
        > If you're just one dude that has no experience in creating operating systems then you shouldn't seriously try to write one from scratch. In the alternate reality following your dictum, Linux does not exist.
roenxi 6y ago
CockroachDB sounds like it has a target market of a couple of hundred to low thousands companies though. How many people are in the market for a "[A] SQL database for building global, scalable cloud services that survive disasters." [0]? It sounds like a pretty good product, lots of potential, etc. But there are not that many people who need to run their own cloud and maintain uptime in the event of a data centre being taken out. If I ever use CockroachDB it is probably because some cloud provider decided to offer it and not Postgres. Those core features are targeted at problems that only a very small subset of people have. Compare that to embedded/immutable/syncable/relational. Everybody in software has dealt with software where that would be a compelling feature set in a database. These are solutions to problems any developer has encountered. It is, in that sense, a much easier project for people to understand and get behind. OP probably doesn't have useful opinions on how his software should handle a direct nuclear strike on his production machine. He probably does have useful opinions on when he should cheat and let a little bit of mutability into the otherwise 'immutable' system. [0] https://www.cockroachlabs.com/docs/stable/ https://www.cockroachlabs.com/docs/stable/

by yashap 6y ago

marta_morena_25 6y ago

You are assuming that one needs to build a new database to achieve the requirements. That is unlikely. This sounds more like a glue-code project, where you combine different existing solutions, like Git, local SQL and probably a SQL server to form a whole, pre-made solution. What the author wants is implemented essentially by many many teams. Each time from scratch. All we need is someone baking this into a re-usable framework. That is STILL ambitious for a side project, but far from impossible.

StavrosK 6y ago

He doesn't have to start from scratch, he can implement syncing/atoms/whatever else on top of SQLite.

yashap 6y ago
I don't think so. For one, he wants it to run in the browser, and SQLite doesn't run in the browser. In his "embedded" section, he makes it pretty clear that he doesn't want to build on SQLite: > SQLite is a great example of that: it is a very powerful relational database that runs almost anywhere. What I miss from it that SQLite doesn’t provide is the ability to run it on the browser > Mentat was an interesting project, but its reliance on SQLite makes it inherit most of the downsides (and benefits too) of SQLite itself. Also, I'm sceptical you could make SQLite efficient for an immutable DB, IF you want to support full SQL. A DB like Datomic has to do some pretty crazy stuff to be remotely efficient, and it doesn't even try to support full SQL - a great read on Datomic internals: https://tonsky.me/blog/unofficial-guide-to-datomic-internals https://tonsky.me/blog/unofficial-guide-to-datomic-internals Mentat, the DB he links to, is much less ambitious than this proposal, and had a couple years of work by a number of devs before eventually being shuttered/abandoned. It's about 35K lines of Rust, despite being built on top of SQLite, isn't remotely done, doesn't run in the browser, etc. Even if you did build this proposed DB on top of another DB, to make it efficient, and have all of the features discussed, I strongly suspect that it would be an absolutely massive amount of work.

kieckerjan 6y ago

What does "10,000x-100,000x as challenging" mean? If the songbook cost him one week, this will cost him at least 200 years? I love a good hyperbole, but this is just disparaging.

inopinatus 6y ago
That could also be written as 40-50dB more challenging.
brulard 6y ago
A comma is being used as decimal separator in some locales...
bufferoverflow 6y ago
Most databases that we use are written by hundreds to thousands of people and take years to become reliable and mostly bug-free. It can easily take much more than 200 programmer-years.
- kieckerjan 6y ago
  By that yardstick nobody should ever attempt to write an OS or programming language or game engine from scratch. I am glad there are plenty of individuals who try.
  - Quarrelsome 6y ago
    ye but be realistic. If you're just one dude that has no experience in creating operating systems then you shouldn't seriously try to write one from scratch. For giggles, sure go into the project with the idea that it will fail, that's not an issue. However if you're serious then what the poster is saying is that its over a lifetime's worth of work.
    - majewsky 6y ago
      > If you're just one dude that has no experience in creating operating systems then you shouldn't seriously try to write one from scratch. In the alternate reality following your dictum, Linux does not exist.

roenxi 6y ago

CockroachDB sounds like it has a target market of a couple of hundred to low thousands companies though. How many people are in the market for a "[A] SQL database for building global, scalable cloud services that survive disasters." [0]? It sounds like a pretty good product, lots of potential, etc. But there are not that many people who need to run their own cloud and maintain uptime in the event of a data centre being taken out. If I ever use CockroachDB it is probably because some cloud provider decided to offer it and not Postgres. Those core features are targeted at problems that only a very small subset of people have. Compare that to embedded/immutable/syncable/relational. Everybody in software has dealt with software where that would be a compelling feature set in a database. These are solutions to problems any developer has encountered. It is, in that sense, a much easier project for people to understand and get behind. OP probably doesn't have useful opinions on how his software should handle a direct nuclear strike on his production machine. He probably does have useful opinions on when he should cheat and let a little bit of mutability into the otherwise 'immutable' system. [0] https://www.cockroachlabs.com/docs/stable/ https://www.cockroachlabs.com/docs/stable/