3 ms·

Yes, we need this - we need more people exploring and building databases. The more people we have tackling interesting problems the better solutions we will dev

Yes, we need this - we need more people exploring and building databases. The more people we have tackling interesting problems the better solutions we will develop. No, building a database is not hard. We need to stop propagating this message, because it deters newcomers from trying. And the less people that try, the more of an echo chamber we get (like everybody building on top of Postgres again and again), with no new ideas. This comment on another current HN homepage article sums it up the best: https://news.ycombinator.com/item?id=13933572 https://news.ycombinator.com/item?id=13933572 . I'll share my story here, since I seem to be an outlier: - In less than 2 years from building a database from scratch (no forking other database), we had a prototype that could save 100M records for $10/day: https://youtu.be/x_WqBuEA7s8 https://youtu.be/x_WqBuEA7s8 . - In 2 years we had a system that can outperform Redis on the same test on the same hardware: https://github.com/amark/gun/wiki/100000-ops-sec-in-IE6-on-2GB-Atom-CPU https://github.com/amark/gun/wiki/100000-ops-sec-in-IE6-on-2... - Within 2.5 years we've built a system that can do on low end hardware across a federated system 1750 table inserts a second (we hope to get this up to 10K inserts/sec across a distributed system), a slower demo here: https://www.dropbox.com/s/amjlr5gqk23et51/load.gif?dl=0 https://www.dropbox.com/s/amjlr5gqk23et51/load.gif?dl=0 - To do this, we even had to build a fully featured distributed "coordination testing" framework so others can build tests that verify things work even in the harshest of conditions (it is for doing Jepsen like testing), check it out here: https://github.com/gundb/panic-server https://github.com/gundb/panic-server - And we've managed this without locking the data to any particular model. Graphs let you do key/value, relational, table, document, or graph type data. This story is important, because it shows what can happen when people just try. Referencing the quote from the other thread: Ivan Sutherland replied, "Well, I didn't know it was hard."

polskibus 10y ago
Slightly offtopic, but I was wondering what is the paradigm at the core of gunjs? Is it CRDTs or operational transform? I'd like to understand the theory part before I dive in. I'd be grateful for some pointers.
- marknadal 10y ago
  Great question, CRDT is the answer, as OT requires centralized servers and gun is decentralized/federated/P2P. Here is a really good (but long) overview of what is going on, including the tradeoffs of where gun does not ideally fit: http://gun.js.org/distributed/matters.html http://gun.js.org/distributed/matters.html A quick glance of the CAP Theorem tradeoffs: https://github.com/amark/gun/wiki/CAP-Theorem https://github.com/amark/gun/wiki/CAP-Theorem Some lower level implementation details of the CRDT: https://github.com/amark/gun/wiki/Conflict-Resolution-with-Guns https://github.com/amark/gun/wiki/Conflict-Resolution-with-G... It is a hybrid vector/timestamp/lexical CRDT, which means it is very naive but has deterministic guarantees. It is a "stupid" enough base CRDT that you can actually build other CRDTs on top of it by combining it with other data structures, to build DAGs (directed acyclic graphs, like git), Merkle trees, blockchains, linked lists, etc. which is where most of your "business logic" value/use-case is going to come from. Which means, it is powerful enough you can build OT-like alternatives on top, here is a demo of collaborative text editing: https://youtu.be/rci89p0o2wQ https://youtu.be/rci89p0o2wQ (note this is an old demo, I'm able to delete text now, and do other things) Anything else I can answer? Great question.
mamcx 10y ago
Ok, but where is the material to learn from? How see the "baby steps"? With compilers you have a lot to get from, but database building are more hard to find. -- One of the problems is that "databases" are assumed to be a ALL-OR-NOTHING system, with a lot of parts (like storage, transactions, invisible-indexes, triggers, cross-table validations, crash-recovery, query-optimizer) so is assumed a huge undertaking, instead of something more small and contained. -- BTW, I'm on the hunt because I wish to build a relational language (alike kdb+ but with more "normal" syntax), for build the same kind of apps as with c#, python, etc. I only need on-memory processing and not storage or transactions (or maybe transactions? I don't see how practical it could be for normal coding)
- marknadal 10y ago
  This is a good point, there is not much good/accessible material on databases - because a lot of that material is considered academic. Honestly, I learned mostly by experience of what frustrated me with current databases and then trying to come up with workaround solutions. Then I practiced by playing around with new ideas and making small prototypes in code. And then I would whiteboard out all the possible combinations of those ideas. Once I had all combinations listed, I reduced them all into their generic rules and which ones depended upon others. Finally, based off those constraints I worked out the mathematics of the system, the constraints, and tradeoffs. Maybe I should write a more intensive article on this to help other people. In the meantime, I think if anybody can read/watch classes on data structures, algorithms, etc. that will benefit them the most. Recently, we have worked on a some interactive animated explainers. Not quite database stuff, more like security (cryptography) and algorithms / data structures. Let me know if they are helpful: Cartoon Cryptography - http://gun.js.org/explainers/data/security.html http://gun.js.org/explainers/data/security.html Cartoon Conflict - http://gun.js.org/explainers/conflict/lexical.html http://gun.js.org/explainers/conflict/lexical.html Cartoon Sorting Algorithms - http://gun.js.org/explainers/basketball/basketball.html http://gun.js.org/explainers/basketball/basketball.html Cartoon Concurrency - http://gun.js.org/distributed/matters.html http://gun.js.org/distributed/matters.html And hopefully more to come. :) I know those are probably not immediately useful for your kdb+ like language. But, I'd love to learn more about what you are trying to do. :D
assface 10y ago
> No, building a database is not hard. Cool. Do you support existing reporting tools and libraries? What about standard APIs for data discovery (i.e., INFORMATION_SCHEMA)? What about transactions? What about tools for examining log files for when your checkpoints get corrupted?

polskibus 10y ago

Slightly offtopic, but I was wondering what is the paradigm at the core of gunjs? Is it CRDTs or operational transform? I'd like to understand the theory part before I dive in. I'd be grateful for some pointers.

marknadal 10y ago
Great question, CRDT is the answer, as OT requires centralized servers and gun is decentralized/federated/P2P. Here is a really good (but long) overview of what is going on, including the tradeoffs of where gun does not ideally fit: http://gun.js.org/distributed/matters.html http://gun.js.org/distributed/matters.html A quick glance of the CAP Theorem tradeoffs: https://github.com/amark/gun/wiki/CAP-Theorem https://github.com/amark/gun/wiki/CAP-Theorem Some lower level implementation details of the CRDT: https://github.com/amark/gun/wiki/Conflict-Resolution-with-Guns https://github.com/amark/gun/wiki/Conflict-Resolution-with-G... It is a hybrid vector/timestamp/lexical CRDT, which means it is very naive but has deterministic guarantees. It is a "stupid" enough base CRDT that you can actually build other CRDTs on top of it by combining it with other data structures, to build DAGs (directed acyclic graphs, like git), Merkle trees, blockchains, linked lists, etc. which is where most of your "business logic" value/use-case is going to come from. Which means, it is powerful enough you can build OT-like alternatives on top, here is a demo of collaborative text editing: https://youtu.be/rci89p0o2wQ https://youtu.be/rci89p0o2wQ (note this is an old demo, I'm able to delete text now, and do other things) Anything else I can answer? Great question.

mamcx 10y ago

Ok, but where is the material to learn from? How see the "baby steps"? With compilers you have a lot to get from, but database building are more hard to find. -- One of the problems is that "databases" are assumed to be a ALL-OR-NOTHING system, with a lot of parts (like storage, transactions, invisible-indexes, triggers, cross-table validations, crash-recovery, query-optimizer) so is assumed a huge undertaking, instead of something more small and contained. -- BTW, I'm on the hunt because I wish to build a relational language (alike kdb+ but with more "normal" syntax), for build the same kind of apps as with c#, python, etc. I only need on-memory processing and not storage or transactions (or maybe transactions? I don't see how practical it could be for normal coding)

marknadal 10y ago
This is a good point, there is not much good/accessible material on databases - because a lot of that material is considered academic. Honestly, I learned mostly by experience of what frustrated me with current databases and then trying to come up with workaround solutions. Then I practiced by playing around with new ideas and making small prototypes in code. And then I would whiteboard out all the possible combinations of those ideas. Once I had all combinations listed, I reduced them all into their generic rules and which ones depended upon others. Finally, based off those constraints I worked out the mathematics of the system, the constraints, and tradeoffs. Maybe I should write a more intensive article on this to help other people. In the meantime, I think if anybody can read/watch classes on data structures, algorithms, etc. that will benefit them the most. Recently, we have worked on a some interactive animated explainers. Not quite database stuff, more like security (cryptography) and algorithms / data structures. Let me know if they are helpful: Cartoon Cryptography - http://gun.js.org/explainers/data/security.html http://gun.js.org/explainers/data/security.html Cartoon Conflict - http://gun.js.org/explainers/conflict/lexical.html http://gun.js.org/explainers/conflict/lexical.html Cartoon Sorting Algorithms - http://gun.js.org/explainers/basketball/basketball.html http://gun.js.org/explainers/basketball/basketball.html Cartoon Concurrency - http://gun.js.org/distributed/matters.html http://gun.js.org/distributed/matters.html And hopefully more to come. :) I know those are probably not immediately useful for your kdb+ like language. But, I'd love to learn more about what you are trying to do. :D

assface 10y ago

> No, building a database is not hard. Cool. Do you support existing reporting tools and libraries? What about standard APIs for data discovery (i.e., INFORMATION_SCHEMA)? What about transactions? What about tools for examining log files for when your checkpoints get corrupted?