4 ms·

Agreed. Reaching back into my memory banks. When bigtable first became available for app engine I was fascinated by the prospect of working with Google scale ca

Agreed. Reaching back into my memory banks. When bigtable first became available for app engine I was fascinated by the prospect of working with Google scale capable database. But there were so many odd hard to reason about vagaries of squeezing that performance out of the database by means of key structure. The developer ergonomics seemed simple at first but in the end were not at all what I expected.

ddorian43 9y ago
.. is that magic that you want in any other product ? the `simplest` I know are redis commands that show the taken.
- foobarchu 9y ago
  To me, it's not about magic, it's about the abstraction. Most implementations of this style of database demand that you consider internals of the engine before storing anything. If you don't design your keys the right way, then you're screwed down the road, and there's no way to fix it without just copying the data to a new table (and losing metadata like TTL). When I use something like Redis, or any SQL databse, I don't need to consider the inner workings of it unless I'm trying to absolutely max out my performance (one slight exception being atomicity of Redis commands). But with LSM style databases, you are going to have a really bad time if you don't have a team member who has dedicated serious time to understanding the internal workings of LSM itself, and the details of your chosen implementation. That's a real mark against it, IMO. tldr; LSM databases are like a colander in the world of leaky abstractions.
  - ddorian43 9y ago
    There is MyRocks, you work like usual mysql. When you want distrubted-db, yeah you need to design. But comparing RocksDB vs LMDB, the second is easier but not by much ? Think citusdb without consideration you're gonna expect good performance in sharded environment?
    - foobarchu 9y ago
      > When you want distrubted-db, yeah you need to design But one shouldn't be designing for implementation details. Usually, we start technologies out with leaky abstractions, and gradually get better at it. A good example is game development, where it used to be that you always used the drawing method of the display and the clock speed of the cpu to your advantage. Nowadays, we've moved past that, because it was working on horrible abstractions, and because the technology underneath improved. I'm not saying I have a solution, and I agree that this problem rears it's head the most when you start bringing in distributed storage. But my point stands: these databases run on a highly leaky abstraction, and that's a big problem going forward.
      - ddorian43 9y ago
        Cost of message passing. It's probably cause you don't care `enough` for performance. See the difference in 1 box of `1 process per core` `scylladb`,`voltdb`,`redis` compared to all other dbs `1-process-for-all-cores`.

ddorian43 9y ago

.. is that magic that you want in any other product ? the `simplest` I know are redis commands that show the taken.

foobarchu 9y ago
To me, it's not about magic, it's about the abstraction. Most implementations of this style of database demand that you consider internals of the engine before storing anything. If you don't design your keys the right way, then you're screwed down the road, and there's no way to fix it without just copying the data to a new table (and losing metadata like TTL). When I use something like Redis, or any SQL databse, I don't need to consider the inner workings of it unless I'm trying to absolutely max out my performance (one slight exception being atomicity of Redis commands). But with LSM style databases, you are going to have a really bad time if you don't have a team member who has dedicated serious time to understanding the internal workings of LSM itself, and the details of your chosen implementation. That's a real mark against it, IMO. tldr; LSM databases are like a colander in the world of leaky abstractions.
- ddorian43 9y ago
  There is MyRocks, you work like usual mysql. When you want distrubted-db, yeah you need to design. But comparing RocksDB vs LMDB, the second is easier but not by much ? Think citusdb without consideration you're gonna expect good performance in sharded environment?
  - foobarchu 9y ago
    > When you want distrubted-db, yeah you need to design But one shouldn't be designing for implementation details. Usually, we start technologies out with leaky abstractions, and gradually get better at it. A good example is game development, where it used to be that you always used the drawing method of the display and the clock speed of the cpu to your advantage. Nowadays, we've moved past that, because it was working on horrible abstractions, and because the technology underneath improved. I'm not saying I have a solution, and I agree that this problem rears it's head the most when you start bringing in distributed storage. But my point stands: these databases run on a highly leaky abstraction, and that's a big problem going forward.
    - ddorian43 9y ago
      Cost of message passing. It's probably cause you don't care `enough` for performance. See the difference in 1 box of `1 process per core` `scylladb`,`voltdb`,`redis` compared to all other dbs `1-process-for-all-cores`.