5 ms·

When the application would be restarted, reading back the log would recreate the in memory data structures. I thought that it was cool, and that databases thems

When the application would be restarted, reading back the log would recreate the in memory data structures. I thought that it was cool, and that databases themselves could be that way instead of using the programming language data structures without a networked API. That is literally how databases work. In Memory + WAL + Data Files on disk. You could, in theory, live without the Data Files and just a big WAL.

kijin 8y ago
Except that most databases don't store their data in anything resembling "programming language data structures". You get tables, rows, and columns (or maybe a bit of JSON if you're lucky) instead of native integers, strings, lists, sets, and dictionaries.
- threeseed 8y ago
  Object and document databases have been around for decades. Likewise ORMs which allow for higher order types etc. have been around since WebObjects i.e. also decades.
  - kijin 8y ago
    The primary purpose of an ORM is to overcome the "impedance mismatch" between relational databases and programming language data structures. There's no need for an ORM if you can store your data structures directly in the database.
    - TeMPOraL 8y ago
      If this is a primary purpose of an ORM, then I wish I knew of one that isn't utterly failing at that. One thing I've learned about this "impedance mismatch" is that it isn't a syntax thing, it's a fundamental difference in the way of viewing the world. The way you store data about the world is different from the way you model that world dynamically, with objects. I find it safer to always split out the "business model" from the storage layer, so that those different views don't interfere - and once you do that, you may as well implement the storage layer in a relational way.
      - icebraining 8y ago
        .. and the code you implement to connect that business layer to that storage layer is an ORM. The idea that the ORM forces the storage layer to a particular representation of the business layer is only true if they implement the ActiveRecord pattern, which isn't universal.
    - neokantian 8y ago
      > ... if you can store your data structures directly in the database. ABSTRACT. Future users of large data banks must be protected from having to know how the data is organized in the machine (the internal representation). It provides a means of describing data with its natural structure only—that is, without superimposing any additional structure for machine representation purposes. Accordingly, it provides a basis for a high level data language which will yield maximal independence between programs on the one hand and machine representation and organization of data on the other. E.F. Codd. 1970. A relational model of data for large shared data banks. Commun. ACM 13, 6 (June 1970), 377-387. You propose to reintroduce a problem that they absolutely wanted to get rid off 40 years ago. Just imagine that you first have to figure out how to painstakingly parse serialized Python dictionaries before you can access the data in another program written in e.g. Rust. It clearly amounts to UNSOLVING a problem that is now SOLVED ALREADY.
      - kijin 8y ago
        Well, Redis allows me to store a JavaScript array, add some more with a Ruby client, remove some items with a PHP client, and finally read it back as a Python list just fine. What's the problem that has been unsolved? :)
        neokantian 8y ago
        https://msgpack.org https://msgpack.org is absolutely fine. I wasn't criticizing msgpack. You can also serialize the bytes of an array of C structs, if you see what I mean.
    - deleted 8y ago
      [deleted]
    - mamcx 8y ago
      That is an angle. Other way to say it is that ORM is a workaround to the fact most languages are VERY poor at manipulate data. Exist 2 main reasons for the "impedance mismatch": - Paradigms. 2 different paradigms will be at odds. Example: Functional and OO. This is ok. - Limitations: The relational model is absolutely superior and more expressive at manipulate data than OO/Functional. You need A LOT of machinery to recover that power. This is not ok. However, this not change the fact that OO is ok.Similar how a KV store is fine, but certainly, a RDBMS store is much more capable.
      - heavenlyblue 8y ago
        How is relational more powerful than OO/Functional if you can’t define an infinite dataset with it?
        mamcx 8y ago
        You can't? Most people only see the relational model as is inside a RDBMS. That is how judge the OO for what you can do on Mongo. RDBMS have some weird and well considered restrictions for their case. But read a little about the relational model, and none of it depend on SQL or say anything about how is the storage. P.D: Is important to note for where is MORE powerful. Remember that the creator of Pascal say: https://en.wikipedia.org/wiki/Algorithms_%2B_Data_Structures_%3D_Programs https://en.wikipedia.org/wiki/Algorithms_%2B_Data_Structures... Algorithms + Data Structures = Programs You can say OO/Functional lean more to the "Algorithms" side but RM to the "Data Structures". OO/Functional not say much how operate on data, most is a exercise for the reader. Instead, RM give a clear answer and defined operations for that. You need to "spice up" things to make the one be more useful to the other part of the equation. RM, too pure, certainly is incredible limited, not even can print "hello world!", but that is just too give a solution about how transform data...
      - notduncansmith 8y ago
        > Other way to say it is that ORM is a workaround to the fact most languages are VERY poor at manipulate data. This is why I love Clojure (and a particular style of Javascript). Destructuring and a good library of object/array manipulation functions make an environment well-suited to transforming data structures (which is precisely what I want to do in most programs I write), and I find I do not need an ORM where this type of data-focused programming is supported.
- jinjin2 8y ago
  It does if you use an object database like Realm.
elvinyung 8y ago
I think you're slightly missing the point -- for me, a certain unique "awesomeness" lies in specifically being able to literally pipe stdout back to stdin and get back the exact same data structures.
- snr 8y ago
  Can be easily done by printing out the stdin to stdout in the first place?
  - elvinyung 8y ago
    ok, without doing that :p
Illniyar 8y ago
Relational databases (except maybe MemQL) treat the file system as the source of truth. And usually the file system is bigger then the memory so they need to constantly update their cache with more relevant data. Redis loads everything to memory. And doesn't keep the structure in the file system, only the log, recreating it from log+snapshot.

kijin 8y ago

Except that most databases don't store their data in anything resembling "programming language data structures". You get tables, rows, and columns (or maybe a bit of JSON if you're lucky) instead of native integers, strings, lists, sets, and dictionaries.

threeseed 8y ago
Object and document databases have been around for decades. Likewise ORMs which allow for higher order types etc. have been around since WebObjects i.e. also decades.
- kijin 8y ago
  The primary purpose of an ORM is to overcome the "impedance mismatch" between relational databases and programming language data structures. There's no need for an ORM if you can store your data structures directly in the database.
  - TeMPOraL 8y ago
    If this is a primary purpose of an ORM, then I wish I knew of one that isn't utterly failing at that. One thing I've learned about this "impedance mismatch" is that it isn't a syntax thing, it's a fundamental difference in the way of viewing the world. The way you store data about the world is different from the way you model that world dynamically, with objects. I find it safer to always split out the "business model" from the storage layer, so that those different views don't interfere - and once you do that, you may as well implement the storage layer in a relational way.
    - icebraining 8y ago
      .. and the code you implement to connect that business layer to that storage layer is an ORM. The idea that the ORM forces the storage layer to a particular representation of the business layer is only true if they implement the ActiveRecord pattern, which isn't universal.
  - neokantian 8y ago
    > ... if you can store your data structures directly in the database. ABSTRACT. Future users of large data banks must be protected from having to know how the data is organized in the machine (the internal representation). It provides a means of describing data with its natural structure only—that is, without superimposing any additional structure for machine representation purposes. Accordingly, it provides a basis for a high level data language which will yield maximal independence between programs on the one hand and machine representation and organization of data on the other. E.F. Codd. 1970. A relational model of data for large shared data banks. Commun. ACM 13, 6 (June 1970), 377-387. You propose to reintroduce a problem that they absolutely wanted to get rid off 40 years ago. Just imagine that you first have to figure out how to painstakingly parse serialized Python dictionaries before you can access the data in another program written in e.g. Rust. It clearly amounts to UNSOLVING a problem that is now SOLVED ALREADY.
    - kijin 8y ago
      Well, Redis allows me to store a JavaScript array, add some more with a Ruby client, remove some items with a PHP client, and finally read it back as a Python list just fine. What's the problem that has been unsolved? :)
      - neokantian 8y ago
        https://msgpack.org https://msgpack.org is absolutely fine. I wasn't criticizing msgpack. You can also serialize the bytes of an array of C structs, if you see what I mean.
  - deleted 8y ago
    [deleted]
  - mamcx 8y ago
    That is an angle. Other way to say it is that ORM is a workaround to the fact most languages are VERY poor at manipulate data. Exist 2 main reasons for the "impedance mismatch": - Paradigms. 2 different paradigms will be at odds. Example: Functional and OO. This is ok. - Limitations: The relational model is absolutely superior and more expressive at manipulate data than OO/Functional. You need A LOT of machinery to recover that power. This is not ok. However, this not change the fact that OO is ok.Similar how a KV store is fine, but certainly, a RDBMS store is much more capable.
    - heavenlyblue 8y ago
      How is relational more powerful than OO/Functional if you can’t define an infinite dataset with it?
      - mamcx 8y ago
        You can't? Most people only see the relational model as is inside a RDBMS. That is how judge the OO for what you can do on Mongo. RDBMS have some weird and well considered restrictions for their case. But read a little about the relational model, and none of it depend on SQL or say anything about how is the storage. P.D: Is important to note for where is MORE powerful. Remember that the creator of Pascal say: https://en.wikipedia.org/wiki/Algorithms_%2B_Data_Structures_%3D_Programs https://en.wikipedia.org/wiki/Algorithms_%2B_Data_Structures... Algorithms + Data Structures = Programs You can say OO/Functional lean more to the "Algorithms" side but RM to the "Data Structures". OO/Functional not say much how operate on data, most is a exercise for the reader. Instead, RM give a clear answer and defined operations for that. You need to "spice up" things to make the one be more useful to the other part of the equation. RM, too pure, certainly is incredible limited, not even can print "hello world!", but that is just too give a solution about how transform data...
    - notduncansmith 8y ago
      > Other way to say it is that ORM is a workaround to the fact most languages are VERY poor at manipulate data. This is why I love Clojure (and a particular style of Javascript). Destructuring and a good library of object/array manipulation functions make an environment well-suited to transforming data structures (which is precisely what I want to do in most programs I write), and I find I do not need an ORM where this type of data-focused programming is supported.
jinjin2 8y ago
It does if you use an object database like Realm.

elvinyung 8y ago

I think you're slightly missing the point -- for me, a certain unique "awesomeness" lies in specifically being able to literally pipe stdout back to stdin and get back the exact same data structures.

snr 8y ago
Can be easily done by printing out the stdin to stdout in the first place?
- elvinyung 8y ago
  ok, without doing that :p

Illniyar 8y ago

Relational databases (except maybe MemQL) treat the file system as the source of truth. And usually the file system is bigger then the memory so they need to constantly update their cache with more relevant data. Redis loads everything to memory. And doesn't keep the structure in the file system, only the log, recreating it from log+snapshot.