4 ms·

"Transparent" and "opaque" seem better terms than "good" and "bad". ORMs usually let you fall back to SQL if you need to (ActiveRecord does), so I would classi

by ntoshev 17y ago

"Transparent" and "opaque" seem better terms than "good" and "bad". ORMs usually let you fall back to SQL if you need to (ActiveRecord does), so I would classify them as transparent. Relational databases themselves are opaque though, they basically present you with a subset of relational algebra and how its operations will be carried out depends on a myriad of parameters in a non-trivial way. And you cannot possibly push some little modification in the inner loop. Another good example is Larry Page asking how to set a custom user-agent for what became the googlebot in the then-current-JDK (and having no way to do it): http://groups.google.com/group/comp.lang.java/browse_thread/thread/6923c024ed392c85/88fa10845061c8ba http://groups.google.com/group/comp.lang.java/browse_thread/...

andreyf 17y ago
Re: ORM/SQL, compare Django's ORM/MySQL to how PG implemented storage in news.YC - I've devoted several orders of magnitude more time working with the former, but I wouldn't be able to describe what happens in-hardware nearly as well as I could on the latter, which I read once (in a language I never seriously used) nearly a year ago.
- ntoshev 17y ago
  I don't have experience with your specific example, but it seems you are complaining against the opacity of relational databases, not the ORM itself. I agree, relational databases are very opaque.
  - andreyf 17y ago
    If an ORM user can be fairly confident fairly quickly about the SQL generated from each of his ORM calls, it's a good abstraction. In my experience with Django's ORM, that hasn't been the case. Fault may be mine :)
  - nwatson 17y ago
    RDBMSs aren't that opaque ... A relational database exists to store rows of data in tables and to relate them, making CRUD operations possible in this context. At that level it's very abstract. But it also starts to expose some implementation details as appropriate, and that's why some DBAs make a very good living (I'm not a DBA myself). Transactions are required once there can be more than one DB client (well, even before that when considering atomicity/durability). And most DBs expose different transaction isolation levels because the various options have different speed vs semantic reliability implications (e.g. dirty reads). These make sense only in light of lower-level concepts that leak through from the notion that data is stored on disk, buffered in memory, handled by threads, protected by locks, etc. And then there are details of indexing, clustering, distributing table data among available disks, where to store the transaction log, how to secure/encrypt data on disk, disaster recovery, support for high availability, etc. These all are exposed leaks that relate to what the little heads on the disk and CPUs and network all do in light of your desired performance and reliability. Some DBs also abstract out the back end, where you can plug in different low-level storage engines or even expose your own data structures as virtual tables (e.g. flat files or network data streams). You're able to write your own back end if you want. And I've heard of at least one effort to rewrite RDBMSs to run well on and take advantage of the nature of SSDs vs. hard drives -- I'll be curious as to how that turns out. I also believe the nature of such a system is fundamentally different enough to make adapting existing RDBMSs to SSDs very difficult. I'd say then that many RDBMSs have done a very good job exposing the low detail levels to the level that's appropriate.
    - ntoshev 17y ago
      Well, in my opinion nothing is completely opaque or transparent, but to me ORMs (given their ability to fall back to arbitrary SQL, which their main task is to abstract away) are way more transparent than relational databases (that constrain you to their relational model). Indeed, RDBMSes give you a myriad parameters to tune, but if you think about it almost all of them affect the behavior of the system in a global way and almost none of them is really an escape hatch that lets you do things in your non-relational non-ACID way (e.g. you can't choose not to persist some of your writes to a given table immediately, although you can choose to keep the entire table in a memory-based storage engine in mysql). SSDs are not that different from hdds I believe. The nature of hard disks is that you should minimize random access. With SSDs you can do all random access you want on reads, but you still better write continuous blocks. There are storage systems like Cassandra that were developed for hdds but conform well to this SSD-optimal pattern, it would be interesting to see how they perform on SSDs.

by ntoshev 17y ago

andreyf 17y ago

Re: ORM/SQL, compare Django's ORM/MySQL to how PG implemented storage in news.YC - I've devoted several orders of magnitude more time working with the former, but I wouldn't be able to describe what happens in-hardware nearly as well as I could on the latter, which I read once (in a language I never seriously used) nearly a year ago.

ntoshev 17y ago
I don't have experience with your specific example, but it seems you are complaining against the opacity of relational databases, not the ORM itself. I agree, relational databases are very opaque.
- andreyf 17y ago
  If an ORM user can be fairly confident fairly quickly about the SQL generated from each of his ORM calls, it's a good abstraction. In my experience with Django's ORM, that hasn't been the case. Fault may be mine :)
- nwatson 17y ago
  RDBMSs aren't that opaque ... A relational database exists to store rows of data in tables and to relate them, making CRUD operations possible in this context. At that level it's very abstract. But it also starts to expose some implementation details as appropriate, and that's why some DBAs make a very good living (I'm not a DBA myself). Transactions are required once there can be more than one DB client (well, even before that when considering atomicity/durability). And most DBs expose different transaction isolation levels because the various options have different speed vs semantic reliability implications (e.g. dirty reads). These make sense only in light of lower-level concepts that leak through from the notion that data is stored on disk, buffered in memory, handled by threads, protected by locks, etc. And then there are details of indexing, clustering, distributing table data among available disks, where to store the transaction log, how to secure/encrypt data on disk, disaster recovery, support for high availability, etc. These all are exposed leaks that relate to what the little heads on the disk and CPUs and network all do in light of your desired performance and reliability. Some DBs also abstract out the back end, where you can plug in different low-level storage engines or even expose your own data structures as virtual tables (e.g. flat files or network data streams). You're able to write your own back end if you want. And I've heard of at least one effort to rewrite RDBMSs to run well on and take advantage of the nature of SSDs vs. hard drives -- I'll be curious as to how that turns out. I also believe the nature of such a system is fundamentally different enough to make adapting existing RDBMSs to SSDs very difficult. I'd say then that many RDBMSs have done a very good job exposing the low detail levels to the level that's appropriate.
  - ntoshev 17y ago
    Well, in my opinion nothing is completely opaque or transparent, but to me ORMs (given their ability to fall back to arbitrary SQL, which their main task is to abstract away) are way more transparent than relational databases (that constrain you to their relational model). Indeed, RDBMSes give you a myriad parameters to tune, but if you think about it almost all of them affect the behavior of the system in a global way and almost none of them is really an escape hatch that lets you do things in your non-relational non-ACID way (e.g. you can't choose not to persist some of your writes to a given table immediately, although you can choose to keep the entire table in a memory-based storage engine in mysql). SSDs are not that different from hdds I believe. The nature of hard disks is that you should minimize random access. With SSDs you can do all random access you want on reads, but you still better write continuous blocks. There are storage systems like Cassandra that were developed for hdds but conform well to this SSD-optimal pattern, it would be interesting to see how they perform on SSDs.