14 ms·

Reading through that documentation page I must say I'm extremely underwhelmed. Is that doc-page outdated/incomplete or did it really take them 4 years and a 21

by moe 17y ago

Reading through that documentation page I must say I'm extremely underwhelmed. Is that doc-page outdated/incomplete or did it really take them 4 years and a 21kloc patch to tie a model to a database instance? I mean, I don't see anything about proper partitioning/sharding, not a glimpse of cross-db queries. Is this a joke? SqlAlchemy has had proper multi-db support (vertical and horizontal) since around 2007. Perhaps the time would be better spent porting django to a real ORM? http://www.sqlalchemy.org/docs/06/session.html#partitioning-strategies http://www.sqlalchemy.org/docs/06/session.html#partitioning-...

simonw 17y ago
The original plan was to add higher level support for certain types of sharding (e.g. tying a specific model to an individual database) and earlier versions of the code actually provided a method for doing so. This was removed because it wasn't judged to be a good enough solution - Django is very keen on the concept of reusable applications, and hard coding a decision of which database should be used for an application's models in to that application would conflict with that concept. Obviously the solution is to specify a mapping somewhere, but there's no need to hold up the feature while waiting for a design for that particular bikeshed. Instead, the plan is to release an initial version with low level primitives: the ability to configure multiple databases and specify which database should be used for a given query. This gives us a chance to see what kind of patterns are most widely used and figure out the correct way to support those at a higher level of the API. In my opinion, the SQLAlchemy example demonstrates why handling sharding automatically is a distinctly non-trivial problem: http://www.sqlalchemy.org/trac/browser/sqlalchemy/trunk/examples/sharding/attribute_shard.py#L129 http://www.sqlalchemy.org/trac/browser/sqlalchemy/trunk/exam... - it's not clear to me that that's a better solution than just manually picking the database to execute a query on at the point of execution.
- moe 17y ago
  Well, SA provides all the primitives required for setting proper sharding up - admittedly it could use some more documentation and an out-of-the-box anonymous sharding lib. The example is fully functional, though, and once you've wrapped your head around it, it all makes sense and does indeed work (I've used it in production). Sorry, but "seeing what patterns are most widely used" is a lame excuse. You could just look at what real world sites do today (and did 4 years ago) - or simply jump right to implementing the only anonymous sharding model that can possibly work (see how mongodb does it). I'm not meaning to bash django as a whole here, but I had a true WTF-moment looking at that page. It seems the home-grown ORM is becoming a ball on a chain if something like that takes this long... Anyways, at least it's progress in the right direction, looking forward to real partitioning support, hopefully in earlier than another 4 years.
jbronn 17y ago
There are plenty of reasons one may prefer Django's ORM over SQLAlchemy. For example, SQLAlchemy doesn't have robust support for the most popular spatial database implementations: PostGIS, MySQL, Oracle, and SpatiaLite. Django's ORM does. Moreover, it did not take four years to develop multi-db support. This was a GSoC 2009 project that was merged in 2009.

by moe 17y ago

simonw 17y ago

The original plan was to add higher level support for certain types of sharding (e.g. tying a specific model to an individual database) and earlier versions of the code actually provided a method for doing so. This was removed because it wasn't judged to be a good enough solution - Django is very keen on the concept of reusable applications, and hard coding a decision of which database should be used for an application's models in to that application would conflict with that concept. Obviously the solution is to specify a mapping somewhere, but there's no need to hold up the feature while waiting for a design for that particular bikeshed. Instead, the plan is to release an initial version with low level primitives: the ability to configure multiple databases and specify which database should be used for a given query. This gives us a chance to see what kind of patterns are most widely used and figure out the correct way to support those at a higher level of the API. In my opinion, the SQLAlchemy example demonstrates why handling sharding automatically is a distinctly non-trivial problem: http://www.sqlalchemy.org/trac/browser/sqlalchemy/trunk/examples/sharding/attribute_shard.py#L129 http://www.sqlalchemy.org/trac/browser/sqlalchemy/trunk/exam... - it's not clear to me that that's a better solution than just manually picking the database to execute a query on at the point of execution.

moe 17y ago
Well, SA provides all the primitives required for setting proper sharding up - admittedly it could use some more documentation and an out-of-the-box anonymous sharding lib. The example is fully functional, though, and once you've wrapped your head around it, it all makes sense and does indeed work (I've used it in production). Sorry, but "seeing what patterns are most widely used" is a lame excuse. You could just look at what real world sites do today (and did 4 years ago) - or simply jump right to implementing the only anonymous sharding model that can possibly work (see how mongodb does it). I'm not meaning to bash django as a whole here, but I had a true WTF-moment looking at that page. It seems the home-grown ORM is becoming a ball on a chain if something like that takes this long... Anyways, at least it's progress in the right direction, looking forward to real partitioning support, hopefully in earlier than another 4 years.

jbronn 17y ago

There are plenty of reasons one may prefer Django's ORM over SQLAlchemy. For example, SQLAlchemy doesn't have robust support for the most popular spatial database implementations: PostGIS, MySQL, Oracle, and SpatiaLite. Django's ORM does. Moreover, it did not take four years to develop multi-db support. This was a GSoC 2009 project that was merged in 2009.