4 ms·

Writing a non-relational Django backend

47 points by wkornewald 17y ago

wkornewald 17y ago
We need help to get official Django support for non-relational / NoSQL DBs. Currently, we only have an App Engine backend and a half-finished MongoDB backend. We have to demonstrate to the Django team that our approach works with a wider variety of DBs. If we achieve that goal Django would have a way to write portable code for a lot of DB types. We'd also have the foundation for much more powerful features like automatic denormalization. Please help us find backend contributors.
- rbanffy 17y ago
  I am decided to give Cassandra a shot. It will take some work to get up to speed, but you can count on me for that.
  - joshfinnie 17y ago
    I know this if off-topic, but what is the best resource to learn Cassandra? I can't find anything good searching through Google.
    - kingkilr 17y ago
      There's a PDF from digg from a NoSQL meetup (NoSQL east I think) and it has a link to a document somewhere there wrote up on understanding the cassandra data model, it's pretty good. You'll have to google around for it though.
  - twanschik 17y ago
    Nice to hear, let us know if you need some help writing the backend for django.
  - wkornewald 17y ago
    Great! Let's continue the discussion on the django-nonrel discussion group: http://groups.google.com/group/django-non-relational http://groups.google.com/group/django-non-relational
kingkilr 17y ago
I (author of multiple-database support for django) am extremely uncomfortable with the approach these guys are taking. The proper approach would be to work with Django itself to make things more flexible (when you're subclassing something named SQLCompiler for a non-SQL backend it should be a hint you've done something wrong).
- twanschik 17y ago
  When you are naming something like the SQLCompiler "SQLCompiler" even if it fits non-relational database's needs than naming it "SQLCompiler" was wrong!
  - kingkilr 17y ago
    As I said on reddit, the name isn't misleading. SQLCompiler is used for generating SQL, and assumes relational database concepts like joins, table aliases, and aggregation.
    - wkornewald 17y ago
      Now, here comes the surprise: That's exactly what we want to emulate with Django nonrel. It's exactly the feature set we badly want to have. This is the reason why the SQLQuery/SQLCompiler stuff fits so well (though, you're right, we're only using a fraction of SQLCompiler's code, so we could actually implement a simplified base class which is used by both, but we can discuss this later on the django-developers group).
- wkornewald 17y ago
  First of all, we have a few modifications in Django itself that make Django more flexible. What exactly are you thinking of that we're missing? Regarding SQLCompiler: Maybe the name is just misleading? We had several attempts at this problem and one of them was a complete refactoring of the ORM as you try to suggest here, but in the end we had to face the facts: We'd reproduce pretty much the same functionality that's already there. It would just be named Query instead of SQLQuery and it would take a huge amount of effort to get there. So, we decided instead to reuse as much of the code that just happens to be named "SQLSomething" as possible. And IMHO this is the best of the approaches we've taken. It's simple and it works really well, as you will soon see for yourself. Then we'll talk again. BTW, we also tried to work with the Django team, but in the end it was all discussion with the hidden message "show us that it works". That's what we are doing now.
m0th87 17y ago
This article begs the question "what for?" * Document stores map pretty easily to object oriented systems, and thus negate the need for an ORM * Distributed hashtables don't have the featuresets available to warrant an ORM abstraction I guess it might work for Cassandra et al. But IMO ORMs barely work for SQL databases. How well will they map on to column stores?
- enki 17y ago
  i think full blown ORM-style is the wrong abstraction, at least for now. until the abstractions are good enough, staying close to the metal is a good idea. also querying is easy (there's not a lot of options), but keeping multiple indexes updated and partitioning them etc. is where the fun begins. for my own needs, i've started building a thin (transparent) abstraction layer for cassandra . take a look here for a code example: http://github.com/enki/tragedy http://github.com/enki/tragedy
  - thomaspaine 17y ago
    Weird, I wrote something almost identical to this. Still waiting on permission to open source though.
- wkornewald 17y ago
  It's for creating an abstraction layer that allows to write portable apps and that provides a much more powerful API than boring hashmaps (as you might have noticed, lots of APIs for those DBs try to get away from the hashmap representation). How often have you written manual denormalization code or manual in-memory JOIN code? How often did you manually keep track of counters? What are you trying to achieve with that, anyway? You're "manualating" features you get for free with SQL. That's waste. It's something you shouldn't have to do. It can be automated and that's our long-term goal. These DBs are like writing with Assembler just to get more speed (or scale in this case). The point is, you can use a high-level language (here, Django's ORM plus a few "optimization" specifications) to achieve the same result much faster and in a portable way. For anything that goes beyond the ORM's features we can still provide MapReduce and other mechanisms, but again with a higher-level API. Do you really believe 10 years from now we'll still be using hashmaps to access those DBs? (just trying to provoke some thought; I don't really believe that you think that way)
urlwolf 17y ago
If you use RDF stores/graph databases, this python ORM is worth watching: http://code.google.com/p/surfrdf/ http://code.google.com/p/surfrdf/ . Like activeRDF in the ruby world, but with the advantage of having solid RDF libs (RDFlibs for python is 3-4 years old now) compared to the buzzing mess of new RDF libs for ruby, most of them still very preliminary.

47 points by wkornewald 17y ago

We need help to get official Django support for non-relational / NoSQL DBs. Currently, we only have an App Engine backend and a half-finished MongoDB backend. We have to demonstrate to the Django team that our approach works with a wider variety of DBs. If we achieve that goal Django would have a way to write portable code for a lot of DB types. We'd also have the foundation for much more powerful features like automatic denormalization. Please help us find backend contributors.

rbanffy 17y ago
I am decided to give Cassandra a shot. It will take some work to get up to speed, but you can count on me for that.
- joshfinnie 17y ago
  I know this if off-topic, but what is the best resource to learn Cassandra? I can't find anything good searching through Google.
  - kingkilr 17y ago
    There's a PDF from digg from a NoSQL meetup (NoSQL east I think) and it has a link to a document somewhere there wrote up on understanding the cassandra data model, it's pretty good. You'll have to google around for it though.
- twanschik 17y ago
  Nice to hear, let us know if you need some help writing the backend for django.
- wkornewald 17y ago
  Great! Let's continue the discussion on the django-nonrel discussion group: http://groups.google.com/group/django-non-relational http://groups.google.com/group/django-non-relational

kingkilr 17y ago

I (author of multiple-database support for django) am extremely uncomfortable with the approach these guys are taking. The proper approach would be to work with Django itself to make things more flexible (when you're subclassing something named SQLCompiler for a non-SQL backend it should be a hint you've done something wrong).

twanschik 17y ago
When you are naming something like the SQLCompiler "SQLCompiler" even if it fits non-relational database's needs than naming it "SQLCompiler" was wrong!
- kingkilr 17y ago
  As I said on reddit, the name isn't misleading. SQLCompiler is used for generating SQL, and assumes relational database concepts like joins, table aliases, and aggregation.
  - wkornewald 17y ago
    Now, here comes the surprise: That's exactly what we want to emulate with Django nonrel. It's exactly the feature set we badly want to have. This is the reason why the SQLQuery/SQLCompiler stuff fits so well (though, you're right, we're only using a fraction of SQLCompiler's code, so we could actually implement a simplified base class which is used by both, but we can discuss this later on the django-developers group).
wkornewald 17y ago
First of all, we have a few modifications in Django itself that make Django more flexible. What exactly are you thinking of that we're missing? Regarding SQLCompiler: Maybe the name is just misleading? We had several attempts at this problem and one of them was a complete refactoring of the ORM as you try to suggest here, but in the end we had to face the facts: We'd reproduce pretty much the same functionality that's already there. It would just be named Query instead of SQLQuery and it would take a huge amount of effort to get there. So, we decided instead to reuse as much of the code that just happens to be named "SQLSomething" as possible. And IMHO this is the best of the approaches we've taken. It's simple and it works really well, as you will soon see for yourself. Then we'll talk again. BTW, we also tried to work with the Django team, but in the end it was all discussion with the hidden message "show us that it works". That's what we are doing now.

m0th87 17y ago

This article begs the question "what for?" * Document stores map pretty easily to object oriented systems, and thus negate the need for an ORM * Distributed hashtables don't have the featuresets available to warrant an ORM abstraction I guess it might work for Cassandra et al. But IMO ORMs barely work for SQL databases. How well will they map on to column stores?

enki 17y ago
i think full blown ORM-style is the wrong abstraction, at least for now. until the abstractions are good enough, staying close to the metal is a good idea. also querying is easy (there's not a lot of options), but keeping multiple indexes updated and partitioning them etc. is where the fun begins. for my own needs, i've started building a thin (transparent) abstraction layer for cassandra . take a look here for a code example: http://github.com/enki/tragedy http://github.com/enki/tragedy
- thomaspaine 17y ago
  Weird, I wrote something almost identical to this. Still waiting on permission to open source though.
wkornewald 17y ago
It's for creating an abstraction layer that allows to write portable apps and that provides a much more powerful API than boring hashmaps (as you might have noticed, lots of APIs for those DBs try to get away from the hashmap representation). How often have you written manual denormalization code or manual in-memory JOIN code? How often did you manually keep track of counters? What are you trying to achieve with that, anyway? You're "manualating" features you get for free with SQL. That's waste. It's something you shouldn't have to do. It can be automated and that's our long-term goal. These DBs are like writing with Assembler just to get more speed (or scale in this case). The point is, you can use a high-level language (here, Django's ORM plus a few "optimization" specifications) to achieve the same result much faster and in a portable way. For anything that goes beyond the ORM's features we can still provide MapReduce and other mechanisms, but again with a higher-level API. Do you really believe 10 years from now we'll still be using hashmaps to access those DBs? (just trying to provoke some thought; I don't really believe that you think that way)

urlwolf 17y ago

If you use RDF stores/graph databases, this python ORM is worth watching: http://code.google.com/p/surfrdf/ http://code.google.com/p/surfrdf/ . Like activeRDF in the ruby world, but with the advantage of having solid RDF libs (RDFlibs for python is 3-4 years old now) compared to the buzzing mess of new RDF libs for ruby, most of them still very preliminary.