4 ms·

This is some really useful stuff. I learned a lot of this over time, and wished I had known it sooner. The point about schema evolution is important. I've tak

This is some really useful stuff. I learned a lot of this over time, and wished I had known it sooner. The point about schema evolution is important. I've taken to using a pickled python object in a text field in every model. That flexibility to store arbitrary data in a hash is amazing. I've used it to, for example, represent twitter accounts that are oauthed, with password, following the business account, or just nothing special. 4 kinds of accounts, with logic in the model to deal with it properly, all without evolving the schema. I only started with the first 2, and added the latter two without a problem. I'm pushing this a bit further now, and transitioning away from models altogether, and experimenting with a non-relational database like couchdb. I'm debating moving to google app engine too, but found some details (like getting on your own domain) to be pretty annoying.

seiji 17y ago
> I've taken to using a pickled python object in a text field in every model. That's basically what App Engine does right there. It stores by an id, then the value is your object serialized as a protocol buffer. The App Engine datastore allows you to look up by other properties than id because it indexes every field of your object in the background without you having to deal with it. You can query by pair-wise or more-wise indexes too, but you have to create those manually.
stanleydrew 17y ago
Storing pickled python objects is fine as long as you don't care about relational data. But if you aren't careful, data integrity will probably suffer over time as the complexity of your application increases. And that is not good. Anyway I'm sure a lot of people here know this stuff already. Just do your research and make sure the route you go makes sense for your application and your scaling requirements.
matclayton 17y ago
Really don't like the sound of that, pickling allows you to serialize any python code, and consequently deserialize it. Presumably you are placing user submitted data into this pickle and therefore without careful sanity checking wouldn't this potentially allow the user to inject code which may be run on deserialization? Someone else may be able to shed more light on this and correct me (please do!) but thats my concern and would love to hear if it was a valid one or not.
- limmeau 17y ago
  It is possible to build a poisonous pickle which, on depickling, calls arbitrary standard library functions, so accepting pickles from untrusted source is a No. One may think that if only the internal model of the user input is pickled, and the user only gets to update this internal model through web requests in defined ways, thus never getting to upload pickles or construct arbitrary complicated python objects, it should be relatively safe. However, what remains is that if someone manages to blindly inject SQL, he can inject executable code into your application, instead of just injecting data.
- patio11 17y ago
  Whoa! I was assuming this worked like doing a JSON grab-bag column in Rails or J2EE, which I am a fan of. Executable code would make me much, muuuuuuuuuch cooler on the idea. Can you just do the same thing with JSON, Yaml, or your favorite serialization method of choice that does not have catastrophic consequences if parsing gets subverted?
  - RyanMcGreal 17y ago
    > Can you just do the same thing with JSON, Yaml, or your favorite serialization method of choice that does not have catastrophic consequences if parsing gets subverted? Sure. You can even extend the `json` module to serialize custom data types not defined in the JSON spec.
- ivankirigin 17y ago
  No, I'm just serializing dicts. Any user input is already in a string, so sanitized. I also usually make helper functions to get/set things in the dict, so I don't operate on the field outside the model often. This snippet is really helpful: http://www.djangosnippets.org/snippets/513/ http://www.djangosnippets.org/snippets/513/
- ivankirigin 17y ago
  Come to think of it, I think Hacker News works like this. When you upvote and need to sign in, the code to execute the upvote is stored to be executed once you sign in.
FraaJad 17y ago
Pickling python objects and storing is an very old practice in Python world. Zope has been persisting objects in ZODB (a giant pickle jar with some transactional features) since 1998.
- ivankirigin 17y ago
  New to me :) I love it
banjiewen 17y ago
Would be interested to hear your take on Django+CouchDB - are you using a library like Couchdbkit, or rolling your own? Are you connecting it to the built-in admin?
- ivankirigin 17y ago
  Still exploring. I'll post something when it actually works.

seiji 17y ago

> I've taken to using a pickled python object in a text field in every model. That's basically what App Engine does right there. It stores by an id, then the value is your object serialized as a protocol buffer. The App Engine datastore allows you to look up by other properties than id because it indexes every field of your object in the background without you having to deal with it. You can query by pair-wise or more-wise indexes too, but you have to create those manually.

stanleydrew 17y ago

Storing pickled python objects is fine as long as you don't care about relational data. But if you aren't careful, data integrity will probably suffer over time as the complexity of your application increases. And that is not good. Anyway I'm sure a lot of people here know this stuff already. Just do your research and make sure the route you go makes sense for your application and your scaling requirements.

matclayton 17y ago

Really don't like the sound of that, pickling allows you to serialize any python code, and consequently deserialize it. Presumably you are placing user submitted data into this pickle and therefore without careful sanity checking wouldn't this potentially allow the user to inject code which may be run on deserialization? Someone else may be able to shed more light on this and correct me (please do!) but thats my concern and would love to hear if it was a valid one or not.

limmeau 17y ago
It is possible to build a poisonous pickle which, on depickling, calls arbitrary standard library functions, so accepting pickles from untrusted source is a No. One may think that if only the internal model of the user input is pickled, and the user only gets to update this internal model through web requests in defined ways, thus never getting to upload pickles or construct arbitrary complicated python objects, it should be relatively safe. However, what remains is that if someone manages to blindly inject SQL, he can inject executable code into your application, instead of just injecting data.
patio11 17y ago
Whoa! I was assuming this worked like doing a JSON grab-bag column in Rails or J2EE, which I am a fan of. Executable code would make me much, muuuuuuuuuch cooler on the idea. Can you just do the same thing with JSON, Yaml, or your favorite serialization method of choice that does not have catastrophic consequences if parsing gets subverted?
- RyanMcGreal 17y ago
  > Can you just do the same thing with JSON, Yaml, or your favorite serialization method of choice that does not have catastrophic consequences if parsing gets subverted? Sure. You can even extend the `json` module to serialize custom data types not defined in the JSON spec.
ivankirigin 17y ago
No, I'm just serializing dicts. Any user input is already in a string, so sanitized. I also usually make helper functions to get/set things in the dict, so I don't operate on the field outside the model often. This snippet is really helpful: http://www.djangosnippets.org/snippets/513/ http://www.djangosnippets.org/snippets/513/
ivankirigin 17y ago
Come to think of it, I think Hacker News works like this. When you upvote and need to sign in, the code to execute the upvote is stored to be executed once you sign in.

FraaJad 17y ago

Pickling python objects and storing is an very old practice in Python world. Zope has been persisting objects in ZODB (a giant pickle jar with some transactional features) since 1998.

ivankirigin 17y ago
New to me :) I love it

banjiewen 17y ago

Would be interested to hear your take on Django+CouchDB - are you using a library like Couchdbkit, or rolling your own? Are you connecting it to the built-in admin?

ivankirigin 17y ago
Still exploring. I'll post something when it actually works.