11 ms·

Mixpanel went down, so we're open sourcing a better pure Python memcache client

97 points by notknifescience 14y ago

deleted 14y ago
[deleted]
lamby 14y ago
Perhaps it could be named better; all the python memcache clients have almost identical names.
jscottmiller 14y ago
> Because we use eventlet, we need a pure python memcache client so that eventlet can patch the socket operations to be non-blocking. Not true. ultramemcache (https://github.com/esnme/ultramemcache https://github.com/esnme/ultramemcache) is an example of a c++ memcached client that is friendly to gevent's monkeypatching. It does this by importing the socket library via the python runtime (https://github.com/esnme/ultramemcache/blob/master/python/umemcache.cpp https://github.com/esnme/ultramemcache/blob/master/python/um...), rather than using a native construct. My experience is with gevent, but I suspect that eventlet's monkeypatching is similar to gevent.
- gtaylor 14y ago
  pylibmc may be fine, as well.
latchkey 14y ago
This is the type of response that makes me a happy Mixpanel user. Hit a snag; fix it, talk about it, open source it.
tebeka 14y ago
Yet again, instead of trying to fix the existing, someone writes yet another library.
- rhizome 14y ago
  This was my thought, too: there seems to be a limit beyond which memcached simply breaks. I learned this 5 years ago, and it had nothing to do with Python.
- catch23 14y ago
  Well, if there are too many users of an existing library, it makes it hard to do large changes. So in their case, it seems like the original library would silently fail on exceptions, they want to make things bubble up to the top which could be bad for a large number of users who have been depending on the silent-fail operation.
evan2m 14y ago
I would argue that python-memcached has been replaced as "the de-facto standard" by pylibmc. I'm not sure if pylibmc works with eventlet or not. Companies open sourcing code is always good. But, I would be surprised if there wasn't already a python memcached client that would suit your needs.
- seunosewa 14y ago
  I've been unable to compile pylibmc Centos 5.
kapso 14y ago
"It turns out that socket operations that fail in python-memcached don’t return an error or raise any exceptions." ...Whats surprising is that this went untested for so long.
- suhail 14y ago
  We've been very well aware of it and safely worked around it at the time. It was also something we just disliked and so we changed it across our entire codebase when we decided to write our own library. When we said "it turns out", we didn't mean that we had suddenly learned that after failure. It's just something we meant to note that we decided to fix.
jwegan 14y ago
You might want to set TCP_NODELAY on the socket to disable Nagle's algorithm (http://en.wikipedia.org/wiki/Nagles_algorithm http://en.wikipedia.org/wiki/Nagles_algorithm). We ran into some issues with memcached where if Nagle's algorithm was not turned of, GETs would often take several milliseconds longer than was necessary.
keypusher 14y ago
> We enforce the calling convention with code in our library that looks like this: if not isinstance(val, str): raise ValidationException('value must be str', val) I love Python, but the more I used it the more I come to see dynamic typing as a mistake. On large projects with shared library code and deeply nested calls, one of the most pervasive problems seems to be wrongly typed data. And building validation exceptions into all of your code seems like a poor solution.
- __alexs 14y ago
  They could have implemented it using a simple decorator for non var-args methods which personally I think is more attractive. Obviously you still pay the cost at run-time but at least the actual source looks more sensible that way.
bifrost 14y ago
I'm confused, why would a cache being down cause data loss? Two months is also not battle tested, perhaps the terminology used here needs to be adjusted...
- mryan 14y ago
  > Calling .get() on a dead server returns None on failure, the same thing that is returned when there’s no value. I suppose they were doing a .get() to check if the server was still up, and assuming None meant "there are no values to get, but the server is up". If .set() works in the same way (silently failing) they were essentially using /dev/null as their task queue. :)
  - illumen 14y ago
    Yeah. It's a cache, it says so in the name. Why would you make not-a-cache on a cache protocol and library? In-sane.

deleted 14y ago

[deleted]

lamby 14y ago

Perhaps it could be named better; all the python memcache clients have almost identical names.

jscottmiller 14y ago

> Because we use eventlet, we need a pure python memcache client so that eventlet can patch the socket operations to be non-blocking. Not true. ultramemcache (https://github.com/esnme/ultramemcache https://github.com/esnme/ultramemcache) is an example of a c++ memcached client that is friendly to gevent's monkeypatching. It does this by importing the socket library via the python runtime (https://github.com/esnme/ultramemcache/blob/master/python/umemcache.cpp https://github.com/esnme/ultramemcache/blob/master/python/um...), rather than using a native construct. My experience is with gevent, but I suspect that eventlet's monkeypatching is similar to gevent.

gtaylor 14y ago
pylibmc may be fine, as well.

latchkey 14y ago

This is the type of response that makes me a happy Mixpanel user. Hit a snag; fix it, talk about it, open source it.

tebeka 14y ago

Yet again, instead of trying to fix the existing, someone writes yet another library.

rhizome 14y ago
This was my thought, too: there seems to be a limit beyond which memcached simply breaks. I learned this 5 years ago, and it had nothing to do with Python.
catch23 14y ago
Well, if there are too many users of an existing library, it makes it hard to do large changes. So in their case, it seems like the original library would silently fail on exceptions, they want to make things bubble up to the top which could be bad for a large number of users who have been depending on the silent-fail operation.

evan2m 14y ago

I would argue that python-memcached has been replaced as "the de-facto standard" by pylibmc. I'm not sure if pylibmc works with eventlet or not. Companies open sourcing code is always good. But, I would be surprised if there wasn't already a python memcached client that would suit your needs.

seunosewa 14y ago
I've been unable to compile pylibmc Centos 5.

kapso 14y ago

"It turns out that socket operations that fail in python-memcached don’t return an error or raise any exceptions." ...Whats surprising is that this went untested for so long.

suhail 14y ago
We've been very well aware of it and safely worked around it at the time. It was also something we just disliked and so we changed it across our entire codebase when we decided to write our own library. When we said "it turns out", we didn't mean that we had suddenly learned that after failure. It's just something we meant to note that we decided to fix.

jwegan 14y ago

You might want to set TCP_NODELAY on the socket to disable Nagle's algorithm (http://en.wikipedia.org/wiki/Nagles_algorithm http://en.wikipedia.org/wiki/Nagles_algorithm). We ran into some issues with memcached where if Nagle's algorithm was not turned of, GETs would often take several milliseconds longer than was necessary.

keypusher 14y ago

> We enforce the calling convention with code in our library that looks like this: if not isinstance(val, str): raise ValidationException('value must be str', val) I love Python, but the more I used it the more I come to see dynamic typing as a mistake. On large projects with shared library code and deeply nested calls, one of the most pervasive problems seems to be wrongly typed data. And building validation exceptions into all of your code seems like a poor solution.

__alexs 14y ago
They could have implemented it using a simple decorator for non var-args methods which personally I think is more attractive. Obviously you still pay the cost at run-time but at least the actual source looks more sensible that way.

bifrost 14y ago

I'm confused, why would a cache being down cause data loss? Two months is also not battle tested, perhaps the terminology used here needs to be adjusted...

mryan 14y ago
> Calling .get() on a dead server returns None on failure, the same thing that is returned when there’s no value. I suppose they were doing a .get() to check if the server was still up, and assuming None meant "there are no values to get, but the server is up". If .set() works in the same way (silently failing) they were essentially using /dev/null as their task queue. :)
- illumen 14y ago
  Yeah. It's a cache, it says so in the name. Why would you make not-a-cache on a cache protocol and library? In-sane.