4 ms·

It seems to me that this submission is getting a lot of blowback in the comments for 1) the style and 2) the implication that wiring up Python services with HTT

It seems to me that this submission is getting a lot of blowback in the comments for 1) the style and 2) the implication that wiring up Python services with HTTP is bad engineering. I don’t think this is productive. On the first point, yeah Rachel’s posts are kinda snarky sometimes, but some of us find that entertaining particularly when they are highly detailed and thoroughly researched. I’ve worked with Rachel and she’s among the best “deep-dive” userspace-to-network driver problem solvers around. She knows her shit and we’re lucky she takes the time to put hard-earned lessons on the net for others to benefit from. As for “microservices written in Python trading a bunch of sloppy JSON around via HTTP” is bad engineering: it is bad engineering, sometimes the flavor of the month is rancid (CORBA, multiple implementation inheritance, XSLT, I could go on). Introducing network boundaries where function calls would work is a bad idea, as anyone who’s dealt seriously with distributed systems for a living knows. JSON-over-HTTP for RPC is lazy, inefficient in machine time and engineering effort, and trivially obsolete in a world where Protocol Buffers/gRPC or Thrift and their ilk are so mature. Now none of this is to say you should rewrite your system if it’s built that way, legacy stuff is a thing. But Rachel wrote a detailed piece on why you are asking for trouble if you build new stuff like this and people are, in my humble opinion, shooting the messenger.

ahuang 7y ago
I think the main issue is it seems really one-sided and the intent was to be snarky, vs educational. I posted a comment here detailing some ways to work around some of the pitfalls. I think if she devoted more time in the article to solutions vs. complaining, her points would come across more productively.
diebeforei485 7y ago
> we’re lucky she takes the time to put hard-earned lessons on the net for others to benefit from. I genuinely don't see much of a lesson to learn from this particular blogpost, and it appears neither did many others in HN. If there is one, beyond "don't use x", it's hard to find it. I get the impression that this particular post is being upvoted to the top of HN because of who the author is, not necessarily because this post itself has value. This results in a whole bunch of others reading it, wondering why they're wasting their time with such a rambling post.
- dfox 7y ago
  The takeaway should be: don't do green threads/event loops for anything that involves any kind of non-trivial processing or even better yet, don't do that unless you really need to do such things (and “better performance” is not valid reason)
- kragen 7y ago
  Here are some lessons you might learn: 1. Load before forking. 2. Remember that green threads tend to have problems with fairness of scheduling. 3. JSON decoding gobbles CPU. 4. Scheduling fairness problems increase response time variance. 4½. Green threads also increase it. 5. Don't forget about retries of timed-out requests into account in protocol design; idempotence is the simplest solution when you can use it. 6. Wake-one semantics to avoid the thundering herd are important for performance when you have multiple threads, and Gunicorn has that thundering herd problem, so you probably don't want to be running it this way on a 64-core box with hyperthreading. (The problem is of course less severe than it was for Apache because the green threads don't thunder.) 7. Gevent uses epoll, not select, poll, or RT signals 8. EAGAIN and SIGPIPE if you didn't know about those. (Somebody is in today's lucky ten thousand.) 9. What kinds of mechanisms “tend to show up given time in a battle-tested [network server] system.” 10. Your systems don't have to be fragile pieces of shit.
  - rachelbythebay 7y ago
    Very nice. Thank you.
    - kragen 7y ago
      I'm not sure whether the person I was replying to is The One to whom all these things are too obvious to be worth mentioning, or if these were too implicit for them to notice, or a combination. Either way, no, thank you for writing it.
seemslegit 7y ago
"JSON-over-HTTP for RPC is lazy, inefficient in machine time and engineering effort, and trivially obsolete in a world where Protocol Buffers/gRPC or Thrift and their ilk are so mature." Laziness is a virtue in our profession :) huge "citation required" on it being more efficient in engineering effort and most importantly it's ubiquitous and interoperable with any language without having to rely on grpc or thrift implementation and tools for that language/platform. Re machine time efficiency - to the extent I understood what she was actually complaining about simply none of those issues are attributable to json-over-http being used.
- p_l 7y ago
  On the digression of efficiency... treating efficiency as something you solve by throwing more machine resources at is causing recognizable problems today. And the calculus of cost doesn't even always work out the same.
  - seemslegit 7y ago
    That's a prime example of survivorship bias, projects that spent their runway optimizing for machine resources are not around to have their problems recognized.
    - p_l 7y ago
      I had more than one project where people were cheaper than machines, and going tight on budgets for machine resources led to the project surviving instead of dying. Hell, a big example of that is Stack Overflow, which runs a very busy site on much less hw than they would have needed otherwise, just by taking high level optimization questions up front.
      - seemslegit 7y ago
        And can you spot a non json/http based service in their stack ? (no redis and sqlserver don't count because they don't own those) https://nickcraver.com/blog/2016/02/17/stack-overflow-the-architecture-2016-edition/ https://nickcraver.com/blog/2016/02/17/stack-overflow-the-ar... don't know if there's a later one. EDIT: I see they store their redis values in protobuf which is probably more due to memory utilization rather than serialization performance concerns
anaphor 7y ago
One of the people who designed Protobuf has criticized it (Edit: to clarify, one of the authors of v2, see below), so that doesn't really inspire much confidence in it for me.[1] Your general point is correct though, there's much more to a well designed RPC system than what HTTP based systems can do, but protobuf/gRPC is very much lacking in ideas that are decades old at this point, like promise pipelining, etc. Also, I feel like she (intentionally maybe) is conflating concurrency and parallelism. These "green thread" systems provide concurrency, but not parallelism. That should be something people are aware of when they use them. [1] https://capnproto.org/faq.html#how-do-i-make-a-field-required-like-in-protocol-buffers https://capnproto.org/faq.html#how-do-i-make-a-field-require...
- kentonv 7y ago
  > One of the people who designed Protobuf has said it's awful Hmm, you seem to be citing me. To clarify: 1. I didn't design Protobuf. I just rewrote the implementation (created version 2) and open sourced it. 2. I don't think it's awful. In fact, I think it's best-of-breed for what it is, and certainly a much better choice than JSON-over-HTTP. Yes, in Cap'n Proto I've added a bunch of features which Protobuf doesn't have, like zero-copy and promise pipelining, and I obviously think these features make it better tech. But, to be fair, these ideas make Cap'n Proto more complicated, and whether that complexity is worth it is still very much unproven at the scale that Google uses Protobuf and gRPC/Stubby. > These "green thread" systems provide concurrency, but not parallelism. I'm not sure if these specific definitions of "concurrency" and "parallelism" are universal. I wasn't aware of them, at least.
  - anaphor 7y ago
    Thanks for the response, I was being a bit tongue in cheek. I don't think you actually think it's awful :) And thanks for clarifying that. About concurrency vs. parallelism, I think it is fairly standard to think of them as two different concepts that overlap somewhat. You can have concurrency with parallelism (e.g. pthreads, or M:N threading where you map "green threads" on to processes that can run in parallel). You can also have concurrency without parallelism. The difference between the two is that parallelism can be deterministic, whereas concurrency is always going to be non-deterministic. I recognize that it's not very clear, but it isn't a distinction that I made up, e.g. https://existentialtype.wordpress.com/2014/04/09/parallelism-and-concurrency-revisited/ https://existentialtype.wordpress.com/2014/04/09/parallelism...
deleted 7y ago
[deleted]

ahuang 7y ago

I think the main issue is it seems really one-sided and the intent was to be snarky, vs educational. I posted a comment here detailing some ways to work around some of the pitfalls. I think if she devoted more time in the article to solutions vs. complaining, her points would come across more productively.

diebeforei485 7y ago

> we’re lucky she takes the time to put hard-earned lessons on the net for others to benefit from. I genuinely don't see much of a lesson to learn from this particular blogpost, and it appears neither did many others in HN. If there is one, beyond "don't use x", it's hard to find it. I get the impression that this particular post is being upvoted to the top of HN because of who the author is, not necessarily because this post itself has value. This results in a whole bunch of others reading it, wondering why they're wasting their time with such a rambling post.

dfox 7y ago
The takeaway should be: don't do green threads/event loops for anything that involves any kind of non-trivial processing or even better yet, don't do that unless you really need to do such things (and “better performance” is not valid reason)
kragen 7y ago
Here are some lessons you might learn: 1. Load before forking. 2. Remember that green threads tend to have problems with fairness of scheduling. 3. JSON decoding gobbles CPU. 4. Scheduling fairness problems increase response time variance. 4½. Green threads also increase it. 5. Don't forget about retries of timed-out requests into account in protocol design; idempotence is the simplest solution when you can use it. 6. Wake-one semantics to avoid the thundering herd are important for performance when you have multiple threads, and Gunicorn has that thundering herd problem, so you probably don't want to be running it this way on a 64-core box with hyperthreading. (The problem is of course less severe than it was for Apache because the green threads don't thunder.) 7. Gevent uses epoll, not select, poll, or RT signals 8. EAGAIN and SIGPIPE if you didn't know about those. (Somebody is in today's lucky ten thousand.) 9. What kinds of mechanisms “tend to show up given time in a battle-tested [network server] system.” 10. Your systems don't have to be fragile pieces of shit.
- rachelbythebay 7y ago
  Very nice. Thank you.
  - kragen 7y ago
    I'm not sure whether the person I was replying to is The One to whom all these things are too obvious to be worth mentioning, or if these were too implicit for them to notice, or a combination. Either way, no, thank you for writing it.

seemslegit 7y ago

"JSON-over-HTTP for RPC is lazy, inefficient in machine time and engineering effort, and trivially obsolete in a world where Protocol Buffers/gRPC or Thrift and their ilk are so mature." Laziness is a virtue in our profession :) huge "citation required" on it being more efficient in engineering effort and most importantly it's ubiquitous and interoperable with any language without having to rely on grpc or thrift implementation and tools for that language/platform. Re machine time efficiency - to the extent I understood what she was actually complaining about simply none of those issues are attributable to json-over-http being used.

p_l 7y ago
On the digression of efficiency... treating efficiency as something you solve by throwing more machine resources at is causing recognizable problems today. And the calculus of cost doesn't even always work out the same.
- seemslegit 7y ago
  That's a prime example of survivorship bias, projects that spent their runway optimizing for machine resources are not around to have their problems recognized.
  - p_l 7y ago
    I had more than one project where people were cheaper than machines, and going tight on budgets for machine resources led to the project surviving instead of dying. Hell, a big example of that is Stack Overflow, which runs a very busy site on much less hw than they would have needed otherwise, just by taking high level optimization questions up front.
    - seemslegit 7y ago
      And can you spot a non json/http based service in their stack ? (no redis and sqlserver don't count because they don't own those) https://nickcraver.com/blog/2016/02/17/stack-overflow-the-architecture-2016-edition/ https://nickcraver.com/blog/2016/02/17/stack-overflow-the-ar... don't know if there's a later one. EDIT: I see they store their redis values in protobuf which is probably more due to memory utilization rather than serialization performance concerns

anaphor 7y ago

One of the people who designed Protobuf has criticized it (Edit: to clarify, one of the authors of v2, see below), so that doesn't really inspire much confidence in it for me.[1] Your general point is correct though, there's much more to a well designed RPC system than what HTTP based systems can do, but protobuf/gRPC is very much lacking in ideas that are decades old at this point, like promise pipelining, etc. Also, I feel like she (intentionally maybe) is conflating concurrency and parallelism. These "green thread" systems provide concurrency, but not parallelism. That should be something people are aware of when they use them. [1] https://capnproto.org/faq.html#how-do-i-make-a-field-required-like-in-protocol-buffers https://capnproto.org/faq.html#how-do-i-make-a-field-require...

kentonv 7y ago
> One of the people who designed Protobuf has said it's awful Hmm, you seem to be citing me. To clarify: 1. I didn't design Protobuf. I just rewrote the implementation (created version 2) and open sourced it. 2. I don't think it's awful. In fact, I think it's best-of-breed for what it is, and certainly a much better choice than JSON-over-HTTP. Yes, in Cap'n Proto I've added a bunch of features which Protobuf doesn't have, like zero-copy and promise pipelining, and I obviously think these features make it better tech. But, to be fair, these ideas make Cap'n Proto more complicated, and whether that complexity is worth it is still very much unproven at the scale that Google uses Protobuf and gRPC/Stubby. > These "green thread" systems provide concurrency, but not parallelism. I'm not sure if these specific definitions of "concurrency" and "parallelism" are universal. I wasn't aware of them, at least.
- anaphor 7y ago
  Thanks for the response, I was being a bit tongue in cheek. I don't think you actually think it's awful :) And thanks for clarifying that. About concurrency vs. parallelism, I think it is fairly standard to think of them as two different concepts that overlap somewhat. You can have concurrency with parallelism (e.g. pthreads, or M:N threading where you map "green threads" on to processes that can run in parallel). You can also have concurrency without parallelism. The difference between the two is that parallelism can be deterministic, whereas concurrency is always going to be non-deterministic. I recognize that it's not very clear, but it isn't a distinction that I made up, e.g. https://existentialtype.wordpress.com/2014/04/09/parallelism-and-concurrency-revisited/ https://existentialtype.wordpress.com/2014/04/09/parallelism...

deleted 7y ago

[deleted]