5 ms·

The developer interface was elegant, but the internal code left something to be desired. That is no longer the case.

Goranek 14y ago
Dude you are my hero. I've printed requests source code and use it as an example of great python code. You and guys like Armin Ronacher are the reason why I use and like Python so much !!! (when jerking stops) Question: I have a feeling that with Requests you're just hiding the problem and not trying to solve it (Urllib is still in the back). Shouldn't urllib(1,2,3...) be rewritten instead?
- jevinskie 14y ago
  Probably, but rewriting a core module (this would be urllib4!?) is a lot harder (politically at least) than creating a 3rd party wrapper module.
- kenneth_reitz 14y ago
  Thanks! Urllib isn't used at all, actually. Internally, I use a project called 'urllib3', but it's totally unrelated to urlilb. It's name is actually a joke. It is essentially a light wrapper around httplib that provides connection pooling :)
  - ak217 14y ago
    By the way, I found a bug in httplib where, when sending a request body, if the peer closes the socket early with an error (HTTP/1.1 compliant behavior), the request will time out instead of returning the correct error information. To my knowledge, this is also present in httplib2. This is because the socket write in both libraries is blocking (which seems simply wrong). I'm hoping to send a patch sometime soon. Regardless, httplib2 seems to have lots of other technical advantages over httplib - have you considered using it?
    - kenneth_reitz 14y ago
      http://code.google.com/p/httplib2/source/browse/python2/httplib2/__init__.py#1598 http://code.google.com/p/httplib2/source/browse/python2/http... The entire process of sending a Request is wrapped in a 'try/except Exception' block. wtf. It also just aimlessly retries failed requests. This is not configurable. Httplib2 is terrible. I actually created Requests partially in spite of it.
      - ak217 14y ago
        Makes sense, thanks. I'll have to work on patching up httplib, then.
    - RyanMcGreal 14y ago
      httplib2 is not thread-safe. That pretty much makes it a deal-breaker, even if the API is a significant improvement over the standard library (though not over requests).
- Goranek 14y ago
  Well that's a different story then :) Good job!
- ak217 14y ago
  I really respect both Kenneth and Armin for their work, but I find that Armin sometimes puts way too much trust in magic :)
  - the_mitsuhiko 14y ago
    Where did I do that?
    - ak217 14y ago
      http://dev.pocoo.org/~mitsuhiko/didntknow.pdf http://dev.pocoo.org/~mitsuhiko/didntknow.pdf "Custom Ducks" and "Cached Properties" terrify me a bit. Not Pythonic (as in, not easily readable). But on the whole, I take that back. I was looking through your talks and realized that was the only place I got that feeling, and also found http://dev.pocoo.org/~mitsuhiko/badideas.pdf http://dev.pocoo.org/~mitsuhiko/badideas.pdf which takes a stand against magic. Edit: I still think arguments would be better (more explicit/readable) than globals (request, etc.) in Flask, but it's your module and I know you've considered the reasons to make it that way. And globals don't exactly qualify as magic.
      - asksol 14y ago
        I love cached_property, and I don't see what makes it unpythonic. I don't consider it magic, it's just a small addition to the built-in property descriptor, and it makes the often repeated code easier to read. I'm sure it will be a common idiom very soon.
  - eric_bullington 14y ago
    Are you perhaps confusing Armin with the creator of another Python web framework? One of the things I love about werkzeug and Flask is the relative absence of magic. Compared to many other popular web frameworks, his projects are extremely Pythonic.
    - ak217 14y ago
      No, I mean the right Armin. See the other reply though. I only got that feeling once when looking through his talks, and I otherwise take that sentiment back.
tbatterii 14y ago
i just spent 20 minutes reviewing the code. it is indeed very clean compared to the last time I remember looking at it. great job kenneth
RyanZAG 14y ago
Leaves the following interesting question: If the developer interface was elegant, and the library itself ran very well without any errors, is anything actually gained by making the internal code elegant? What happens if the new version, while more elegant internally, has a new complex bug in it (such as a race condition, that only triggers once every billion requests)? Still worthwhile to fix up the inelegant code that was working correctly? (Disclaimer: I feel it's better to clean it up regardless, but I have a feeling a study would show something different in terms of lost productivity...)
- kenneth_reitz 14y ago
  It does surface to the lower-level user interfaces, as shown in the blog post's connection adapter example. For advanced users, these changes will mean the world. The new code is also much easier to debug and pleasure to work with. Bugs will be significantly easier to fix :)
- nailer 14y ago
  > What happens if the new version, while more elegant internally, has a new complex bug in it (such as a race condition, that only triggers once every billion requests)? Still worthwhile to fix up the inelegant code that was working correctly? Then it's easier to find. All software has bugs, readability is one important part of fixing them.
- deleted 14y ago
  [deleted]
- dalke 14y ago
  The "developer interface" includes at least three different parts: the API calls, the calling sequence, and the validity of the results. That last part is borderline. Some wouldn't call it part of the interface, but I do, because if there are problems then interface users end up wrapping the interface with another interface to work around those problems. I therefore argue that the "more elegant but buggier" internals don't have the same interface, even if the API calls look the same. As to why it should be more elegant? There are at least four parts here . How does the developer, or any interface user, review the code to see if there is a complex bug inside? That is, your scenario about the bug in the elegant code also applies to the inelegant. And how easy is it to carry out extensions to the project in the old code base vs. the new, even if those new features aren't yet implemented? Third, the type of person who strives for an elegant interface also probably strives for elegant internals. Fourth, there are people who don't like closed boxes, and want to understand some of the insides of a package before they use it. If the insides don't look good, then they might be adverse to using it. A term for the negative aspects of this tendency to clean up internal code is called "gold plating." A positive term is "technical debt", as in the inelegant code incurs a debt which must be paid off in the future by cleaning up the code.

Dude you are my hero. I've printed requests source code and use it as an example of great python code. You and guys like Armin Ronacher are the reason why I use and like Python so much !!! (when jerking stops) Question: I have a feeling that with Requests you're just hiding the problem and not trying to solve it (Urllib is still in the back). Shouldn't urllib(1,2,3...) be rewritten instead?

jevinskie 14y ago
Probably, but rewriting a core module (this would be urllib4!?) is a lot harder (politically at least) than creating a 3rd party wrapper module.
kenneth_reitz 14y ago
Thanks! Urllib isn't used at all, actually. Internally, I use a project called 'urllib3', but it's totally unrelated to urlilb. It's name is actually a joke. It is essentially a light wrapper around httplib that provides connection pooling :)
- ak217 14y ago
  By the way, I found a bug in httplib where, when sending a request body, if the peer closes the socket early with an error (HTTP/1.1 compliant behavior), the request will time out instead of returning the correct error information. To my knowledge, this is also present in httplib2. This is because the socket write in both libraries is blocking (which seems simply wrong). I'm hoping to send a patch sometime soon. Regardless, httplib2 seems to have lots of other technical advantages over httplib - have you considered using it?
  - kenneth_reitz 14y ago
    http://code.google.com/p/httplib2/source/browse/python2/httplib2/__init__.py#1598 http://code.google.com/p/httplib2/source/browse/python2/http... The entire process of sending a Request is wrapped in a 'try/except Exception' block. wtf. It also just aimlessly retries failed requests. This is not configurable. Httplib2 is terrible. I actually created Requests partially in spite of it.
    - ak217 14y ago
      Makes sense, thanks. I'll have to work on patching up httplib, then.
  - RyanMcGreal 14y ago
    httplib2 is not thread-safe. That pretty much makes it a deal-breaker, even if the API is a significant improvement over the standard library (though not over requests).
Goranek 14y ago
Well that's a different story then :) Good job!
ak217 14y ago
I really respect both Kenneth and Armin for their work, but I find that Armin sometimes puts way too much trust in magic :)
- the_mitsuhiko 14y ago
  Where did I do that?
  - ak217 14y ago
    http://dev.pocoo.org/~mitsuhiko/didntknow.pdf http://dev.pocoo.org/~mitsuhiko/didntknow.pdf "Custom Ducks" and "Cached Properties" terrify me a bit. Not Pythonic (as in, not easily readable). But on the whole, I take that back. I was looking through your talks and realized that was the only place I got that feeling, and also found http://dev.pocoo.org/~mitsuhiko/badideas.pdf http://dev.pocoo.org/~mitsuhiko/badideas.pdf which takes a stand against magic. Edit: I still think arguments would be better (more explicit/readable) than globals (request, etc.) in Flask, but it's your module and I know you've considered the reasons to make it that way. And globals don't exactly qualify as magic.
    - asksol 14y ago
      I love cached_property, and I don't see what makes it unpythonic. I don't consider it magic, it's just a small addition to the built-in property descriptor, and it makes the often repeated code easier to read. I'm sure it will be a common idiom very soon.
- eric_bullington 14y ago
  Are you perhaps confusing Armin with the creator of another Python web framework? One of the things I love about werkzeug and Flask is the relative absence of magic. Compared to many other popular web frameworks, his projects are extremely Pythonic.
  - ak217 14y ago
    No, I mean the right Armin. See the other reply though. I only got that feeling once when looking through his talks, and I otherwise take that sentiment back.

tbatterii 14y ago

i just spent 20 minutes reviewing the code. it is indeed very clean compared to the last time I remember looking at it. great job kenneth

RyanZAG 14y ago

Leaves the following interesting question: If the developer interface was elegant, and the library itself ran very well without any errors, is anything actually gained by making the internal code elegant? What happens if the new version, while more elegant internally, has a new complex bug in it (such as a race condition, that only triggers once every billion requests)? Still worthwhile to fix up the inelegant code that was working correctly? (Disclaimer: I feel it's better to clean it up regardless, but I have a feeling a study would show something different in terms of lost productivity...)

kenneth_reitz 14y ago
It does surface to the lower-level user interfaces, as shown in the blog post's connection adapter example. For advanced users, these changes will mean the world. The new code is also much easier to debug and pleasure to work with. Bugs will be significantly easier to fix :)
nailer 14y ago
> What happens if the new version, while more elegant internally, has a new complex bug in it (such as a race condition, that only triggers once every billion requests)? Still worthwhile to fix up the inelegant code that was working correctly? Then it's easier to find. All software has bugs, readability is one important part of fixing them.
deleted 14y ago
[deleted]
dalke 14y ago
The "developer interface" includes at least three different parts: the API calls, the calling sequence, and the validity of the results. That last part is borderline. Some wouldn't call it part of the interface, but I do, because if there are problems then interface users end up wrapping the interface with another interface to work around those problems. I therefore argue that the "more elegant but buggier" internals don't have the same interface, even if the API calls look the same. As to why it should be more elegant? There are at least four parts here . How does the developer, or any interface user, review the code to see if there is a complex bug inside? That is, your scenario about the bug in the elegant code also applies to the inelegant. And how easy is it to carry out extensions to the project in the old code base vs. the new, even if those new features aren't yet implemented? Third, the type of person who strives for an elegant interface also probably strives for elegant internals. Fourth, there are people who don't like closed boxes, and want to understand some of the insides of a package before they use it. If the insides don't look good, then they might be adverse to using it. A term for the negative aspects of this tendency to clean up internal code is called "gold plating." A positive term is "technical debt", as in the inelegant code incurs a debt which must be paid off in the future by cleaning up the code.