12 ms·
Why I forked httpx
- swiftcoder 6mo agoSomehow I confused httpx with htmlx
- croemer 6mo agoSame! Only just realized it thanks to your comment.
- Tade0 6mo agoI thought your comment was starting with "Samuel". Plenty of people on sick leave as of late - must be difficult for many to focus their sight.
- eknkc 6mo agoAnd also htmlx with htmx I guess?
- g947o 6mo agoI guess you mean htmx. Same here. I read the article for a while, and was confused by "HTTPX is a very popular HTTP client for Python." and wondering "why is OpenAI using htmx", until I eventually realized what's going on.
- jordiburgos 6mo agoI've been reading the whole article wrong too.
- globular-toast 6mo agoIt's a shame, httpx has so much potential to be the default Python http library. It's crazy that there isn't one really. I contributed some patches to the project some years ago now and it was a nice and friendly process. I was expecting a v1 release imminently. It looks like the author is having some issues which seem to afflict so many in this field for some reason. I notice they've changed their name since I last interacted with the project...
- WesolyKubeczek 6mo agoYou try to touch low level HTTP with Python, and once you dive into both RFC2616 and Python deep enough, your brain is cooked, basically. Look at what happened to the author of requests, a textbook example. Or maybe it is that your brain is cooked already, or is on the brink, and your condition attracts you to HTTP and Python, after which it basically has you. The only way to not go bonkers is to design a library by commitee, so that the disease spreads evenly and doesn't hit any one individual with full force. The result will be ugly, but hopefully free of drama.
- mesahm 6mo agothe http landscape is rather scary lately in Python. instead of forking join forces... See Niquests https://github.com/jawah/niquests https://github.com/jawah/niquests I am trying to resolve what you've seen. For years of hard work.
- Orelus 6mo agoCan confirm, more features, a breeze to switch.
- greatgib 6mo agoThe basis of httpx is not very good at all. I think that it owes its success to be first "port" of python requests to support async, that was a strong need. But otherwise it is bad: API is not that great, performance is not that great, tweaking is not that great, and the maintainer mindset is not that great also. For the last point, few points were referenced in the article, but it can easily put your production project to suddenly break in a bad way without valid reason. Without being perfect, I would advise everyone to switch to Aiohttp.
- mesahm 6mo agoaiohttp is an excellent library. very stable. I concurs, but! it's too heavily tied to HTTP/1, and well, I am not a fan of opening thousands of TCP conn just to keep up with HTTP/2 onward. niquests easily beat aiohttp just using 10 conn and crush httpx see https://gist.github.com/Ousret/9e99b07e66eec48ccea5811775ec116d https://gist.github.com/Ousret/9e99b07e66eec48ccea5811775ec1... fwiw, HTTP/2 is twelve years old, just saying.
- sammy2255 6mo agoaiohttp is for asynchronous contexts only
- sgt 6mo agoI literally the other week had the choice between using requests and httpx. I chose httpx after deliberating a bit. I don't need async capabilities right now but I figured it'll be more consistent if that changes later.
- cies 6mo agoHi Michiel! Just a small headsup: clicking on the Leiden Python link in your About Me page give not the expected results. And a small nitpick: it's "Michiel's" in English (where it's "Michiels" in Dutch). Thanks for devoting time to opensource... <3
- roywashere 6mo agothanks, I hope I fixed the https://pythonleiden.nl https://pythonleiden.nl website now
- sdovan1 6mo agoI guess the Discussion on Hacker News href should be "https://news.ycombinator.com/item?id=47514603 https://news.ycombinator.com/item?id=47514603" instead of "news.ycombinator.com/item?id=47514603"
- ayhanfuat 6mo agoMore related drama: The Slow Collapse of MkDocs (https://fpgmaas.com/blog/collapse-of-mkdocs/ https://fpgmaas.com/blog/collapse-of-mkdocs/)
- znpy 6mo agoOh i recognised one of the involved people immediately, drama person. I still think that hijacking the mkdocs package was the wrong way to go though. The foss landscape has become way too much fork-phobic. Just fork mkdocs and go over your merry way.
- rglullis 6mo agoDrama around Starlette. Drama around httpx. Drama around MkDocs. I just hope that DRF is not next, I still have some projects that depend on it.
- forkerenok 6mo agoWhat's the drama around starlette? (Can't find anything)
- mananaysiempre 6mo agohttps://github.com/Kludex/starlette/issues/3180 https://github.com/Kludex/starlette/issues/3180 and before that https://github.com/Kludex/starlette/issues/3042 https://github.com/Kludex/starlette/issues/3042
- noirscape 6mo agoI think that may be the first time I've seen licensing drama over something as minor as adding another author to the copyright list. Pretty sure those are completely standard for major changes in maintainers/hostile forks/acknowledging major contributors. I've seen a lot of abandoned MIT/BSD projects add a new line for forks/maintainers being active again in order to acknowledge that the project is currently being headed by someone else. From my "I am not a lawyer" view, Kludex is basically correct, although I suppose to do it "properly", he might need to just duplicate the license text in order to make it clear both contributors licensed under BSD 3-clause. Probably unnecessary though, given it's not a license switch (you see that style more for ie. switching from MIT to BSD or from MIT/BSD to GPL, since that's a more substantial change); the intent of the license remains the same regardless and it's hard to imagine anyone would get confused. I suspect (given the hammering on it in responses), that Kludex asking ChatGPT if it was correct is what actually pissed off the original developer, rather than the addition of Kludex to the list in and of itself.
- paseante 6mo ago[dead]
- nathell 6mo agoCongratulations on forking! Always remember that open-source is an author’s gift to the world, and the author doesn’t owe anything to anyone. Thus, if you need a feature that for whatever reason can’t or won’t go upstream, forking is just about the only viable option. Fingers crossed!
- cachius 6mo agoThis is not merely open-source, but taking part in a huge package ecosystem in a foundational role in an XKCD 2347 type of way for HTTP requests. Put your side project on your personal homepage and walk away - fine. Make it central infrastructure - respond to participants or extend or cede maintainership.
- troad 6mo agoIf "taking part in a huge ecosystem in a foundational role" means 'other people choosing to use your FOSS software', and I can't think of what else it would mean, then no, you have no obligation to do any of that. FOSS means the right to use and fork. That's all it means. That's all it ever meant. Any social expectations beyond that live entirely in your imagination.
- duskdozer 6mo agoA foundational role in a huge open-source package ecosystem? I wonder what such an esteemed position pays.
- Yokohiii 6mo agoA (hypothetical) professional propriety project at same scale would probably feed a handful of people, with much less stress. FOSS version is zero cash and exaggerated community demands. Dream job.
- Yokohiii 6mo agoI guess frustration speaks here? There is simply no responsibility an OSS maintainer has. They can choose to be responsible, but no one can force them. Eventually OSS licensing is THE solution at heart to solve this problem. Maintainers go rogue? Fork and move on. But surprise, who is going to fork AND maintain? Filling in all the demands from the community, for potentially no benefit? No one can force him to take the responsibility, just like no one can force anyone else to.
- mettamage 6mo ago> Visitor 4209 since we started counting Loved that little detail, reminds me of the old interwebs :)
- croemer 6mo agoIt's gone from 45 when I looked at it an hour ago to 261 just now.
- glaucon 6mo agoGood line from the blog post ... "So what is the plan now?" - "Move a little faster and not break things"
- deleted 6mo ago[deleted]
- Kwpolska 6mo agoWhat is it about Python that makes developers love fragmentation so much? Sending HTTP requests is a basic capability in the modern world, the standard library should include a friendly, fully-featured, battle-tested, async-ready client. But not in Python, stdlib only has the ugly urllib.request, and everyone is using third party stuff like requests or httpx, which aren't always well maintained. (See also: packaging)
- maccard 6mo ago> Then I found out it was broken. I contributed a fix. The fix was ignored and there was never any release since November 2024. This seems like a pretty good reason to fork to me. > Sending HTTP requests is a basic capability in the modern world, the standard library should include a friendly, fully-featured, battle-tested, async-ready client. But not in Python, Or Javascript (well node), or golang (http/net is _worse_ than urllib IMO), Rust , Java (UrlRequest is the same as python's), even dotnet's HttpClient is... fine. Honestly the thing that consistently surprises me is that requests hasn't been standardised and brought into the standard library
- lenkite 6mo agoYour java knowledge is outdated. Java's JDK has a nice, modern HTTP Client https://docs.oracle.com/en/java/javase/11/docs/api/java.net.http/java/net/http/HttpClient.html https://docs.oracle.com/en/java/javase/11/docs/api/java.net....
- ffsm8 6mo agoAhh, java. You never change, even if you're modern HttpClient client = HttpClient.newBuilder() .version(Version.HTTP_1_1) .followRedirects(Redirect.NORMAL) .connectTimeout(Duration.ofSeconds(20)) .proxy(ProxySelector.of( new InetSocketAddress("proxy.example.com", 80) )) .authenticator(Authenticator.getDefault()) .build(); HttpResponse<String> response = client.send(request, BodyHandlers.ofString()); System.out.println(response.statusCode()); System.out.println(response.body()); For the record, you're most likely not even interacting with that API directly if you're using any current framework, because most just provide automagically generated clients and you only define the interface with some annotations
- eats_indigo 6mo agosmells like supply chain attack
- souvlakius 6mo agoYeah, it's a shame because otherwise the library is really nice and could have become the default HTTP library, but it feels like someone will manage to inject some weird behaviour soon and half the planet will be compromised
- localuser13 6mo agoI'm not a lawyer, but are there any potential trademark issues? AFAIK in general you HAVE to change the name to something clearly different. I consider it morally OK, and it's probably fine, but HTTPXYZ is cutting it close. It's too late for a rebrand, but IMO open-source people often ignore this topic a bit too much.
- Gander5739 6mo agoIs httpx trademarked? I couldn't find anything indicating it was.
- CorrectHorseBat 6mo agoDon't you need to register and actively defend you trademark for it to apply?
- sushibowl 6mo agoThere are unregistered trademarks as well as registered ones. Usually the "TM" symbol is applied to unregistered trademarks, and the ® symbol for registered ones. Both enjoy protection, although it's generally an easier time in court when your trademark is registered. Whether actively defending your trademark is actually required is a bit of a nuanced topic. Generally, trademarks can be lost through genericide (the mark becomes a generic term for the type of product) or abandonment. Abandonment happens when either the mark owner stops using the mark itself, or takes an action that weakens the mark. The question, then, is whether failing to defend infringing use constitutes a weakening action. Courts differ on this, and there is a large gray area between "we didn't immediately sue a local mom-and-pop shop" and "we allowed a rival company to use the mark erroneously across several states for years without taking action."
- nwellnhof 6mo agoIn this case, the name is already so generic that you might even be denied a trademark in the first place.
- ahoka 6mo ago
- zeeshana07x 6mo agoThe lack of a well-maintained async HTTP client in Python's stdlib has been a pain point for a while. Makes sense someone eventually took it into their own hands
- WhyNotHugo 6mo agoAn async HTTP client in the stdlib would also be great for tools like pip, which could really benefit from doing more async work. One of the reasons that uv is much faster is precisely this.
- notatallshaw 6mo agoAs a pip maintainer I don't think that's really true. The resolver in both pip and uv are fundamentally sequential and single threaded, you can't really queue up or split out jobs. What uv does is parallelize the final download of packages after resolution, and batch pre-fetch metadata during resolution. I don't think these benefit from async, due to their batch nature classic multi-threaded download pools are probably the better solution, but I could be wrong! Experiments have been done on the former in pip and didn't find much/any improvement in CPython, this may change in free threaded CPython. For the latter we currently don't have the information from the resolver to extract a range of possible metadata versions we could pre-range, I am working on this but it requires new APIs in packaging (the Python library) and changes to the resolver, and again we will need to benchmark to see if adding pre-fetching actually improves things.
- cachius 6mo agoAnother abandoned project hurting users: https://github.com/benweet/stackedit https://github.com/benweet/stackedit
- Spivak 6mo agoDo you see yourself taking over httpcore as well as it's likely to have the same maintainership problem? It would certainly instill more confidence that this is a serious fork. This certainly wouldn't be the first time an author of a popular library got a little too distracted on the sequel to their library that the current users are left to languish a bit.
- fede_dp 6mo ago[dead]
- maltyxxx 6mo ago[flagged]
- leontloveless 6mo ago[dead]
- joouha 6mo agoThis sounds like an ideal use case for modshim [0] One of its intended use cases is bridging contribution gaps: while contributing upstream is ideal, maintainers may be slow to merge contributions for various reasons. Forking in response creates a permanent schism and a significant maintenance burden for what might be a small change. Modshim would allow you to create a new Python package containing only the fixes for your bugbears, while automatically inheriting the rest from upstream httpx. [0] https://github.com/joouha/modshim https://github.com/joouha/modshim
- robmccoll 6mo agoSince modshim isn't money patching and appears to only be wrapping the external API of a package, if the change is deep enough inside the package, wouldn't you end up reimplementing most of the package from the outside?
- joouha 6mo agoModshim does more than just wrap the external API of a package - it allows you to tweak something internal to the module while leaving its interface alone, without having to re-implement most of the package in order to re-bind new versions of objects. There are a couple of example of this readme: (1) modifing the TextWrapper object but then use it through the textwrap library's wrap() function, and (2) modifing the requests Session object, but then just using the standard requests.get(). Without modshim (using standard monkey-patching) you would have to re-implement the wrap and get methods in order to bind the new TextWrapper / Session classes.
- ial32 6mo agoI'm not necessarily interested in using modshim, but it got me wondering a couple of things out of curiosity: - what's the performance like for big packages (say, pytorch)? Have you done some benchmarking? - is typing kept for the shims? My immediate guess, again, without even trying it out, is not. If yes, how? edit: formatting
- bustah 6mo ago[flagged]
- zahlman 6mo ago> The fix was ignored and there was never any release since November 2024. Me, and others, asked repeatedly for a release containing my fix. I sent email to the author personally. I got response when I added that I was considering forking. The author replied “1.0 development is on course”.... I do understand about maintainer burnout, and preferring to work on ‘next’, and that there is life outside of Python, but I think not doing anything for maintenance and also not letting other people help out in maintaining, for such a high profile module, is problematic. I feel like it's counterproductive in situations like this to mention forking. It will come across like a threat, when there isn't really anything intrinsically aggressive about it. So just do it; and when you have a decent amount of separate development, you can decide whether to make PRs back, advertise your fork, etc.
- nateb2022 6mo agoI'll plug Pyreqwest here: https://github.com/MarkusSintonen/pyreqwest https://github.com/MarkusSintonen/pyreqwest It's been a pleasure to use, has a httpx compatibility layer for gradually migrating to its API, and it's a lot more performant (right now, I think it's the most performant Python http client out there: https://github.com/MarkusSintonen/pyreqwest/blob/main/docs/benchmarks.md https://github.com/MarkusSintonen/pyreqwest/blob/main/docs/b...)
- renegat0x0 6mo agoThere are many nice http clients: - httpx - curl cffi - httpmorph - httpcloak - stealth crawler I wrote a framework, link below, which uses them all. You can compare each to verify crawling speed. Some sites can be cleanly crawled with a one particular framework. Having read the article I am in a pain. I do break things while development. I rewrite stuff. Maybe some day I will find a way to develop things "stable". One thing I try to keep in good shape is 'docker' image. I update it once everything seems to be quite stable. https://github.com/rumca-js/crawler-buddy https://github.com/rumca-js/crawler-buddy