7 ms·

I really feel like Python has an embarrassment of riches in terms of web frameworks. Django, Flask, and even FastAPI are master classes in how to build great op

by madrox 4y ago

I really feel like Python has an embarrassment of riches in terms of web frameworks. Django, Flask, and even FastAPI are master classes in how to build great open source projects that will survive the test of time. In my opinion they're a huge reason why Python continues to be a popular language for backend development.

pid-1 4y ago
Something not often mentioned: WSGI and ASGI do a great job at providing a common, well-thought abstraction to library writers.
deaddodo 4y ago
Definitely, the biggest hiccup has been the py3 migration. But generally, once a framework is stable on a major version, it seems to have very minor necessities for upkeep (though, changes for new features definitely can improve the functionality). As to flask, I would love to see it reach the milestone that Openbox has achieved of “being done” and only really being updated for bug+security fixes or base language feature updates. As a “micro framework”, it has perfectly filled it’s necessary niche.
- JustSomeNobody 4y ago
  Sadly, if it were “done”, a lot of devs would complain that it was dead and stop using it.
  - Supermancho 4y ago
    > Sadly, if it were “done”, a lot of devs would complain that it was dead and stop using it Both pessimistic and incorrect. You can observe the vast number of libraries across all ecosystems that are in use, as reliable libraries, well into functional obsolescence. Not to say some niche software doesn't suffer the fate of "nothing new, so it's dead" but that's not the norm. Web frameworks are never feature-complete, per se. There are always new technologies and workflows to support. Frameworks are usually considered dead when the development halts in exhaustion or in transition to another project (as individuals, the development team members drift to these modalities), not when they are "done" by some finite collection of features.
0xFACEFEED 4y ago
Why not the other way around? What did Python have relative to other languages that inspired these developers to devote so much energy into their projects? They could have chosen PHP, Ruby, Node, Go, Java, etc. If I think back to ~2012 I remember that compared to other languages Python had: 1) Decent package management and "virtualization" (via virtualenv) 2) Nice balance of expressiveness and strictness 3) Was fun to write, hard to explain but it was the vibe 4) Decent enough performance for most things 5) Lots of developer tooling for debugging and stuff It felt like the least messy of the scripting languages. - Everyone knows all the issues PHP had, no need to list it all here - Ruby was cool but there were a thousand ways to do the same thing - Node was inheriting that weird web language with all its quirks - Go was just in its infancy - Java was the kitchen sink
- throwaway894345 4y ago
  Sadly Python has stood still in the interim, focusing mostly on completing the migration from 2 to 3, by which point the competition was much farther along and continuing to innovate at a rapid pace (performance, package management, type system, tooling, etc).
  - 0xFACEFEED 4y ago
  - jonnycomputer 4y ago
    Honestly, I'm having a hard time buying this opinion. Python3 has changed a lot in the last 10 years, and much of those changes have been for the better.
    - throwaway894345 4y ago
      I guess I’ll preface by saying I’ve been developing and operating Python services for about a decade, and I did a bunch of Python before that in various capacities. I’ve made a lot of money with Python, and I’m not one of the people who tars a language without having used it in anger. Those changes have largely been incremental and/or questionable. Python has async now, which is cool, but anyone calling a sync call or some poorly optimized algorithm anywhere in the stack can bring the entire application down. Type hints are moving slowly in the right direction, but they’re still really cumbersome and mypy is still a pain to use (publishing type hints, recursive types, loading type hints, etc). Last I tried it (admittedly it’s been a year since I was deep in the Python game) it was still pretty beta quality. Package management still sucks. Deployment still sucks. Yeah, Docker helps, but it also makes some things worse, like local dev and build times and so on. And there aren’t many good solutions for distributing CLIs and other apps to users—some of the zip things work okay but they don’t close over the runtime, stdlib, or shared objects. Moreover, Python dependency zips can be huuuge—we were busting the 250MB lambda limit (compressed—this was back when lambda had a limit, pretty sure they upped/removed it since) meanwhile a comparable Go binary weighed in at 6MB. Performance still sucks. Yeah, the 3.11 improvements will feel big to a Python programmer, but 40% (made up number) faster than 1000X slower than Go is still really slow (yeah, “just write the slow parts in C/Numpy/multiprocessing/whatever” works once in a while, but very often it doesn’t and it’s basically impossible to know from the project’s outset whether all of its bottlenecks will be amenable to that particular optimization). So to summarize areas where Python has been left in the dust (probably an incomplete list): * Performance * Package management * Static type system * Deployment * Async
Larrikin 4y ago
Django was the first web framework I took the time to learn about a decade ago. I've learned a lot of different languages and done various types of projects, but whenever I get free time to start a personal project that has a web component in my favorite language at the time, I'm always a little bit sad I inevitably have to build out something that was just included in Django a decade ago.
- Noughmad 4y ago
  Why not build the web component in Django, and the rest in your language of choice?
pictur 4y ago
nodejs is still in bad shape in my opinion in this regard. There are many frameworks, but most of them are things that do not work stable or their features are not fully settled. maybe the problem is directly related to javascript, i don't know.
- venomsnake 4y ago
  The larger the codebase, the greater the need for stronger typing. When dealing with JS this typing should be in the head of the developer. Too hard and too unreliable. Typescript seems to be having it's heart in the right place though.
- 0xFACEFEED 4y ago
  Not JavaScript but the way Node was originally designed. When your business application reaches a point of "non-triviality" (for lack of a better term) you'd (hopefully) realize that you need to ditch Node ASAP or face growing pains. To understand why requires some historical context. During the time when Node was becoming popular most scripting languages had weird bottlenecks at the request level. PHP: When a request was received, your whole application would load into memory. You know how an application might read a config file on startup? Yea, every single request would load that file. There was no concept of "global variables across all requests". That's one OS process per request. Sometimes process forking was used to make that faster but it didn't matter that much because your entire application needed to load for every request to be handled. Python: You'd use a toolkit like uWSGI to launch N number of processes when your server started. The processes would stay up and continue to serve requests. You'd avoid the overhead of initializing your app on every request, but you were still locked to one process per request because of the GIL. Ruby: Basically the same as Python. Java, C++, Go, etc: Too slow to develop in, too complicated for rapid iteration. Type systems scary. Weak dynamic typing fun. Productivity was king and you needed some time in the hard languages to git gud. Okay so the fundamental problem with PHP/Python/Ruby was that you needed to tie up a single OS process per request. Let's say you wrote an HTTP API that would fetch the current time from time.gov or something. While your HTTP handler was fetching data from time.gov that OS process would be STUCK. It's sitting around and doing nothing while holding your whole application state in memory just for that one request you're serving. This wasn't actually a problem, not really. You could serve an ridiculous amount of web requests using this model. Computers are fast! Until... you needed concurrency... like a chat app. Because in a chat app you need to keep a TCP request open for every single user in the system. 500k active users? 500k OS processes just sitting there doing nothing with all of your application state in memory. Not a good fit for PHP/Python/Ruby where every open request was tying up all those resources. So how would you solve it in C++/Java/etc? Easy, you'd use non-blocking IO. With some fancy connection pooling, threading, and use of the epoll_wait syscall (or whatever) you could handle all 500k users with a SINGLE OS process. That's because your chat app is really just routing bytes between client applications. Most of the time is spent in I/O, not in your application. Except non-blocking IO is not easy. Threading is not easy. Connection pooling is not easy. Type systems, code compilation, etc etc are not easy. Enter NodeJS. The premise is simple. Node is a single threaded runtime environment that does all IO using non-blocking IO. All accessible via a well known language called JavaScript. No memory management, no types - oh and functions are first class primitives. Nice. Oh and don't try to calculate the Nth prime number because that'll block your whole single threaded application even if it's serving 500k requests. All of the early demos of Node (that I saw) were basically chat apps. Applications with very little business logic that spend most of their time doing IO. It was more of a glorified router of bytes... But then people realized that CPUs are actually really fast and you could start modeling basic web apps. All you're doing in most web apps is pulling some data out of a DB, decorating that data, and pushing it out of NodeJS. Single threading is no problem. So people starting building on it... and building on it... until they needed to scale for real. Some unfortunate souls didn't realize any of this and built complex business applications in Node. They struggled a lot with the single-threaded nature of this environment. That's when you started seeing things like Node Cluster which basically used threading under the hood to distribute some of that CPU load. Over time better alternatives came along. So the reason you don't see big complex frameworks in Node is because Node doesn't need them. Using a big complex framework means you're shoving way too much business logic into a technology that wasn't designed for it. Express and Koa are basically peak Node "frameworks". They're effectively just syntax sugar for cleanly routing your requests somewhere else. Need something more than that? Look elsewhere or you'll regret it later! EDIT: Fixed a bunch of typos. Did not expect to write so much.
arcturus17 4y ago
I built a production app in Django last year, and I loved a lot of things about it, but I felt some important shortcomings. Async support was clunky at best... The Django way of doing things has traditionally been to either use a queueing system like Celery or multi-threading, but I find first-class async like in Node is much easier for many typical web tasks (database I/O and calling external services). I also faced issues with libraries for menial tasks. My setup was Django + React and JWT was poorly supported - the most popular auth library needed quite a bit of manual patching in code to make it all work. This was very surprising... I walked away with the feeling that some of the libs for what should be common use cases nowadays are not "production-grade". DRF while powerful and well-documented also had a decent amount of quirks for things that were not so far at the edges. Raw deployment on PaaS like Heroku and Elastic Beanstalk (tried both during the project, ended up settling on the latter) without Docker or K8s, also a bit messier than I'd have liked at times, but ultimately OK. Things I really liked: models and the ORM (again the occasional niggle at the edges but good 98% of the time), most of DRF, Python itself, Python types, the admin panel, management commands, the custom commands "framework", testing. But as it stands, with my qualms above, I think for the next project that deserves a full-batteries framework, I might give something like .Net or Spring + Kotlin a chance.
- g0ran 4y ago
  > the most popular auth library needed quite a bit of manual patching in code to make it all work Which one did you use?
  - arcturus17 4y ago
    It was dj-rest-auth, the version that continues to be supported, with the jwt plugin.
- bkovacev 4y ago
  Just a few observations: - Async... The Django way of doing things... Celery -> it's a background task system which is like bringing a bazooka to kill a fly for what you wanted to do (I assume proxying the request to an external API). - database I/O and calling external services -> Use gunicorn gevent worker type that brings up the pseudo threads. - JWT was poorly supported -> There's at least 5 battle tested libraries that you can use to implement JWT alone (simplejwt comes to mind first) - the most popular auth library needed quite a bit of manual patching in code to make it all work -> Which one? There's few that are standard and most are easy to customize via global settings. - Raw deployment on PaaS like Heroku and Elastic Beanstalk -> that's more on the platform rather than Django itself - DRF -> I agree, DRF has it's own way of doing things and deviating from them can give you massive headaches.
mmcnl 4y ago
I disagree about FastAPI in this list of frameworks. Django and Flask have great documentation, FastAPI does not. There is no API reference for FastAPI, so often you will have to read the source code to understand what options are available and how to use them. Also the FastAPI explanation about concurrency (https://fastapi.tiangolo.com/async/#concurrent-burgers https://fastapi.tiangolo.com/async/#concurrent-burgers) is terrible. It's filled with emojis and seems to be written for children.
- floppydiskette 4y ago
  It’s the worst thing I’ve ever seen in documentation. It made it really hard to take the entire project seriously.
alberth 4y ago
Call me old but I remember in the early 00s just unbelievably fragmented and difficult it was to use python on the web. And then you had PHP, where you didn’t need any middleware and could just drop a file into your webroot and it just worked.

by madrox 4y ago

pid-1 4y ago

Something not often mentioned: WSGI and ASGI do a great job at providing a common, well-thought abstraction to library writers.

deaddodo 4y ago

Definitely, the biggest hiccup has been the py3 migration. But generally, once a framework is stable on a major version, it seems to have very minor necessities for upkeep (though, changes for new features definitely can improve the functionality). As to flask, I would love to see it reach the milestone that Openbox has achieved of “being done” and only really being updated for bug+security fixes or base language feature updates. As a “micro framework”, it has perfectly filled it’s necessary niche.

JustSomeNobody 4y ago
Sadly, if it were “done”, a lot of devs would complain that it was dead and stop using it.
- Supermancho 4y ago
  > Sadly, if it were “done”, a lot of devs would complain that it was dead and stop using it Both pessimistic and incorrect. You can observe the vast number of libraries across all ecosystems that are in use, as reliable libraries, well into functional obsolescence. Not to say some niche software doesn't suffer the fate of "nothing new, so it's dead" but that's not the norm. Web frameworks are never feature-complete, per se. There are always new technologies and workflows to support. Frameworks are usually considered dead when the development halts in exhaustion or in transition to another project (as individuals, the development team members drift to these modalities), not when they are "done" by some finite collection of features.

0xFACEFEED 4y ago

Why not the other way around? What did Python have relative to other languages that inspired these developers to devote so much energy into their projects? They could have chosen PHP, Ruby, Node, Go, Java, etc. If I think back to ~2012 I remember that compared to other languages Python had: 1) Decent package management and "virtualization" (via virtualenv) 2) Nice balance of expressiveness and strictness 3) Was fun to write, hard to explain but it was the vibe 4) Decent enough performance for most things 5) Lots of developer tooling for debugging and stuff It felt like the least messy of the scripting languages. - Everyone knows all the issues PHP had, no need to list it all here - Ruby was cool but there were a thousand ways to do the same thing - Node was inheriting that weird web language with all its quirks - Go was just in its infancy - Java was the kitchen sink

throwaway894345 4y ago
Sadly Python has stood still in the interim, focusing mostly on completing the migration from 2 to 3, by which point the competition was much farther along and continuing to innovate at a rapid pace (performance, package management, type system, tooling, etc).
- 0xFACEFEED 4y ago
- jonnycomputer 4y ago
  Honestly, I'm having a hard time buying this opinion. Python3 has changed a lot in the last 10 years, and much of those changes have been for the better.
  - throwaway894345 4y ago
    I guess I’ll preface by saying I’ve been developing and operating Python services for about a decade, and I did a bunch of Python before that in various capacities. I’ve made a lot of money with Python, and I’m not one of the people who tars a language without having used it in anger. Those changes have largely been incremental and/or questionable. Python has async now, which is cool, but anyone calling a sync call or some poorly optimized algorithm anywhere in the stack can bring the entire application down. Type hints are moving slowly in the right direction, but they’re still really cumbersome and mypy is still a pain to use (publishing type hints, recursive types, loading type hints, etc). Last I tried it (admittedly it’s been a year since I was deep in the Python game) it was still pretty beta quality. Package management still sucks. Deployment still sucks. Yeah, Docker helps, but it also makes some things worse, like local dev and build times and so on. And there aren’t many good solutions for distributing CLIs and other apps to users—some of the zip things work okay but they don’t close over the runtime, stdlib, or shared objects. Moreover, Python dependency zips can be huuuge—we were busting the 250MB lambda limit (compressed—this was back when lambda had a limit, pretty sure they upped/removed it since) meanwhile a comparable Go binary weighed in at 6MB. Performance still sucks. Yeah, the 3.11 improvements will feel big to a Python programmer, but 40% (made up number) faster than 1000X slower than Go is still really slow (yeah, “just write the slow parts in C/Numpy/multiprocessing/whatever” works once in a while, but very often it doesn’t and it’s basically impossible to know from the project’s outset whether all of its bottlenecks will be amenable to that particular optimization). So to summarize areas where Python has been left in the dust (probably an incomplete list): * Performance * Package management * Static type system * Deployment * Async

Larrikin 4y ago

Django was the first web framework I took the time to learn about a decade ago. I've learned a lot of different languages and done various types of projects, but whenever I get free time to start a personal project that has a web component in my favorite language at the time, I'm always a little bit sad I inevitably have to build out something that was just included in Django a decade ago.

Noughmad 4y ago
Why not build the web component in Django, and the rest in your language of choice?

pictur 4y ago

nodejs is still in bad shape in my opinion in this regard. There are many frameworks, but most of them are things that do not work stable or their features are not fully settled. maybe the problem is directly related to javascript, i don't know.

venomsnake 4y ago
The larger the codebase, the greater the need for stronger typing. When dealing with JS this typing should be in the head of the developer. Too hard and too unreliable. Typescript seems to be having it's heart in the right place though.
0xFACEFEED 4y ago
Not JavaScript but the way Node was originally designed. When your business application reaches a point of "non-triviality" (for lack of a better term) you'd (hopefully) realize that you need to ditch Node ASAP or face growing pains. To understand why requires some historical context. During the time when Node was becoming popular most scripting languages had weird bottlenecks at the request level. PHP: When a request was received, your whole application would load into memory. You know how an application might read a config file on startup? Yea, every single request would load that file. There was no concept of "global variables across all requests". That's one OS process per request. Sometimes process forking was used to make that faster but it didn't matter that much because your entire application needed to load for every request to be handled. Python: You'd use a toolkit like uWSGI to launch N number of processes when your server started. The processes would stay up and continue to serve requests. You'd avoid the overhead of initializing your app on every request, but you were still locked to one process per request because of the GIL. Ruby: Basically the same as Python. Java, C++, Go, etc: Too slow to develop in, too complicated for rapid iteration. Type systems scary. Weak dynamic typing fun. Productivity was king and you needed some time in the hard languages to git gud. Okay so the fundamental problem with PHP/Python/Ruby was that you needed to tie up a single OS process per request. Let's say you wrote an HTTP API that would fetch the current time from time.gov or something. While your HTTP handler was fetching data from time.gov that OS process would be STUCK. It's sitting around and doing nothing while holding your whole application state in memory just for that one request you're serving. This wasn't actually a problem, not really. You could serve an ridiculous amount of web requests using this model. Computers are fast! Until... you needed concurrency... like a chat app. Because in a chat app you need to keep a TCP request open for every single user in the system. 500k active users? 500k OS processes just sitting there doing nothing with all of your application state in memory. Not a good fit for PHP/Python/Ruby where every open request was tying up all those resources. So how would you solve it in C++/Java/etc? Easy, you'd use non-blocking IO. With some fancy connection pooling, threading, and use of the epoll_wait syscall (or whatever) you could handle all 500k users with a SINGLE OS process. That's because your chat app is really just routing bytes between client applications. Most of the time is spent in I/O, not in your application. Except non-blocking IO is not easy. Threading is not easy. Connection pooling is not easy. Type systems, code compilation, etc etc are not easy. Enter NodeJS. The premise is simple. Node is a single threaded runtime environment that does all IO using non-blocking IO. All accessible via a well known language called JavaScript. No memory management, no types - oh and functions are first class primitives. Nice. Oh and don't try to calculate the Nth prime number because that'll block your whole single threaded application even if it's serving 500k requests. All of the early demos of Node (that I saw) were basically chat apps. Applications with very little business logic that spend most of their time doing IO. It was more of a glorified router of bytes... But then people realized that CPUs are actually really fast and you could start modeling basic web apps. All you're doing in most web apps is pulling some data out of a DB, decorating that data, and pushing it out of NodeJS. Single threading is no problem. So people starting building on it... and building on it... until they needed to scale for real. Some unfortunate souls didn't realize any of this and built complex business applications in Node. They struggled a lot with the single-threaded nature of this environment. That's when you started seeing things like Node Cluster which basically used threading under the hood to distribute some of that CPU load. Over time better alternatives came along. So the reason you don't see big complex frameworks in Node is because Node doesn't need them. Using a big complex framework means you're shoving way too much business logic into a technology that wasn't designed for it. Express and Koa are basically peak Node "frameworks". They're effectively just syntax sugar for cleanly routing your requests somewhere else. Need something more than that? Look elsewhere or you'll regret it later! EDIT: Fixed a bunch of typos. Did not expect to write so much.

arcturus17 4y ago

I built a production app in Django last year, and I loved a lot of things about it, but I felt some important shortcomings. Async support was clunky at best... The Django way of doing things has traditionally been to either use a queueing system like Celery or multi-threading, but I find first-class async like in Node is much easier for many typical web tasks (database I/O and calling external services). I also faced issues with libraries for menial tasks. My setup was Django + React and JWT was poorly supported - the most popular auth library needed quite a bit of manual patching in code to make it all work. This was very surprising... I walked away with the feeling that some of the libs for what should be common use cases nowadays are not "production-grade". DRF while powerful and well-documented also had a decent amount of quirks for things that were not so far at the edges. Raw deployment on PaaS like Heroku and Elastic Beanstalk (tried both during the project, ended up settling on the latter) without Docker or K8s, also a bit messier than I'd have liked at times, but ultimately OK. Things I really liked: models and the ORM (again the occasional niggle at the edges but good 98% of the time), most of DRF, Python itself, Python types, the admin panel, management commands, the custom commands "framework", testing. But as it stands, with my qualms above, I think for the next project that deserves a full-batteries framework, I might give something like .Net or Spring + Kotlin a chance.

g0ran 4y ago
> the most popular auth library needed quite a bit of manual patching in code to make it all work Which one did you use?
- arcturus17 4y ago
  It was dj-rest-auth, the version that continues to be supported, with the jwt plugin.
bkovacev 4y ago
Just a few observations: - Async... The Django way of doing things... Celery -> it's a background task system which is like bringing a bazooka to kill a fly for what you wanted to do (I assume proxying the request to an external API). - database I/O and calling external services -> Use gunicorn gevent worker type that brings up the pseudo threads. - JWT was poorly supported -> There's at least 5 battle tested libraries that you can use to implement JWT alone (simplejwt comes to mind first) - the most popular auth library needed quite a bit of manual patching in code to make it all work -> Which one? There's few that are standard and most are easy to customize via global settings. - Raw deployment on PaaS like Heroku and Elastic Beanstalk -> that's more on the platform rather than Django itself - DRF -> I agree, DRF has it's own way of doing things and deviating from them can give you massive headaches.

mmcnl 4y ago

I disagree about FastAPI in this list of frameworks. Django and Flask have great documentation, FastAPI does not. There is no API reference for FastAPI, so often you will have to read the source code to understand what options are available and how to use them. Also the FastAPI explanation about concurrency (https://fastapi.tiangolo.com/async/#concurrent-burgers https://fastapi.tiangolo.com/async/#concurrent-burgers) is terrible. It's filled with emojis and seems to be written for children.

floppydiskette 4y ago
It’s the worst thing I’ve ever seen in documentation. It made it really hard to take the entire project seriously.

alberth 4y ago

Call me old but I remember in the early 00s just unbelievably fragmented and difficult it was to use python on the web. And then you had PHP, where you didn’t need any middleware and could just drop a file into your webroot and it just worked.