5 ms·

Python 2.7 will be around for a while due to legacy code, but it's only a matter of time before Python 3.x is more widely used. Don't invest time learning skill

Python 2.7 will be around for a while due to legacy code, but it's only a matter of time before Python 3.x is more widely used. Don't invest time learning skills that will only dwindle in value. Some stuff is broken in the ecosystem. That's true of all ecosystems. If you're coming from the JS world it's far, far better. But if you are importing every 0.x versioned library that vaguely solves a problem close to your problem, no ecosystem will save you. Python is a solid choice of language for the back ends of web applications.

lrem 11y ago
> Python is a solid choice of language for the back ends of web applications. With caution advised when it comes to scaling. You might, or might not, pay a pretty hefty premium in the amount of hardware needed.
- ianamartin 11y ago
  I don't want to come across as a Python zealot here, but in my experience, Python has almost never been the problem in scaling. Certainly not to the extent where I would say that it costs a premium in hardware. Though, I suppose it depends on the application. My experience has been that Python will give you more than enough rope to hang yourself with, and lots of apps that are meant to scale have some homegrown crap of an architecture, or people try to get fancy when they don't need to and try to write go or scala in Python. If you have a clear understanding of your data model, Python backends scale quite well vertically. Run one instance of your app per core you have available and use IPTables to round robin (or whatever. you can get fancier if you feel like it) to each app instance. Boom. You are now banging on all 4 or 8 or however many cores your server has without the context switch overhead of the multiprocessing module and also without the headaches that go along with the shared memory space of the multithreading libs. Not to mention cleaner, nicer code. If you build your product to work well in this way, then scaling horizontally also becomes a non-issue as far as Python code is concerned. You just put haproxy in front of multiple machines, set up health checks, and for the vast majority of use cases, you're done until you need to scale the storage horizontally. Then you have some serious decisions to make. But that's a different topic. But back to my original point: every case where I've seen Python being all slow has really been an artifact of a crap architecture, or bolting on crap front end frameworks, or depending too much on an ORM like SQLAlchemy without really understanding your data and depending on it to do everything for you instead of thinking rationally about what your data is. Or some combination of all three above. Yes, I know that there are languages that execute much faster than Python. I work with C#/.NET/SQL Server almost as much as I use the Python/Pyramid/PostgreSQL stack. I can write a crappy, slow, messy architecture on either stack. And in fact, when I was first getting to know C# and still writing Python code in that language, I did that. The bottom line is that if you understand the strengths, weaknesses, and idioms of the language you are working with right now, almost every high-level language in common use today is sufficient for massive-scale deployments for the most common types of web apps. Heavy computing or scientific purposes obviously excepted. But most web apps that scale to hundreds of millions of users are basically "window-on-data" type applications. I can't really think of a case where some other language besides Python would be noticeably cheaper hardware-wise. Maybe you have to know your entire stack a bit better if you want scalable Python apps, maybe you have to do a little more Jiggery-Pokery with your Linux servers, but if you are serving at massive scale, you are going to have to go there anyway, especially when you start dealing with storage. That's part of the game.
  - nawitus 11y ago
    > Run one instance of your app per core you have available and use IPTables to round robin (or whatever. you can get fancier if you feel like it) to each app instance. But if you're using Flask and you have 4 cores with 4 instances, and you're making 4 database requests at the same time you're whole application is suddenly locked. And it's not uncommon for a single view to make up to 4 separate database requests. Most modern web applications are I/O bound, therefore a single Node.js process would have better performance here.
    - ianamartin 11y ago
      That hasn't been my experience. If your ACLs are set up properly, you won't encounter a full table lock, and therefore won't have problems like you described. I agree that this could be a problem in theory. In practice, it has not been. Again, it boils down to how you architect your code. EDIT: you need to stick your sessions to a specific instance. Otherwise, you are correct that there's a problem. But as long as you stick your sessions, there's no problem with a database lock.
      - nawitus 11y ago
        I wasn't talking about database locks. My point was that making a database request takes a really long time relative to processing a request in the web application. Therefore the web application spends most of the time doing nothing and waiting for the database request to complete.
        ianamartin 11y ago
        So, write data-backed applications without calling a database? Not sure what you are getting at here. If you are talking about the raw scalability of your database, yes, you are correct. The speed with which you are able to get info from your DB will be a limiting factor. But that's a totally different point than the one I was making, which was about how to scale Python, not your database. If I understand you correctly, the time spent in retrieving stuff from your database will be a limiting factor no matter what language you are using. The DB I/O will eventually limit C# or go or node.js or whatever. I really don't see how this is relevant. But since you brought it up, you can really cut down on data access times by relying less on ORMs and getting your fingers dirty with stored procedures and building tables that make sense to SQL instead of pretending everything in your RDBMS is a class.
- MichaelBurge 11y ago
  You can always rewrite it if needed. A tactical rewrite of specific heavily-used routes that cuts out unnecessary abstractions often doesn't take too long and gets most of the benefit. I can't see even a slow interpreted language being unable to process 100 requests/second. That leaves like 40 million clock cycles per request. Especially since they tend to come with built-in bytecode compilers, and most of the pure algorithmic code is written in C. Most web requests just don't take that much work to answer. Most performance issues I see with web applications are somebody writing ORMTable->find(x) in a loop or the equivalent, and then papering it over with a cache(or 3). Or memory leaks. Or badly-written SQL queries(no index, too many indices, disabling seq scans, using CTEs to elegantly 'refactor' SQL, etc.). Or doing blocking calculations/service requests as part of the original web request. The underlying language is almost never what's holding them back.
- CuriouslyC 11y ago
  In addition to the things other people have mentioned, you also have the ability to easily call C code from python (compared to the unpleasantness that is JNI, for instance), and there are tools like Cython, Numba, Blaze, etc that let you rewrite your hot code into a more performant variant. Of course, at megascale these optimizations might still not be sufficient, but there is no reason you can't deliver hundreds of thousands of requests per second very affordably with a python stack.
carlosrg 11y ago
> it's only a matter of time before Python 3.x is more widely used I've been hearing that for a long time...

lrem 11y ago

> Python is a solid choice of language for the back ends of web applications. With caution advised when it comes to scaling. You might, or might not, pay a pretty hefty premium in the amount of hardware needed.

ianamartin 11y ago
I don't want to come across as a Python zealot here, but in my experience, Python has almost never been the problem in scaling. Certainly not to the extent where I would say that it costs a premium in hardware. Though, I suppose it depends on the application. My experience has been that Python will give you more than enough rope to hang yourself with, and lots of apps that are meant to scale have some homegrown crap of an architecture, or people try to get fancy when they don't need to and try to write go or scala in Python. If you have a clear understanding of your data model, Python backends scale quite well vertically. Run one instance of your app per core you have available and use IPTables to round robin (or whatever. you can get fancier if you feel like it) to each app instance. Boom. You are now banging on all 4 or 8 or however many cores your server has without the context switch overhead of the multiprocessing module and also without the headaches that go along with the shared memory space of the multithreading libs. Not to mention cleaner, nicer code. If you build your product to work well in this way, then scaling horizontally also becomes a non-issue as far as Python code is concerned. You just put haproxy in front of multiple machines, set up health checks, and for the vast majority of use cases, you're done until you need to scale the storage horizontally. Then you have some serious decisions to make. But that's a different topic. But back to my original point: every case where I've seen Python being all slow has really been an artifact of a crap architecture, or bolting on crap front end frameworks, or depending too much on an ORM like SQLAlchemy without really understanding your data and depending on it to do everything for you instead of thinking rationally about what your data is. Or some combination of all three above. Yes, I know that there are languages that execute much faster than Python. I work with C#/.NET/SQL Server almost as much as I use the Python/Pyramid/PostgreSQL stack. I can write a crappy, slow, messy architecture on either stack. And in fact, when I was first getting to know C# and still writing Python code in that language, I did that. The bottom line is that if you understand the strengths, weaknesses, and idioms of the language you are working with right now, almost every high-level language in common use today is sufficient for massive-scale deployments for the most common types of web apps. Heavy computing or scientific purposes obviously excepted. But most web apps that scale to hundreds of millions of users are basically "window-on-data" type applications. I can't really think of a case where some other language besides Python would be noticeably cheaper hardware-wise. Maybe you have to know your entire stack a bit better if you want scalable Python apps, maybe you have to do a little more Jiggery-Pokery with your Linux servers, but if you are serving at massive scale, you are going to have to go there anyway, especially when you start dealing with storage. That's part of the game.
- nawitus 11y ago
  > Run one instance of your app per core you have available and use IPTables to round robin (or whatever. you can get fancier if you feel like it) to each app instance. But if you're using Flask and you have 4 cores with 4 instances, and you're making 4 database requests at the same time you're whole application is suddenly locked. And it's not uncommon for a single view to make up to 4 separate database requests. Most modern web applications are I/O bound, therefore a single Node.js process would have better performance here.
  - ianamartin 11y ago
    That hasn't been my experience. If your ACLs are set up properly, you won't encounter a full table lock, and therefore won't have problems like you described. I agree that this could be a problem in theory. In practice, it has not been. Again, it boils down to how you architect your code. EDIT: you need to stick your sessions to a specific instance. Otherwise, you are correct that there's a problem. But as long as you stick your sessions, there's no problem with a database lock.
    - nawitus 11y ago
      I wasn't talking about database locks. My point was that making a database request takes a really long time relative to processing a request in the web application. Therefore the web application spends most of the time doing nothing and waiting for the database request to complete.
      - ianamartin 11y ago
        So, write data-backed applications without calling a database? Not sure what you are getting at here. If you are talking about the raw scalability of your database, yes, you are correct. The speed with which you are able to get info from your DB will be a limiting factor. But that's a totally different point than the one I was making, which was about how to scale Python, not your database. If I understand you correctly, the time spent in retrieving stuff from your database will be a limiting factor no matter what language you are using. The DB I/O will eventually limit C# or go or node.js or whatever. I really don't see how this is relevant. But since you brought it up, you can really cut down on data access times by relying less on ORMs and getting your fingers dirty with stored procedures and building tables that make sense to SQL instead of pretending everything in your RDBMS is a class.
MichaelBurge 11y ago
You can always rewrite it if needed. A tactical rewrite of specific heavily-used routes that cuts out unnecessary abstractions often doesn't take too long and gets most of the benefit. I can't see even a slow interpreted language being unable to process 100 requests/second. That leaves like 40 million clock cycles per request. Especially since they tend to come with built-in bytecode compilers, and most of the pure algorithmic code is written in C. Most web requests just don't take that much work to answer. Most performance issues I see with web applications are somebody writing ORMTable->find(x) in a loop or the equivalent, and then papering it over with a cache(or 3). Or memory leaks. Or badly-written SQL queries(no index, too many indices, disabling seq scans, using CTEs to elegantly 'refactor' SQL, etc.). Or doing blocking calculations/service requests as part of the original web request. The underlying language is almost never what's holding them back.
CuriouslyC 11y ago
In addition to the things other people have mentioned, you also have the ability to easily call C code from python (compared to the unpleasantness that is JNI, for instance), and there are tools like Cython, Numba, Blaze, etc that let you rewrite your hot code into a more performant variant. Of course, at megascale these optimizations might still not be sufficient, but there is no reason you can't deliver hundreds of thousands of requests per second very affordably with a python stack.

carlosrg 11y ago

> it's only a matter of time before Python 3.x is more widely used I've been hearing that for a long time...