5 ms·

A trap for your dream world were you suddenly get google size ? Because I have a 1 million unique users video streaming service still running python 2.7, using

A trap for your dream world were you suddenly get google size ? Because I have a 1 million unique users video streaming service still running python 2.7, using a few servers. The thing has a mobile version serving a different media on the fly, encodes user uploaded videos, features comments, tagging, and even has machine learning detection of content now. It is still maintained by one single person, and he is not a professional dev. And I'd say it's already very rare to reach that size in any project, in any company. Hell my last 3 paid projects as a freelancer have less than 20 concurrent users. And that's professional. That's people's real life. 2 out of 3 IT projects fail, no code is even shipped. I would worry about the scaling later. Once you do have the problem, pay the price for more hardware or a rewrite, you made it! And if you do know for sure you will have the problem (you already work for a GAFAM), then not choosing python is ok, it's not a trap, it's a trade off. But for the vast majority of us out there, having an easy, solid, battle tested and clean language with a huge ecosystem is worth a 10000 times more than some scaling potential in 5 years down the road. I have seen so many big C++ or Java project fail I don't associate any language with big and failure anyway. I've seen devs arguing about the purity of this in Haskell, or that in Ocaml, and nothing gets to the end user because it's never perfect, or it's a toy, and doesn't handle IRL. But what powers most companies ? Rail apps that you won't upgrade ever, but are still running. Spaghetti PHP code and wordpress that just won't dies. Horrible SAP scripts, VBA macros and excel sheets that actually deal with your real data. Because they shipped. They did the job.

Cthulhu_ 5y ago
It's a variation on cargo cult; you WISH you had the scaling problems that could be solved by using something a bit more performant. But solve the problem first, and solve it fast. It's why so many startups from the 2010's on used Ruby; they were productive in it, solving a real problem, rolling out features fast. I've worked in a few projects where they dove onto the cargo cult, of wishing they had the kind of requests and load that e.g. a microservices architecture might help with. Massively overpriced projects too, because they hired lots of consultants and self-employed people. The one time there was a guy who rocked up in a Maserati (used, lol); he took two weeks to build a page that was just a centered bit of text and a button, and it didn't even work.
- takeda 5y ago
  Also, a lot of developers forget to it, but even if you start with a fast language down the line you will end up rewriting it. The problem is that requirements change a lot and because of that your application will often end up being a mess. For similar reason RDBMS is also better than NoSQL when starting a project. When rewriting you already know how the project is now, so you can design your app around that, which makes things simpler and faster.
- bcrosby95 5y ago
  Just write a stateless app tier. Doesn't matter what language you use at that point if you need to scale it out. It doesn't require any extra work either. Just requires knowing what you're doing.
ithrow 5y ago
Even faster for 0->1, use Javascript, big ecosystem as Python but same language in front-end and back-end. Faster runtime also.
- m_mueller 5y ago
  Depends. Mainly on what libraries you can make use of. For anything numerical or also ETL processes that deal with various document formats python is way more mature.
  - ithrow 5y ago
    Since you don't know if your product will workout and you want to go fast you use a third-party service first. Of course, for some type of stuff, you might have to go with python but if both can equally do the job well in the backend and you need a web[mobile] frontend the...
- BiteCode_dev 5y ago
  There are still no "rails" or "django" for JS. There are frameworks emerging with the same features, but they don't have close to the ecosystem. Meaning every time you want auth, social integration, etc. you have to reinvent the wheel. There is also the stdlib. JS doesn't have simple things built in like left trim a string, csv loading, uuid generation, etc. Then there is the API. Sorting in JS requires this weird -1/1 function. Getting the last element or truthiness to manipulate length. Suddenly not only you have to install tons of things, but libs and frameworks can't rely on a common ground, nothing seems to integrate well. You glue a lot of things manually. The popularity of JS is because it has a monopoly on the browser, and so a lot of people must use it, and a lot of users know it by default, and look for it elsewhere. But when you have Ruby, Python, or even modern PHP frameworks in front of it, it's not a great deal. Of course, not all programming tasks are about the Web. And this is where JS collapses completely. Sysadmin, data analysis, pen testing or automation using JS is a pain compared to the fluent versatility of alternatives. You know, things like connect to this ftp, download the excel files every hour, put the average of every column in the mysql db then export the thing as a PDF and send that to the boss.
willcipriano 5y ago
I haven't worked at Google's scale, but I've heard a lot of critism about tools and programming languages from people who create architectures that are inherently slow. When a user login requires 8 http calls to distinct microservices or something else silly like that the problem isn't a lack of static types. So far in my career I haven't had a programming language ever be the reason why something can't scale, most often the problem is the initial developers lacking a understanding of relational databases and cache locality.
dzonga 5y ago
link to the site ?
Abishek_Muthian 5y ago
> Because I have a 1 million unique users video streaming service still running python 2.7, using a few servers. Congratulations. •How many servers 2, 5, 10? •How many pure python libraries did you have to replace with a Cython extension after you realized that it cannot support your scale? •How many external dependencies? How many of them did you replace after they were abandoned? Surely not all actively developed library are still using Python2.7. > It is still maintained by one single person, and he is not a professional dev. So he cannot upgrade the project to Python3.x along with all matching dependencies either, So the project stays with all bugs and vulnerabilities. Not everyone would be happy with such a setup and aforementioned drawbacks, that's why I think the parent comment says Python is a trap at scale and I concur. I've faced issues at at scale (few hundred thousand) and there's no way a python project can meet that scale without relying upon Cython dependencies. > A trap for your dream world were you suddenly get google size ? So only Google size is a benchmark for scale? If I were to start a web project which has just 10 concurrent users I would choose Go because I know I can develop the entire project with little to no dependencies, it can scale to N users without touching the code and with the confidence that the number of servers required will be < than that of python. I still use Python every day, for where it's good i.e. small scripts for personal use, prototyping algorithms. For professional products I use Python only as an Machine Learning endpoint(With all the bandages) because I have no other choice.
- BiteCode_dev 5y ago
  > •How many servers 2, 5, 10? 7 But the question is rather, whats the cost / ratio ? The 7 servers costs 7 less times money that they bring. The solo dev makes $150k/years with it, and doesn't need to work full time, have flexible hours and its own business. > How many pure python libraries did you have to replace with a Cython extension after you realized that it cannot support your scale? Many. Being extensible with compiled extension is a feature of Python. Numpy and co are part of the ecosystem. That's an awesome thing about it: I can code in Python and get c speed anyway. > •How many external dependencies? How many of them did you replace after they were abandoned? Surely not all actively developed library are still using Python2.7. Many. And the project still runs, with so little resource, bring the money in and serve its purpose, after 10 years of service. > So he cannot upgrade the project to Python3.x along with all matching dependencies either, So the project stays with all bugs and vulnerabilities. Sure he can, it's just an expense he won't do. Bugs are workaround, vulnerabilities are shielded by other parts of the system. Essential things like injections are dealt with. Some trade off are made, like always. > So only Google size is a benchmark for scale? If I were to start a web project which has just 10 concurrent users I would choose Go because I know I can develop the entire project with little to no dependencies, No you can't. Not if you want the first version to come out in a few months. Because you need a scrapper, and a bdd, and registration, and auth, and permissions, and screenshotting, and upload, and videos encoding, with background tasks, and load balancing all the videos and pics, tagging, i18n, xss protection, and search, etc. So either you pay to use cloud services to do all those things for you, or you leverage a rich ecosystem. Or you take a year to do it. Even if Go were the best language in the world, it can't beat 10 years of django plugins, 25 years of python modules and the iterative speed of a dynamic syntax. Well, it can, with AWS :) In fact, Go is a very niche language. If you take it out of the I/O and concurrency specialty, it's quite average. Rust is better at raw perfs. At same perf, java has a better ecosystem. For simplicity and easy compilation, you have nim and zig that are more expressive. Resilience ? Erlang. And for speed of dev, python is better. So for our use case it is not were it would shine, because: > it can scale to N users without touching the code and with the confidence that the number of servers required will be < than that of python. 1 server only run the python code, the others are for the video hosting and the db. Python is not the bottleneck in this site. It almost never is in web dev. nginx serves the content, postgres and redis deals with the data.
blub 5y ago
Many (most?) web projects allow hiding performance issues by scaling horizontally, caching and offloading work to native implementations where necessary (like video encoding). Python can work great in such circumstances. The best testament to this is the reality that popular web services were able to limp along on a slow language for a long time before they had to rewrite their code. But if one is doing something that requires raw power or precise control over resources, or precise timing then not choosing Python is not merely a tradeoff to be considered, but an obvious architecture decision.

Cthulhu_ 5y ago

It's a variation on cargo cult; you WISH you had the scaling problems that could be solved by using something a bit more performant. But solve the problem first, and solve it fast. It's why so many startups from the 2010's on used Ruby; they were productive in it, solving a real problem, rolling out features fast. I've worked in a few projects where they dove onto the cargo cult, of wishing they had the kind of requests and load that e.g. a microservices architecture might help with. Massively overpriced projects too, because they hired lots of consultants and self-employed people. The one time there was a guy who rocked up in a Maserati (used, lol); he took two weeks to build a page that was just a centered bit of text and a button, and it didn't even work.

takeda 5y ago
Also, a lot of developers forget to it, but even if you start with a fast language down the line you will end up rewriting it. The problem is that requirements change a lot and because of that your application will often end up being a mess. For similar reason RDBMS is also better than NoSQL when starting a project. When rewriting you already know how the project is now, so you can design your app around that, which makes things simpler and faster.
bcrosby95 5y ago
Just write a stateless app tier. Doesn't matter what language you use at that point if you need to scale it out. It doesn't require any extra work either. Just requires knowing what you're doing.

ithrow 5y ago

Even faster for 0->1, use Javascript, big ecosystem as Python but same language in front-end and back-end. Faster runtime also.

m_mueller 5y ago
Depends. Mainly on what libraries you can make use of. For anything numerical or also ETL processes that deal with various document formats python is way more mature.
- ithrow 5y ago
  Since you don't know if your product will workout and you want to go fast you use a third-party service first. Of course, for some type of stuff, you might have to go with python but if both can equally do the job well in the backend and you need a web[mobile] frontend the...
BiteCode_dev 5y ago
There are still no "rails" or "django" for JS. There are frameworks emerging with the same features, but they don't have close to the ecosystem. Meaning every time you want auth, social integration, etc. you have to reinvent the wheel. There is also the stdlib. JS doesn't have simple things built in like left trim a string, csv loading, uuid generation, etc. Then there is the API. Sorting in JS requires this weird -1/1 function. Getting the last element or truthiness to manipulate length. Suddenly not only you have to install tons of things, but libs and frameworks can't rely on a common ground, nothing seems to integrate well. You glue a lot of things manually. The popularity of JS is because it has a monopoly on the browser, and so a lot of people must use it, and a lot of users know it by default, and look for it elsewhere. But when you have Ruby, Python, or even modern PHP frameworks in front of it, it's not a great deal. Of course, not all programming tasks are about the Web. And this is where JS collapses completely. Sysadmin, data analysis, pen testing or automation using JS is a pain compared to the fluent versatility of alternatives. You know, things like connect to this ftp, download the excel files every hour, put the average of every column in the mysql db then export the thing as a PDF and send that to the boss.

willcipriano 5y ago

I haven't worked at Google's scale, but I've heard a lot of critism about tools and programming languages from people who create architectures that are inherently slow. When a user login requires 8 http calls to distinct microservices or something else silly like that the problem isn't a lack of static types. So far in my career I haven't had a programming language ever be the reason why something can't scale, most often the problem is the initial developers lacking a understanding of relational databases and cache locality.

dzonga 5y ago

link to the site ?

Abishek_Muthian 5y ago

> Because I have a 1 million unique users video streaming service still running python 2.7, using a few servers. Congratulations. •How many servers 2, 5, 10? •How many pure python libraries did you have to replace with a Cython extension after you realized that it cannot support your scale? •How many external dependencies? How many of them did you replace after they were abandoned? Surely not all actively developed library are still using Python2.7. > It is still maintained by one single person, and he is not a professional dev. So he cannot upgrade the project to Python3.x along with all matching dependencies either, So the project stays with all bugs and vulnerabilities. Not everyone would be happy with such a setup and aforementioned drawbacks, that's why I think the parent comment says Python is a trap at scale and I concur. I've faced issues at at scale (few hundred thousand) and there's no way a python project can meet that scale without relying upon Cython dependencies. > A trap for your dream world were you suddenly get google size ? So only Google size is a benchmark for scale? If I were to start a web project which has just 10 concurrent users I would choose Go because I know I can develop the entire project with little to no dependencies, it can scale to N users without touching the code and with the confidence that the number of servers required will be < than that of python. I still use Python every day, for where it's good i.e. small scripts for personal use, prototyping algorithms. For professional products I use Python only as an Machine Learning endpoint(With all the bandages) because I have no other choice.

BiteCode_dev 5y ago
> •How many servers 2, 5, 10? 7 But the question is rather, whats the cost / ratio ? The 7 servers costs 7 less times money that they bring. The solo dev makes $150k/years with it, and doesn't need to work full time, have flexible hours and its own business. > How many pure python libraries did you have to replace with a Cython extension after you realized that it cannot support your scale? Many. Being extensible with compiled extension is a feature of Python. Numpy and co are part of the ecosystem. That's an awesome thing about it: I can code in Python and get c speed anyway. > •How many external dependencies? How many of them did you replace after they were abandoned? Surely not all actively developed library are still using Python2.7. Many. And the project still runs, with so little resource, bring the money in and serve its purpose, after 10 years of service. > So he cannot upgrade the project to Python3.x along with all matching dependencies either, So the project stays with all bugs and vulnerabilities. Sure he can, it's just an expense he won't do. Bugs are workaround, vulnerabilities are shielded by other parts of the system. Essential things like injections are dealt with. Some trade off are made, like always. > So only Google size is a benchmark for scale? If I were to start a web project which has just 10 concurrent users I would choose Go because I know I can develop the entire project with little to no dependencies, No you can't. Not if you want the first version to come out in a few months. Because you need a scrapper, and a bdd, and registration, and auth, and permissions, and screenshotting, and upload, and videos encoding, with background tasks, and load balancing all the videos and pics, tagging, i18n, xss protection, and search, etc. So either you pay to use cloud services to do all those things for you, or you leverage a rich ecosystem. Or you take a year to do it. Even if Go were the best language in the world, it can't beat 10 years of django plugins, 25 years of python modules and the iterative speed of a dynamic syntax. Well, it can, with AWS :) In fact, Go is a very niche language. If you take it out of the I/O and concurrency specialty, it's quite average. Rust is better at raw perfs. At same perf, java has a better ecosystem. For simplicity and easy compilation, you have nim and zig that are more expressive. Resilience ? Erlang. And for speed of dev, python is better. So for our use case it is not were it would shine, because: > it can scale to N users without touching the code and with the confidence that the number of servers required will be < than that of python. 1 server only run the python code, the others are for the video hosting and the db. Python is not the bottleneck in this site. It almost never is in web dev. nginx serves the content, postgres and redis deals with the data.

blub 5y ago

Many (most?) web projects allow hiding performance issues by scaling horizontally, caching and offloading work to native implementations where necessary (like video encoding). Python can work great in such circumstances. The best testament to this is the reality that popular web services were able to limp along on a slow language for a long time before they had to rewrite their code. But if one is doing something that requires raw power or precise control over resources, or precise timing then not choosing Python is not merely a tradeoff to be considered, but an obvious architecture decision.