32 ms·

I have become increasingly convinced Python as a language is a "trap" for any use of notable scale, be the scale about number of developers, codebase size, or p

by ctur 5y ago

I have become increasingly convinced Python as a language is a "trap" for any use of notable scale, be the scale about number of developers, codebase size, or performance requirements. It's a great 0->1 language and great at simple glue, but eventually you hit a wall and have to keep investing larger and larger amounts of people or computing resources to get continued returns... all due to fundamental design decisions in the language from two decades ago that are quite difficult to fix. Unless you go all-in on typing -- which is difficult today with any meaningfully sized existing codebase -- maintenance is largely a "hope manual testing and unit tests catch anything resembling type errors" which is a major challenge for, say, structural change to a code base like refactoring. Plus typing is still young, the tooling somewhat immature, and can lead to false senses of security if you aren't very careful and opt into the strictest modes. This makes large number of developers and codebase size a major stumbling block. The interpreter performance and GIL are fundamental issues as well. Multiprocessing and hacks around the GIL are quite painful if you even glance at any native threading code (say in a C++ library) and even when you stick to pure Python, you have a debugging mess when anything goes wrong. But if someone can improve performance, that'd be great, and has massive impact potential. It is incremental though and doesn't solve fundamental issues with the language. I'm also skeptical of the "5x" plan referenced in the blog, and very skeptical we can ever see meaningful removal of the GIL due to library and baked in design decisions in existing code. This means performance will fall further and further behind compiled languages. (I write all of this having been a part of supporting Python at massive scale for over two decades, including at two FAANG companies who invest heavily in it. I've seen the curve and the pain it's caused, and would never use it for any code that needs to be performant or actively developed on the multi-month or year timescale).

_wldu 5y ago
This is my experience as well (but not at a FAANG). You can do things quickly in Python and it is a joy to write, but down the road (as you grow), you will regret it and find yourself re-writing it.
SoylentOrange 5y ago
What do you use nowadays?
chriswarbo 5y ago
Adding types after-the-fact can certainly be painful, but at least it's not an all-or-nothing choice, we can opt in to types for selected parts of a codebase. Personally I've got a lot of mileage out of Hypothesis for property-based testing. It's good at exercising edge-cases, and works particularly well when we sprinkle assertions through a codebase (where the "property" we're testing is simply "calling Foo doesn't throw an exception").
- throwaway894345 5y ago
  > Adding types after-the-fact can certainly be painful, but at least it's not an all-or-nothing choice, we can opt in to types for selected parts of a codebase. One of the benefits of typing (that is rarely discussed) is that it deters people from writing code whose type would otherwise be crazy (e.g., "if someone passes the string 'foo' in for a parameter, then the return type is a string, otherwise it's a bytestring"). In other words, it deters a lot of crappy code. When you annotate after the fact, the annotation becomes really difficult and people get crabby that they're having to make this really complicated annotation and they blame typing (rather than their own poor coding). To your point, typing after-the-fact is still better than nothing, but you're missing out on a lot by waiting. Moreover, Python's typing story is still really immature with respect to syntax and tooling.
  - joconde 5y ago
    If a person like that is able to harm the team, I'd argue that it's a management problem. Lazy coding is only an issue when you allow people to be lazy...
    - throwaway894345 5y ago
      I think you could make the same argument about code styling or really just about anything. If you had sufficiently quality coworkers, these things probably wouldn't be an issue, but it's a lot easier to just implement the technical solution (a code formatter or type checker).
nicolaslem 5y ago
Isn't it that maintaining large codebases in the long run is just hard? What language is not a trap?
- ctur 5y ago
  Fair, maybe it's always a matter of picking the trap you want :) If you are optimizing for 0->1, Python is great. But if you want code to live and evolve over time with multiple authors, or ever will care about runtime performance, it's a dead end almost from the first line. But sometimes the productivity benefit is all that matters and you accept you may have to throw it all out later (or invest insanely in making it work)... it's all about making an informed decision.
  - cglace 5y ago
    As with any language it's how you end up architecting your codebase. I do believe python lacks an authoritative resource for what is "good architecture" which leads to a lot of the code scaling problems.
    - meltedcapacitor 5y ago
      What widely adopted language achieved this? Even Rust, despite the head start, is already a mess on its way to inevitable cobolization.
- sorokod 5y ago
  Digging a well is hard, doesn't mean you have to this with a spoon.
UnpossibleJim 5y ago
I'm curious, as you seem very versed in the Python ecosystem, where you stand on projects like Cython and integration of C libraries with Python using projects like Cython? Or would you consider that more of a stop-gap measure and not "real" Python (which would be valid)?
- ctur 5y ago
  Things like Cython are great until they aren't, so yes, I think of them as a stop gap (albeit a practical and useful one). Any time you change interpreters or the execution model, you risk compatibility challenges with third party libraries and native modules. You also risk getting painted into a corner by their limitations. They definitely can help with hot code paths when their trade-offs don't prevent it but it still ends up being a kluge working around fundamental language issues. But when Cython fits, it's great, and a good tool to have in your toolbox if you're working with Python.
dunefox 5y ago
It was never intended as a stand-alone language for large projects, rather as a glue and prototyping language. All the tacked-on "static typing" won't change this. I think it would be best if Python was used according to its strengths and not as a 'unicorn' language.
toolslive 5y ago
Also, typing in python has a "kluge" feel about it. It's like most things in python: it works nicely on toy examples, but is rather frail on the edges (if you're not convinced try to define the json type). I also think they chose the wrong strategy (type inference would have been way better) To use a metaphor: Python is the duplo of programming languages.
- mixmastamyk 5y ago
  Python has always had strong, inferred typing. It’s just that you can rebind variables.
  - toolslive 5y ago
    _static_ typing.
klyrs 5y ago
I'm not terribly familiar with the options for typing, mostly because I'm salty that Python rolled out annotations without a clear plan for how they'd be useful*. Aside from Cython, does typing provide any mechanism to improve performance, or is it only there for static analysis? Worse yet, is it being used to do dynamic analysis that slows the whole thing even further? * specifically, this happened long after the tragically many-ways-to-do-it of packaging systems and virtual environments, which survive to this day.
- mgraczyk 5y ago
  There are some tools that use types for optimization, but I'd say it's still pretty early. For example, mypyc is a python to C compiler that is part of mypy. It uses types for both correctness and optimization: https://mypyc.readthedocs.io/en/latest/using_type_annotations.html https://mypyc.readthedocs.io/en/latest/using_type_annotation...
mrfusion 5y ago
You may be right but I’m convinced most problems don’t need to scale.
- ctur 5y ago
  Absolutely true.
wil421 5y ago
Do you have the opposite experience? A language used by FAANG that doesn’t have performance issues at their scale. They seem to invest a lot in JavaScript which is not what I’d call a performant language and as you’ve said Python. I’ve used Python for more scripting type tasks and creating basic APIs to get specific jobs done. I’m very weary to go “all in” on Python for an app based on all the feedback about performance. All that being said I’ve never heard someone say we used X language and it scaled remarkably without issues. I’ve noticed lots of companies eventually go to Java once they get big but nobody really likes to brag about Java apps.
- ctur 5y ago
  Everything has performance issues but Python is at least one, if not two, orders of magnitude off of compiled languages like C++, Rust, and Go (despite Go having a garbage collector). They have various usability tradeoffs but once you get things working, at least you're not spending 10x as many cycles to do the same work as another language would need. (Java probably is also fine; I've less personal experience there so I can't say).
  - mda 5y ago
    My experience, Java is usually 10-100x faster than python in general.
  - bcrosby95 5y ago
    Statically typed, garbage collected languages are all roughly in the same ballpark, on average (Go, C#, Java).
- pyentropy 5y ago
  > They seem to invest a lot in JavaScript which is not what I’d call a performant language JavaScript (V8 at least) is extremely performant, near native code performance. Considering how dynamic it is, it's not an easy feat but Google, Apple and Mozilla work a lot on it. Here's a benchmark where JS is 50x faster than Python: https://github.com/kostya/benchmarks https://github.com/kostya/benchmarks Note that PyPy does much better.
chuckcode 5y ago
Thanks for this comment, sums up exactly my experience with python. Do you have a pointer to a larger write up of these issues? I really enjoy python, but agree with you that it isn't well suited for performance or scale necessarily. I find that people really get attached to it though and I'd like to have a good reference for them to help explain why issues like interpreter performance, GIL and typing make a difference on large projects even if not their pet ones.
qsort 5y ago
Pretty much everything you're saying matches my experience, so, playing the devil's advocate: - The typing argument is impossible to argue against, but isn't it the same problem you'd have with any other dynamic language? How is, say, Javascript any better? I don't see this as something that's specifically Python's fault, rather, yet another argument as to why starting a large project in 2021 without seriously considering static types is bordering professional malpractice. - As for the GIL thing, I'm with you that it's never going to be lifted in any meaningful way, but wouldn't the subinterpreters idea along the lines of Ruby's ractors fix most of the pain in most cases? Also, why are you skeptical about "the 5x plan"? I don't really have an opinion about it, but I'm interested in hearing any input.
- azeirah 5y ago
  For your first point, I don't think a language such as JavaScript -is- much better, which is why Typescript was invented as an alternative for larger codebases/organizations
  - igouy 5y ago
    Also Dart.
  - dragonwriter 5y ago
    TypeScript:JS is a lot like mypy:Python; the only real difference is that Python adapted so that mypy (originally envisioned as its own language) code is also valid Python as well as the reverse, obviating the need for a compilation step to run mypy in the Python runtime (whereas TS has to compile to JS to run); also, mypy has mypyc to compile (currently, only a subset of) mypy code to something more efficient than normal python.
    - whimsicalism 5y ago
      Having worked extensively with both, I don't think this is an apt comparison. Type inference & hinting at the IDE level is just not as developed either
tristor 5y ago
This mirrors my experience as well. At one (at the time major) cloud provider we had to internally re-implement 100% API / behavior compatible Java applications for the publicly available Python applications for our OpenStack deployment, due to performance and scaling requirements. OpenStack Foundation required all projects to be only written in Python, which I think was a major factor in why OpenStack adoption was never significant. Scaling and deploying it was an absolute nightmare using public code. Anything done at significant enough scale /must/ be multi-threaded to take full advantage of modern hardware which makes languages which don't provide a reasonable threading model effectively non-starters or at best a toy language for internal prototyping. I shifted from Python to Ruby, simply because Ruby provided a realistic way to interface with real threads, and then to Go, and haven't looked back. Simple applications are massively more performant in Go than in Python, and it's not just due to typing and compilation. I love how simple Python is to learn and how it brings more people into the fold in approaching solving complex problems, but at the end of the day you should treat it like runnable pseudocode to get shared understanding so you can implement in a real language.
Lucasoato 5y ago
Ok, let's admit it, writing bad Python code is very easy, as it is in every dinamically typed language, like Javascript... ... but I think you're forgetting the real motivations that drive people to use Python: it's easy to learn, it's very readable, there's a large (and not so much toxic) community behind, it's versatile. No one is saying that it will replace every other language, but it still can find its space even among large projects.
- taeric 5y ago
  I think the "dynamic" jab is a bit of a trap, itself. It is easy to write bad code in most any language. Anyone that tells you otherwise is trying to sell you something.
- capitainenemo 5y ago
  Readable varies by person. Myself, I find the decision on significant whitespace to make reading Python like reading English with no punctuation or capitalisation.
- BiteCode_dev 5y ago
  It's not that easy actually. Python already does so much for you, and provides so much tooling to do even more, it's hard to write bad code if your daily job is to be a coder. The iterator protocol means you rarely get off by one errors, context managers and GC deals with resource handling, the GIL, for all its faults, helps a lot of concurrency errors... Now, add a good IDE, and syntax errors, names errors and attribute errors go away. Play a little in the REPL, and use cases become more obvious, behaviors more clear. Want more security? You can add type hints and unit tests (which are incredibly easy to write, thanks to pytest). Even without all that, the size of the ecosystem means you won't write most of the code anyway, and less code, means less potential for mistakes. Adds to that the language bare bone syntax, and you really has something that actively helps with writing good code. Not to say you can't write bad code in Python, but unless you are not a professional dev (in that case, you can write bad code in any lang), you have a lot to help you there.
- dragonwriter 5y ago
  > Ok, let's admit it, writing bad Python code is very easy, as it is in every dinamically typed language Writing bad code is easy in most statically typed languages too.
j1elo 5y ago
I'm a C++ dev and was taught Python as a tool for prototyping and validation of ideas, not as a language for final implementations. Later with years I've read endless histories about the "traps" you mention, so I understood the notion I was taught wasn't without its merits. So I still look suspiciously to all mid to large projects what are written in Python. It takes some serious effort and good engineering to maintain good quality, and still the tooling is not there for error detection, like in other better typed languages would be. I wouldn't feel confident relying on such project for a critical infrastructure piece. The idea should have been validated quickly with Python, then implemented in a more appropriate language. EDIT: Obviously that's just my possibly wrong opinion. There are huge projects out there proving otherwise, but the issues discussed here are _very_ real.
indymike 5y ago
> This means performance will fall further and further behind compiled languages. This is exactly as it should be: the nature of the interpreter is that it has to do a lot of the work a compiler does when it compiles whenever a new command is run (and in Python's case, that is before you cover the whole "everything is an object" part). Yes, some of it can be mitigated, but the expectation that an interpreter keep pace with compiled code is somewhat of a pipe dream. What I'd love is a language that can be compiled (and be highly optimized) and interpreted without changing it's behavior. The developer experience with an interpreter is fantastic, the performance of compiled code is... better.
- colejohnson66 5y ago
  > What I'd love is a language that can be compiled (and be highly optimized) and interpreted without changing it's behavior. The developer experience with an interpreter is fantastic, the performance of compiled code is... better. You mean something like a byte code like language with a JIT system? Like Java and C#? C# in particular now supports AOT compilation, but can still be run as “interpreted” CIL.
  - indymike 5y ago
    Having a VM / JIT in the middle is a way to get there, but ultimately, different. At least in Java's case, my code targets an abstraction instead of hardware, so I don't get the full benefit of compilation. Performance is part of the story... access to the OS and hardware is another.
    - smallnamespace 5y ago
      Even native compilers like clang will use LLVM, and LLVM IR can look a lot like JVM bytecode. The VM abstraction is not a barrier to fully optimizing for your architecture, but rather how much time you can spend converting IR/bytecode into assembly, whether you do it at runtime, and whether runtime information lets you optimize even further.
- giovannibonetti 5y ago
stinos 5y ago
I have become increasingly convinced Python as a language is a "trap" for any use of notable scale I assume you're saying this out of experience, so can you give a practical example of what you call 'notable scale'? And are you talking about desktop or web or mobile, or just all of them? Just so that we know what you are talking about in a concrete way. Not the 'large software with large number of developers is always a problem' way. edit I also see others here using 'scale' in a 'performance / number of request per second' etc meaning - I thought we were talking codebase size though. I mean if you know on beforehand this type of performace is a requirement it seems weird to turn to Python, of all things?
- ck_one 5y ago
  +1 Would be cool if some experienced dev could share some estimates when the Python scaling issues start. When you build a backend using Python + Django/FastAPI, I assume in most cases the DB and not Python is the limiting factor. Moreover, you could always spin up more workers to mitigate scaling issues. When you train ML models, your Python code just calls C++ functions. Python is not a limiting factor here either.
  - ferdowsi 5y ago
    I worked for a startup that built their services using Python and Twisted. The codebase was a monolith broken up into several Twisted services, by the time I left it was probably 200k LOC of Python. For us, the scaling problems started when we had two independent teams working in the same codebase. The extreme dynamism of the language meant that classes and data structures were being mutated willy-nilly in ridiculous ways across the execution flow. The lack of static typing made onboarding new developers difficult as they had to parse generations of excessively clever code and magic left behind by departed developers. This problem has only gotten worse in Python 3, which keeps piling on more ways to accomplish the same task. The deployment story was also awful, but I don't think that's a surprise to anyone who has deployed Python at scale. In terms of raw performance, at one point we estimated that our Python stack was adding 3-400ms of request latency compared to a comparable system written in Go. With the Python 3 deadline coming, we convinced management to invest in rewriting performance-critical parts of the service in Go, instead of the migration to Python 3. I left before the project was completed but we were already seeing massive improvements.
dom96 5y ago
If anybody is looking for an alternative to Python that is also a great 0->1 language but doesn't have the same wall as Python described here, check out Nim[1]. 1 - https://nim-lang.org https://nim-lang.org
- amelius 5y ago
  People's main reason for using Python is the ecosystem.
  - arc619 5y ago
    Nim can use Python's ecosystem in both directions with nimpy https://github.com/yglukhov/nimpy https://github.com/yglukhov/nimpy and nimporter https://github.com/yglukhov/nimpy https://github.com/yglukhov/nimpy This lets you gradually transition hot path Python modules to Nim, get compiled performance generally on par with C and Rust, whilst enjoying strong, static typing with great type inference. In my experience (6-7 years 4 of which are full time) Nim strikes the perfect balance of the productivity you get with Python with high performance at the same time. Also the metaprogramming features are incredible and, importantly, don't use a language subset but use the base Nim language itself.
    - amelius 5y ago
      That's interesting but it looks like Nim has no support for Qt yet (where Python has PyQt and PySide).
      - arc619 5y ago
        I've never used Qt myself, but the Status messaging app written in Nim does, so it's definitely doable: https://github.com/status-im/status-desktop https://github.com/status-im/status-desktop There's also a QML wrapper here: https://github.com/filcuc/nimqml https://github.com/filcuc/nimqml
    - arc619 5y ago
      Oops wrong link for nimporter. Actual link: https://github.com/Pebaz/Nimporter https://github.com/Pebaz/Nimporter
spamizbad 5y ago
Most problems do not need scale. Some problems require it. For those: use one of many proven "fast" languages. For everything else Python (or similar language) is more than adequate.
nickjj 5y ago
> I have become increasingly convinced Python as a language is a "trap" for any use of notable scal At what point do you embrace the trap and happily live with it? For example Dropbox runs many millions of lines of Python, has massive traffic and their server costs aren't completely out of line for the service they offer. Their code base is also over 10 years old. It seems to be working very well for them. I talked to one of their engineers a few months ago. He gave me a complete run down of how they build and deploy Dropbox at https://runninginproduction.com/podcast/82-dropbox-gives-you-secure-access-to-all-of-your-files https://runninginproduction.com/podcast/82-dropbox-gives-you....
- ferdowsi 5y ago
  My understanding is that Dropbox has been building performance-critical parts of their services in Go since 2014 at the least. https://twitter.com/jamwt/status/629727590782099456 https://twitter.com/jamwt/status/629727590782099456 Like many companies, they had initial scaling successes with Python,hit performance bottlenecks and looked for a way out with faster languages.
  - coldtea 5y ago
    So? If they started with those faster languages they'd have slowed down their rolling of features initially. They might not even be here now.
  - nickjj 5y ago
    Sure, Rust too for some of the desktop app components. But the takeaway is you only need to do that when you're operating at mega scale and actually hit these bottlenecks. In a ton of cases you'll be completely fine serving a few million monthly page views of your SAAS app on a single $20-40 a month server using Flask, Django or whatever Python web framework you prefer. Performance will be really good too. Just talking about most apps where it's mainly reading and writing data to a DB.
    - wickoff 5y ago
      You may be better off using rails instead to fully capitalize on rapid development through automagic.
Pxtl 5y ago
Likewise. I cut my teeth on Python 20 years ago and I loved it back then - even getting my hands dirty in the guts of the interpreter. But now in hindsight, I'm shocked that Python 3.0 wasn't seized as an opportunity to lose more of its baggage - the under-optimized interpreter, the poor parallelization story, the excessive surface area of the interpreter exposed to Python itself, making it impractical to re-implement in other interpreters, better typing, etc. They broke backwards compatibility and all we got for it was Unicode?
CuriouslyC 5y ago
Python is great for developing tools for infrastructure management, for scientific computation, and for small-medium web applications. Given its best in breed interoperability with C/C++, it is also good as a shell language for a top level project that incorporates other sub projects.
rowanseymour 5y ago
I feel pretty similarly. Our product used to be a Python/Django monolith and over the years we've ended up pulling so much of the functionality out into services written in golang. It was fantastic for getting the product up and running relatively quickly, and we're still very happy to let Django take care of the frontend and database migrations. But when your codebase has gotten big, refactoring in Python feels like Russian roulette, and when you're doing a lot of background processing, celery feels like the wrong tool for the job.
- ck_one 5y ago
  How do you you handle the communication between the Django codebase and the Go codebase? Does it produce a lot of overhead?
  - rowanseymour 5y ago
    All the communication is initiated on the Django side and it's either HTTP and tasks queued in Redis depending on whether it needs to be synchronous. It adds a little extra complexity but it's also nice being able to stop and start services without bringing the whole system down.
joconde 5y ago
We started a project using typing annotations for everything, and it's really a pleasure to use. It's like the type safety of C++ without the insane language complexity; the "advanced" parts of the language usually make intuitive sense. For our use case of deep learning, REST APIs, and image processing (often a mix of these), Python seems like the best choice we have. The GIL isn't a problem at all because all our computations are done with NumPy or PyTorch, which release the GIL during most of their operations.
mynameisash 5y ago
> eventually you hit a wall and have to keep investing larger and larger amounts of people or computing resources to get continued returns. That's certainly my impression. I'm not a professional Python programmer, though I have used it in a few projects that went to prod. I really want to like Python, and it is really nice for some quick one-offs. However, in my project, where the codebase was 30-40% Python, I estimate that 80-90% of bugs and time spent were in Python. The other side of things was more or less write once, never revisit.
coldtea 5y ago
>It's a great 0->1 language and great at simple glue, but eventually you hit a wall and have to keep investing larger and larger amounts of people or computing resources to get continued returns... If you're a millionaire by that time, or have a strong established business, then it doesn't matter, it's a nice problem to have. Have your engineers have a go at it. See: Dropbox, Disqus, AirBnB, and others... And if you're not, then going from 0->100 in 2 years instead of 0->1 in a few months with Python wasn't worth it anyway (and you also don't have enough customers/users to make use of those 100x improvement).
- nightski 5y ago
  I work as a consultant for companies with billions in revenue and from what I see it definitely matters. Re-writes simply don't happen that often. Maybe for the pure venture capital SV tech companies you mentioned, but many companies are not in that bucket. Making good engineering choices from the beginning matters a lot.
  - jacobolus 5y ago
    Rewrites may not happen that often, but plenty of large-scale corporate tech projects flail for years without ever reaching a functional state. If the choice is between a sub-optimally performant but working system, or a boondoggle that never gets off the ground, the “better engineering choice” is probably the former.
- gizdan 5y ago
  Also, pretty sure Python was called out as one of the main reasons why some hundred of Google Video engineers couldn't keep up with YouTube prior to Google's purchase. Python might not be the most performant, but once you need the scale, it's a good problem to have because at that point, your product has already made it.
  - takeda 5y ago
    Do you have an article about that? I would be interested to learn more about it.
wenc 5y ago
As someone with nearly two decades of experience with Python and who has deployed it in production in high impact scenarios, I’d like to nuance the above for people who think Python is only good for prototyping and that you have to rewrite in a “proper” programming language. This is the kind of blanket thinking to avoid. I think the rewriting part is only necessary if there’s some characteristic in your use case that you need that Python doesn’t provide — like raw computational speed, low latencies etc. If you’re a senior engineer familiar with Python, these trade offs are always front of mind. For the majority of software I’ve developed, these characteristics were not essential. Python was a good choice for 80-90% of all of the important software products I’ve ever written. For many data science projects, maintaining large code bases in Python is often the optimal decision (rather than reinventing the wheel and writing your own data frame and machine learning libraries, or using immature poorly maintained ones in other languages — for data manipulation and scientific algorithms, these libraries are highly optimized in Python anyway since the underlying code is in C or Fortran). Python is also a good choice for data engineering pipelines. Where I might hesitate to recommend Python is when you have to write web services or desktop apps or any kind of application which requires speed or scale that cannot be handed down to a lower level library in Python. Also for very large code bases, static typing truly helps to keep things sane, especially when you have different teams working on different parts of the codebase — with statically typed languages, no type checks in unit tests are needed, and refactoring is much more solid and error free. Python’s type annotations are an attempt to move in this direction, but static languages truly excel at type integrity (which are sometimes the cause of subtle errors in Python). Otherwise my experience has often been that Python is a good first or second choice, depending on what you’re doing. Making that choice correctly is what sets senior/principal engineers apart from junior folks.
- takeda 5y ago
  > Python’s type annotations are an attempt to move in this direction, but static languages truly excel at type integrity (which are sometimes the cause of subtle errors in Python). I wouldn't call it an attempt. I mean, sure it has no chance against for example Rust, but Python's type system is actually quite decent and I think it is more powerful than the one from Go. Also you have freedom of using both nominal and structural typing if you chose. The only problem I have with it when I have to use a package that don't have types, but fortunately that is happening less and less. Some package authors refuse to add types, but others provide stubs to solve that problem. For example boto3-stubs provides types for boto3. I really like that thanks to types I can also easily refactor code without worrying about breaking something in the process. I suspect if types existed and were popular with Python 2 then the whole Python 2 -> Python 3 migration would be a non issue.
runT1ME 5y ago
Would love to get your take on if python is popular in large organizations because of the existing libraries or if it is the language asethetic itself? To an outsider who knows a bit of python because of Airflow and Spark, it seems Python has become a popular metaprogramming language that various disparate ecosystems have all adapated. Pyspark is python but kind of its own language and much of the processing is happening outside the python runtime. I think you could say the same for people doing Numpy or Pandas. Tensorflow probably even more so where I believe Python is the most popular interface, but you're really just programming a program to run somewhere else. Again this is the same with Airflow conceptually, though I believe the runtime is python. If my hypothesis is overall correct, and that a majority (or large share) of python programmers aren't sharing the same ecosystem, libraries and packages, then shifting to a different runtime or language that just encapsulates a subset of the language is much less challenging. This is the opposite problem of the JVM, where no one really likes Java the langauge, so everyone tries to create enjoyable (and productive) languages for the jvm to keep using the libraries and the runtime optimizations.
BiteCode_dev 5y ago
A trap for your dream world were you suddenly get google size ? Because I have a 1 million unique users video streaming service still running python 2.7, using a few servers. The thing has a mobile version serving a different media on the fly, encodes user uploaded videos, features comments, tagging, and even has machine learning detection of content now. It is still maintained by one single person, and he is not a professional dev. And I'd say it's already very rare to reach that size in any project, in any company. Hell my last 3 paid projects as a freelancer have less than 20 concurrent users. And that's professional. That's people's real life. 2 out of 3 IT projects fail, no code is even shipped. I would worry about the scaling later. Once you do have the problem, pay the price for more hardware or a rewrite, you made it! And if you do know for sure you will have the problem (you already work for a GAFAM), then not choosing python is ok, it's not a trap, it's a trade off. But for the vast majority of us out there, having an easy, solid, battle tested and clean language with a huge ecosystem is worth a 10000 times more than some scaling potential in 5 years down the road. I have seen so many big C++ or Java project fail I don't associate any language with big and failure anyway. I've seen devs arguing about the purity of this in Haskell, or that in Ocaml, and nothing gets to the end user because it's never perfect, or it's a toy, and doesn't handle IRL. But what powers most companies ? Rail apps that you won't upgrade ever, but are still running. Spaghetti PHP code and wordpress that just won't dies. Horrible SAP scripts, VBA macros and excel sheets that actually deal with your real data. Because they shipped. They did the job.
- Cthulhu_ 5y ago
  It's a variation on cargo cult; you WISH you had the scaling problems that could be solved by using something a bit more performant. But solve the problem first, and solve it fast. It's why so many startups from the 2010's on used Ruby; they were productive in it, solving a real problem, rolling out features fast. I've worked in a few projects where they dove onto the cargo cult, of wishing they had the kind of requests and load that e.g. a microservices architecture might help with. Massively overpriced projects too, because they hired lots of consultants and self-employed people. The one time there was a guy who rocked up in a Maserati (used, lol); he took two weeks to build a page that was just a centered bit of text and a button, and it didn't even work.
zohch 5y ago
> I've seen the curve and the pain it's caused, and would never use it for any code that needs to be performant or actively developed on the multi-month or year timescale Typed python is safer and more maintainable than Go. Slow as a dog, but it's really a hard choice when choosing performance and a bad type system with a half baked language vs a good language with a good type system that is a bit of a drag on performance. And you really don't have to go all in typing. I have worked on gradually typing code bases and it pays off from day one almost.
njharman 5y ago
The GIL has already been removed in at least Jython and PyPy. PyPy is 4x faster (on benchmarks), no xp with Jython.
prionassembly 5y ago
I recently learned just enough Rust to use its ndarray crate and rewrite bottleneck bits of code. I used to think the whole "write inner loops in $fastlang, use $scriptlang as glue" was something of a lame "cope" (and I say this as someone who doesn't know $fastlang), but I'm seeing upwards of 100X time performance improvements for code that can't be hammered into numpy idioms. I can provide code examples. Edit: code example https://gist.github.com/asemic-horizon/2830ed3637cfd278e7937f56450fd271 https://gist.github.com/asemic-horizon/2830ed3637cfd278e7937...
- wenc 5y ago
  Thanks for providing code. I’ve written code like that before and the first thing that jumps out at me is that you have for loops in Python code which isn’t slow but will never be as fast as loops in a static compiled language. Sometimes for loops are the most readable way of expressing a calculation but as a numerical computation person my instinct is to look for opportunities to vectorize by rewriting loops into matrix notation on paper and then expressing them as array calculations. Otherwise you’re right — whenever you have “for” loops you’re always going to end up ahead if you rewrite those in C, Rust, Julia or Fortran. And that’s exactly what authors of high performance numerical codes do.
  - fasterbynight 5y ago
    I don't know about other languages, but in Julia I've heard people often say that loops end up faster than the equivalent vectorized code. So while this is true for Python/Matlab I don't think it is good universal advice. That said, matrix notation can sometimes be the more readable way of expressing a calculation.
    - prionassembly 5y ago
      I might have used an early beta of Julia circa 2018 or something, but the chorus that it performs like $static_fastlang doesn't match the experience I had.
      - 5y ago
bilater 5y ago
uhmm Instagram runs on Django
max-ibel 5y ago
As a counterpoint: Micropython seems to make embedded code more readable and bug free than Arduino C-based programs, judging from a few projects I looked into. Yes, it would probably be a mistake to write a large distributed, high performance database (say, for a login system handling billions of users) in python, but many, if not most, projects never hit the wall you are referring to. In that sense, I feel that writing something in the most scalable language (if such a thing existed) would just be premature optimization at the highest level.
brundolf 5y ago
To add to this: any time somebody at my company who doesn't work on our Python services every day has to dip into them for a one-off task, it takes a day or two of troubleshooting their environment just to get to where they can start working with the actual code. And this isn't necessarily from scratch: any Python environment that's sat still for more than a few weeks becomes inevitably broken. It's a huge time-sink and I have to think it offsets any other productivity gains that Python may be giving us. Personally I refuse to touch our Python services with a ten-foot pole. My local app hits the testing environment and that's that.
- manfre 5y ago
  I recommend learning to use tox and virtualenvs to avoid this headache. A properly laid out project (of any language) should have a fast time from checkout to running tests. For most python projects, that can be as simple as `pip install tox & tox`
  - brundolf 5y ago
    We use pip and virtual envs; not sure about tox
8589934591 5y ago
Is/Are there alternative programming language(s) you would suggest instead of python? It would be great if you could list the use cases also where one language does it better than python.
comeonseriously 5y ago
Last I checked there were 37,373.7 programming languages. Pick one. Shrug.

by ctur 5y ago

_wldu 5y ago

This is my experience as well (but not at a FAANG). You can do things quickly in Python and it is a joy to write, but down the road (as you grow), you will regret it and find yourself re-writing it.

SoylentOrange 5y ago

What do you use nowadays?

chriswarbo 5y ago

Adding types after-the-fact can certainly be painful, but at least it's not an all-or-nothing choice, we can opt in to types for selected parts of a codebase. Personally I've got a lot of mileage out of Hypothesis for property-based testing. It's good at exercising edge-cases, and works particularly well when we sprinkle assertions through a codebase (where the "property" we're testing is simply "calling Foo doesn't throw an exception").

throwaway894345 5y ago
> Adding types after-the-fact can certainly be painful, but at least it's not an all-or-nothing choice, we can opt in to types for selected parts of a codebase. One of the benefits of typing (that is rarely discussed) is that it deters people from writing code whose type would otherwise be crazy (e.g., "if someone passes the string 'foo' in for a parameter, then the return type is a string, otherwise it's a bytestring"). In other words, it deters a lot of crappy code. When you annotate after the fact, the annotation becomes really difficult and people get crabby that they're having to make this really complicated annotation and they blame typing (rather than their own poor coding). To your point, typing after-the-fact is still better than nothing, but you're missing out on a lot by waiting. Moreover, Python's typing story is still really immature with respect to syntax and tooling.
- joconde 5y ago
  If a person like that is able to harm the team, I'd argue that it's a management problem. Lazy coding is only an issue when you allow people to be lazy...
  - throwaway894345 5y ago
    I think you could make the same argument about code styling or really just about anything. If you had sufficiently quality coworkers, these things probably wouldn't be an issue, but it's a lot easier to just implement the technical solution (a code formatter or type checker).

nicolaslem 5y ago

Isn't it that maintaining large codebases in the long run is just hard? What language is not a trap?

ctur 5y ago
Fair, maybe it's always a matter of picking the trap you want :) If you are optimizing for 0->1, Python is great. But if you want code to live and evolve over time with multiple authors, or ever will care about runtime performance, it's a dead end almost from the first line. But sometimes the productivity benefit is all that matters and you accept you may have to throw it all out later (or invest insanely in making it work)... it's all about making an informed decision.
- cglace 5y ago
  As with any language it's how you end up architecting your codebase. I do believe python lacks an authoritative resource for what is "good architecture" which leads to a lot of the code scaling problems.
  - meltedcapacitor 5y ago
    What widely adopted language achieved this? Even Rust, despite the head start, is already a mess on its way to inevitable cobolization.
sorokod 5y ago
Digging a well is hard, doesn't mean you have to this with a spoon.

UnpossibleJim 5y ago

I'm curious, as you seem very versed in the Python ecosystem, where you stand on projects like Cython and integration of C libraries with Python using projects like Cython? Or would you consider that more of a stop-gap measure and not "real" Python (which would be valid)?

ctur 5y ago
Things like Cython are great until they aren't, so yes, I think of them as a stop gap (albeit a practical and useful one). Any time you change interpreters or the execution model, you risk compatibility challenges with third party libraries and native modules. You also risk getting painted into a corner by their limitations. They definitely can help with hot code paths when their trade-offs don't prevent it but it still ends up being a kluge working around fundamental language issues. But when Cython fits, it's great, and a good tool to have in your toolbox if you're working with Python.

dunefox 5y ago

It was never intended as a stand-alone language for large projects, rather as a glue and prototyping language. All the tacked-on "static typing" won't change this. I think it would be best if Python was used according to its strengths and not as a 'unicorn' language.

toolslive 5y ago

Also, typing in python has a "kluge" feel about it. It's like most things in python: it works nicely on toy examples, but is rather frail on the edges (if you're not convinced try to define the json type). I also think they chose the wrong strategy (type inference would have been way better) To use a metaphor: Python is the duplo of programming languages.

mixmastamyk 5y ago
Python has always had strong, inferred typing. It’s just that you can rebind variables.
- toolslive 5y ago
  _static_ typing.

klyrs 5y ago

I'm not terribly familiar with the options for typing, mostly because I'm salty that Python rolled out annotations without a clear plan for how they'd be useful*. Aside from Cython, does typing provide any mechanism to improve performance, or is it only there for static analysis? Worse yet, is it being used to do dynamic analysis that slows the whole thing even further? * specifically, this happened long after the tragically many-ways-to-do-it of packaging systems and virtual environments, which survive to this day.

mgraczyk 5y ago
There are some tools that use types for optimization, but I'd say it's still pretty early. For example, mypyc is a python to C compiler that is part of mypy. It uses types for both correctness and optimization: https://mypyc.readthedocs.io/en/latest/using_type_annotations.html https://mypyc.readthedocs.io/en/latest/using_type_annotation...

mrfusion 5y ago

You may be right but I’m convinced most problems don’t need to scale.

ctur 5y ago
Absolutely true.

wil421 5y ago

Do you have the opposite experience? A language used by FAANG that doesn’t have performance issues at their scale. They seem to invest a lot in JavaScript which is not what I’d call a performant language and as you’ve said Python. I’ve used Python for more scripting type tasks and creating basic APIs to get specific jobs done. I’m very weary to go “all in” on Python for an app based on all the feedback about performance. All that being said I’ve never heard someone say we used X language and it scaled remarkably without issues. I’ve noticed lots of companies eventually go to Java once they get big but nobody really likes to brag about Java apps.

ctur 5y ago
Everything has performance issues but Python is at least one, if not two, orders of magnitude off of compiled languages like C++, Rust, and Go (despite Go having a garbage collector). They have various usability tradeoffs but once you get things working, at least you're not spending 10x as many cycles to do the same work as another language would need. (Java probably is also fine; I've less personal experience there so I can't say).
- mda 5y ago
  My experience, Java is usually 10-100x faster than python in general.
- bcrosby95 5y ago
  Statically typed, garbage collected languages are all roughly in the same ballpark, on average (Go, C#, Java).
pyentropy 5y ago
> They seem to invest a lot in JavaScript which is not what I’d call a performant language JavaScript (V8 at least) is extremely performant, near native code performance. Considering how dynamic it is, it's not an easy feat but Google, Apple and Mozilla work a lot on it. Here's a benchmark where JS is 50x faster than Python: https://github.com/kostya/benchmarks https://github.com/kostya/benchmarks Note that PyPy does much better.

chuckcode 5y ago

Thanks for this comment, sums up exactly my experience with python. Do you have a pointer to a larger write up of these issues? I really enjoy python, but agree with you that it isn't well suited for performance or scale necessarily. I find that people really get attached to it though and I'd like to have a good reference for them to help explain why issues like interpreter performance, GIL and typing make a difference on large projects even if not their pet ones.

qsort 5y ago

Pretty much everything you're saying matches my experience, so, playing the devil's advocate: - The typing argument is impossible to argue against, but isn't it the same problem you'd have with any other dynamic language? How is, say, Javascript any better? I don't see this as something that's specifically Python's fault, rather, yet another argument as to why starting a large project in 2021 without seriously considering static types is bordering professional malpractice. - As for the GIL thing, I'm with you that it's never going to be lifted in any meaningful way, but wouldn't the subinterpreters idea along the lines of Ruby's ractors fix most of the pain in most cases? Also, why are you skeptical about "the 5x plan"? I don't really have an opinion about it, but I'm interested in hearing any input.

azeirah 5y ago
For your first point, I don't think a language such as JavaScript -is- much better, which is why Typescript was invented as an alternative for larger codebases/organizations
- igouy 5y ago
  Also Dart.
- dragonwriter 5y ago
  TypeScript:JS is a lot like mypy:Python; the only real difference is that Python adapted so that mypy (originally envisioned as its own language) code is also valid Python as well as the reverse, obviating the need for a compilation step to run mypy in the Python runtime (whereas TS has to compile to JS to run); also, mypy has mypyc to compile (currently, only a subset of) mypy code to something more efficient than normal python.
  - whimsicalism 5y ago
    Having worked extensively with both, I don't think this is an apt comparison. Type inference & hinting at the IDE level is just not as developed either

tristor 5y ago

This mirrors my experience as well. At one (at the time major) cloud provider we had to internally re-implement 100% API / behavior compatible Java applications for the publicly available Python applications for our OpenStack deployment, due to performance and scaling requirements. OpenStack Foundation required all projects to be only written in Python, which I think was a major factor in why OpenStack adoption was never significant. Scaling and deploying it was an absolute nightmare using public code. Anything done at significant enough scale /must/ be multi-threaded to take full advantage of modern hardware which makes languages which don't provide a reasonable threading model effectively non-starters or at best a toy language for internal prototyping. I shifted from Python to Ruby, simply because Ruby provided a realistic way to interface with real threads, and then to Go, and haven't looked back. Simple applications are massively more performant in Go than in Python, and it's not just due to typing and compilation. I love how simple Python is to learn and how it brings more people into the fold in approaching solving complex problems, but at the end of the day you should treat it like runnable pseudocode to get shared understanding so you can implement in a real language.

Lucasoato 5y ago

Ok, let's admit it, writing bad Python code is very easy, as it is in every dinamically typed language, like Javascript... ... but I think you're forgetting the real motivations that drive people to use Python: it's easy to learn, it's very readable, there's a large (and not so much toxic) community behind, it's versatile. No one is saying that it will replace every other language, but it still can find its space even among large projects.

taeric 5y ago
I think the "dynamic" jab is a bit of a trap, itself. It is easy to write bad code in most any language. Anyone that tells you otherwise is trying to sell you something.
capitainenemo 5y ago
Readable varies by person. Myself, I find the decision on significant whitespace to make reading Python like reading English with no punctuation or capitalisation.
BiteCode_dev 5y ago
It's not that easy actually. Python already does so much for you, and provides so much tooling to do even more, it's hard to write bad code if your daily job is to be a coder. The iterator protocol means you rarely get off by one errors, context managers and GC deals with resource handling, the GIL, for all its faults, helps a lot of concurrency errors... Now, add a good IDE, and syntax errors, names errors and attribute errors go away. Play a little in the REPL, and use cases become more obvious, behaviors more clear. Want more security? You can add type hints and unit tests (which are incredibly easy to write, thanks to pytest). Even without all that, the size of the ecosystem means you won't write most of the code anyway, and less code, means less potential for mistakes. Adds to that the language bare bone syntax, and you really has something that actively helps with writing good code. Not to say you can't write bad code in Python, but unless you are not a professional dev (in that case, you can write bad code in any lang), you have a lot to help you there.
dragonwriter 5y ago
> Ok, let's admit it, writing bad Python code is very easy, as it is in every dinamically typed language Writing bad code is easy in most statically typed languages too.

j1elo 5y ago

I'm a C++ dev and was taught Python as a tool for prototyping and validation of ideas, not as a language for final implementations. Later with years I've read endless histories about the "traps" you mention, so I understood the notion I was taught wasn't without its merits. So I still look suspiciously to all mid to large projects what are written in Python. It takes some serious effort and good engineering to maintain good quality, and still the tooling is not there for error detection, like in other better typed languages would be. I wouldn't feel confident relying on such project for a critical infrastructure piece. The idea should have been validated quickly with Python, then implemented in a more appropriate language. EDIT: Obviously that's just my possibly wrong opinion. There are huge projects out there proving otherwise, but the issues discussed here are _very_ real.

indymike 5y ago

> This means performance will fall further and further behind compiled languages. This is exactly as it should be: the nature of the interpreter is that it has to do a lot of the work a compiler does when it compiles whenever a new command is run (and in Python's case, that is before you cover the whole "everything is an object" part). Yes, some of it can be mitigated, but the expectation that an interpreter keep pace with compiled code is somewhat of a pipe dream. What I'd love is a language that can be compiled (and be highly optimized) and interpreted without changing it's behavior. The developer experience with an interpreter is fantastic, the performance of compiled code is... better.

colejohnson66 5y ago
> What I'd love is a language that can be compiled (and be highly optimized) and interpreted without changing it's behavior. The developer experience with an interpreter is fantastic, the performance of compiled code is... better. You mean something like a byte code like language with a JIT system? Like Java and C#? C# in particular now supports AOT compilation, but can still be run as “interpreted” CIL.
- indymike 5y ago
  Having a VM / JIT in the middle is a way to get there, but ultimately, different. At least in Java's case, my code targets an abstraction instead of hardware, so I don't get the full benefit of compilation. Performance is part of the story... access to the OS and hardware is another.
  - smallnamespace 5y ago
    Even native compilers like clang will use LLVM, and LLVM IR can look a lot like JVM bytecode. The VM abstraction is not a barrier to fully optimizing for your architecture, but rather how much time you can spend converting IR/bytecode into assembly, whether you do it at runtime, and whether runtime information lets you optimize even further.
giovannibonetti 5y ago

stinos 5y ago

I have become increasingly convinced Python as a language is a "trap" for any use of notable scale I assume you're saying this out of experience, so can you give a practical example of what you call 'notable scale'? And are you talking about desktop or web or mobile, or just all of them? Just so that we know what you are talking about in a concrete way. Not the 'large software with large number of developers is always a problem' way. edit I also see others here using 'scale' in a 'performance / number of request per second' etc meaning - I thought we were talking codebase size though. I mean if you know on beforehand this type of performace is a requirement it seems weird to turn to Python, of all things?

ck_one 5y ago
+1 Would be cool if some experienced dev could share some estimates when the Python scaling issues start. When you build a backend using Python + Django/FastAPI, I assume in most cases the DB and not Python is the limiting factor. Moreover, you could always spin up more workers to mitigate scaling issues. When you train ML models, your Python code just calls C++ functions. Python is not a limiting factor here either.
- ferdowsi 5y ago
  I worked for a startup that built their services using Python and Twisted. The codebase was a monolith broken up into several Twisted services, by the time I left it was probably 200k LOC of Python. For us, the scaling problems started when we had two independent teams working in the same codebase. The extreme dynamism of the language meant that classes and data structures were being mutated willy-nilly in ridiculous ways across the execution flow. The lack of static typing made onboarding new developers difficult as they had to parse generations of excessively clever code and magic left behind by departed developers. This problem has only gotten worse in Python 3, which keeps piling on more ways to accomplish the same task. The deployment story was also awful, but I don't think that's a surprise to anyone who has deployed Python at scale. In terms of raw performance, at one point we estimated that our Python stack was adding 3-400ms of request latency compared to a comparable system written in Go. With the Python 3 deadline coming, we convinced management to invest in rewriting performance-critical parts of the service in Go, instead of the migration to Python 3. I left before the project was completed but we were already seeing massive improvements.

dom96 5y ago

If anybody is looking for an alternative to Python that is also a great 0->1 language but doesn't have the same wall as Python described here, check out Nim[1]. 1 - https://nim-lang.org https://nim-lang.org

amelius 5y ago
People's main reason for using Python is the ecosystem.
- arc619 5y ago
  Nim can use Python's ecosystem in both directions with nimpy https://github.com/yglukhov/nimpy https://github.com/yglukhov/nimpy and nimporter https://github.com/yglukhov/nimpy https://github.com/yglukhov/nimpy This lets you gradually transition hot path Python modules to Nim, get compiled performance generally on par with C and Rust, whilst enjoying strong, static typing with great type inference. In my experience (6-7 years 4 of which are full time) Nim strikes the perfect balance of the productivity you get with Python with high performance at the same time. Also the metaprogramming features are incredible and, importantly, don't use a language subset but use the base Nim language itself.
  - amelius 5y ago
    That's interesting but it looks like Nim has no support for Qt yet (where Python has PyQt and PySide).
    - arc619 5y ago
      I've never used Qt myself, but the Status messaging app written in Nim does, so it's definitely doable: https://github.com/status-im/status-desktop https://github.com/status-im/status-desktop There's also a QML wrapper here: https://github.com/filcuc/nimqml https://github.com/filcuc/nimqml
  - arc619 5y ago
    Oops wrong link for nimporter. Actual link: https://github.com/Pebaz/Nimporter https://github.com/Pebaz/Nimporter

spamizbad 5y ago

Most problems do not need scale. Some problems require it. For those: use one of many proven "fast" languages. For everything else Python (or similar language) is more than adequate.

nickjj 5y ago

> I have become increasingly convinced Python as a language is a "trap" for any use of notable scal At what point do you embrace the trap and happily live with it? For example Dropbox runs many millions of lines of Python, has massive traffic and their server costs aren't completely out of line for the service they offer. Their code base is also over 10 years old. It seems to be working very well for them. I talked to one of their engineers a few months ago. He gave me a complete run down of how they build and deploy Dropbox at https://runninginproduction.com/podcast/82-dropbox-gives-you-secure-access-to-all-of-your-files https://runninginproduction.com/podcast/82-dropbox-gives-you....

ferdowsi 5y ago
My understanding is that Dropbox has been building performance-critical parts of their services in Go since 2014 at the least. https://twitter.com/jamwt/status/629727590782099456 https://twitter.com/jamwt/status/629727590782099456 Like many companies, they had initial scaling successes with Python,hit performance bottlenecks and looked for a way out with faster languages.
- coldtea 5y ago
  So? If they started with those faster languages they'd have slowed down their rolling of features initially. They might not even be here now.
- nickjj 5y ago
  Sure, Rust too for some of the desktop app components. But the takeaway is you only need to do that when you're operating at mega scale and actually hit these bottlenecks. In a ton of cases you'll be completely fine serving a few million monthly page views of your SAAS app on a single $20-40 a month server using Flask, Django or whatever Python web framework you prefer. Performance will be really good too. Just talking about most apps where it's mainly reading and writing data to a DB.
  - wickoff 5y ago
    You may be better off using rails instead to fully capitalize on rapid development through automagic.

Pxtl 5y ago

Likewise. I cut my teeth on Python 20 years ago and I loved it back then - even getting my hands dirty in the guts of the interpreter. But now in hindsight, I'm shocked that Python 3.0 wasn't seized as an opportunity to lose more of its baggage - the under-optimized interpreter, the poor parallelization story, the excessive surface area of the interpreter exposed to Python itself, making it impractical to re-implement in other interpreters, better typing, etc. They broke backwards compatibility and all we got for it was Unicode?

CuriouslyC 5y ago

Python is great for developing tools for infrastructure management, for scientific computation, and for small-medium web applications. Given its best in breed interoperability with C/C++, it is also good as a shell language for a top level project that incorporates other sub projects.

rowanseymour 5y ago

I feel pretty similarly. Our product used to be a Python/Django monolith and over the years we've ended up pulling so much of the functionality out into services written in golang. It was fantastic for getting the product up and running relatively quickly, and we're still very happy to let Django take care of the frontend and database migrations. But when your codebase has gotten big, refactoring in Python feels like Russian roulette, and when you're doing a lot of background processing, celery feels like the wrong tool for the job.

ck_one 5y ago
How do you you handle the communication between the Django codebase and the Go codebase? Does it produce a lot of overhead?
- rowanseymour 5y ago
  All the communication is initiated on the Django side and it's either HTTP and tasks queued in Redis depending on whether it needs to be synchronous. It adds a little extra complexity but it's also nice being able to stop and start services without bringing the whole system down.

joconde 5y ago

We started a project using typing annotations for everything, and it's really a pleasure to use. It's like the type safety of C++ without the insane language complexity; the "advanced" parts of the language usually make intuitive sense. For our use case of deep learning, REST APIs, and image processing (often a mix of these), Python seems like the best choice we have. The GIL isn't a problem at all because all our computations are done with NumPy or PyTorch, which release the GIL during most of their operations.

mynameisash 5y ago

> eventually you hit a wall and have to keep investing larger and larger amounts of people or computing resources to get continued returns. That's certainly my impression. I'm not a professional Python programmer, though I have used it in a few projects that went to prod. I really want to like Python, and it is really nice for some quick one-offs. However, in my project, where the codebase was 30-40% Python, I estimate that 80-90% of bugs and time spent were in Python. The other side of things was more or less write once, never revisit.

coldtea 5y ago

>It's a great 0->1 language and great at simple glue, but eventually you hit a wall and have to keep investing larger and larger amounts of people or computing resources to get continued returns... If you're a millionaire by that time, or have a strong established business, then it doesn't matter, it's a nice problem to have. Have your engineers have a go at it. See: Dropbox, Disqus, AirBnB, and others... And if you're not, then going from 0->100 in 2 years instead of 0->1 in a few months with Python wasn't worth it anyway (and you also don't have enough customers/users to make use of those 100x improvement).

nightski 5y ago
I work as a consultant for companies with billions in revenue and from what I see it definitely matters. Re-writes simply don't happen that often. Maybe for the pure venture capital SV tech companies you mentioned, but many companies are not in that bucket. Making good engineering choices from the beginning matters a lot.
- jacobolus 5y ago
  Rewrites may not happen that often, but plenty of large-scale corporate tech projects flail for years without ever reaching a functional state. If the choice is between a sub-optimally performant but working system, or a boondoggle that never gets off the ground, the “better engineering choice” is probably the former.
gizdan 5y ago
Also, pretty sure Python was called out as one of the main reasons why some hundred of Google Video engineers couldn't keep up with YouTube prior to Google's purchase. Python might not be the most performant, but once you need the scale, it's a good problem to have because at that point, your product has already made it.
- takeda 5y ago
  Do you have an article about that? I would be interested to learn more about it.

wenc 5y ago

As someone with nearly two decades of experience with Python and who has deployed it in production in high impact scenarios, I’d like to nuance the above for people who think Python is only good for prototyping and that you have to rewrite in a “proper” programming language. This is the kind of blanket thinking to avoid. I think the rewriting part is only necessary if there’s some characteristic in your use case that you need that Python doesn’t provide — like raw computational speed, low latencies etc. If you’re a senior engineer familiar with Python, these trade offs are always front of mind. For the majority of software I’ve developed, these characteristics were not essential. Python was a good choice for 80-90% of all of the important software products I’ve ever written. For many data science projects, maintaining large code bases in Python is often the optimal decision (rather than reinventing the wheel and writing your own data frame and machine learning libraries, or using immature poorly maintained ones in other languages — for data manipulation and scientific algorithms, these libraries are highly optimized in Python anyway since the underlying code is in C or Fortran). Python is also a good choice for data engineering pipelines. Where I might hesitate to recommend Python is when you have to write web services or desktop apps or any kind of application which requires speed or scale that cannot be handed down to a lower level library in Python. Also for very large code bases, static typing truly helps to keep things sane, especially when you have different teams working on different parts of the codebase — with statically typed languages, no type checks in unit tests are needed, and refactoring is much more solid and error free. Python’s type annotations are an attempt to move in this direction, but static languages truly excel at type integrity (which are sometimes the cause of subtle errors in Python). Otherwise my experience has often been that Python is a good first or second choice, depending on what you’re doing. Making that choice correctly is what sets senior/principal engineers apart from junior folks.

takeda 5y ago
> Python’s type annotations are an attempt to move in this direction, but static languages truly excel at type integrity (which are sometimes the cause of subtle errors in Python). I wouldn't call it an attempt. I mean, sure it has no chance against for example Rust, but Python's type system is actually quite decent and I think it is more powerful than the one from Go. Also you have freedom of using both nominal and structural typing if you chose. The only problem I have with it when I have to use a package that don't have types, but fortunately that is happening less and less. Some package authors refuse to add types, but others provide stubs to solve that problem. For example boto3-stubs provides types for boto3. I really like that thanks to types I can also easily refactor code without worrying about breaking something in the process. I suspect if types existed and were popular with Python 2 then the whole Python 2 -> Python 3 migration would be a non issue.

runT1ME 5y ago

Would love to get your take on if python is popular in large organizations because of the existing libraries or if it is the language asethetic itself? To an outsider who knows a bit of python because of Airflow and Spark, it seems Python has become a popular metaprogramming language that various disparate ecosystems have all adapated. Pyspark is python but kind of its own language and much of the processing is happening outside the python runtime. I think you could say the same for people doing Numpy or Pandas. Tensorflow probably even more so where I believe Python is the most popular interface, but you're really just programming a program to run somewhere else. Again this is the same with Airflow conceptually, though I believe the runtime is python. If my hypothesis is overall correct, and that a majority (or large share) of python programmers aren't sharing the same ecosystem, libraries and packages, then shifting to a different runtime or language that just encapsulates a subset of the language is much less challenging. This is the opposite problem of the JVM, where no one really likes Java the langauge, so everyone tries to create enjoyable (and productive) languages for the jvm to keep using the libraries and the runtime optimizations.

BiteCode_dev 5y ago

A trap for your dream world were you suddenly get google size ? Because I have a 1 million unique users video streaming service still running python 2.7, using a few servers. The thing has a mobile version serving a different media on the fly, encodes user uploaded videos, features comments, tagging, and even has machine learning detection of content now. It is still maintained by one single person, and he is not a professional dev. And I'd say it's already very rare to reach that size in any project, in any company. Hell my last 3 paid projects as a freelancer have less than 20 concurrent users. And that's professional. That's people's real life. 2 out of 3 IT projects fail, no code is even shipped. I would worry about the scaling later. Once you do have the problem, pay the price for more hardware or a rewrite, you made it! And if you do know for sure you will have the problem (you already work for a GAFAM), then not choosing python is ok, it's not a trap, it's a trade off. But for the vast majority of us out there, having an easy, solid, battle tested and clean language with a huge ecosystem is worth a 10000 times more than some scaling potential in 5 years down the road. I have seen so many big C++ or Java project fail I don't associate any language with big and failure anyway. I've seen devs arguing about the purity of this in Haskell, or that in Ocaml, and nothing gets to the end user because it's never perfect, or it's a toy, and doesn't handle IRL. But what powers most companies ? Rail apps that you won't upgrade ever, but are still running. Spaghetti PHP code and wordpress that just won't dies. Horrible SAP scripts, VBA macros and excel sheets that actually deal with your real data. Because they shipped. They did the job.

Cthulhu_ 5y ago
It's a variation on cargo cult; you WISH you had the scaling problems that could be solved by using something a bit more performant. But solve the problem first, and solve it fast. It's why so many startups from the 2010's on used Ruby; they were productive in it, solving a real problem, rolling out features fast. I've worked in a few projects where they dove onto the cargo cult, of wishing they had the kind of requests and load that e.g. a microservices architecture might help with. Massively overpriced projects too, because they hired lots of consultants and self-employed people. The one time there was a guy who rocked up in a Maserati (used, lol); he took two weeks to build a page that was just a centered bit of text and a button, and it didn't even work.

zohch 5y ago

> I've seen the curve and the pain it's caused, and would never use it for any code that needs to be performant or actively developed on the multi-month or year timescale Typed python is safer and more maintainable than Go. Slow as a dog, but it's really a hard choice when choosing performance and a bad type system with a half baked language vs a good language with a good type system that is a bit of a drag on performance. And you really don't have to go all in typing. I have worked on gradually typing code bases and it pays off from day one almost.

njharman 5y ago

The GIL has already been removed in at least Jython and PyPy. PyPy is 4x faster (on benchmarks), no xp with Jython.

prionassembly 5y ago

I recently learned just enough Rust to use its ndarray crate and rewrite bottleneck bits of code. I used to think the whole "write inner loops in $fastlang, use $scriptlang as glue" was something of a lame "cope" (and I say this as someone who doesn't know $fastlang), but I'm seeing upwards of 100X time performance improvements for code that can't be hammered into numpy idioms. I can provide code examples. Edit: code example https://gist.github.com/asemic-horizon/2830ed3637cfd278e7937f56450fd271 https://gist.github.com/asemic-horizon/2830ed3637cfd278e7937...

wenc 5y ago
Thanks for providing code. I’ve written code like that before and the first thing that jumps out at me is that you have for loops in Python code which isn’t slow but will never be as fast as loops in a static compiled language. Sometimes for loops are the most readable way of expressing a calculation but as a numerical computation person my instinct is to look for opportunities to vectorize by rewriting loops into matrix notation on paper and then expressing them as array calculations. Otherwise you’re right — whenever you have “for” loops you’re always going to end up ahead if you rewrite those in C, Rust, Julia or Fortran. And that’s exactly what authors of high performance numerical codes do.
- fasterbynight 5y ago
  I don't know about other languages, but in Julia I've heard people often say that loops end up faster than the equivalent vectorized code. So while this is true for Python/Matlab I don't think it is good universal advice. That said, matrix notation can sometimes be the more readable way of expressing a calculation.
  - prionassembly 5y ago
    I might have used an early beta of Julia circa 2018 or something, but the chorus that it performs like $static_fastlang doesn't match the experience I had.
    - 5y ago

bilater 5y ago

uhmm Instagram runs on Django

max-ibel 5y ago

As a counterpoint: Micropython seems to make embedded code more readable and bug free than Arduino C-based programs, judging from a few projects I looked into. Yes, it would probably be a mistake to write a large distributed, high performance database (say, for a login system handling billions of users) in python, but many, if not most, projects never hit the wall you are referring to. In that sense, I feel that writing something in the most scalable language (if such a thing existed) would just be premature optimization at the highest level.

brundolf 5y ago

To add to this: any time somebody at my company who doesn't work on our Python services every day has to dip into them for a one-off task, it takes a day or two of troubleshooting their environment just to get to where they can start working with the actual code. And this isn't necessarily from scratch: any Python environment that's sat still for more than a few weeks becomes inevitably broken. It's a huge time-sink and I have to think it offsets any other productivity gains that Python may be giving us. Personally I refuse to touch our Python services with a ten-foot pole. My local app hits the testing environment and that's that.

manfre 5y ago
I recommend learning to use tox and virtualenvs to avoid this headache. A properly laid out project (of any language) should have a fast time from checkout to running tests. For most python projects, that can be as simple as `pip install tox & tox`
- brundolf 5y ago
  We use pip and virtual envs; not sure about tox

8589934591 5y ago

Is/Are there alternative programming language(s) you would suggest instead of python? It would be great if you could list the use cases also where one language does it better than python.

comeonseriously 5y ago

Last I checked there were 37,373.7 programming languages. Pick one. Shrug.