6 ms·

I tried writing a production software system in Python in about 2009 using TurboGears (RoR but for Python). The whole system suffered from the well-known GIL p

by smoyer 5y ago

I tried writing a production software system in Python in about 2009 using TurboGears (RoR but for Python). The whole system suffered from the well-known GIL problems so the project was rewritten in JavaEE. My understanding is that the GIL problem has been eliminated (or at least minimized) but I've moved on to Go for backend software. I still love Python but now only use it for scripting system tasks and data science.

itamarst 5y ago
The article is about using Rust (for a Python memory profiler, to be fair).
29athrowaway 5y ago
There has been a lot of evolution since 2009. Async I/O, type annotations, etc.
- klyrs 5y ago
  What do type annotations actually get you? So far, cpython only stores them and makes them available for retrieval. This is a sore point for me -- batteries not included; so there's an ever-growing list of third party bolt-ons that are mutually incompatible, a vast departure from the Zen "one obvious way to do it".
  - klodolph 5y ago
    I use type annotations. They make it a lot easier to do things like refactoring. I've written a ton of Python since the early 2000s, and my experience is that projects above a certain size (and it's not really that large, like a few thousand loc) start getting hard to modify. There are also a few types of errors that reliably get caught by type checkers that you can easily encounter in Python code... for example, misspelling an exception name, or constructing an exception wrong. Simple typos in code paths that are not exercised in tests. The annotations also help you acclimate faster to code you haven't seen before (you can navigate to definitions / call sites more reliably), and lets Emacs / Vim / VS Code make better autocompletion suggestions. In short, it helps me code faster.
    - klyrs 5y ago
      Do you not test exceptions? It's really easy°, and I strongly encourage testing them. with pytest.raises(ZeroDivisionError): 1/0 ° note, it's not always easy to exercise all exceptions -- just, it's easy to make tests that fail if an exception isn't raised
      - klodolph 5y ago
        I do test exceptions, I just don't test 100% of code paths. Do you encourage 100% code coverage? Because I think that 100% code coverage is insane... The problem is that as you approach 100% code coverage, the amount of effort it takes to test the remaining code paths skyrockets. At some point, you decide to stop writing tests. Then static typing steps in, and catches some bugs that were not cost effective to catch with testing. The exceptions are sometimes there because I am programming defensively... as in, "I don't think that it's possible for this condition to happen, but if it does, I'd rather raise an exception." Then I run mypy, and it tells me that there's no such thing as a VauleError. The other factor here is that when I exercise poorly tested code paths (which again, I'm going to assume exist), it's often because I'm in the middle of working on something else. The simple typo or bad call is an interruption.
        deleted 5y ago
        [deleted]
        klyrs 5y ago
        > Do you encourage 100% code coverage? Because I think that 100% code coverage is insane... [ed. note, wow you added a lot as I was typing out my response to this] Not as a hard and fast rule, but as a target to aim for. If you're finding bugs in code that isn't covered by tests, I'd take that as evidence that your coverage isn't sufficient. I do a lot of my work in strongly typed languages; believe me when I say that code without type errors or variable misspellings can still be chockablock with bugs. In some codebases, 100% test coverage is equivalent to solving the Halting Problem. Don't sweat it.
        5y ago
  - 29athrowaway 5y ago
    Type inference is something that computers are better at doing given enough information. So, let computers do it. Use human time for more important things, like things that require human intelligence. If you don't have type safety you will end up writing 10x more unit tests anyways, while getting less certainty about your code.
    - klyrs 5y ago
      Sounds nice, but my gripe is that type annotations do not provide type safety. For that, you need one of many type checkers, which aren't mutually compatible. How certain are you of your certainty?
lostmsu 5y ago
GIL is hindering data science too. For instance, PyTorch DataLoader has somewhat limited throughput for any datasets where any part of item retrieval is implemented in Python.
plainnoodles 5y ago
I agree, I really like both Java and Python and I've typically used Java for anything that's a service and I've never been let down. I've owned a few python services and have been... disappointed with them overall but generally never because of *python* specifically. Usually the code just being an enormous untyped spaghetti blob and also being just bad code on top of it (the kind of thing where you look at it for 5 mins and could probably cut the runtime by 70% but you don't because nobody asked you to and this internal web page being slow is not high on your list of things that annoy you overall). I recently started a side project and decided to use python for the backend and I'm not too far into it so far, but I can already tell I'm going to miss 2 main things: 1. the ridiculous amount of information a jvm provides by default (e.g. jmx) 2. java's speed/jit So far, python seems fast enough, and I feel like I'll be able to make anything that I need to do scale in python, and that if I end up with enough users to make that not the case, I have bigger problems (like "my side project's customer support is competing materially with my day job for my time"). So I don't even think the tech, much less python, will be the scaling weak point with this project. But I'm still keeping an eye on it.
fulafel 5y ago
For web backends people just use process/vm level parallelism behind load balancers, like with Node, Ruby, Lua etc.

by smoyer 5y ago

itamarst 5y ago

The article is about using Rust (for a Python memory profiler, to be fair).

29athrowaway 5y ago

There has been a lot of evolution since 2009. Async I/O, type annotations, etc.

klyrs 5y ago
What do type annotations actually get you? So far, cpython only stores them and makes them available for retrieval. This is a sore point for me -- batteries not included; so there's an ever-growing list of third party bolt-ons that are mutually incompatible, a vast departure from the Zen "one obvious way to do it".
- klodolph 5y ago
  I use type annotations. They make it a lot easier to do things like refactoring. I've written a ton of Python since the early 2000s, and my experience is that projects above a certain size (and it's not really that large, like a few thousand loc) start getting hard to modify. There are also a few types of errors that reliably get caught by type checkers that you can easily encounter in Python code... for example, misspelling an exception name, or constructing an exception wrong. Simple typos in code paths that are not exercised in tests. The annotations also help you acclimate faster to code you haven't seen before (you can navigate to definitions / call sites more reliably), and lets Emacs / Vim / VS Code make better autocompletion suggestions. In short, it helps me code faster.
  - klyrs 5y ago
    Do you not test exceptions? It's really easy°, and I strongly encourage testing them. with pytest.raises(ZeroDivisionError): 1/0 ° note, it's not always easy to exercise all exceptions -- just, it's easy to make tests that fail if an exception isn't raised
    - klodolph 5y ago
      I do test exceptions, I just don't test 100% of code paths. Do you encourage 100% code coverage? Because I think that 100% code coverage is insane... The problem is that as you approach 100% code coverage, the amount of effort it takes to test the remaining code paths skyrockets. At some point, you decide to stop writing tests. Then static typing steps in, and catches some bugs that were not cost effective to catch with testing. The exceptions are sometimes there because I am programming defensively... as in, "I don't think that it's possible for this condition to happen, but if it does, I'd rather raise an exception." Then I run mypy, and it tells me that there's no such thing as a VauleError. The other factor here is that when I exercise poorly tested code paths (which again, I'm going to assume exist), it's often because I'm in the middle of working on something else. The simple typo or bad call is an interruption.
      - deleted 5y ago
        [deleted]
      - klyrs 5y ago
        > Do you encourage 100% code coverage? Because I think that 100% code coverage is insane... [ed. note, wow you added a lot as I was typing out my response to this] Not as a hard and fast rule, but as a target to aim for. If you're finding bugs in code that isn't covered by tests, I'd take that as evidence that your coverage isn't sufficient. I do a lot of my work in strongly typed languages; believe me when I say that code without type errors or variable misspellings can still be chockablock with bugs. In some codebases, 100% test coverage is equivalent to solving the Halting Problem. Don't sweat it.
        5y ago
- 29athrowaway 5y ago
  Type inference is something that computers are better at doing given enough information. So, let computers do it. Use human time for more important things, like things that require human intelligence. If you don't have type safety you will end up writing 10x more unit tests anyways, while getting less certainty about your code.
  - klyrs 5y ago
    Sounds nice, but my gripe is that type annotations do not provide type safety. For that, you need one of many type checkers, which aren't mutually compatible. How certain are you of your certainty?

lostmsu 5y ago

GIL is hindering data science too. For instance, PyTorch DataLoader has somewhat limited throughput for any datasets where any part of item retrieval is implemented in Python.

plainnoodles 5y ago

I agree, I really like both Java and Python and I've typically used Java for anything that's a service and I've never been let down. I've owned a few python services and have been... disappointed with them overall but generally never because of *python* specifically. Usually the code just being an enormous untyped spaghetti blob and also being just bad code on top of it (the kind of thing where you look at it for 5 mins and could probably cut the runtime by 70% but you don't because nobody asked you to and this internal web page being slow is not high on your list of things that annoy you overall). I recently started a side project and decided to use python for the backend and I'm not too far into it so far, but I can already tell I'm going to miss 2 main things: 1. the ridiculous amount of information a jvm provides by default (e.g. jmx) 2. java's speed/jit So far, python seems fast enough, and I feel like I'll be able to make anything that I need to do scale in python, and that if I end up with enough users to make that not the case, I have bigger problems (like "my side project's customer support is competing materially with my day job for my time"). So I don't even think the tech, much less python, will be the scaling weak point with this project. But I'm still keeping an eye on it.

fulafel 5y ago

For web backends people just use process/vm level parallelism behind load balancers, like with Node, Ruby, Lua etc.