7 ms·

Python's been my main language for 22 years now. Whenever I think I should switch to faster, more efficient language (C#/Go/Rust/...), I am repeatedly blown aw

Python's been my main language for 22 years now. Whenever I think I should switch to faster, more efficient language (C#/Go/Rust/...), I am repeatedly blown away by what Python lets me do, with an incredibly ecosystem of packages, and all the ergonomics of incremental development from a REPL. Case in point is my current project: building a parser for PDF bank statements from all UK banks. I can match a 10page bank statement against a library of 100 templates, and then extract its data in a standardised format, in around 50ms per statement. That's broadly comparable to the time for load the source file from cloud storage. And that 50ms is processing pages sequentially, using a single core on my little laptop. Plenty of scope to parallelise if I wanted to. The reason why I'm still using Python, to borrow a slogan from Bruce Eckel long ago, "Python fits [my] brain".

elabajaba 4y ago
My problem with Python, especially when reading someone else's code, or my own old bad code, is that I find that Python doesn't fit [my] brain. The lack of typing (in my experience you always end up with a bunch of libraries that don't have type hints) means I spend most of my time just figuring out what I'm supposed to pass to a function (I hate the django docs. Give me something like docs.rs where I can quickly scan what functions something implements, and see what they accept and what they return instead of forcing me to read an 800 page novel that STILL doesn't give you that basic info), and writing tests and trying to guard against passing the wrong thing around.
- stevesimmons 4y ago
  In my own code, I always use typehints and write good docstrings (nearly) everywhere. Visual Studio Code's type checker saves so much time and has improved my code quality to no end. It is especially powerful on polymorphic inputs, making sure the different code paths operate on the input type you expect.
  - shanebellone 4y ago
    PyCharm is good for this too. I've recently started using Type Hints by default. At the very least, it makes your code more readable. Nothing bad comes from that.
    - type-r 4y ago
      I started enforcing type hinting in my personal projects. 99% of the time it's easy and useful to add. When I can't figure out how to make a mypy error go away within 30 seconds, I just disable it for that line. I feel this makes for very good 80-20 value/effort tradeoff. I will say that more than one bug has been found after I disabled the single line hint though! Always surprising to me how powerful types can be.
  - mharig 4y ago
    I do not see the benefit of type hints. Good docstrings (and naming of the function and arguments) are superior, at least for the human. Type hints are too much clutter.
    - baq 4y ago
      There’s a certain program size from which they start to make sense and further still the benefits are obvious - you have a computer to help solve puzzles for you, so why not use it if the puzzle is ‘will this peg fit into that hole?’ when you have thousands of pegs and holes.
      - mharig 4y ago
        And so you end up writing climb_the_wooden_ladder(l: WoodenLadder) plus climb_the_aluminium_ladder(l: AluminiumLadder) or press all into an OO hierarchy instead of climb(thing): '''Climb a thing with steps''' # prevent run time crash: if not hasattr(thing, 'getSteps'): hitDeveloper()
        baq 4y ago
        No, you define a Ladder (or Climbable if you’re feeling fancy today) Protocol and all remains good in the world.
        cfu28 4y ago
        No need for OO hierarchy, just use a protocol
        mharig 4y ago
        Oh, I missed the Protocol class of the typing module. Have to further investigate the usage of this. Thanks to you and baq!
    - kbelder 4y ago
      I have a fear that Python will be improved to the point it offers no more advantage over other languages.
- dehrmann 4y ago
  > The lack of typing It's gotten a lot better with type hints and typeshed, but I agree, not knowing what type an arg should be and hoping duck typing works out isn't great.
Ultimatt 4y ago
50ms doesn't sound especially fast. If you had a million of them to process thats 4 hours, maybe an hour with some parallelism? A 100x speedup from going native then becomes quite welcome. Python has its place, but the fact you even think the tens of ms domain is "fast" or even fast enough on modern CPUs shows the real strength of Python, which is most people dont actually care at all about performance. Thats not to say its performant though. Just that no one cares anymore. Some rando boss is happy to wait 4 hours for your script to chug through a million statements, because no one actually told them it can take 2.5 minutes in a native language. If they did maybe the boss would suddenly speak about their dream to not only process the data but have a near real-time BI panel instead of a batch report so they can react within one business day. The issue with Python is missed value, not the value it can deliver.
- squeaky-clean 4y ago
  Why limit parallelism to 4x? Spin up a ton of lambdas and get it done in 10 seconds. You're also forgetting that 50ms time to load each file from cloud storage. It's still 4 hours and 2.5 minutes with your native code compared to 8 hours with Python. Suddenly not such a massive improvement. Even then I don't see the issue with it taking 4 hours to do a million of them. Do you need to do a million per hour? Is it even likely they'll need to do a million of them total? Do bank statements even come in frequently enough to do a realtime dashboard with? I get my bank statement every 30 days. How much longer does it take to develop the native version? How much longer does it take to modify when a bug is found or a bank changes their statement layout? How much more do you have to pay a native code engineer compared to a Python dev and how easy are they going to be to replace eventually?
  - blindhippo 4y ago
    One dimension to consider here is cost of compute. Going from 8 hours to 4 hours is a 50% reduction in the time to compute and we're assuming this is occurring on the same relative hardware/instance size. At scale, that could translate into hundreds of thousands of dollars in savings. But your points are relevant - as with anything related to development, "it depends" rules the day. There isn't a clear cut "x is objectively better than y" in general.
shanebellone 4y ago
I have Python analytics capturing HTTP Headers, processing them, and finally storing them in less than 8ms on a single core dev instance. Python is plenty fast for many things. Also, I love the syntax. It reads like a sentence.
- Someone 4y ago
  > in less than 8ms on a single core dev instance. Python is plenty fast for many things. What’s that, nowadays? 25 million cycles, 75 million instructions? Python isn’t plenty fast, modern hardware is.
  - pdonis 4y ago
    More precisely, modern CPUs are so much faster than network or file I/O that if your program has to wait on I/O, the CPU is going to sit idle most of the time anyway, so trying to optimize CPU time is pointless. The GP's use case is an example: he's capturing HTTP headers, so the bottleneck is network I/O, not CPU. So Python for his use case is, as he says, plenty fast. For use cases where CPU is the bottleneck, such as numeric simulations, Python users reach for something like numpy, where all the CPU intensive stuff is being done in C extensions, not Python bytecode.
  - KyeRussell 4y ago
    The person you’re replying to is saying that Python meets their needs. Judging by what they’ve said about their use case, I doubt it’s a meaningful affirm to their overall pipeline. Your counterpoint is that it’d be slower if it were running on a Pentium 4. Do you honestly think that they don’t know this? Yes, this is precisely the point. Things are at a point where Python’s benefits are worth more than what another language would bring in speed improvements. I imagine that there would be less of a benefit to busses if we had to Flintstone them everywhere instead using a combustion engine. That’s not the current reality though.
- iLoveOncall 4y ago
  That sounds insanely slow. I think it's fine to like Python as a scripting language, but arguing that Python is fast is simply laughable. I forsee Python's reputation going the way of PHP's (< 5, not modern), and statements like this will definitely contribute to it.
lowbloodsugar 4y ago
Plenty of languages have a huge package ecosystem. Your problem is that you don’t know them. I’m not saying the distinction is helpful for your situation, but your problem is “I have too much time invested in learning Pythons ecosystem” and not “other languages don’t have all the packages I need.” For REPL that’s another mindset. You use a REPL to experiment and prototype, I use unit tests. You use python because you already learned it. Your brain has been fit to python, not the other way around. You can fit it to other languages if you choose. I am currently learning my 7th “use in production” language.
- stevesimmons 4y ago
  You're partly right. However Python's sweet spot for me is when writing code to process complex, messy, poorly understood data. Using a Jupyter notebook lets me keep partially processed inputs as live objects, examine and change their in-memory representations, and quickly prototype the next stage of processing. It feels like sculpting with playdoh. Once that's working, the code moves from the Jupyter notebook into a proper code module with unit tests. Then the emphasis is on building the production system rather than manipulating the data. I get what you mean with unit tests. Just for me it feels further away from the data, which is where most the complexity in my business domain lies.
  - lowbloodsugar 4y ago
    Sure. I do use Jupyter in a decent IDE for some adhoc data diving or for a spike prototype before “downcoding” into a cluster.
- deleted 4y ago
  [deleted]
roflyear 4y ago
How are you matching a file to a template like that?
deleted 4y ago
[deleted]
MrVandemar 4y ago
>Case in point is my current project: building a parser for PDF bank statements from all UK banks. I can match a 10page bank statement against a library of 100 templates, and then extract its data in a standardised format, in around 50ms per statement. I'm interested in management of domestic documents and similar institutional correspondence, and I'd love to hear more about how you approached and solved your problems with the PDF bank statements. I have lots of bank statements ... and other documents like utility bills, rates invoices, etc ... and I'd love to just be able to feed them to a script and automatically tease out salient details.
cosmicradiance 4y ago
When building a large system, the choice of language is always non-binary. You do not have to commit to one and only one language for the entire system. Python offers a great stating point for most cases. When you scale, certain modules will start becoming the bottlenecks. At this point one can just re-write the given module as a sidecar and Python by design is easy to bind external libraries.

elabajaba 4y ago

My problem with Python, especially when reading someone else's code, or my own old bad code, is that I find that Python doesn't fit [my] brain. The lack of typing (in my experience you always end up with a bunch of libraries that don't have type hints) means I spend most of my time just figuring out what I'm supposed to pass to a function (I hate the django docs. Give me something like docs.rs where I can quickly scan what functions something implements, and see what they accept and what they return instead of forcing me to read an 800 page novel that STILL doesn't give you that basic info), and writing tests and trying to guard against passing the wrong thing around.

stevesimmons 4y ago
In my own code, I always use typehints and write good docstrings (nearly) everywhere. Visual Studio Code's type checker saves so much time and has improved my code quality to no end. It is especially powerful on polymorphic inputs, making sure the different code paths operate on the input type you expect.
- shanebellone 4y ago
  PyCharm is good for this too. I've recently started using Type Hints by default. At the very least, it makes your code more readable. Nothing bad comes from that.
  - type-r 4y ago
    I started enforcing type hinting in my personal projects. 99% of the time it's easy and useful to add. When I can't figure out how to make a mypy error go away within 30 seconds, I just disable it for that line. I feel this makes for very good 80-20 value/effort tradeoff. I will say that more than one bug has been found after I disabled the single line hint though! Always surprising to me how powerful types can be.
- mharig 4y ago
  I do not see the benefit of type hints. Good docstrings (and naming of the function and arguments) are superior, at least for the human. Type hints are too much clutter.
  - baq 4y ago
    There’s a certain program size from which they start to make sense and further still the benefits are obvious - you have a computer to help solve puzzles for you, so why not use it if the puzzle is ‘will this peg fit into that hole?’ when you have thousands of pegs and holes.
    - mharig 4y ago
      And so you end up writing climb_the_wooden_ladder(l: WoodenLadder) plus climb_the_aluminium_ladder(l: AluminiumLadder) or press all into an OO hierarchy instead of climb(thing): '''Climb a thing with steps''' # prevent run time crash: if not hasattr(thing, 'getSteps'): hitDeveloper()
      - baq 4y ago
        No, you define a Ladder (or Climbable if you’re feeling fancy today) Protocol and all remains good in the world.
      - cfu28 4y ago
        No need for OO hierarchy, just use a protocol
        mharig 4y ago
        Oh, I missed the Protocol class of the typing module. Have to further investigate the usage of this. Thanks to you and baq!
  - kbelder 4y ago
    I have a fear that Python will be improved to the point it offers no more advantage over other languages.
dehrmann 4y ago
> The lack of typing It's gotten a lot better with type hints and typeshed, but I agree, not knowing what type an arg should be and hoping duck typing works out isn't great.

Ultimatt 4y ago

50ms doesn't sound especially fast. If you had a million of them to process thats 4 hours, maybe an hour with some parallelism? A 100x speedup from going native then becomes quite welcome. Python has its place, but the fact you even think the tens of ms domain is "fast" or even fast enough on modern CPUs shows the real strength of Python, which is most people dont actually care at all about performance. Thats not to say its performant though. Just that no one cares anymore. Some rando boss is happy to wait 4 hours for your script to chug through a million statements, because no one actually told them it can take 2.5 minutes in a native language. If they did maybe the boss would suddenly speak about their dream to not only process the data but have a near real-time BI panel instead of a batch report so they can react within one business day. The issue with Python is missed value, not the value it can deliver.

squeaky-clean 4y ago
Why limit parallelism to 4x? Spin up a ton of lambdas and get it done in 10 seconds. You're also forgetting that 50ms time to load each file from cloud storage. It's still 4 hours and 2.5 minutes with your native code compared to 8 hours with Python. Suddenly not such a massive improvement. Even then I don't see the issue with it taking 4 hours to do a million of them. Do you need to do a million per hour? Is it even likely they'll need to do a million of them total? Do bank statements even come in frequently enough to do a realtime dashboard with? I get my bank statement every 30 days. How much longer does it take to develop the native version? How much longer does it take to modify when a bug is found or a bank changes their statement layout? How much more do you have to pay a native code engineer compared to a Python dev and how easy are they going to be to replace eventually?
- blindhippo 4y ago
  One dimension to consider here is cost of compute. Going from 8 hours to 4 hours is a 50% reduction in the time to compute and we're assuming this is occurring on the same relative hardware/instance size. At scale, that could translate into hundreds of thousands of dollars in savings. But your points are relevant - as with anything related to development, "it depends" rules the day. There isn't a clear cut "x is objectively better than y" in general.

shanebellone 4y ago

I have Python analytics capturing HTTP Headers, processing them, and finally storing them in less than 8ms on a single core dev instance. Python is plenty fast for many things. Also, I love the syntax. It reads like a sentence.

Someone 4y ago
> in less than 8ms on a single core dev instance. Python is plenty fast for many things. What’s that, nowadays? 25 million cycles, 75 million instructions? Python isn’t plenty fast, modern hardware is.
- pdonis 4y ago
  More precisely, modern CPUs are so much faster than network or file I/O that if your program has to wait on I/O, the CPU is going to sit idle most of the time anyway, so trying to optimize CPU time is pointless. The GP's use case is an example: he's capturing HTTP headers, so the bottleneck is network I/O, not CPU. So Python for his use case is, as he says, plenty fast. For use cases where CPU is the bottleneck, such as numeric simulations, Python users reach for something like numpy, where all the CPU intensive stuff is being done in C extensions, not Python bytecode.
- KyeRussell 4y ago
  The person you’re replying to is saying that Python meets their needs. Judging by what they’ve said about their use case, I doubt it’s a meaningful affirm to their overall pipeline. Your counterpoint is that it’d be slower if it were running on a Pentium 4. Do you honestly think that they don’t know this? Yes, this is precisely the point. Things are at a point where Python’s benefits are worth more than what another language would bring in speed improvements. I imagine that there would be less of a benefit to busses if we had to Flintstone them everywhere instead using a combustion engine. That’s not the current reality though.
iLoveOncall 4y ago
That sounds insanely slow. I think it's fine to like Python as a scripting language, but arguing that Python is fast is simply laughable. I forsee Python's reputation going the way of PHP's (< 5, not modern), and statements like this will definitely contribute to it.

lowbloodsugar 4y ago

Plenty of languages have a huge package ecosystem. Your problem is that you don’t know them. I’m not saying the distinction is helpful for your situation, but your problem is “I have too much time invested in learning Pythons ecosystem” and not “other languages don’t have all the packages I need.” For REPL that’s another mindset. You use a REPL to experiment and prototype, I use unit tests. You use python because you already learned it. Your brain has been fit to python, not the other way around. You can fit it to other languages if you choose. I am currently learning my 7th “use in production” language.

stevesimmons 4y ago
You're partly right. However Python's sweet spot for me is when writing code to process complex, messy, poorly understood data. Using a Jupyter notebook lets me keep partially processed inputs as live objects, examine and change their in-memory representations, and quickly prototype the next stage of processing. It feels like sculpting with playdoh. Once that's working, the code moves from the Jupyter notebook into a proper code module with unit tests. Then the emphasis is on building the production system rather than manipulating the data. I get what you mean with unit tests. Just for me it feels further away from the data, which is where most the complexity in my business domain lies.
- lowbloodsugar 4y ago
  Sure. I do use Jupyter in a decent IDE for some adhoc data diving or for a spike prototype before “downcoding” into a cluster.
deleted 4y ago
[deleted]

roflyear 4y ago

How are you matching a file to a template like that?

deleted 4y ago

[deleted]

MrVandemar 4y ago

>Case in point is my current project: building a parser for PDF bank statements from all UK banks. I can match a 10page bank statement against a library of 100 templates, and then extract its data in a standardised format, in around 50ms per statement. I'm interested in management of domestic documents and similar institutional correspondence, and I'd love to hear more about how you approached and solved your problems with the PDF bank statements. I have lots of bank statements ... and other documents like utility bills, rates invoices, etc ... and I'd love to just be able to feed them to a script and automatically tease out salient details.

cosmicradiance 4y ago

When building a large system, the choice of language is always non-binary. You do not have to commit to one and only one language for the entire system. Python offers a great stating point for most cases. When you scale, certain modules will start becoming the bottlenecks. At this point one can just re-write the given module as a sidecar and Python by design is easy to bind external libraries.