4 ms·
I remember writing a hash join in awk at my first job because I got tired of waiting hours for python to finish. Was probably two orders of magnitude faster in
by marcinzm 6y ago
I remember writing a hash join in awk at my first job because I got tired of waiting hours for python to finish. Was probably two orders of magnitude faster in awk.
- acdha 6y agoHow did that work? I'd expect some difference but orders of magnitude sounds like there was a substantial difference in the amount of work being performed.
- marcinzm 6y agoPython is fairly slow and I'm pretty sure it was even slower back then. Parsing tens of millions of lines of text, splitting them, converting to numbers, looking things up in a giant dictionary, etc. adds up. I remember inlining functions by hand to increase performance in Python back then. AWK is optimized for parsing text and has the performance of a compiled language.
- hedora 6y agoBased on my experience with python, the python used global variables (which is idiomatic for scripts), making it about 10x slower than normal python. The other 10x? That also matches my experience with other people’s python. Someone probably tried to implement an abstraction, or something foolish like that.
- saila 6y agoI'm curious what you mean when you say global variables are idiomatic in Python scripts. I work on Python scripts on a daily basis and very rarely use globals except for the occasional "const" value. If globals ever do cause a noticeable slowdown due to dynamic lookup, this can easily be fixed by pointing a local var at the global at the start of a function.
- hedora 6y agoGlobals are idiomatic in all scripting languages except python. You don’t define a function called main, then manually invoke it at the bottom of your bash, perl, etc... script. In python, you do, partially because there is a severe performance penalty for not doing that.
- saila 6y agoAh, I see. I read "which is idiomatic for scripts" as applying to Python scripts and not scripts generally.
- RHSeeger 6y agoDefining a main function and then calling it from the bottom of your script is a mainstay of many people who program in scripting languages. It's certainly how I did things in Tcl. Really, it depends on the type of thing you're writing in the language, moreso than the language itself.