4 ms·
This is an old example, but - date/time parsing. A coworker of mine years ago was trying to parse out some large logfiles and it was running incredibly slowly
by danudey 2y ago
This is an old example, but - date/time parsing.
A coworker of mine years ago was trying to parse out some large logfiles and it was running incredibly slowly (because the log file was huge).
Just for fun he profiled the code and found that 90% of the time was spent taking the timestamp ("2019-04-22 15:24:41") into a Python datetime. It was a slow morning, so we went back and forth trying to come up with new methods of optimizing this parsing, including (among other things) creating a dict to map date strings to datetime objects (since there were a lot of repeats).
After some even more profiling, I found that most of the slowdown happened because most of Python's strptime() implementation is written in Python so that it can handle timezones correctly; this prevented them from just calling out to the C library's strptime() implementation.
Since our timestamps didn't have a timezone specified anyway, I wrote my first ever C module[0] for Python, which simply takes two strings (the format and the timestamp) and runs them through strptime and returns a python datetime of the result.
I lost the actual benchmark data before I had a chance to record it somewhere to reproduce, and the Python 3 version in my repo doesn't have as much of a speedup compared to the default Python code, but the initial code that I wrote provided a 47x performance boost to the parsing compared to the built-in Python strptime().
Anyone who had a similar Python script and converted it wholesale to Rust (or C or Golang, probably) would have seen a similarly massive increase in performance.
[0] https://github.com/danudey/pystrptime/ https://github.com/danudey/pystrptime/
- jcgrillo 2y agoOne could argue that writing a timestamp as a string which then has to be parsed is silly and instead it should be delta-of-delta encoded and packed into variable width integers, but even then double integrating and constructing a datetime for each one would still be expensive in python, only less so.