3 ms·
I'm not knowledgeable on Python and Redis, but very likely you're doing something wrong with either of them. Parsing fixed-length fields should take no time at
by nandemo 14y ago
I'm not knowledgeable on Python and Redis, but very likely you're doing something wrong with either of them.
Parsing fixed-length fields should take no time at all, so the bulk of the time should be taken by reading the file and inserting the parsed data into the data store. Reading and writing a 150MB file from disk should take only a couple of seconds. So before bothering with another language, you should try to pinpoint where the problem is. Python has a profiler, use it. Also, you appear to be using multiple threads. That probably won't help you much since your task is IO-bound, so try making it single-threaded.
If you cannot find the problem then I suggest asking for advice on StackOverflow (or CodeReview).
- bsg75 14y agoRemoving the single call to update Redis takes execution time from 3 minutes to ~5 seconds. Profiling the execution shows the most of the time is taken up inside the Redis driver, but I see similar times if I use the MongoDB Python driver. The drivers are open-source, so I could invest some time in looking for bottlenecks there, but I believe the authors of those drivers are better Python developers than I. Interest in a different language comes with some thought that drivers written in faster languages will perform faster.