3 ms·
Why is this regex so slow? Am I missing something obvious?
I’ve been trying to optimize a script that processes a huge text file, but my regex seems to be the bottleneck. Here’s the snippet:
python
Copy code
import re
pattern = re.compile(r"(?:https?|ftp)://[^\s/$.?#].[^\s]*")
with open("large_file.txt", "r") as f:
matches = [pattern.search(line) for line in f]
It works, but it’s painfully slow on large datasets. I tried using find all instead of search, but it didn’t help much.
- zahlman 2y ago(HN doesn't support full Markdown. Please see https://news.ycombinator.com/formatdoc https://news.ycombinator.com/formatdoc for details.) Can you quantify "painfully slow", "large dataset" etc.? Have you tried running through with a simpler regex as a baseline? What about just reading the file in the first place? Does this "huge text file" have excessively long lines, or just a lot of them?
- Terr_ 2y agoOne thing that stands out is that you are basically duplicating the entire file into memory, which might be a problem if your files are bigger than your free RAM. This is because pattern.search(subject) returns a Match object which contains the original line, and you're keeping all of them in the resulting list. It should be easy to check if that's the issue by monitoring system memory usage, or by testing with a version that does the search but immediately discards the result.