4 ms·
When I was working on a link parser in python for a crawler. I had two choices: 1. use some form of regex 2. use libxml and find links 1. was faster than 2.
by polote 6y ago
When I was working on a link parser in python for a crawler. I had two choices:
1. use some form of regex
2. use libxml and find links
1. was faster than 2. by a factor or two
Does a link-only parser would have been faster ? Yeah probably but it is much more complex to do
- benlivengood 6y agoWhat was the average difference in accuracy? It's a fair tradeoff especially for a crawler where it's never guaranteed to reach all documents anyway.
- contravariant 6y agoMaybe I'm confused about the problem you're solving but regexp was faster than just: from lxml import html doc = html.from_string(response.content) doc.xpath('//a/@href') ?
- polote 6y agoYes exactly. I'm not even sure XPath was faster than iter on a tags. But I would need to check on that one