3 ms·
Yes, I expect full text extraction and normalization to be computationally expensive. I did some simple minded experiments on Usenet feeds back in the 90ies and
by jcynix 3y ago
Yes, I expect full text extraction and normalization to be computationally expensive. I did some simple minded experiments on Usenet feeds back in the 90ies and it was quite some work for the VAX it ran on.
- marginalia_nu 3y agoYou can get it to run fairly fast on modern hardware. Like run a text extraction, tokenization and POS-tagging workflow on a quarter billion documents on PC hardware, takes like 24-36 hours. That's doable and affordable. But ML-adjacent methods are not. Requires far too much GPU compute, have no A100s :-/