4 ms·
I think I would just start with indexing the latest Common Crawl WET files along with a server and a couple of Exos X20 20TB drives. Wondering out loud if ther
by aantix 5y ago
I think I would just start with indexing the latest Common Crawl WET files along with a server and a couple of Exos X20 20TB drives.
Wondering out loud if there's a crawl that will produce real-time updates via common news outlets..
- supernovae 5y agoCrawling is the easy part. The 20tb drives would be too slow for query though. Query really needs everything in memory to be fast enough.