4 ms·
There is a standard set of tooling for indexing archives: CDX files. [1] They index WARC archives and can be used to quickly find records. You can build on top
by uniqueuid 5y ago
There is a standard set of tooling for indexing archives: CDX files. [1]
They index WARC archives and can be used to quickly find records. You can build on top of this (and some systems do) to make a proper search front-end.
But in general, these archives are NOT geared towards full-blown search because it would be pretty expensive to keep the indexes in hot cache. Plus you would need to deal with historical versions of records, which is not normally done in search UX.
[1] https://wiki.archiveteam.org/index.php/The_WARC_Ecosystem#CDX_File_Format https://wiki.archiveteam.org/index.php/The_WARC_Ecosystem#CD...
- camtarn 5y agoAh, is the WARC format the reason it's called 'Warrior'? It seems like a very strange name for an archival program.
- myself248 5y agoArchiveTeam seems very guerrilla in their operations. I always imagined the Warrior as a camo-faced archivist operating under cover of darkness, preserving data even in the most hostile Yahoo-occupied territory.
- prox 5y agoThank you for that information!