2 ms·
I tried to follow well-known formats first, such as WARC and ZIM from Kiwix, so we could benefit from existing tooling support. For my own custom data format,
by tamnd 4mo ago
I tried to follow well-known formats first, such as WARC and ZIM from Kiwix, so we could benefit from existing tooling support.
For my own custom data format, I have a lot of private code that I plan to release soon. It is optimized for compression, fast lookups, and more. I have been working on it for two years.
This is part of a larger, ambitious umbrella project: I am building Google from scratch (all open source), something that anyone can host, including the crawler, indexer, storage, and serving layers. Stay tuned!
- sanqui 4mo agoI'm a fan of compatibility with established formats! Sounds awesome. There is a lot of untapped potential with respect to efficiently archiving and indexing websites. I saw the impressive things Marginalia Search is doing in this area (the blog is great when it gets technical). There is also a lot of very complete archives of websites out there which are not being indexed at all, and I would love to make them available for researchers. In any case, I'm interested in your project!
- Prime_Axiom 4mo agoLooking forward to the next project! I love these kinds of archiving tools.
- threecheese 4mo agoOK, sounds fascinating; following! (your GH)
- tamnd 4mo agoThanks ;)