3 ms·
WARC-GPT: An open-source tool for exploring web archives using AI
- Cedricgc 2y agoLooking into this, could be very useful for working with personal Zotero archives of webpages
- deleted 2y ago[deleted]
- jszymborski 2y agoAs previously seen on HN: https://github.com/Frost-group/The-Oracle-of-Zotero https://github.com/Frost-group/The-Oracle-of-Zotero
- westurner 2y agopaper-qa: https://github.com/whitead/paper-qa https://github.com/whitead/paper-qa : > LLM Chain querying documents with citations [e.g. a scientific Zotero library] > This is a minimal package for doing question and answering from PDFs or text files (which can be raw HTML). It strives to give very good answers, with no hallucinations, by grounding responses with in-text citations. pip install paper-qa > If you use Zotero to organize your personal bibliography, you can use the paperqa.contrib.ZoteroDB to query papers from your library, which relies on pyzotero. Install pyzotero to use this feature: pip install pyzotero > If you want to use [ paperqa with pyzotero ] in an jupyter notebook or colab, you need to run the following command: import nest_asyncio nest_asyncio.apply() ... there is also neuml/paper-ai w/ paperetl; https://news.ycombinator.com/item?id=39363115 https://news.ycombinator.com/item?id=39363115 : gh topic: pdfgpt, chatpdf, neuml/paperai has a YAML report definition schema: https://github.com/neuml/paperai https://github.com/neuml/paperai : > Semantic search and workflows for medical/scientific paper python -m paperai.report report.yml 50 md <path to model directory> > The following [columns and answers] report [output] formats are supported: [Markdown, CSV], Annotation - Columns and answers are extracted from articles with the results annotated over the original PDF files. Requires passing in a path with the original PDF files.
- szvsw 2y agoThere’s also ARIA [1]. I actually think the one linked above seems more interesting. At the same time, doesn’t seem all that different from just rolling your own solution. [1] https://github.com/lifan0127/ai-research-assistant https://github.com/lifan0127/ai-research-assistant
- ttpphd 2y agoIf you just want semantic search, check out Semantra: https://github.com/freedmand/semantra https://github.com/freedmand/semantra
- deleted 2y ago[deleted]
- monkeydust 2y agoNice, been looking for something like this. In case you - like me - are wondering on the embedding models options: https://github.com/freedmand/semantra/blob/main/docs/guide_models.md https://github.com/freedmand/semantra/blob/main/docs/guide_m...
- brunoqc 2y agoAny nice tool to archive web sites using the warc format?
- aussieguy1234 2y agoThe Wikipedia article on the warc format references wget
- ricopags 2y agoMay be some here https://github.com/iipc/awesome-web-archiving/blob/main/README.md#warc-io-libraries https://github.com/iipc/awesome-web-archiving/blob/main/READ...
- toomuchtodo 2y agohttps://github.com/ArchiveTeam/grab-site https://github.com/ArchiveTeam/grab-site https://wiki.archiveteam.org/index.php/The_WARC_Ecosystem https://wiki.archiveteam.org/index.php/The_WARC_Ecosystem
- andrewmackrodt 2y agoIf you want a full fidelity warc file, browsertrix-crawler is nice. It is slower than wget on account of using chrome, but it works better for sites with highly dynamic content and can generate wacz files which can be efficiently served via a file/objectstore when used with something like replayweb.page.
- nikisweeting 2y agoWebrecorder's are by far the best imo. https://archiveweb.page https://archiveweb.page and Browsertrix Crawler. ArchiveBox uses wget internally for it's WARC generation but I'd love to integrate it with Webrecorder in the future for this part.
- scottmcdot 2y agoIs this something that could be used to create a chatbot with a knowledge base based on all pages and all snapshots of those pages of a specific domain archived by web.archive.org/?
- yoouareperfect 2y agoI first read it as "War-GPT" and was very relieved to find it was otherwise.
- toomuchtodo 2y agoWOPR-GPT
- throw46365 2y ago[flagged]
- koolala 2y agoweb 4.0?
- 0xdeadbeefbabe 2y ago> it is not meant to replace keyword and filter-based search, but instead to complement it, providing a different starting point for scholarly inquiry and potentially expanding access to otherwise hard-to-reach information.