3 ms·
Show HN: Simple CLI to aggregate repos, papers and docs for LLM ingestion
I created a tool that consolidates information from the following inputs:
GitHub repository URL (e.g., https://github.com/jimmc414/onefilellm https://github.com/jimmc414/onefilellm)
arXiv abstract URL (e.g., https://arxiv.org/abs/2401.14295 https://arxiv.org/abs/2401.14295)
Local folder path (e.g., C:\python\PipMyRide)
Youtube video URL (e.g., https://www.youtube.com/watch?v=KZ_NlnmPQYk https://www.youtube.com/watch?v=KZ_NlnmPQYk)
Webpage URL (e.g., https://llm.datasette.io/en/stable/ https://llm.datasette.io/en/stable/)
It outputs the repo, web documentation, arXiv paper or YT transcript to a text file and the clipboard, displaying a token count. It also produces an additional file of the same content that reduces token usage by removing stopwords and converting to lowercase.
The purpose of this tool is to make it easier to manually pass repos, papers, transcripts or web documentation into an LLM for inference.
For accessing Github repos you just need to generate a github personal token as described in the readme.
I needed this tool for myself and thought it might be of use to someone else.