13 ms·
Show HN: Turn any website into a knowledge base for LLMs
I built this tool because I wanted a way to just take a bunch of URLs or domains, and query their content in RAG applications.
It takes away the pain of crawling, extracting content, chunking, vectorizing, and updating periodically.
I'm curious to see if it can be useful to others. I meant to launch this six months ago but life got in the way...
- cranberryturkey 2y agohow does this work?
- tompec 2y agoGive it URLs or domains, and it will crawl and extract their content, embed them in a vector database, and give you an endpoint that you can then query when doing RAG stuff or semantic search.
- xiconfjs 2y agoBut how does it work in the background? What‘s the tech stack?
- ramon156 2y agoIn another comment: > Tech stack is a mix of serverless Laravel, with Cloudflare and AWS functions, and some Pinecone for vector storage. Still experimenting on a few things but don't want to over-engineer unless I know where I'm going.
- kordlessagain 2y agoI do this with https://mitta.ai https://mitta.ai by using a Playwright container that does a callback to a pipeline that uses either meta data from the PDF or sends it to an EasyOCR deployment on a GPU instance on Google for text extraction. Then I use a custom chunker and instructor/xl embeddings. All of that code is Open Source, and works well for most sites. Some sites block Google IPs, but the Playwright container can run locally, so should be able to work around it with some minimal effort.
- sunir 2y agoThere are a few ways. I built something similar huckai.com on top of vectara.com. They have open sourced their versions https://github.com/vectara https://github.com/vectara You can also do this on AWS now fairly easily. https://medium.com/data-reply-it-datatech/how-to-build-a-custom-rag-chatbot-in-aws-0d9250d5f8bd https://medium.com/data-reply-it-datatech/how-to-build-a-cus... The lablab.ai Discord community is a pretty good place to learn how this product category is evolving.
- deleted 2y ago[deleted]
- samuria 2y agoInteresting, I wanted to do this for a personal use case (mostly learning), but with PDFs. What's tech stack? I have explored using the AWS AI tools, but it seems a bit overkill for what I want it to do.
- tompec 2y agoTech stack is a mix of serverless Laravel, with Cloudflare and AWS functions, and some Pinecone for vector storage. Still experimenting on a few things but don't want to over-engineer unless I know where I'm going.
- stevenicr 2y agoGiven that cloudflare spies on traffic and reports to multiple agencies on it's findings, perhaps a breakdown of the chain and the privacy implications of each block in the stack would be beneficial?
- stevenicr 2y agoYa know, a downvote on this pre-aug 2019 would be fine. people still being ignorant about their publicly posted policies 5 years later is annoying.
- lou1306 2y agoIf the PDFS are textual or have OCR, then pdf2text from the Poppler suite ought to be enough? If not, add Tesseract/ocrmypdf to the pipeline?
- kordlessagain 2y agoHere's some code to deal with that: https://github.com/MittaAI/SlothAI/blob/main/SlothAI/lib/processor.py#L2069 https://github.com/MittaAI/SlothAI/blob/main/SlothAI/lib/pro... https://github.com/MittaAI/mitta-community/tree/main/services/gpu/ocr https://github.com/MittaAI/mitta-community/tree/main/service... There's code in there that just reads PDF meta data as well, but you can't always guarantee it's there in a PDF.
- 2y ago
- blackeyeblitzar 2y agoIs there a way to deal with websites where you need to login? Like subscription based sites?
- tompec 2y agoUnless you own those sites, I'm afraid that's not going to be possible.
- onetokeoverthe 2y ago[dead]
- brianfryer 2y agoBut if I do own those sites, there’s a way?
- lopatamd 2y ago[dead]
- ashwinnair99 2y agoHow are you deciding on the best RAG configuration for your app? How you decide chunking strategy, embedding and retrievers for your app? Check out our open source tool-RAgBuilder that allows developers get to the top performing RAG for their data https://news.ycombinator.com/item?id=41145093 https://news.ycombinator.com/item?id=41145093
- toomuchtodo 2y agoCould you support ingesting WARC files? https://github.com/harvard-lil/warc-gpt https://github.com/harvard-lil/warc-gpt https://lil.law.harvard.edu/blog/2024/02/12/warc-gpt-an-open-source-tool-for-exploring-web-archives-with-ai/ https://lil.law.harvard.edu/blog/2024/02/12/warc-gpt-an-open...
- deleted 2y ago[deleted]
- ancras 2y agoThis is interesting. Can it work with any website, even say document repositories hosted on standard servers like gitbook?
- tompec 2y agoIt works with pretty much any website, and works well with docs hosted on GitBook yes, I have embedded a website that's hosted there.
- dazbradbury 2y agoNice! What's the underlying model / RAG approach being used? Be good to understand that part as presumably it will have a big impact on performance / usability of the results.
- r0b05 2y agoDoes it hallucinate much?
- michaelmior 2y agoThis looks interesting, but I get a 404 on the iframe when I try to go into the chat.
- deleted 2y ago[deleted]
- tompec 2y agoSorry about that, a bit too much load at the moment
- 23B1 2y agoI tried it out. This would be extremely useful to me to the point I'd be willing to happily pay for it, as it's something I would have otherwise had to spend a long time hacking together. 1) The returned output from a query seems pretty limited in length and breadth. 2) No apparent way to adjust my prompts to improve/adjust the output e.g. not really 'conversational' (not sure if that is your intent) Otherwise keep developing and be sure to push update notifications to your new mailing list! ;-)
- dmje 2y agoAgree with this. I also think the emphasis here (to OP) should be "I'd be willing to happily pay for it" - ie I'd rather be paying a reasonable amount each month for something that is going to remain active that have the large (current) disparity between "free" and "enterprise". I'd say make some middle tiers of (I don't know?) $5 / $10 / $20 a month for reasonable numbers of queries or whatever. Keep the "enterprise" offering there for the biggies, but offer us small players some hope that this will be sufficiently funded / supported. Brilliant idea, btw, I like it :-)
- tompec 2y agoThanks! I'm still figuring things out about pricing, but there will be small plans available.
- tompec 2y agoThanks! The chat demo is actually just a small thing I put together as a preview of what can be done, but the main product is the API. But seeing that most users seem to like that, there's probably something there... If you want to email me at support at embedding.io with some requirements, I can see how to make that work for you.
- pryelluw 2y agoDoes this respect robots.txt?
- deleted 2y ago[deleted]
- srameshc 2y agoValid question and I am sure it doesn't.
- danirod 2y agoI hope this gets answered. Also I've checked their docs to see if there is any mention about the user agents or IP ranges they use for scraping, with no luck.
- tompec 2y agoIt does respect robots.txt when crawling. I'll add more details about this in the docs.
- pryelluw 2y agoI appreciate the reply. As someone who runs multiple CMSs it’s painful to deal with the ai crawlers these days. Specially the ones that don’t respect my terms.
- mkw5053 2y agoI made a similar open source app a year ago or so https://github.com/mkwatson/chat_any_site https://github.com/mkwatson/chat_any_site
- deleted 2y ago[deleted]
- crowcroft 2y agoI like this. Abstracting away the management of embeddings and vector database is something I desperately want, and adding in website crawling is useful as well.
- rcarmo 2y agoHow do I feed it a sitemap?
- tompec 2y agoIt currently will try to find a sitemap on its own. But I have on the roadmap to let users add their own.
- muggermuch 2y agoI like this a lot! But: I feel the more of these services come to being, the more likely it is that every website starts putting up gates to keep the bots away. Sort of like a weird GenAI take on Cixin Liu's Dark Forest hypothesis (https://en.wikipedia.org/wiki/Dark_forest_hypothesis https://en.wikipedia.org/wiki/Dark_forest_hypothesis). (Edited to add a reference.)
- marcellus23 2y agoResponding just because it's a pet peeve of mine: Cixin Liu did not invent the dark forest hypothesis. People were discussing it, and writing science fiction books about it, for decades before the 3BP books were published. Nothing against him, and he definitely helped popularize the concept, but I think it's incorrect to refer to it as "Cixin Liu's hypothesis".
- dankwizard 2y agoBut he is responsible for the name, not the concept. So yes it is Cixin Liu's Dark Forest hypothesis.
- diggan 2y agoJust like Amerigo Vespucci put the name "America" on a map and people starting referring to the New World as such, although he didn't discover it himself.
- sooheon 2y agoWas curious as a lover of the 3BP series, google gave me this: "We've been sitting in our tree chirping like foolish birds for over a century now, wondering why no other birds answered. The galactic skies are full of hawks, that's why." (The Forge of God, Legend edition, 1989, pg 315). Yeah, same concept and even the same imagery. Source: https://warwick.ac.uk/fac/sci/physics/research/astro/people/stanway/sciencefiction/cosmicstories/the_dark_forest https://warwick.ac.uk/fac/sci/physics/research/astro/people/...
- khanan 2y agoCan this be deployed on-prem or is it an cloud-toy?
- tompec 2y agoCurrently just a cloud-toy.
- danirogerc 2y agoCan I query multiple vectorized websites at once? Can I export vectorized websites and host them myself? Any chance to export them to a no-code format, like PDF?
- tompec 2y agoYou can group as many websites as you want into a collection. Then query that collection. Not sure what you mean by exporting; you would like to export the vectors themselves? Or just the chunks of text from the websites?
- nhggfu 2y ago[flagged]
- Cynddl 2y agoI find it interesting that as an (edit: UK) academic researcher, I would be likely be forbidden to use tools like this, that fail basic ethics standards, regulations such as GDPR, and practical standards such as respecting robots.txt [given there's no information on embedding.io, it's unlikely I can block the crawler when designing a website]. There's still room for an ethical development of such crawlers and technologies, but it needs to be consent-first, with strong ethical and legal standards. The crazy development of such tools has been a massive issue for a number of small online organisations that struggle with poorly implemented or maintained bots (as discussed for OpenStreetMap or Read The Docs).
- popcorncowboy 2y agoI'm less convinced. Are you saying it's unethical to automate browsing a site? Because if you save the pages you browse on some site, they're yours (authors don't own your cache). Perhaps you're arguing that if you wrote a lightweight script/browser (which is just your user agent) to save some website for offline use, that'd be unethical and GDPR violating? Again, I don't think so but maybe I'm missing something. But perhaps this turns on what defines a "user agent". Perhaps this becomes a "depth of pre-fetch" question. If your browser prefetches linked pages, that's "automated" downloading, akin to the script approach above. Downloading. To your cache. Which you own. (Where I struggle to see an ethical violation) Genuinely curious where the line is, or what exactly here is triggering ethics, GDPR and practical standards?
- Cynddl 2y agoMaybe a good illustration would be ClearView AI. They are scraping websites, extracting information (images), and training ML models to learn embeddings (distance between faces). They indiscriminately collect personal data without opt-in, but a limited opt-out mechanism. In this case, if this tool is used to scrape a website, there are too direct issues: 1/ no immediate way for the website owner to exclude this particular scraper (what is the useragent?) 2/ no way for data subjects (whose data is present on the website) to search whether the scraper learned their personal data in the embeddings. Data being available publicly doesn't mean it can be widely used [at least outside the US, where we have much stricter rules on scraping].
- rvz 2y ago> Enterprise: Contact Us If there is no certifications or compliance information then I don't think there is anything to discuss about any enterprise plan.
- tompec 2y agoGotta start somewhere :)
- kaycebasques 2y agoI spent a lot of time thinking about how to manage embeddings for docs sites. This is basically the same solution that I landed on but never got around to shipping as a general-purpose product. A key question that the docs should answer (and perhaps the "How it works" page too): chunking. You generate an embedding for the entire page? Or do you generate embeddings for sections? And what's the size limit per page? Some of our docs pages have thousands of words per page. I'm doubtful you can ingest all that, let alone whether the embedding would be that useful in practice.
- tompec 2y agoI chunk pages and generate embeddings for each chunk. So there's no real size limit per page.
- kaycebasques 2y agoThe more detail, the better. If `<section>` elements are found you chunk those? Do you do it recursively or do you stop after a certain level? And when section elements don't exist, you use `<h1>`, `<h2>`, etc. to infer logical chunks?
- tompec 2y agoHaving looked at a lot of HTMLs, I noticed that sections are not really the default. I rely on headings (h1, h2, ...) to chunk each pages. Each chunk has its heading hierarchy attached to it. There are a lot of optimizations that could be done at that level.
- chasd00 2y agoi'm just guessing but i would think following whatever semantics leads to the highest search rank in google's algorithm would be what you're most likely to find out in the wild.
- olalonde 2y agoWhich LLM model is it using?
- deleted 2y ago[deleted]
- MattDaEskimo 2y agoIn my opinion this is a transitional niche. Soon websites/apps whatever you want to call them will have their own built-in handling for AI. It's inefficient and rude to be scraping pages for content. Especially for profit.
- muratsu 2y agoI doubt. For larger players, data is valuable - so they are preventing scraping already (eg reddit, linkedin). For smaller websites there’s also not much of an incentive.. Maybe hosting providers will help with preventing scraping? like ddos protection
- obilgic 2y agoI think each website/data-source having their own built-in AI is also a transitional period. It is like every website having search engine vs google.
- sooheon 2y agoWe already have the google analogue (llm that's seen all the websites), so are we going in circles?
- temporary_name 2y agoIn agree that this niche is DOA. No offence to OP but the barrier for entry to this stuff is low. I built basically the same thing over a weekend for personal use. React frontend, python server, chroma for embeddings, sqlite cache, switch between open AI and anthropic (I want to add llama for full local execution when I get a better pc). I have a local SPA with named "projects", can configure crawl depth from a start page, I can set my crawl rate, don't have to pay to use it, can choose any provider I want... I'm just one guy and that took a day to get working plus a bit of polish. I would guess the hardest thing by far in developing the advertised product would be user management, authentication, payments and wrapping the subscription model's business logic around the core loop. And probably scaling, as running embeddings over hundreds of scraped pages adds up quickly when free tier users start hammering you. My question when deciding to sell something I've built is, if building the service model is harder than building the actual service, where is the value add? My take on the natural evolution is that collating and caching documents, websites etc for search (with source attribution ideally) is a problem that will I think ultimately be solved by OS vendors. Why sign up for SaaS and expose all your content to untrustworthy 3rd parties, when it's built right in and handled by your "trusty" OS. In the meantime, I reckon someone more dedicated than me will (or probably already has) open source something like I built but better, probably as a CLI tool, which will eventually reach maturity and be stolen cough I mean adopted by the top end of town. Ethically I think nothing's changed for centuries in regards to plagiarism and attribution. It gets easier to copy work and thinking, but it also ultimately gets easier to acknowledge sources. Good folk will do the right thing as they always have done. Regarding efficiency, I think tools like this have a place in making access to relevant and summarised knowledge during general research more efficient, when doing the broad strokes to find areas of interest to zoom in on, when more traditional approaches take over. Interesting times anyway. I have to give credit to people that try, but I'm taking a back seat in thinking of ideas to productise in this space, as by the time I've thought it through, something new comes along that instantly makes it obsolete.
- breck 2y ago#1. Gratuitous self promotion (but also my honest best advice): The future of knowledge bases is ScrollSets: https://sets.scroll.pub/ https://sets.scroll.pub/ #2. If you are interested in knowledge bases, see #1
- jakubsuchy 2y agoI feel like this is unethical. You built yet another bot scraper. It would only be an ethical tool if it validated I own the website I am scraping before it starts.
- corn13read2 2y agoyes only big conglomerates now can scrape pages. If you're not google stealing the info then...... Right?
- jakubsuchy 2y agoI didn't say that but a site owner should have the right to decide. In addition, this scraper doesn't even identify itself (I checked). It pretends to be a normal browser, without saying it's a scraper.
- talldatethrow 2y agoI think a website owner can decide. They can take the site down, or they can put it behind some kind of user wall.
- visarga 2y agoThis is probably a losing direction - protecting your little island of content in the sea of internet and LLM outputs. Get more value by exposure. This is the trend of open source, wikipedia and open scientific publication. LLMs double down on the same collaborative approach to intelligence. You can of course decouple from the big discussion and isolate your content with access restrictions, but the real interesting activity will be outside. Look for example the llama.cpp and other open source AI tools we have gotten recently. So much energy and enthusiasm, so much collaboration. Closed stuff doesn't get that level of energy. I think IP laws are in for a reckoning, protecting creativity by restricting it is not the best idea in the world. There are better models. Copyright is anachronic, it was invented in the era of the printing press when copying was made easy, LLMs remix they don't simply copy, even the name is unfitting for the new reality. We need to rename it remixright.
- boredemployee 2y agodoes it embbed images as well? if not, do you plan to do so?
- tompec 2y agoIt doesn't embed images, no. But that's a great idea for the roadmap!
- boredemployee 2y agogreat. I really want a feature like that! I'd like to query my knowledge base about images as well!
- mattfrommars 2y agoSo i provide a URL, your service does the crawling of the site?
- deleted 2y ago[deleted]
- vladde 2y agoThe example API key on the page is decoded to "WOW YOU'RE A HACKER" :)
- soundblaster 2y agothat == at the end looked like a base64 encoded string ;)
- oars 2y agoInterested in seeing whether this will be widespread in 5 years or whether sites will have fought back.
- dartos 2y agoSites are already fighting back. Twitter and Reddit locked down their APIs. Soon enough, you’ll need an account to even access any content
- replete 2y agoDoes anyone know of a way to do this locally with Ollama? The 'chat with documentation' thing is something I was thinking of a week ago when dealing with hallucinating cloud AI. I think it'd be worth the energy to embed a set of documentation locally to help with development
- dartos 2y agohttps://github.com/open-webui/open-webui https://github.com/open-webui/open-webui
- navbaker 2y agoYes, Langchain has tooling specifically for connecting to Ollama that can be chained with other tooling in their library to pull in your documents, chunk them, and store them for RAG. See here for a good example notebook: https://github.com/langchain-ai/langchain/blob/master/cookbook/Semi_structured_multi_modal_RAG_LLaMA2.ipynb https://github.com/langchain-ai/langchain/blob/master/cookbo...
- jeanloolz 2y agoI built a similar thing as a python library that does just that: https://github.com/philippe2803/contentmap https://github.com/philippe2803/contentmap Blog post that explains the rationale behind the library: https://philippeoger.com/pages/can-we-rag-the-whole-web https://philippeoger.com/pages/can-we-rag-the-whole-web Just submit your XML sitemap into a python class, and it will do the crawling, chunking, vectorizing and storage in an SQLite file for you. It's using SQLiteVSS integration with Langchain, but thinking of moving away from it, and do an integration with the new sqlite-vec instead.
- samstave 2y agoThis is part of a dream of a tool I would like: A relational crawler on a particular subject with nuanced, opaque, seemingly-temporally-unrelated connections that show a particular MIC conduction of acts:: "Follow all the congress members who have been a part of a particular committee, track their signatory/support for particular ACTs that have been passed, and look at their investment history from open data, quiver, etc - and show language in any public speaking talking about conflicts and arms deals occurring whereby their support of the funding for said conflicts are traceable to their ACTs, committee seat, speaking engagements, investment profit and reporting as compared to their stated net worth over each year as compared to the stated gains stated by their filings for investment. Apply this pattern to all congress, and their public-profile orbit of folks, without violating their otherwise private-related actions." And give it a series of URLs with known content for which these nuances may be gleaned. Or have a trainer bot that will constantly only consume this context from the open internet over time such that you can just have a graph over time for the data... PYTHON: Run it all through txtai / your library ? nodes and ask questions of the data in real time? (And it reminds me of the work of this fine person/it:: https://mlops.systems/#category=isafpr https://mlops.systems/#category=isafpr https://mlops.systems/#category=afghanistan https://mlops.systems/#category=afghanistan
- xrd 2y agoI know sqlite-vss has been upgraded lately. But, it was unstable for a while prior. Are you having good experiences with it?
- alok-g 2y agoWould be great to use for developer documentation for various languages, frameworks and libraries.
- have_faith 2y agoLooks cool! anything about how it compares to similar RAG-as-a-service products? something I've been researching a little. FWIW, the pricing model of jumping from free to "contact us" is slightly ominous.
- crngefest 2y agoWith these early stage startups it often means they haven’t really figured out how to price their product and will cut you a very generous deal if you push a bit
- notatoad 2y agobut what does "a generous deal" even mean. they have no pricing at all listed, so there's no reference point. if i want to index ten pages more than the free tier is it going to cost me $10/mo or $10000/mo
- barrenko 2y agoWill this work for forums?
- hluska 2y agoI like the concept, the documentation is very good and I even enjoy the domain name. This is an excellent launch and congratulations on getting it out.
- ckluis 2y agoHow much does it cost?
- is_true 2y agoDo you plan on doing revenue sharing with the site owners?
- vulture916 2y agoExperiencing many Internal Server Errors.
- suyash 2y agoCan it get content that is gated/behind login ?
- onemoresoop 2y agoHow would expect it to do that??
- suyash 2y agothere are ways to do that, just wondering if this tool has the capability or not.
- suyash 2y agoany open source tools for doing just this?
- nashashmi 2y agoWould you share the source? I want to use this for a private internal network of pages. How would that work?
- dmitrygr 2y ago> Turn any website into a knowledge base for LLMs I would pay for the opposite product: make your website completely unusable/unreadable by LLMs while readable by real humans, with low false positive rates.
- lua-steve 2y agoHow does this handle changes to the website? Does it re-crawl the whole site periodically and regenerate the embeddings? Or is there some sort of diff-checker that only picks up pages that have changed, added, or deleted?