3 ms·
I'm not that familiar with llms, but surely we are already at the point where web pages can be easily scrapped? Is markdown really an easier format to understan
by bawolff 2y ago
I'm not that familiar with llms, but surely we are already at the point where web pages can be easily scrapped? Is markdown really an easier format to understand than html? If this is actually useful wouldn't .txt be supperior to markdown for this usecase?
Does this solve a problem llms actually have?
Not trying to be negative, i'm honestly curious.
- autoexec 2y agoYeah, I'm not sure what the point of markdown is here either. I would expect that anything that looks remotely like a URL will be collected and scraped no matter what format it's in.
- jph00 2y agoContext windows for LLM inference are limited. You can't just throw everything into it -- it won't all fit, and larger amounts of context are slower and more expensive. So it's important to have a carefully curated set of well-formatted documents to work with.
- idf00 2y agoYes. Converting docs to markdown and using them in claude projects, for example, makes a big difference.
- dada78641 2y agoI hate to be this person, but... it's scraping, not scrapping. You're scraping the information off the page regardless of what its structure is.