3 ms·
For LLM scrapers, it doesn't even matter if LLMs would be able to understand the raw text or not because it's extremely easy to just strip junk unicode characte
by logicprog 11mo ago
For LLM scrapers, it doesn't even matter if LLMs would be able to understand the raw text or not because it's extremely easy to just strip junk unicode characters. It's literally a single regex, and, like, that kind of sanitization regex is something they should already be using, and that I'd use by default if I were writing one.
- layer8 11mo agoThere are no “junk” Unicode characters. There are just nonsensical combinations of characters. Stripping out characters blindly is not a solution, because you have no way of knowing what was intended.