3 ms·
502 - Bad Gateway Another cool thing about Mistral 7b is it runs offline, and I don't need to download any massive URL list that may or may not be current. It
by cyanregiment 2mo ago
502 - Bad Gateway
Another cool thing about Mistral 7b is it runs offline, and I don't need to download any massive URL list that may or may not be current.
It does the work of finding URLs at runtime, offline (until it verifies the URL exists).
- esseph 2mo agoAnother option: https://www.certkit.io/tools/ct-logs/ https://www.certkit.io/tools/ct-logs/
- cyanregiment 2mo agoLooks like they only give you 100 on a trial account
- esseph 2mo agoSorry, let me be more obvious. I am not pointing to a solution, but a pointer to the solution. Showing what is possible. The solution is pulling/parsing the CT logs yourself.
- cyanregiment 2mo agoIf you’re talking about SSL cert logs, that excludes a ton of web content. And you haven’t shown me that you can even get all those for free anyway
- esseph 2mo ago> If you’re talking about SSL cert logs, that excludes a ton of web content. I guess it depends on what you are looking for? It's far less brute force than trying to randomly generate hostnames and then do lookups, and your solution also misses everything that is just random alphanumeric combinations, of which there are tons. > And you haven’t shown me that you can even get all those for free anyway Free? Maybe.
- cyanregiment 2mo agoYeah, but because it's an LLM it makes very good guesses. It's not so random, it's seen URLs in training data. One of the points of the experiment was to see how much of a shortcut using an LLM could give me, as the developer. I've done other LLM search experiments where I basically just prompt to "predict 25 search results for this query" and it's surprisingly good. You can also run a check to verify the URLs to splice hallucinations out of the result, but yeah, LLM is already doing a lot of the work for search.