5 ms·
We need it - I’d use 3 search engines. It’s not like MMOs or social media where 1 winner takes all. I think people want more search options, cloud, email, and s
by cyanregiment 2mo ago
We need it - I’d use 3 search engines. It’s not like MMOs or social media where 1 winner takes all. I think people want more search options, cloud, email, and so-on.
I once tried to index the web with local Mistral 7b. Here’s what I did:
1. Copy/pasted an English dictionary and removed the definitions, so I just had an Array of about 500k words.
2. Wrote a simple loop to iterate over the words, with the instruction to Mistral to try to guess 10 URLs related to that word. I would typically get 2-3 sites per word.
3. When visiting a site, also check that site for links to other URLs, and summarize it with hash tags, visiting the new sites as well and recursively doing that.
4. Keep a tally of how many sites are being linked to for the page’s “rank”.
In the end you should have a massive list of objects with valid URLs ranked by how popular they are (to each other, by link), and a set of searchable hash tags for each.
I ran this for days and I never got past B, and almost every site was adult content. Most of the sites came from other sites. But it does work! Janky page crawling and ranking.
Anyway more people should take on web search, I agree. LLMs offer shortcuts.
- benj111 2mo agoWouldn't you do better say indexing Wikipedia, then what it links directly, then what those link directly? As you note, you got into an adult content 'bubble'. Going to random sites doesn't mean that site is good in anyway. Wikipedia, or some other proxy for quality at least gives you a good baseline. If you're aiming to index all of the web, you should have the same endpoint, but in the meantime my way, you are indexing 'good' sites
- cyanregiment 2mo agoMy crawling approach did hit Wikipedia several times, and crawled a number of pages, but not the entire site at once. With either approach, we would both eventually end up with all the websites - that being a surprisingly small number (~1 billion of which we only ever access ~1000). I got in an adult content bubble from starting with "A" ("Adult") and would have hit another one presumably at "X". But "B" had quite a few too, lol. Most of the words were bubbles - where it spent a decent amount of time on each one and I wondered if it would ever move on or was stuck. There are just a lot of backlinks and tangents for every word you can think of. Another fun thing about this experiment is you will find a lot more international websites that no other search engine will find - some weird personal sites and some surprisingly good forums. But yeah, you could enter anywhere - Wikipedia, or some other random Array of words. It doesn't have to be a dictionary.
- deleted 2mo ago[deleted]
- esseph 2mo agoYou could just pull tls cert registrations CT logs, way easier. Or https://crt.sh https://crt.sh
- cyanregiment 2mo ago502 - Bad Gateway Another cool thing about Mistral 7b is it runs offline, and I don't need to download any massive URL list that may or may not be current. It does the work of finding URLs at runtime, offline (until it verifies the URL exists).
- esseph 2mo agoAnother option: https://www.certkit.io/tools/ct-logs/ https://www.certkit.io/tools/ct-logs/
- cyanregiment 2mo agoLooks like they only give you 100 on a trial account
- esseph 2mo agoSorry, let me be more obvious. I am not pointing to a solution, but a pointer to the solution. Showing what is possible. The solution is pulling/parsing the CT logs yourself.
- cyanregiment 2mo agoIf you’re talking about SSL cert logs, that excludes a ton of web content. And you haven’t shown me that you can even get all those for free anyway
- esseph 2mo ago> If you’re talking about SSL cert logs, that excludes a ton of web content. I guess it depends on what you are looking for? It's far less brute force than trying to randomly generate hostnames and then do lookups, and your solution also misses everything that is just random alphanumeric combinations, of which there are tons. > And you haven’t shown me that you can even get all those for free anyway Free? Maybe.