5 ms·

Yeah I was looking into the firehose as a potential way to source to discover new domains for my search engine. Even though it didn't pan out, I really appreci

Yeah I was looking into the firehose as a potential way to source to discover new domains for my search engine. Even though it didn't pan out, I really appreciate how accessible the data is.

morkalork 2y ago
Why didn't it work out?
- marginalia_nu 2y ago
  I decided agains it because it had an incredibly bad signal to noise ratio. Almost all links I saw were either to big websites like newspapers, patreon, onlyfans; or behind url shorteners. Dunno, I may explore it further down the line, but for now the juice didn't seem worth the squeeze.
  - deleted 2y ago
    [deleted]
  - dools 2y ago
    I created a website like 10 years ago called birdmine that indexed every link you or one of your followers shared on Twitter, in a Solr search engine so you could search stuff that had been curated to an extent. It was pretty cool, I think I’m the only person that ever used it though.
  - djbusby 2y ago
    BSky could use a better search, or a better curation of feeds. I'm certain there is more room in the discovery play.
mariusor 2y ago
I might have asked this before, but did you look at adding this type of ingestion for ActivityPub?
- marginalia_nu 2y ago
  Briefly, but I've come to learn there's a contingent of aggressively search-engine hostile people that has made a home on the fediverse. The federated nature of it makes it somewhat tricky to untangle the search engine friendly people from the hostile. I don't need the inevitable DDOS:es and death threats you get when upsetting a clique of mentally ill people online.
  - mariusor 2y ago
    You're right, I forgot about the backlash that this type of efforts got from people that don't understand the technology and just make assumptions about element visibility in the network, though maybe calling them mentally ill is a bit much.
    - marginalia_nu 2y ago
      The sorts threatening physical violence almost certainly are. But it's a minority to be sure.

Yeah I was looking into the firehose as a potential way to source to discover new domains for my search engine. Even though it didn't pan out, I really appreciate how accessible the data is.

morkalork 2y ago

Why didn't it work out?

marginalia_nu 2y ago
I decided agains it because it had an incredibly bad signal to noise ratio. Almost all links I saw were either to big websites like newspapers, patreon, onlyfans; or behind url shorteners. Dunno, I may explore it further down the line, but for now the juice didn't seem worth the squeeze.
- deleted 2y ago
  [deleted]
- dools 2y ago
  I created a website like 10 years ago called birdmine that indexed every link you or one of your followers shared on Twitter, in a Solr search engine so you could search stuff that had been curated to an extent. It was pretty cool, I think I’m the only person that ever used it though.
- djbusby 2y ago
  BSky could use a better search, or a better curation of feeds. I'm certain there is more room in the discovery play.

mariusor 2y ago

I might have asked this before, but did you look at adding this type of ingestion for ActivityPub?

marginalia_nu 2y ago
Briefly, but I've come to learn there's a contingent of aggressively search-engine hostile people that has made a home on the fediverse. The federated nature of it makes it somewhat tricky to untangle the search engine friendly people from the hostile. I don't need the inevitable DDOS:es and death threats you get when upsetting a clique of mentally ill people online.
- mariusor 2y ago
  You're right, I forgot about the backlash that this type of efforts got from people that don't understand the technology and just make assumptions about element visibility in the network, though maybe calling them mentally ill is a bit much.
  - marginalia_nu 2y ago
    The sorts threatening physical violence almost certainly are. But it's a minority to be sure.