4 ms·

> some random person decided to start indexing every post on the mastodon network. The backlash to this was incredible. A lot of servers blocked this instance,

by jefftk 4y ago

> some random person decided to start indexing every post on the mastodon network. The backlash to this was incredible. A lot of servers blocked this instance, and the guy got banned within hours of announcing his indexer The next person will do this without announcing it, in a way that just looks like a server full of lurkers. When you have a protocol that's all about broadcasting things to anyone who asks to be notified, it's very hard to enforce "no indexing". I also see Mastodon posts showing up in search results, so it looks to me like indexing is already happening: https://www.google.com/search?q=%22How+I+write+chord+charts%22 https://www.google.com/search?q=%22How+I+write+chord+charts%...

WorldMaker 4y ago
A lot of lurker-only servers tend to get blocked, too. Some instances check every follow/follow request and block if anything looks off, including "lurker only". > I also see Mastodon posts showing up in search results, so it looks to me like indexing is already happening Mastodon has a configurable robots.txt. Some instances are fine with search indexing their public timelines. The big thing is instances can want their public timelines indexed but not their "followers only" timelines.
- jefftk 4y ago
  > Some instances check every follow/follow request and block if anything looks off, including "lurker only". That's rare, though, right? Since there's often a ton of following going around, especially now? It's probably also not that hard to make something that easily passes those sorts of quick checks? > Some instances are fine with search indexing their public timelines. Which then means anything of yours that gets boosted by someone on a server like that will be indexed.
  - WorldMaker 4y ago
    > That's rare, though, right? Not in my experience. I'm on a tiny instance and follows are so rare in general, I can watch them all individually. There are many users that require follow requests where they verify/vet every single follow. > It's probably also not that hard to make something that easily passes those sorts of quick checks? Maybe such that one person doesn't notice, but remember the scale here: if someone is trying to "follow the world" maliciously, they have to make sure that they pass the quick checks of every paranoid person they try to follow. Once a malicious site is "discovered" the #fediblock hash tag moves pretty virally to encourage other paranoid users/instances to (re-)evaluate the bad actor. > Which then means anything of yours that gets boosted by someone on a server like that will be indexed (Courteous) Instances don't allow you to boost a Followers Only post to the Public timeline. Discourteous instances may exist, but when they are found out, they are generally blocked (again, the #fediblock hashtag sometimes moves swiftly and virally here).
    - fediblock 4y ago
      Fediblock just re-centralizes blacklisting to the (opaque) maintainers of that list, which defeats much of the point of federation
      - WorldMaker 4y ago
        There's no "Fediblock central list". At the moment it's just a search hashtag that federates with other toots (it federates as toots because it is just toots). It is entirely up to instance admins what they do with what they see in that hashtag.
- gs17 4y ago
  > A lot of lurker-only servers tend to get blocked, too. Some instances check every follow/follow request and block if anything looks off, including "lurker only". Well, there goes thinking I can always just host my own instance, I probably wouldn't be active enough for their tastes. And it adds another concern to joining a smaller instance, seems like I have to rely on other users to make it look more legit if I don't want to be so centralized. Or are they looking for very strictly, literally "lurker only" instances?
  - WorldMaker 4y ago
    There's always an "everyone blocks me and no one likes me" fear in hosting your own instance, and yes you will run the risk of preemptive blocks if people don't like how your instance looks. The particular combo here, though, is "lurker only instance" that also spams follow requests. If you don't spam follow requests and are selective about who you follow (like a human rather than a bot), you have less risk of this specific paranoia. Lurkers are fine to many Mastodon users. Lurkers spamming following requests are a concern.
  - xg15 4y ago
    Not a Mastodon user, but I'd imagine most admins follow common sense. If your server only has a handful of accounts that obviously correspond to you and your friends, I don't see why anyone would have a problem with that. On the other hand, if the server has 100s of accounts, each of which follows a large number of people off-server - yet somehow no one follows anyone else on the server and no one ever talks a single word, that would probably raise some suspicion.
    - merlincorey 4y ago
      > On the other hand, if the server has 100s of accounts, each of which follows a large number of people off-server - yet somehow no one follows anyone else on the server and no one ever talks a single word, that would probably raise some suspicion. Existing Twitter and Facebook connections (names, followers, friends, etc) to create many secret indexer servers made of easily verifiable Google searches of real people with real friends and GPT-like toot generation to fake just enough activity seems like a pretty straightforward route to avoid suspicion. Arms races suck -- public data is public, period.
Kye 4y ago
You say that like you think it hasn't already happened. People noticed, admins compared notes, and they still ended up blocked.
- jefftk 4y ago
  People catching some cases doesn't mean others aren't succeeding unnoticed. A privacy model that depends on "admins would probably notice something fishy" is not a model users should rely on.
  - Kye 4y ago
    All it takes is one person to notice for a #fediblock notice to propagate. The more instances, the harder it is to hide.
    - simfree 4y ago
      With OP's perspective services like Cloudflare, Akamai and anyone else trying to mitigate malicious traffic should just give up. This extremely nihilistic view is not productive, and ignores that 99% success in mitigating bad actors is much better than no mitigation.
2fast4you 4y ago
ActivityPub makes it easy to crawl and index the whole Fediverse, you don't even need to start a server. I played with the idea a bit before I learned more about the community. If you're being intentionally malicious, disguising your traffic as legit shouldn't be impossible
easrng 4y ago
Authorized fetch makes it impossible to do this without either scraping the web interface (which some instances turn off (it's one switch in the settings)) or using a followbot which will get you #fediblocked immediately. RE: Search results, instances have a per-user option to add a noindex meta tag to your posts on the web interface.
sinity 4y ago
> The next person will do this without announcing it, in a way that just looks like a server full of lurkers. When you have a protocol that's all about broadcasting things to anyone who asks to be notified, it's very hard to enforce "no indexing". Yeah, their culture doesn't sound very... sane from these descriptions. Neurotic about weird stuff, like someone linking to them.
- peterashford 4y ago
  I would say that says more about your preconceptions than their sanity. There's nothing wrong with people wanting spaces with different rules. That's why they built their own space, after all
  - simfree 4y ago
    Built, currently host, manage access to, and block malicious actors from too. Some instances don't even allow non-logged in users to view public toots, good luck scraping those toots without sendingba follow request (which will be seen by the account and server admin.

by jefftk 4y ago

WorldMaker 4y ago

A lot of lurker-only servers tend to get blocked, too. Some instances check every follow/follow request and block if anything looks off, including "lurker only". > I also see Mastodon posts showing up in search results, so it looks to me like indexing is already happening Mastodon has a configurable robots.txt. Some instances are fine with search indexing their public timelines. The big thing is instances can want their public timelines indexed but not their "followers only" timelines.

jefftk 4y ago
> Some instances check every follow/follow request and block if anything looks off, including "lurker only". That's rare, though, right? Since there's often a ton of following going around, especially now? It's probably also not that hard to make something that easily passes those sorts of quick checks? > Some instances are fine with search indexing their public timelines. Which then means anything of yours that gets boosted by someone on a server like that will be indexed.
- WorldMaker 4y ago
  > That's rare, though, right? Not in my experience. I'm on a tiny instance and follows are so rare in general, I can watch them all individually. There are many users that require follow requests where they verify/vet every single follow. > It's probably also not that hard to make something that easily passes those sorts of quick checks? Maybe such that one person doesn't notice, but remember the scale here: if someone is trying to "follow the world" maliciously, they have to make sure that they pass the quick checks of every paranoid person they try to follow. Once a malicious site is "discovered" the #fediblock hash tag moves pretty virally to encourage other paranoid users/instances to (re-)evaluate the bad actor. > Which then means anything of yours that gets boosted by someone on a server like that will be indexed (Courteous) Instances don't allow you to boost a Followers Only post to the Public timeline. Discourteous instances may exist, but when they are found out, they are generally blocked (again, the #fediblock hashtag sometimes moves swiftly and virally here).
  - fediblock 4y ago
    Fediblock just re-centralizes blacklisting to the (opaque) maintainers of that list, which defeats much of the point of federation
    - WorldMaker 4y ago
      There's no "Fediblock central list". At the moment it's just a search hashtag that federates with other toots (it federates as toots because it is just toots). It is entirely up to instance admins what they do with what they see in that hashtag.
gs17 4y ago
> A lot of lurker-only servers tend to get blocked, too. Some instances check every follow/follow request and block if anything looks off, including "lurker only". Well, there goes thinking I can always just host my own instance, I probably wouldn't be active enough for their tastes. And it adds another concern to joining a smaller instance, seems like I have to rely on other users to make it look more legit if I don't want to be so centralized. Or are they looking for very strictly, literally "lurker only" instances?
- WorldMaker 4y ago
  There's always an "everyone blocks me and no one likes me" fear in hosting your own instance, and yes you will run the risk of preemptive blocks if people don't like how your instance looks. The particular combo here, though, is "lurker only instance" that also spams follow requests. If you don't spam follow requests and are selective about who you follow (like a human rather than a bot), you have less risk of this specific paranoia. Lurkers are fine to many Mastodon users. Lurkers spamming following requests are a concern.
- xg15 4y ago
  Not a Mastodon user, but I'd imagine most admins follow common sense. If your server only has a handful of accounts that obviously correspond to you and your friends, I don't see why anyone would have a problem with that. On the other hand, if the server has 100s of accounts, each of which follows a large number of people off-server - yet somehow no one follows anyone else on the server and no one ever talks a single word, that would probably raise some suspicion.
  - merlincorey 4y ago
    > On the other hand, if the server has 100s of accounts, each of which follows a large number of people off-server - yet somehow no one follows anyone else on the server and no one ever talks a single word, that would probably raise some suspicion. Existing Twitter and Facebook connections (names, followers, friends, etc) to create many secret indexer servers made of easily verifiable Google searches of real people with real friends and GPT-like toot generation to fake just enough activity seems like a pretty straightforward route to avoid suspicion. Arms races suck -- public data is public, period.

Kye 4y ago

You say that like you think it hasn't already happened. People noticed, admins compared notes, and they still ended up blocked.

jefftk 4y ago
People catching some cases doesn't mean others aren't succeeding unnoticed. A privacy model that depends on "admins would probably notice something fishy" is not a model users should rely on.
- Kye 4y ago
  All it takes is one person to notice for a #fediblock notice to propagate. The more instances, the harder it is to hide.
  - simfree 4y ago
    With OP's perspective services like Cloudflare, Akamai and anyone else trying to mitigate malicious traffic should just give up. This extremely nihilistic view is not productive, and ignores that 99% success in mitigating bad actors is much better than no mitigation.

2fast4you 4y ago

ActivityPub makes it easy to crawl and index the whole Fediverse, you don't even need to start a server. I played with the idea a bit before I learned more about the community. If you're being intentionally malicious, disguising your traffic as legit shouldn't be impossible

easrng 4y ago

Authorized fetch makes it impossible to do this without either scraping the web interface (which some instances turn off (it's one switch in the settings)) or using a followbot which will get you #fediblocked immediately. RE: Search results, instances have a per-user option to add a noindex meta tag to your posts on the web interface.

sinity 4y ago

> The next person will do this without announcing it, in a way that just looks like a server full of lurkers. When you have a protocol that's all about broadcasting things to anyone who asks to be notified, it's very hard to enforce "no indexing". Yeah, their culture doesn't sound very... sane from these descriptions. Neurotic about weird stuff, like someone linking to them.

peterashford 4y ago
I would say that says more about your preconceptions than their sanity. There's nothing wrong with people wanting spaces with different rules. That's why they built their own space, after all
- simfree 4y ago
  Built, currently host, manage access to, and block malicious actors from too. Some instances don't even allow non-logged in users to view public toots, good luck scraping those toots without sendingba follow request (which will be seen by the account and server admin.