3 ms·

We had X-No-Archive: YES throughout the 90s, which thankfully DejaNews honored (up until Google purchased them in 2001, anyways). Search engines are destroying

We had X-No-Archive: YES throughout the 90s, which thankfully DejaNews honored (up until Google purchased them in 2001, anyways). Search engines are destroying the social fabric of humanity, making it possible for haters to hate with perfect precision. We've been on the wrong course ('index everything! archive everything!') for thirty years now, and we've earned today's toxic and deadly Internet that resulted from it. Oh well.

lstamour 5y ago
In defence of search engines, when content disappears from the web, it disappears from the search index within months. And robots.txt has been a thing since forever. But I agree with the overall point, that archiving is distinct from indexing, and we need better mechanisms than copyright to take down content given many services terms allow content to be kept forever and/or make it hard to delete content uploaded by others. This might come down to free speech vs privacy, though.
- floatingatoll 5y ago
  If I ever again post a website, I’m going to post it with basic auth, so that search engines won’t index it to begin with. Exposure doesn’t pay the bills, after all.
- Sanzig 5y ago
  > And robots.txt has been a thing since forever. Unfortunately, both archive.org and the affiliated Archive Team now intentionally ignore robots.txt files. Archive Team, in particular, wrote a very irritating and condescending screed called "ROBOTS.TXT IS A SUICIDE NOTE" [1]. Apparently, they don't understand that not everyone wants their stuff backed up forever and they treat anyone who disagrees with them as a moron. And that manifesto wasn't written by some random team member, it was from Jason Scott himself. I know that Archive Team isn't officially part of IA, but they are tightly integrated with their archiving operations. It's a bad look. [1] http://wiki.archiveteam.org/index.php?title=Robots.txt http://wiki.archiveteam.org/index.php?title=Robots.txt
clairity 5y ago
> "...the wrong course ('index everything! archive everything!') for thirty years now..." that's overly broad. although i particularly dislike dichotomies, we can draw a simple one here for illustrative purposes: there's public and private information, and omni-indexing/-archiving is the "wrong course" only for private information, with private information encompassing more than just sensitive data, including any personal information a person chooses not to divulge publicly. humanity has absolutely advanced from indexing all public information, even considering the exponential rise of disinformation and idle bullshit. instantaneous knowledge coalition and mass multiway communication are the two astonishing advances stemming from the internet. and now we're dealing with the inevitable but unintended negative consequences of these amazing advances.
- floatingatoll 5y ago
  Your chosen dichotomy is both common and insufficient, and I am with you in disliking it. The dichotomy I'm considering is threefold instead: Worldwide, Community, and Secret. This is what the classical public/private dichotomy is missing: the social information strata of "local community only" information that gets lost when we convert everything to the Internet. Without a Community context, there is no safe place from haters. Searching a Community's content should not be possible from a Worldwide context. My comments on HN are Community, not Worldwide, but modern search indexing has no grasp of the difference between those two, and lumps them both together as Worldwide — what your dichotomy calls "Public". Yes, some people will intentionally write content for Worldwide consumption, but the vast majority of my written words are intended for Community consumption only. We see this every day, when some unwitting soul on Twitter gets their off-the-cuff Community-context tweet selected as Trending and the entirety of Worldwide descends upon them. Even celebrities, people who specialize their career in producing Worldwide content, have started to realize that they, too, need Community-context spaces to live and breathe in, and that online social media is not a safe Community space for them. What is an example of a healthy Community space, on today's Internet, that protects against the content discovery efforts of Worldwide haters, while still allowing new participants to join the space? (Certainly it's not HN, which is trying and failing to be a Community space operating in Worldwide context.) ps. I forgot Secrets! Secrets are mine to divulge in the Worldwide context, and no one else's. My biometrics, my medical records, my fetishes would all be good examples of what we would expect to be kept Secret. I understand and accept that some Secrets will be upgraded to Community context in the matter of course of human sociology, and I don't expect idle gossip to end. But I have every right to expect that my Secrets will not be divulged to the world, even if they're known to my Community. Unfortunately, due to search engines deciding that all Public content is therefore Worldwide, all Secrets that are shared within any private Community will end up indexed for all the haters to find. That's a very lonely way to live, not being able to be vulnerable within a Community. No wonder people use Facebook; it offers a well-known Community context (private groups) that's not searchable from a Worldwide context.
actually_a_dog 5y ago
> ...haters to hate with perfect precision. Except... not. All too often, we see stuff people did or wrote years ago being used against them. Those things may not reflect the person they are today, and that's even worse than "hat[ing] with perfect precision."
- floatingatoll 5y ago
  That's certainly one view, but it's not my view. If people don't feel safe participating today on some topic that haters have google alerts set up for, then those future archives will be heavily biased in favor of content that's acceptable to the haters of that time. Sounds like a great way to ensure the spread of hate across social generations to me.
- pvaldes 5y ago
  Searches will always have the problem of the lack of context. A song or a literary work can mean totally different things if we lack the context or understand the years when they were written. The second and more serious problem is people thinking that algorithms are so smart that are infallible. This, when a much more sophisticated human brain would have a serious trouble trying to navigate the long lists of random dictionary words that we enter in a browser. Netizens in general don't understand, or don't care, and just simplify and fill the gaps until they have a caricature that can fit in some "well known stereotype" slot. The result is that our societies are being more and more fractured, radicalized and isolated into different groups.

lstamour 5y ago

In defence of search engines, when content disappears from the web, it disappears from the search index within months. And robots.txt has been a thing since forever. But I agree with the overall point, that archiving is distinct from indexing, and we need better mechanisms than copyright to take down content given many services terms allow content to be kept forever and/or make it hard to delete content uploaded by others. This might come down to free speech vs privacy, though.

floatingatoll 5y ago
If I ever again post a website, I’m going to post it with basic auth, so that search engines won’t index it to begin with. Exposure doesn’t pay the bills, after all.
Sanzig 5y ago
> And robots.txt has been a thing since forever. Unfortunately, both archive.org and the affiliated Archive Team now intentionally ignore robots.txt files. Archive Team, in particular, wrote a very irritating and condescending screed called "ROBOTS.TXT IS A SUICIDE NOTE" [1]. Apparently, they don't understand that not everyone wants their stuff backed up forever and they treat anyone who disagrees with them as a moron. And that manifesto wasn't written by some random team member, it was from Jason Scott himself. I know that Archive Team isn't officially part of IA, but they are tightly integrated with their archiving operations. It's a bad look. [1] http://wiki.archiveteam.org/index.php?title=Robots.txt http://wiki.archiveteam.org/index.php?title=Robots.txt

clairity 5y ago

> "...the wrong course ('index everything! archive everything!') for thirty years now..." that's overly broad. although i particularly dislike dichotomies, we can draw a simple one here for illustrative purposes: there's public and private information, and omni-indexing/-archiving is the "wrong course" only for private information, with private information encompassing more than just sensitive data, including any personal information a person chooses not to divulge publicly. humanity has absolutely advanced from indexing all public information, even considering the exponential rise of disinformation and idle bullshit. instantaneous knowledge coalition and mass multiway communication are the two astonishing advances stemming from the internet. and now we're dealing with the inevitable but unintended negative consequences of these amazing advances.

floatingatoll 5y ago
Your chosen dichotomy is both common and insufficient, and I am with you in disliking it. The dichotomy I'm considering is threefold instead: Worldwide, Community, and Secret. This is what the classical public/private dichotomy is missing: the social information strata of "local community only" information that gets lost when we convert everything to the Internet. Without a Community context, there is no safe place from haters. Searching a Community's content should not be possible from a Worldwide context. My comments on HN are Community, not Worldwide, but modern search indexing has no grasp of the difference between those two, and lumps them both together as Worldwide — what your dichotomy calls "Public". Yes, some people will intentionally write content for Worldwide consumption, but the vast majority of my written words are intended for Community consumption only. We see this every day, when some unwitting soul on Twitter gets their off-the-cuff Community-context tweet selected as Trending and the entirety of Worldwide descends upon them. Even celebrities, people who specialize their career in producing Worldwide content, have started to realize that they, too, need Community-context spaces to live and breathe in, and that online social media is not a safe Community space for them. What is an example of a healthy Community space, on today's Internet, that protects against the content discovery efforts of Worldwide haters, while still allowing new participants to join the space? (Certainly it's not HN, which is trying and failing to be a Community space operating in Worldwide context.) ps. I forgot Secrets! Secrets are mine to divulge in the Worldwide context, and no one else's. My biometrics, my medical records, my fetishes would all be good examples of what we would expect to be kept Secret. I understand and accept that some Secrets will be upgraded to Community context in the matter of course of human sociology, and I don't expect idle gossip to end. But I have every right to expect that my Secrets will not be divulged to the world, even if they're known to my Community. Unfortunately, due to search engines deciding that all Public content is therefore Worldwide, all Secrets that are shared within any private Community will end up indexed for all the haters to find. That's a very lonely way to live, not being able to be vulnerable within a Community. No wonder people use Facebook; it offers a well-known Community context (private groups) that's not searchable from a Worldwide context.

actually_a_dog 5y ago

> ...haters to hate with perfect precision. Except... not. All too often, we see stuff people did or wrote years ago being used against them. Those things may not reflect the person they are today, and that's even worse than "hat[ing] with perfect precision."

floatingatoll 5y ago
That's certainly one view, but it's not my view. If people don't feel safe participating today on some topic that haters have google alerts set up for, then those future archives will be heavily biased in favor of content that's acceptable to the haters of that time. Sounds like a great way to ensure the spread of hate across social generations to me.
pvaldes 5y ago
Searches will always have the problem of the lack of context. A song or a literary work can mean totally different things if we lack the context or understand the years when they were written. The second and more serious problem is people thinking that algorithms are so smart that are infallible. This, when a much more sophisticated human brain would have a serious trouble trying to navigate the long lists of random dictionary words that we enter in a browser. Netizens in general don't understand, or don't care, and just simplify and fill the gaps until they have a caricature that can fit in some "well known stereotype" slot. The result is that our societies are being more and more fractured, radicalized and isolated into different groups.