8 ms·

Are they searchable and indexable? What made PHPBB and all that cool was that it'd turn up in search queries. As a hobbyist, this was an incredible wealth of in

by aliqot 4y ago

Are they searchable and indexable? What made PHPBB and all that cool was that it'd turn up in search queries. As a hobbyist, this was an incredible wealth of information for me that has mostly dried up with the advent of walled gardens like slack, discord, etc. Some might point at BBS and IRC and say that those were the same, but I'd argue that they had lower volume, and less rich media, and I'd go one further to say that newsgroups were well indexed and searchable. We are in a 'dark decade' of lost history search-wise in some hobby communities.

jhgg 4y ago
There is talk about allowing public/discoverable servers to be indexed. I don't know when those plans might materialize, but it is very much top of mind for us that there is ever growing knowledge on Discord that isn't accessible via search indexes.
- MonkeyMalarky 4y ago
  This would be a huge boost in adoption! But how would it rank on google? At first there would be few to any links to content from discord.
  - gpm 4y ago
    That sounds like Google's problem? I imagine they'd figure it out pretty quickly given the size and popularity of discord.
  - SahAssar 4y ago
    I think with the amount of internal linking and original content it would start to rank pretty quickly. At least if the scrape algorithm makes any sense at all.
- danudey 4y ago
  I'm already coming across more and more communities who say "For more information/to download the example files/to see our FAQ just join our Discord!", which is an immediate turnoff. I'm not going to "join a discord" just to see an FAQ or download some prerelease build or something else. Now even entire communities are going to move onto Discord and become completely unreachable for people like me who don't want to "join the community" just to read one post that one person wrote one time.
  - pph 4y ago
    I also feel that "just" is very much an understatement when it means "sign up for a service that doesn't play nice with anonymous networks (Tor) and often requires to share and verify your phone number".
- joe-collins 4y ago
  I'm hugely relieved that you see the black box as a situation to be addressed. For me, that circumstance alone turns Discord from a neat and useful tool into a tragedy that I'm forced to live with, and I'd love to see it move into the former category Hopefully distinct from mere searchability: how about integrating a wiki service for big/boosted servers?
deleted 4y ago
[deleted]
r3trohack3r 4y ago
Apt association I'd never made before. Communication in discord does have similar traits as communication in IRC. IRC is indexed via passive bots. I can still go back and search my old IRC user names in popular search engines and find messages I sent 15 years ago.
EGreg 4y ago
Discourse vs Discord Not PhpBB
- tester756 4y ago
  mehh, I'd rather use PHPBB-like forums instead of Discourse.
freediver 4y ago
Knowledge being accessible via search engines is the main reason we are sticking to a real 'forum' vs the new Discord feature (despite having a very large Discord community). There is a lot of sophistication needed for managing a forum so it could mean that this Discord feature is meant to address a different need.
SteveMoody73 4y ago
I've run a forum and worked on a couple of others. First one was for lactose intolerance when there was very little information available. Others were technical sites. In that case more information became easily available and in others, it's moved to platforms like facebook and twitter. I do mourn the loss of these forums as there was a lot of good information, and more importantly some first hand experience in some of the posts. A lot of these things now are view or click driven so they lose the personality. While there are still a lot of good forum sites around and do well, I think the rise of spam posts and increasing work of moderating the smaller sights was too high and eventually killed them off.
Buttons840 4y ago
It's important for a forum to be both externally searchable and internally. A lot of the old forums had internal search tools that really sucked, and external search engines didn't seem to cover everything.
mjr00 4y ago
On the modern internet, I don't know that being searchable and indexed is always a benefit. I like the idea of a semi-private space. I don't want people on the internet to be able to drudge up out of context statements I made 10 years ago by googling "site:discord.com <my discord name>". I can tell a lot of people are the same way just by their behavior; the discussion that happens on Discord is a lot different than anything on, say, HN or Reddit, and certainly Facebook or LinkedIn. People are a lot more open when they're not worried about being watched. Discussion that happens on public forums usually sounds like everyone's at a job interview getting scrutinized by a potential employer. Which, in fairness, they effectively are.
- culturestate 4y ago
  > I don't want people on the internet to be able to drudge up out of context statements I made 10 years ago by googling "site:discord.com <my discord name>". We’ve had public and private forums - or more often, private sub-forums within larger public forums - forever. I don’t see why you can’t achieve the same separation on Discord, as long as they allow admins to explicitly set indexability(?) for each channel.
  - mmcdermott 4y ago
    That seems reasonable enough technically, but I doubt it would work out that way in practice. Most users would probably treat all Discord channels alike. Those who don't understand the setup would carry on as they always had, many of those who do would find it easier to treat all alike (if for no other reason than that the setting could always change).
    - solveit 4y ago
      I'm fine with only privacy-conscious people getting privacy.
      - bee_rider 4y ago
        Privacy should be the default so that people who go for privacy don't stick out.
kbyatnal 4y ago
I totally agree - I find myself missing the Google discussions filter (if you remember what that was?) very often. It would let you search only across forums, message boards, and other places where actual discussions where taking place (as opposed to SEO and blog spam). I actually built a small tool for myself to try and replicate this. It searches across a few discords, a curated index of forums and message boards, and a little bit of Twitter (while applying some advance parameters to filter out the marketing spam). Works pretty well! If anyone else wants to give it a go (desktop only for now): https://crew-rho.vercel.app https://crew-rho.vercel.app (the forum search takes a little bit of time to load, but you'll see it if you scroll down)
kmeisthax 4y ago
Searchable? Yes. Indexable? Absolutely not. Even public servers are only public to human accounts and only server admins are allowed to invite bots. If you try to scrape Discord with a human account you will be banned.
- jotm 4y ago
  That's AIcist :D But really, you can't tell human accounts from bots.
  - franga2000 4y ago
    With a sophisticated enough crawler, sure, but that would require a significant amount of R&D. I don't know of any organization that both has the resources to do that, is willing to break Discord's ToS and make their index public.
  - kmeisthax 4y ago
    Given what I've heard about people getting banned for "self-bots"[0] they seem to err more on the "ban humans that look too bot-like" side of that dilemma. That being said, while there is some overlap between the two, it's not terribly difficult to detect specific things that bots would want to do, such as scrape a Discord channel, and ban any human accounts that match that profile. A regular human is going to be sending lots of messages and querying message history only rarely; a bot is going to hit the message history API as quickly as the rate limits allow and never send a single message. Browser fingerprinting could also be used to sus out bots that don't look like a web browser enough. Most bot developers are just making HTTP requests rather than firing up a real browser and attaching to it with WebDriver[1]. Once you do that then you have to play walls-and-ladders[2] with the Discord user-interface team. Oops, your bot just clicked on the fake invisible "please ban me immediately" button that we snuck in right next to the link that opens #general! Oops, we decided to obfuscate all message content so that incoming message JSON is unreadable and outgoing HTML requires having your specific Discord CSS loaded to view it! Just as there's unlimited ways to disguise a bot as a human, there's also unlimited ways to unmask them or frustrate them while remaining usable for human users. Turing completeness is a harsh mistress, and scrapers only work because people find economic value in being scraped - not that scrapers have an inherent advantage. [0] Scraping your own Discord client token and using it to authenticate a Discord bot with your own account [1] Writing a WebExtension content script to inject input into a browser would also work to self-bot [2] https://devblogs.microsoft.com/oldnewthing/20120117-00/?p=8523 https://devblogs.microsoft.com/oldnewthing/20120117-00/?p=85...
fny 4y ago
PHPBB:Reddit::AIM:Discord The name of the app is Discord for crying out loud. Have you seen the kinds of things people are using it for? Discord is *wildly* hackable chat, not forum software. There's tons of crap being slung around on Discord that doesn't deserve to be archived because that's not the purpose. Is IRC indexable or searchable? No. Usenet served that purpose. People hacked in query bots to IRC. At least Discord comes with built in search. If you want a forum, go use forum software.
- mediaman 4y ago
  Sure, but the problem is that so many communities are moving discussion around solving domain-specific problems to Discord. And the collective wealth of knowledge - some of which gets accumulated across decades - instead becomes an amnesiac experience where answering one person’s question provides no gain to the next person with the same question in a year’s time. I’m not sure why this move is happening.
  - fny 4y ago
    Exactly. This is a people problem not a software problem.
- devmor 4y ago
  >If you want a forum, go use forum software. Boy have I got some news for you about what Discord just announced. You should check out the link on the post you're commenting on!
holler 4y ago
I created https://sqwok.im https://sqwok.im because I wanted an open, indexable, public chat site tailored to kitchen-table conversations, influenced from my years growing up on IRC. There's been an active discussion on whether to allow gifs/images in the chat, with many saying they like that it's focused just on conversation.

by aliqot 4y ago

jhgg 4y ago

There is talk about allowing public/discoverable servers to be indexed. I don't know when those plans might materialize, but it is very much top of mind for us that there is ever growing knowledge on Discord that isn't accessible via search indexes.

MonkeyMalarky 4y ago
This would be a huge boost in adoption! But how would it rank on google? At first there would be few to any links to content from discord.
- gpm 4y ago
  That sounds like Google's problem? I imagine they'd figure it out pretty quickly given the size and popularity of discord.
- SahAssar 4y ago
  I think with the amount of internal linking and original content it would start to rank pretty quickly. At least if the scrape algorithm makes any sense at all.
danudey 4y ago
I'm already coming across more and more communities who say "For more information/to download the example files/to see our FAQ just join our Discord!", which is an immediate turnoff. I'm not going to "join a discord" just to see an FAQ or download some prerelease build or something else. Now even entire communities are going to move onto Discord and become completely unreachable for people like me who don't want to "join the community" just to read one post that one person wrote one time.
- pph 4y ago
  I also feel that "just" is very much an understatement when it means "sign up for a service that doesn't play nice with anonymous networks (Tor) and often requires to share and verify your phone number".
joe-collins 4y ago
I'm hugely relieved that you see the black box as a situation to be addressed. For me, that circumstance alone turns Discord from a neat and useful tool into a tragedy that I'm forced to live with, and I'd love to see it move into the former category Hopefully distinct from mere searchability: how about integrating a wiki service for big/boosted servers?

deleted 4y ago

[deleted]

r3trohack3r 4y ago

Apt association I'd never made before. Communication in discord does have similar traits as communication in IRC. IRC is indexed via passive bots. I can still go back and search my old IRC user names in popular search engines and find messages I sent 15 years ago.

EGreg 4y ago

Discourse vs Discord Not PhpBB

tester756 4y ago
mehh, I'd rather use PHPBB-like forums instead of Discourse.

freediver 4y ago

Knowledge being accessible via search engines is the main reason we are sticking to a real 'forum' vs the new Discord feature (despite having a very large Discord community). There is a lot of sophistication needed for managing a forum so it could mean that this Discord feature is meant to address a different need.

SteveMoody73 4y ago

I've run a forum and worked on a couple of others. First one was for lactose intolerance when there was very little information available. Others were technical sites. In that case more information became easily available and in others, it's moved to platforms like facebook and twitter. I do mourn the loss of these forums as there was a lot of good information, and more importantly some first hand experience in some of the posts. A lot of these things now are view or click driven so they lose the personality. While there are still a lot of good forum sites around and do well, I think the rise of spam posts and increasing work of moderating the smaller sights was too high and eventually killed them off.

Buttons840 4y ago

It's important for a forum to be both externally searchable and internally. A lot of the old forums had internal search tools that really sucked, and external search engines didn't seem to cover everything.

mjr00 4y ago

On the modern internet, I don't know that being searchable and indexed is always a benefit. I like the idea of a semi-private space. I don't want people on the internet to be able to drudge up out of context statements I made 10 years ago by googling "site:discord.com <my discord name>". I can tell a lot of people are the same way just by their behavior; the discussion that happens on Discord is a lot different than anything on, say, HN or Reddit, and certainly Facebook or LinkedIn. People are a lot more open when they're not worried about being watched. Discussion that happens on public forums usually sounds like everyone's at a job interview getting scrutinized by a potential employer. Which, in fairness, they effectively are.

culturestate 4y ago
> I don't want people on the internet to be able to drudge up out of context statements I made 10 years ago by googling "site:discord.com <my discord name>". We’ve had public and private forums - or more often, private sub-forums within larger public forums - forever. I don’t see why you can’t achieve the same separation on Discord, as long as they allow admins to explicitly set indexability(?) for each channel.
- mmcdermott 4y ago
  That seems reasonable enough technically, but I doubt it would work out that way in practice. Most users would probably treat all Discord channels alike. Those who don't understand the setup would carry on as they always had, many of those who do would find it easier to treat all alike (if for no other reason than that the setting could always change).
  - solveit 4y ago
    I'm fine with only privacy-conscious people getting privacy.
    - bee_rider 4y ago
      Privacy should be the default so that people who go for privacy don't stick out.

kbyatnal 4y ago

I totally agree - I find myself missing the Google discussions filter (if you remember what that was?) very often. It would let you search only across forums, message boards, and other places where actual discussions where taking place (as opposed to SEO and blog spam). I actually built a small tool for myself to try and replicate this. It searches across a few discords, a curated index of forums and message boards, and a little bit of Twitter (while applying some advance parameters to filter out the marketing spam). Works pretty well! If anyone else wants to give it a go (desktop only for now): https://crew-rho.vercel.app https://crew-rho.vercel.app (the forum search takes a little bit of time to load, but you'll see it if you scroll down)

kmeisthax 4y ago

Searchable? Yes. Indexable? Absolutely not. Even public servers are only public to human accounts and only server admins are allowed to invite bots. If you try to scrape Discord with a human account you will be banned.

jotm 4y ago
That's AIcist :D But really, you can't tell human accounts from bots.
- franga2000 4y ago
  With a sophisticated enough crawler, sure, but that would require a significant amount of R&D. I don't know of any organization that both has the resources to do that, is willing to break Discord's ToS and make their index public.
- kmeisthax 4y ago
  Given what I've heard about people getting banned for "self-bots"[0] they seem to err more on the "ban humans that look too bot-like" side of that dilemma. That being said, while there is some overlap between the two, it's not terribly difficult to detect specific things that bots would want to do, such as scrape a Discord channel, and ban any human accounts that match that profile. A regular human is going to be sending lots of messages and querying message history only rarely; a bot is going to hit the message history API as quickly as the rate limits allow and never send a single message. Browser fingerprinting could also be used to sus out bots that don't look like a web browser enough. Most bot developers are just making HTTP requests rather than firing up a real browser and attaching to it with WebDriver[1]. Once you do that then you have to play walls-and-ladders[2] with the Discord user-interface team. Oops, your bot just clicked on the fake invisible "please ban me immediately" button that we snuck in right next to the link that opens #general! Oops, we decided to obfuscate all message content so that incoming message JSON is unreadable and outgoing HTML requires having your specific Discord CSS loaded to view it! Just as there's unlimited ways to disguise a bot as a human, there's also unlimited ways to unmask them or frustrate them while remaining usable for human users. Turing completeness is a harsh mistress, and scrapers only work because people find economic value in being scraped - not that scrapers have an inherent advantage. [0] Scraping your own Discord client token and using it to authenticate a Discord bot with your own account [1] Writing a WebExtension content script to inject input into a browser would also work to self-bot [2] https://devblogs.microsoft.com/oldnewthing/20120117-00/?p=8523 https://devblogs.microsoft.com/oldnewthing/20120117-00/?p=85...

fny 4y ago

PHPBB:Reddit::AIM:Discord The name of the app is Discord for crying out loud. Have you seen the kinds of things people are using it for? Discord is *wildly* hackable chat, not forum software. There's tons of crap being slung around on Discord that doesn't deserve to be archived because that's not the purpose. Is IRC indexable or searchable? No. Usenet served that purpose. People hacked in query bots to IRC. At least Discord comes with built in search. If you want a forum, go use forum software.

mediaman 4y ago
Sure, but the problem is that so many communities are moving discussion around solving domain-specific problems to Discord. And the collective wealth of knowledge - some of which gets accumulated across decades - instead becomes an amnesiac experience where answering one person’s question provides no gain to the next person with the same question in a year’s time. I’m not sure why this move is happening.
- fny 4y ago
  Exactly. This is a people problem not a software problem.
devmor 4y ago
>If you want a forum, go use forum software. Boy have I got some news for you about what Discord just announced. You should check out the link on the post you're commenting on!

holler 4y ago

I created https://sqwok.im https://sqwok.im because I wanted an open, indexable, public chat site tailored to kitchen-table conversations, influenced from my years growing up on IRC. There's been an active discussion on whether to allow gifs/images in the chat, with many saying they like that it's focused just on conversation.