12 ms·

It’s a little unsettling to think about how much information and knowledge is being locked up in walled-garden servers on discord, basically unsearchable (disco

It’s a little unsettling to think about how much information and knowledge is being locked up in walled-garden servers on discord, basically unsearchable (discord has a search feature, but it’s pretty awful). There’s so many communities that end up moving to it because it serves their most engaged members so well, but it’s terrible for everyone else. For example, “Voron” 3D printers are an awesome open-source design, but more and more I am directed to their discord to ask questions - many of which were, in all likelihood, asked dozens of times before. It’s great for their engaged members, who are all super helpful - but if it’s a reddit thread I can get my answer almost immediately, rather than asking, waiting and consuming someone else’s time for trivialities. Sites like reddit at least can be readily searched from a conventional search engine, and can be crawled and stored externally in a pinch. Discord has its place, especially for game communities or other such personal things, but I’m not sure it’s ideal compared to a conventional forum as time passes and more information is built up and either lost or hidden away.

hanniabu 4y ago
> discord has a search feature, but it’s pretty awful I don't think it's bad at all, why do you say it's awful? I feel like you think it's bad because you have a negative bias towards discord
- Panzer04 4y ago
  It only really works for exact matches. If I have a general idea of what I'm looking for, I can usually hunt it down with google. Discord will give me too many results or no results, which sucks. I like discord - It makes up a very significant (honestly, majority) portion of my social life. But it's not good for any kind of real information storage (FAQ, guides, expert answers and so on) compared to a forum.
MH15 4y ago
I remember chatting on the RepRap 3D Printer forums in the early 2010s, everything searchable by Google and static text. That mode is long gone now, Discord's superior UX seems to have swallowed up most of the forums.
charcircuit 4y ago
Discord's search works great for me
- jsheard 4y ago
  The search is per-server so before you even start you need to know which server has the thing you're looking for, which isn't always obvious, and you can't search a server without actively joining it, announcing your presence and using up your finite server slots. There's no equivalent to Googling something and passively pulling answers from wherever.
- Panzer04 4y ago
  Unless i'm missing something (and noting the drawbacks mentioned elsewhere re. no google search), its native search doesn't have any kind of fuzziness. You can't think of some set of terms and have it bring up a particular thing, you have to know an exact word used in it. This means you need to know enough already just to get it to come up, and more to prune out the other 500 results if your term is generic enough. Basically, the search only really works if you want to find a specific conversation you remember (and it better be recent, given how easily you forget the specifics of things as time passes)
  - Springtime 4y ago
    > Unless i'm missing something, its native search doesn't have any kind of fuzziness. For the past couple years it's returned variations of words, eg: `installer` will return `installers`, `installed`, `installs`, `installing`. However it doesn't return synonyms. It's essentially just searching like one would with regular chat logs, except limited to searching only one server at a time. I actually wish I could disable such matching since even with double quotes I can't get only an exact word matched (which is necessary when trying to filter a large number of results).
duxup 4y ago
> it because it serves their most engaged members so well, but it’s terrible for everyone else I join a discord channel and honestly most of the time I’m overwhelmed. There’s often a ton of sub channels for every specific thing… that aren’t very active. It’s hard to get a feel for what is going on. I join some only to find everyone is annoyed by my elementary question, but hell if I can find any answers in discord. I just never know the lay of the land. The handful of highly active people do, but that’s it.
- deleted 4y ago
  [deleted]
- Panzer04 4y ago
  Yup - the most engaged members are usually running the server, so they optimise it for their use. Multiple channels and categories make it easy to remember context if you switch often, and prevent recent relevant conversations from going out of scope too quickly.
Springtime 4y ago
What began as effectively an IRC-like alternative + file hosting and voice support is now being used as a replacement for forums and I think that's where the issue is. IRC isn't publicly searchable either unless someone was logging it and uploading them to some web server. IRC chats similarly often contain very useful info and answers. Discord unfortunately doesn't have any native chat export feature so the best that can be done are third-party exporters, copy-pasting or screenshots which aren't ideal and don't end up being indexed as desired even if communities wanted them to be.
- urthor 4y ago
  Exactly right. We're complaining about a problem that was no different in the heyday of IRC. Ultimately, publicly searchable information is a voluntary act.
  - Dalewyn 4y ago
    IRC can be accessed by any client adhering to IRC standards which are free and open. Discord can be accessed by any client adhering to Discord standards, which are closed and proprietary. Nobody has an obligation to publish information for public access, nor is free necessarily superior to proprietary or vice versa, but Discord is absolutely less accessible than IRC or HTTP(S) as an objective fact.
    - NavinF 4y ago
      99.9% of users don't care about any of that. Users want embedded media, custom emotes, free fully featured clients on every device they use (any client that needs a bouncer doesn't count), integration with desktop software such as games, video streaming / screen sharing, and voice calls. Notice that every single chat software used by normal humans (iMessage, Google chat, FB messenger, etc) has most of these features, just with much lower bitrates than Discord. For the very few users that don't need any feature they didn't have in the 90s, IRC is still around. The rest of us just add one of many text logging bots to our discord servers.
nine_k 4y ago
Cynically speaking, platforms like Discord are not interested in their content being widely searchable. If a,user can satisfy their need for information by reading an existing answer (even within Discord), that user needs Discord less. The user depends less on there being an active community in Discord members of which could provide answers. This all likely means fewer paying subscribers. So, Discord search should be fine for a few months depth, and need not be good further into the past. Exposing historical data to external search engines is even less desirable. Same of course applies to Slack, HipChat and whatever other commercial chat-like software.
- Manuel_D 4y ago
  I don't buy it: making discovery faster and more accessible may result in fewer hours spent on discord, but Discord as a whole becomes more valuable. If slack created some awesome feature that mined your company's chat logs to train something like the Librarian in Snow Crash that would be a massive selling point even if it means fewer chats between human beings. It's s valuable because it produces an answer without taxing another human.
  - lovich 4y ago
    Look at stackoverflow. They made their information extremely accessible and now search engines are riddled with sites ripping the content directly from stackoverflow, rehosting it, and winning SEO on specific questions which bleeds advertising revenue. Discord, and 99.99% of companies under capitalism, do not care if they produce more value unless they get to keep that value.
    - Manuel_D 4y ago
      I think it works perfectly for stack overflow. Rarely is the full answer in the Google blurb. I almost always click through to the actual SO post. Again, discord improving search would make users rely on their software more, since searching discord for answers becomes a part of user's workflows. Slack offering a premium enterprise "Librarian" feature would probably have significant demand. Usually, companies that offer more value can charge more for that value. I don't see the logic I how gimping search functionality somehow ends up improving these companies' bottom lines.
phreack 4y ago
And you can't even filter out spammy channels like bot command channels. On mobile it's particularly awful where if you look for a message's context, it's very likely you'll lose your "place" in the search panel and have to scroll from the top all over.
- Tumblewood 4y ago
  You can filter a discord search by what channel the message is in.
  - phreack 4y ago
    But not if you don't. Large servers tend to have information spread across many different channels and if you want to find out about something that may be in different channels, you have to search individually in each of them and that gets unwieldy fast.
berkle4455 4y ago
IRC was the same way for 15 or so years, all mostly lost.
- wraptile 4y ago
  IRC used to be indexed by Google and such though. Also because irc clients didn't have rich media support a lot of knowledge would make it outside of IRC like code snippets or microblogs that would eventually get indexed.
  - theshrike79 4y ago
    Google didn't "Index IRC". Source: have been on IRC before Google was a thing. What Google indexed was people's logger bots putting channel logs on the world wide web.
    - wraptile 4y ago
      Many channels had official logs
BlueTemplar 4y ago
Reddit is still bad, not only because it's a platform, but also because it tends to lock threads after merely a couple of months (which prevents necroposting, which sometimes IS the right thing to do, while creating a new post is the wrong one) - so even if someone comes in later with a solution, they can't even answer the previous posters ! (and some of the new forums, seems like Discourse has it on by default?)
- bee_rider 4y ago
  Hackernews seems to solve this by simply not notifying us of responses, so an ancient post that somehow gets responded to is just never noticed.
  - BlueTemplar 4y ago
    Actually, HN has the same issue than Reddit here : it locks threads. But now I wonder why duckduckgo never seems to show hn results (unless restricted to it of course) ? I have searched hn before as it contains a lot of "Lore" as GP calls it, some of it quite helpful ! I guess that hn is a bit to reddit what is IRC to Discord : because of its focus and lack of features, it's ironically better in this context because most people won't even try to use it for something more serious than "post-it's on the fridge door".
    - emsixteen 4y ago
      HN may be like the forums for Advanced Custom Fields, which weren't even indexed. I sent a mail to them and submitted added themselves to Bing, which in turn meant being added to DDG afaik.
swatcoder 4y ago
You’re seeing the symptom of something deeper. Conventional search was a ~20 year solution to navigating the “entirety” of online content when the available content was within the scope of that innovation. That era is coming to an end. There’s just too much content to index literally and too much noise too quantify quality and that problem is getting worse much faster than crawl+search technology can scale. So new techniques to navigating content are emerging, some of them calling back to pre-search solutions. LLM chat assistants drop the literal reference requirement by just mushing up all the sources they can and hallucinating something vaguely relevant to incoming questions. They lean into the noise and try to find patterns in it rather than sources. Meanwhile, “walled garden” private communities like Discord, Slack, Whatsapp/iMessage, and the growing list of login-required social content sites commit to sharing literal source content but address the noise problem by regimenting and moderating how content is incorporated. There will almost certainly be a next generation “meta-search” that can help you frame and make queries across these walled gardens, but it’s going to take a long while for the infrastructure and business models around that to establish themselves. In the meantime, this is what we get and what we can expect for a while.
- phendrenad2 4y ago
  This is correct. Google became a monopoly and they stopped caring about surfacing any results that they couldn't immediately monetize. If your content is archived in a forum somewhere, Google won't find it anyway, it'll instead show you results from youtube, ads, and whatever is on top of their cache. Search is effectively dead, so we have to resort to asking other humans directly for answers. Which sucks, but that's the phase of the competition/monopoly cycle that we're in right now.
  - paulmd 4y ago
    ignoring the problem of robots.txt inaccessibility, is it feasible to have Kagi-style "private google" with a more limited number of high-signal-to-noise sites, especially if you drop the concept of e-commerce and some other low-SNR feeds? perhaps one interesting thing is that a decent number of the highest-SNR feeds don't actually need to be crawled at all - wikipedia, reddit, etc are available as dumps and you can ingest their content directly. And the sources in which I am most interested in for my hobbies (technical data around cameras, computer parts, aircraft, etc) tend to be mostly static web-1.0 sites that basically never change. There's some stuff that falls inbetween, like I'm not sure if random other wikis necessarily have takeout dumps, but again, fandom-wiki and a couple other mega-wikis probably contain a majority of the interesting content, or at least a large enough amount of content you could get meaningful results. Another interesting one would be if you could get the Internet Archive to give you "slices" of sites in a google takeout-style format. Like they already have scraped a great deal of content, so, if I want site X and the most recent non-404 versions of all pages in a given domain, it would be fantastic if they could just build that as a zip and dump it over in bulk. In fact a lot of the best technical content is no longer available on the live web unfortunately... (did fh-reddit ever update again? or is there a way to get pushift to give you a bulk dump of everything? they stopped back in like 2019 and I'm not sure if they ever got back into it, it wasn't on bigquery last time I checked. Kind of a bummer too.) I say exclude e-commerce because there's not a lot of informational value in knowing the 27 sites selling a video card (especially as a few megaretailers crush all the competition anyway), but there is lots of informational value in say having a copy of the sites of asus, asrock, gigabyte, MSI, etc for searching (probably don't want full binaries cached though). But basically I think there's probably like, sub-100 TB of content that would even be useful to me if stored in some kind of relatively dense representation (reddit post/comment dumps, not pages, same for other forum content, etc, stored on a gzip level5 filesystem or something). That's easily within reach of a small server, not sure if pagerank would work as well without all the "noise" linking into it and telling you where the signal is, but I think that's well within typical r/datahoarder level builds. And you could dynamically augment that from live internet and internet archive as needed - just treat it as an ever-growing cache and index your hoard.
dfxm12 4y ago
Discord has its place, especially for game communities or other such personal things I'm not sure Discord is necessarily good for "gaming communities". I mean, Discord is live chat. This is good for some aspects: match making, news, etc. - anything that has a short lifespan. However, a lot of things about games don't. Wikis are perfect for publishing this info. Think of a Street Fighter type game. Characters have their move set, that doesn't change. Imagine having to search through discord for how to do a fireball with Ryu. Then there's strategy - that changes, albeit periodically, after tournaments, etc. By all means, this stuff can be discussed in discord, but the consensus strategies have to be published because it's a terrible experience to search through chat logs and follow along with long finished conversations over who knows how many posts, figure out context, etc. compared to just reading it on a wiki.
- the_af 4y ago
  Agreed. I see Discord being used for things where forums (and non realtime conversations) used to rule supreme, like hobbies, tutorials, etc. Why? A tutorial is not a real time chat. A showcase of hobby projects isn't a real time conversation either. And the searchability of these tools like Discord is terrible. I really don't understand why this terrible thing became popular.
- Panzer04 4y ago
  I guess I should be clearer - I'm more referring to multiplayer communities rather than actual game-specific resources. Communities, rather than knowledge resources, are where it's appropriate, basically. Of course, these often end up bleeding into eachother over time.
- Klonoar 4y ago
  You would be surprised by the FGC entities that seemingly gravitate towards this very thing.
- Claude_Shannon 4y ago
  Most FGC servers solve this by having character channels with the best guides pinned.
- 4y ago
ElijahLynn 4y ago
Slack, IRC, Matrix are all like this too. All the services really need a "Make public searchable archive" option.
- avx56 4y ago
  Yes, I know this is opt-in, but Matrix has a static & crawlable service for their chats: https://view.matrix.org/ https://view.matrix.org/
  - ptman 4y ago
    And https://github.com/matrix-org/matrix-public-archive https://github.com/matrix-org/matrix-public-archive
  - ElijahLynn 4y ago
    Oh heck yeah, this I did not know!!!
lmm 4y ago
I'd say it's a symptom of how toxic the public internet (especially Twitter) has become. When there is an angry mob constantly scouring every community for transgressions (real or imagined), being unindexed and unsearchable is a feature not a bug.
rom-antics 4y ago
> discord has a search feature, but it’s pretty awful Their search makes me want to pull my hair out. Why can't I just search the history with grep? That's a feature I would pay for (if anyone at Discord is reading this and wants my $10/month)
- jhgg 4y ago
  Because it would be obscenely expensive (in terms of computational resources) to search history via grep. Hence why we use an inverted index.
  - rom-antics 4y ago
    This is megabytes (at most) of text we're talking about, not gigabytes. And ripgrep is absurdly fast. Grepping a 5MB text file should be pretty much instantaneous.
    - jhgg 4y ago
      Do you know of any service that is running at scale out there that stores their data as 5mb plain text files?
      - selfhoster11 4y ago
        The “text files” don’t have to be the primary backing store - merely a derived cache file that can be periodically refreshed with new content. It doesn’t have to be text either - SQLite is pretty structured if you need more features in a minimal container too.
    - deleted 4y ago
      [deleted]
- Turing_Machine 4y ago
  I'd be happy even if they just didn't hijack Command/Ctrl-F. You can't even search for text on the screen you're viewing without being dumped into their shitful pseudosearch.
Fomite 4y ago
We've had the same thing happen with a local gaming club - we've pretty much wholesale moved from Facebook to Discord. Which is great for current members, but makes recruiting (which is sort of important in a college town) next to impossible.
Klonoar 4y ago
I will say what I feel like I say whenever this comes up: Discord could contribute to a solution here, even partially, by making Discord Forums search indexable. It would (theoretically) help with the "has my question been answered before" and it would (theoretically) make (some) archival efforts simpler.
nicbou 4y ago
Most of German immigration information and advice is locked into private Facebook groups. This information is not good nor reliable, but it's the only way to tell how things actually play out at the immigration office, for example. That information would serve a lot more people if it was available to them with a simple search.
- geraldhh 4y ago
  indeed very disappointing that most games have wikis/subreddits where one can get the gist of it served in a consumer friendly format but public services do not
  - madeofpalk 4y ago
    Well, it's people sharing their personal experiences. I'm sure the government has a public website where they try to be helpful, but personal experiences of immigration are very useful, and just not really something the government can collate (often because they work against each other - people trying to 'game' immigration).
    - nicbou 4y ago
      It's absolutely horrendous, and supplementing this information is how I pay the rent. I'd love to be tasked with improving the official resources, and I keep making new contacts in the city government. This could happen eventually, who knows.
    - geraldhh 4y ago
      > I'm sure the government has a public website where they try to be helpful ime more often than not, the focus is not on being helpful to customers but to please organizational demands. > people trying to 'game' immigration exactly, why assume hostile intent? if somone wants to immigrate it should be welcomed since most industrial nations have a lack of births anyways.
sneak 4y ago
What's far more unsettling is knowing that Discord's sysadmins, as well as their acquirer (which for a minute looked like it might have been MSFT) have the complete plaintext logs of every DM conversation. Every private link, every NDA'd product info, every insider crypto tip, all the passwords and credentials, all the sexting, all the nudes.... and all linked to your real world identity via the non-VoIP phone number you have to add to your account to join most channel groups ("servers"). The trove of blackmail and extortion data alone is worth a few dozen millions. The insider crypto trading that Discord makes possible is worth probably $20-50mm USD each month.
- TeMPOraL 4y ago
  And the mundane reality is that they won't use it for blackmail or in any such personalized fashion (except for government requests). What they will use it for, is training language models. That's the new way of monetizing "user-generated content", especially one you've managed to lock up so it can't be casually scrapped by anyone else.
- doublerabbit 4y ago
  Don't forget that if you do decide to delete your account its simply renamed not purged.
- anticensor 4y ago
  One of Discord's shareholders is Tencent (not Microsoft!!), however, the co-founders are still on the board and are highly occupied with day to day stuff. Message deletion on Discord is traceless from the outset (once deleted, no one, not even database admins, can retrieve deleted messages; the same thing also applies to the metadata, but not messages, of deleted users), in fact, this is one of the features they are bragging with. Traceless deletion is verified by both engineering related blogposts and interactions with support.
8note 4y ago
Do people want to write comments and replies that are publicly searchable?
ehnto 4y ago
Discord is nice because it lets you have small talk and build strong communities with a more natural cadence. In the same way we don't record every spoken conversation, I think we don't need to stress about discord going missing. I imagine any historically relevant outcomes of conversations on discord will be recorded outside of discord. All that said, it's still important to have knowledge bases to reference I agree, but I think that should be an effort separate to Discord. I wouldn't want to search an arbitrarily long chat log to find answers to questions, it's not well suited for the task.
- sciolistse 4y ago
  I would say in something like speedrunning there is tons of information that only exists in discord pinned threads etc., and in the combined heads of community members. To those who care about that hobby a lot of very useful information would vanish if discord went away. That's not a great state of affairs and discord is a very bad place to keep that information, even at the best of times it's not easy to find things in there. But I think it's probably the case for a decent number of communities. You could definitely argue that it's not historically relevant, since, if speedrunning as a whole disappeared it wouldn't really matter.
Razengan 4y ago
> It’s a little unsettling to think about how much information and knowledge is being locked up in walled-garden servers Not just on Discord, in general, everywhere: Facebook, Instagram, TikTok, Reddit and even here HackerNews. It's not just about being searchable or long-lived, but that they can pull the rug anytime (or make stealth edits like the Reddit fiasco with their admin Spez) How to solve this? Should the burden of searchability and archival be a requirement upon companies that provide social media services? Third-party bots already manually crawl all services and provide external search/archive interfaces (like the "Reddit Undelete" services) How to make that better?
poulpy123 4y ago
I discovered one years ago that Facebook had a trove of useful technical groups that are completely invisible because they are not indexable. I wonder how much knowledge we have lost because of discord and FB are the new forum
artagnon 4y ago
Unfortunately, even the LLVM community has chosen a combination of Discord/Discourse, and deprecated their mailing list/IRC channel. This is a very unhealthy trend, and only the most ardent of communities such as Linux/Git/GCC stick to old-fashioned publicly-archived mailing lists. [1]: https://public-inbox.org/README https://public-inbox.org/README
- kaoD 4y ago
  Well, that's because mailing lists are terrible.
  - account42 4y ago
    Not as terrible as Discord.
  - Apocryphon 4y ago
    What if they tried using Substack?
_cenw 4y ago
That is if you get support at all. Many times I have not. At least on an issue tracker I can find other people who might have solved a similar problem on their own.
pilarphosol 4y ago
Walled gardens are unfortunately the future of the internet. The public web is full of bots and adversarial content. Worse, anything you contribute in good faith can be used against you in the future by businesses and governments. Even in the rare case where those institutions are trustworthy, there is no guarantee of them being that way. So, the public web’s only power users are those who seek to influence others, who are therefore adversarial toward any higher minded purpose. Balkanisation and fragmentation, unfortunately, seem to be at least our near future.
- indigochill 4y ago
  But how does this overlap with federated platforms? Then you can still balkanize into tiny groups but federate among compatible groups to rebuild larger networks bottom-up (I like to draw a comparison to how multicellular organisms are composed of many discrete cells). And if you don't federate with hosts that serve businesses, you can fly way under the radar (Pleroma even has built-in onion routing support IIRC).
irthomasthomas 4y ago
Even more unsettling, when you realise there must be a lot of children talking to a lot of adults, about who knows what, on discords invisible to parents and police.
Arrath 4y ago
> more and more I am directed to their discord to ask questions - many of which were, in all likelihood, asked dozens of times before It becomes painfully obvious and a little funny when the question triggers a bot to reply with a link to a pinned comment answering FAQ#36, and you see it happen a dozen times a day.
Abishek_Muthian 4y ago
I wonder if setting up a discourse instance is too much of a friction that businesses are instead choosing a real-time chat inspite of it being awful for knowledge sharing as a whole. May be there's a need-gap for a low friction forum.

hanniabu 4y ago

> discord has a search feature, but it’s pretty awful I don't think it's bad at all, why do you say it's awful? I feel like you think it's bad because you have a negative bias towards discord

Panzer04 4y ago
It only really works for exact matches. If I have a general idea of what I'm looking for, I can usually hunt it down with google. Discord will give me too many results or no results, which sucks. I like discord - It makes up a very significant (honestly, majority) portion of my social life. But it's not good for any kind of real information storage (FAQ, guides, expert answers and so on) compared to a forum.

MH15 4y ago

I remember chatting on the RepRap 3D Printer forums in the early 2010s, everything searchable by Google and static text. That mode is long gone now, Discord's superior UX seems to have swallowed up most of the forums.

charcircuit 4y ago

Discord's search works great for me

jsheard 4y ago
The search is per-server so before you even start you need to know which server has the thing you're looking for, which isn't always obvious, and you can't search a server without actively joining it, announcing your presence and using up your finite server slots. There's no equivalent to Googling something and passively pulling answers from wherever.
Panzer04 4y ago
Unless i'm missing something (and noting the drawbacks mentioned elsewhere re. no google search), its native search doesn't have any kind of fuzziness. You can't think of some set of terms and have it bring up a particular thing, you have to know an exact word used in it. This means you need to know enough already just to get it to come up, and more to prune out the other 500 results if your term is generic enough. Basically, the search only really works if you want to find a specific conversation you remember (and it better be recent, given how easily you forget the specifics of things as time passes)
- Springtime 4y ago
  > Unless i'm missing something, its native search doesn't have any kind of fuzziness. For the past couple years it's returned variations of words, eg: `installer` will return `installers`, `installed`, `installs`, `installing`. However it doesn't return synonyms. It's essentially just searching like one would with regular chat logs, except limited to searching only one server at a time. I actually wish I could disable such matching since even with double quotes I can't get only an exact word matched (which is necessary when trying to filter a large number of results).

duxup 4y ago

> it because it serves their most engaged members so well, but it’s terrible for everyone else I join a discord channel and honestly most of the time I’m overwhelmed. There’s often a ton of sub channels for every specific thing… that aren’t very active. It’s hard to get a feel for what is going on. I join some only to find everyone is annoyed by my elementary question, but hell if I can find any answers in discord. I just never know the lay of the land. The handful of highly active people do, but that’s it.

deleted 4y ago
[deleted]
Panzer04 4y ago
Yup - the most engaged members are usually running the server, so they optimise it for their use. Multiple channels and categories make it easy to remember context if you switch often, and prevent recent relevant conversations from going out of scope too quickly.

Springtime 4y ago

What began as effectively an IRC-like alternative + file hosting and voice support is now being used as a replacement for forums and I think that's where the issue is. IRC isn't publicly searchable either unless someone was logging it and uploading them to some web server. IRC chats similarly often contain very useful info and answers. Discord unfortunately doesn't have any native chat export feature so the best that can be done are third-party exporters, copy-pasting or screenshots which aren't ideal and don't end up being indexed as desired even if communities wanted them to be.

urthor 4y ago
Exactly right. We're complaining about a problem that was no different in the heyday of IRC. Ultimately, publicly searchable information is a voluntary act.
- Dalewyn 4y ago
  IRC can be accessed by any client adhering to IRC standards which are free and open. Discord can be accessed by any client adhering to Discord standards, which are closed and proprietary. Nobody has an obligation to publish information for public access, nor is free necessarily superior to proprietary or vice versa, but Discord is absolutely less accessible than IRC or HTTP(S) as an objective fact.
  - NavinF 4y ago
    99.9% of users don't care about any of that. Users want embedded media, custom emotes, free fully featured clients on every device they use (any client that needs a bouncer doesn't count), integration with desktop software such as games, video streaming / screen sharing, and voice calls. Notice that every single chat software used by normal humans (iMessage, Google chat, FB messenger, etc) has most of these features, just with much lower bitrates than Discord. For the very few users that don't need any feature they didn't have in the 90s, IRC is still around. The rest of us just add one of many text logging bots to our discord servers.

nine_k 4y ago

Cynically speaking, platforms like Discord are not interested in their content being widely searchable. If a,user can satisfy their need for information by reading an existing answer (even within Discord), that user needs Discord less. The user depends less on there being an active community in Discord members of which could provide answers. This all likely means fewer paying subscribers. So, Discord search should be fine for a few months depth, and need not be good further into the past. Exposing historical data to external search engines is even less desirable. Same of course applies to Slack, HipChat and whatever other commercial chat-like software.

Manuel_D 4y ago
I don't buy it: making discovery faster and more accessible may result in fewer hours spent on discord, but Discord as a whole becomes more valuable. If slack created some awesome feature that mined your company's chat logs to train something like the Librarian in Snow Crash that would be a massive selling point even if it means fewer chats between human beings. It's s valuable because it produces an answer without taxing another human.
- lovich 4y ago
  Look at stackoverflow. They made their information extremely accessible and now search engines are riddled with sites ripping the content directly from stackoverflow, rehosting it, and winning SEO on specific questions which bleeds advertising revenue. Discord, and 99.99% of companies under capitalism, do not care if they produce more value unless they get to keep that value.
  - Manuel_D 4y ago
    I think it works perfectly for stack overflow. Rarely is the full answer in the Google blurb. I almost always click through to the actual SO post. Again, discord improving search would make users rely on their software more, since searching discord for answers becomes a part of user's workflows. Slack offering a premium enterprise "Librarian" feature would probably have significant demand. Usually, companies that offer more value can charge more for that value. I don't see the logic I how gimping search functionality somehow ends up improving these companies' bottom lines.

phreack 4y ago

And you can't even filter out spammy channels like bot command channels. On mobile it's particularly awful where if you look for a message's context, it's very likely you'll lose your "place" in the search panel and have to scroll from the top all over.

Tumblewood 4y ago
You can filter a discord search by what channel the message is in.
- phreack 4y ago
  But not if you don't. Large servers tend to have information spread across many different channels and if you want to find out about something that may be in different channels, you have to search individually in each of them and that gets unwieldy fast.

berkle4455 4y ago

IRC was the same way for 15 or so years, all mostly lost.

wraptile 4y ago
IRC used to be indexed by Google and such though. Also because irc clients didn't have rich media support a lot of knowledge would make it outside of IRC like code snippets or microblogs that would eventually get indexed.
- theshrike79 4y ago
  Google didn't "Index IRC". Source: have been on IRC before Google was a thing. What Google indexed was people's logger bots putting channel logs on the world wide web.
  - wraptile 4y ago
    Many channels had official logs

BlueTemplar 4y ago

Reddit is still bad, not only because it's a platform, but also because it tends to lock threads after merely a couple of months (which prevents necroposting, which sometimes IS the right thing to do, while creating a new post is the wrong one) - so even if someone comes in later with a solution, they can't even answer the previous posters ! (and some of the new forums, seems like Discourse has it on by default?)

bee_rider 4y ago
Hackernews seems to solve this by simply not notifying us of responses, so an ancient post that somehow gets responded to is just never noticed.
- BlueTemplar 4y ago
  Actually, HN has the same issue than Reddit here : it locks threads. But now I wonder why duckduckgo never seems to show hn results (unless restricted to it of course) ? I have searched hn before as it contains a lot of "Lore" as GP calls it, some of it quite helpful ! I guess that hn is a bit to reddit what is IRC to Discord : because of its focus and lack of features, it's ironically better in this context because most people won't even try to use it for something more serious than "post-it's on the fridge door".
  - emsixteen 4y ago
    HN may be like the forums for Advanced Custom Fields, which weren't even indexed. I sent a mail to them and submitted added themselves to Bing, which in turn meant being added to DDG afaik.

swatcoder 4y ago

You’re seeing the symptom of something deeper. Conventional search was a ~20 year solution to navigating the “entirety” of online content when the available content was within the scope of that innovation. That era is coming to an end. There’s just too much content to index literally and too much noise too quantify quality and that problem is getting worse much faster than crawl+search technology can scale. So new techniques to navigating content are emerging, some of them calling back to pre-search solutions. LLM chat assistants drop the literal reference requirement by just mushing up all the sources they can and hallucinating something vaguely relevant to incoming questions. They lean into the noise and try to find patterns in it rather than sources. Meanwhile, “walled garden” private communities like Discord, Slack, Whatsapp/iMessage, and the growing list of login-required social content sites commit to sharing literal source content but address the noise problem by regimenting and moderating how content is incorporated. There will almost certainly be a next generation “meta-search” that can help you frame and make queries across these walled gardens, but it’s going to take a long while for the infrastructure and business models around that to establish themselves. In the meantime, this is what we get and what we can expect for a while.

phendrenad2 4y ago
This is correct. Google became a monopoly and they stopped caring about surfacing any results that they couldn't immediately monetize. If your content is archived in a forum somewhere, Google won't find it anyway, it'll instead show you results from youtube, ads, and whatever is on top of their cache. Search is effectively dead, so we have to resort to asking other humans directly for answers. Which sucks, but that's the phase of the competition/monopoly cycle that we're in right now.
- paulmd 4y ago
  ignoring the problem of robots.txt inaccessibility, is it feasible to have Kagi-style "private google" with a more limited number of high-signal-to-noise sites, especially if you drop the concept of e-commerce and some other low-SNR feeds? perhaps one interesting thing is that a decent number of the highest-SNR feeds don't actually need to be crawled at all - wikipedia, reddit, etc are available as dumps and you can ingest their content directly. And the sources in which I am most interested in for my hobbies (technical data around cameras, computer parts, aircraft, etc) tend to be mostly static web-1.0 sites that basically never change. There's some stuff that falls inbetween, like I'm not sure if random other wikis necessarily have takeout dumps, but again, fandom-wiki and a couple other mega-wikis probably contain a majority of the interesting content, or at least a large enough amount of content you could get meaningful results. Another interesting one would be if you could get the Internet Archive to give you "slices" of sites in a google takeout-style format. Like they already have scraped a great deal of content, so, if I want site X and the most recent non-404 versions of all pages in a given domain, it would be fantastic if they could just build that as a zip and dump it over in bulk. In fact a lot of the best technical content is no longer available on the live web unfortunately... (did fh-reddit ever update again? or is there a way to get pushift to give you a bulk dump of everything? they stopped back in like 2019 and I'm not sure if they ever got back into it, it wasn't on bigquery last time I checked. Kind of a bummer too.) I say exclude e-commerce because there's not a lot of informational value in knowing the 27 sites selling a video card (especially as a few megaretailers crush all the competition anyway), but there is lots of informational value in say having a copy of the sites of asus, asrock, gigabyte, MSI, etc for searching (probably don't want full binaries cached though). But basically I think there's probably like, sub-100 TB of content that would even be useful to me if stored in some kind of relatively dense representation (reddit post/comment dumps, not pages, same for other forum content, etc, stored on a gzip level5 filesystem or something). That's easily within reach of a small server, not sure if pagerank would work as well without all the "noise" linking into it and telling you where the signal is, but I think that's well within typical r/datahoarder level builds. And you could dynamically augment that from live internet and internet archive as needed - just treat it as an ever-growing cache and index your hoard.

dfxm12 4y ago

Discord has its place, especially for game communities or other such personal things I'm not sure Discord is necessarily good for "gaming communities". I mean, Discord is live chat. This is good for some aspects: match making, news, etc. - anything that has a short lifespan. However, a lot of things about games don't. Wikis are perfect for publishing this info. Think of a Street Fighter type game. Characters have their move set, that doesn't change. Imagine having to search through discord for how to do a fireball with Ryu. Then there's strategy - that changes, albeit periodically, after tournaments, etc. By all means, this stuff can be discussed in discord, but the consensus strategies have to be published because it's a terrible experience to search through chat logs and follow along with long finished conversations over who knows how many posts, figure out context, etc. compared to just reading it on a wiki.

the_af 4y ago
Agreed. I see Discord being used for things where forums (and non realtime conversations) used to rule supreme, like hobbies, tutorials, etc. Why? A tutorial is not a real time chat. A showcase of hobby projects isn't a real time conversation either. And the searchability of these tools like Discord is terrible. I really don't understand why this terrible thing became popular.
Panzer04 4y ago
I guess I should be clearer - I'm more referring to multiplayer communities rather than actual game-specific resources. Communities, rather than knowledge resources, are where it's appropriate, basically. Of course, these often end up bleeding into eachother over time.
Klonoar 4y ago
You would be surprised by the FGC entities that seemingly gravitate towards this very thing.
Claude_Shannon 4y ago
Most FGC servers solve this by having character channels with the best guides pinned.
4y ago

ElijahLynn 4y ago

Slack, IRC, Matrix are all like this too. All the services really need a "Make public searchable archive" option.

avx56 4y ago
Yes, I know this is opt-in, but Matrix has a static & crawlable service for their chats: https://view.matrix.org/ https://view.matrix.org/
- ptman 4y ago
  And https://github.com/matrix-org/matrix-public-archive https://github.com/matrix-org/matrix-public-archive
- ElijahLynn 4y ago
  Oh heck yeah, this I did not know!!!

lmm 4y ago

I'd say it's a symptom of how toxic the public internet (especially Twitter) has become. When there is an angry mob constantly scouring every community for transgressions (real or imagined), being unindexed and unsearchable is a feature not a bug.

rom-antics 4y ago

> discord has a search feature, but it’s pretty awful Their search makes me want to pull my hair out. Why can't I just search the history with grep? That's a feature I would pay for (if anyone at Discord is reading this and wants my $10/month)

jhgg 4y ago
Because it would be obscenely expensive (in terms of computational resources) to search history via grep. Hence why we use an inverted index.
- rom-antics 4y ago
  This is megabytes (at most) of text we're talking about, not gigabytes. And ripgrep is absurdly fast. Grepping a 5MB text file should be pretty much instantaneous.
  - jhgg 4y ago
    Do you know of any service that is running at scale out there that stores their data as 5mb plain text files?
    - selfhoster11 4y ago
      The “text files” don’t have to be the primary backing store - merely a derived cache file that can be periodically refreshed with new content. It doesn’t have to be text either - SQLite is pretty structured if you need more features in a minimal container too.
  - deleted 4y ago
    [deleted]
Turing_Machine 4y ago
I'd be happy even if they just didn't hijack Command/Ctrl-F. You can't even search for text on the screen you're viewing without being dumped into their shitful pseudosearch.

Fomite 4y ago

We've had the same thing happen with a local gaming club - we've pretty much wholesale moved from Facebook to Discord. Which is great for current members, but makes recruiting (which is sort of important in a college town) next to impossible.

Klonoar 4y ago

I will say what I feel like I say whenever this comes up: Discord could contribute to a solution here, even partially, by making Discord Forums search indexable. It would (theoretically) help with the "has my question been answered before" and it would (theoretically) make (some) archival efforts simpler.

nicbou 4y ago

Most of German immigration information and advice is locked into private Facebook groups. This information is not good nor reliable, but it's the only way to tell how things actually play out at the immigration office, for example. That information would serve a lot more people if it was available to them with a simple search.

geraldhh 4y ago
indeed very disappointing that most games have wikis/subreddits where one can get the gist of it served in a consumer friendly format but public services do not
- madeofpalk 4y ago
  Well, it's people sharing their personal experiences. I'm sure the government has a public website where they try to be helpful, but personal experiences of immigration are very useful, and just not really something the government can collate (often because they work against each other - people trying to 'game' immigration).
  - nicbou 4y ago
    It's absolutely horrendous, and supplementing this information is how I pay the rent. I'd love to be tasked with improving the official resources, and I keep making new contacts in the city government. This could happen eventually, who knows.
  - geraldhh 4y ago
    > I'm sure the government has a public website where they try to be helpful ime more often than not, the focus is not on being helpful to customers but to please organizational demands. > people trying to 'game' immigration exactly, why assume hostile intent? if somone wants to immigrate it should be welcomed since most industrial nations have a lack of births anyways.

sneak 4y ago

What's far more unsettling is knowing that Discord's sysadmins, as well as their acquirer (which for a minute looked like it might have been MSFT) have the complete plaintext logs of every DM conversation. Every private link, every NDA'd product info, every insider crypto tip, all the passwords and credentials, all the sexting, all the nudes.... and all linked to your real world identity via the non-VoIP phone number you have to add to your account to join most channel groups ("servers"). The trove of blackmail and extortion data alone is worth a few dozen millions. The insider crypto trading that Discord makes possible is worth probably $20-50mm USD each month.

TeMPOraL 4y ago
And the mundane reality is that they won't use it for blackmail or in any such personalized fashion (except for government requests). What they will use it for, is training language models. That's the new way of monetizing "user-generated content", especially one you've managed to lock up so it can't be casually scrapped by anyone else.
doublerabbit 4y ago
Don't forget that if you do decide to delete your account its simply renamed not purged.
anticensor 4y ago
One of Discord's shareholders is Tencent (not Microsoft!!), however, the co-founders are still on the board and are highly occupied with day to day stuff. Message deletion on Discord is traceless from the outset (once deleted, no one, not even database admins, can retrieve deleted messages; the same thing also applies to the metadata, but not messages, of deleted users), in fact, this is one of the features they are bragging with. Traceless deletion is verified by both engineering related blogposts and interactions with support.

8note 4y ago

Do people want to write comments and replies that are publicly searchable?

ehnto 4y ago

Discord is nice because it lets you have small talk and build strong communities with a more natural cadence. In the same way we don't record every spoken conversation, I think we don't need to stress about discord going missing. I imagine any historically relevant outcomes of conversations on discord will be recorded outside of discord. All that said, it's still important to have knowledge bases to reference I agree, but I think that should be an effort separate to Discord. I wouldn't want to search an arbitrarily long chat log to find answers to questions, it's not well suited for the task.

sciolistse 4y ago
I would say in something like speedrunning there is tons of information that only exists in discord pinned threads etc., and in the combined heads of community members. To those who care about that hobby a lot of very useful information would vanish if discord went away. That's not a great state of affairs and discord is a very bad place to keep that information, even at the best of times it's not easy to find things in there. But I think it's probably the case for a decent number of communities. You could definitely argue that it's not historically relevant, since, if speedrunning as a whole disappeared it wouldn't really matter.

Razengan 4y ago

> It’s a little unsettling to think about how much information and knowledge is being locked up in walled-garden servers Not just on Discord, in general, everywhere: Facebook, Instagram, TikTok, Reddit and even here HackerNews. It's not just about being searchable or long-lived, but that they can pull the rug anytime (or make stealth edits like the Reddit fiasco with their admin Spez) How to solve this? Should the burden of searchability and archival be a requirement upon companies that provide social media services? Third-party bots already manually crawl all services and provide external search/archive interfaces (like the "Reddit Undelete" services) How to make that better?

poulpy123 4y ago

I discovered one years ago that Facebook had a trove of useful technical groups that are completely invisible because they are not indexable. I wonder how much knowledge we have lost because of discord and FB are the new forum

artagnon 4y ago

Unfortunately, even the LLVM community has chosen a combination of Discord/Discourse, and deprecated their mailing list/IRC channel. This is a very unhealthy trend, and only the most ardent of communities such as Linux/Git/GCC stick to old-fashioned publicly-archived mailing lists. [1]: https://public-inbox.org/README https://public-inbox.org/README

kaoD 4y ago
Well, that's because mailing lists are terrible.
- account42 4y ago
  Not as terrible as Discord.
- Apocryphon 4y ago
  What if they tried using Substack?

_cenw 4y ago

That is if you get support at all. Many times I have not. At least on an issue tracker I can find other people who might have solved a similar problem on their own.

pilarphosol 4y ago

Walled gardens are unfortunately the future of the internet. The public web is full of bots and adversarial content. Worse, anything you contribute in good faith can be used against you in the future by businesses and governments. Even in the rare case where those institutions are trustworthy, there is no guarantee of them being that way. So, the public web’s only power users are those who seek to influence others, who are therefore adversarial toward any higher minded purpose. Balkanisation and fragmentation, unfortunately, seem to be at least our near future.

indigochill 4y ago
But how does this overlap with federated platforms? Then you can still balkanize into tiny groups but federate among compatible groups to rebuild larger networks bottom-up (I like to draw a comparison to how multicellular organisms are composed of many discrete cells). And if you don't federate with hosts that serve businesses, you can fly way under the radar (Pleroma even has built-in onion routing support IIRC).

irthomasthomas 4y ago

Even more unsettling, when you realise there must be a lot of children talking to a lot of adults, about who knows what, on discords invisible to parents and police.

Arrath 4y ago

> more and more I am directed to their discord to ask questions - many of which were, in all likelihood, asked dozens of times before It becomes painfully obvious and a little funny when the question triggers a bot to reply with a link to a pinned comment answering FAQ#36, and you see it happen a dozen times a day.

Abishek_Muthian 4y ago

I wonder if setting up a discourse instance is too much of a friction that businesses are instead choosing a real-time chat inspite of it being awful for knowledge sharing as a whole. May be there's a need-gap for a low friction forum.