7 ms·

Show HN: A free, privacy preserving, archive of public Discord servers

70 points by searchcord 1y ago

Hey HN! I have been working on this project for a while, and I think this solves a problem that a lot of people here have: not being able to easily search Discord servers. Currently, I only scrape servers that are marked as "discoverable" on Discord. However, if there's enough interest in the project, I'm open to adding specific servers by request. I'm primarily focused on informational servers rather than casual hangout spaces, such as open source projects, Minecraft mods, and support communities for tools, services, or platforms (for example, hosting providers). I have placed restrictions on searching directly by user ID to prevent doxing. I also made the opt out process one click, for those who do not want to be archived. This is my first large scale project, so I'd love to hear your feedback!

hofrogs 1y ago
This is really cool and actually useful for peeking behind those annoying login walls. What software do you use to store/index/search in so much data? How did you get the data in the first place? Discord isn't exactly known for letting its data be available easily. Have the administrators of the guilds asked you for this? Have you contacted them and made them aware after the fact?
- searchcord 1y ago
  Hey, Thanks for your feedback. For software, I use ScyllaDB and Elasticsearch. It's split across 6 physical nodes (8 including the CDN). Data collection is handled using standard user accounts, accessing only public, discoverable servers. I plan to write a blog post about the technical aspect of how this was done soon. Admins of these servers weren't contacted, as the content indexed is already publicly accessible, comparable to a forum like this or public subreddit. That said, I understand the sensitivity around data visibility, and I've made it very simple for any user to opt out of indexing at any time. Private or invite-only servers are, of course, completely excluded.
  - hofrogs 1y ago
    That's a lot of compute, how much does it cost to keep it running? I don't see how that project would generate any income on its own
    - searchcord 1y ago
      I already own the hardware, so I only pay for colocation and transit. It's probably a lot less than you think. I hope to find some way to monetize it, but it is cheap enough that I can keep it running for quite a long time without any income.
      - hofrogs 1y ago
        Thanks for this. Well good luck with keeping it up, it's a really useful service.
      - uniqueuid 1y ago
        Wow, that must be quite expensive! You said the files alone are a few PB. So at least 2PB / 8 servers ~= 250TB per server, which would probably put each server at > 20k $ (unless you’re putting it together with duct tape and scraps, but even then the disks will cost a ton).
        searchcord 1y ago
        Hey, Not exactly. Attachments are only fetched from Discord as the user requests them. This means that the vast majority of attachments are never stored on my server. Right now, I only have about 280TB of attachments locally on my own infrastructure. You can see more stats here: https://searchcord.io/about https://searchcord.io/about Thanks for your question!
  - klntsky 1y ago
    I suggest you to remove the opt-out functionality and let it scrape private servers that it discovers via publicly posted invite links. You don't owe anyone posting on a public forum any privacy. Moreover, the most valuable data to search for is probably somewhat obscured.
legionof7 1y ago
I've been looking for something like this for so long, thanks for making! There's so much stuff locked in Discord now that forums have fallen in popularity, think this sort of thing really helps unlock that knowledge again.
- searchcord 1y ago
  Thanks for your feedback! <3
orph 1y ago
Can I download all the messages & attachments?
- searchcord 1y ago
  Sure, but there's a few petabytes of attachments and over 63 billion messages. Feel free to use the API.
treyd 1y ago
Would you consider making regular dumps of the database available in sharded torrents like Anna's Archive does so that users can back up the data themselves for preservation purposes? This would complicate retroactively removing users' activity, but that data could already be scraped. And related, I'd like to be able to run this locally for exports of guilds that I'm on myself. Is that even possible with the architect you've built?
- searchcord 1y ago
  Hey, This is absolutely something I want to do, but at the guild level. The database itself is over 13TB which is much to large to create regular exports of. I will probably provide a SQLite export of each guild, regenerated each week/month. Anyone is free to download whatever they want in real time from the API. Thanks for your question!
  - squigz 1y ago
    Big +1 for dumps You might try reaching out to Anna's Archive and see if this would be a dataset they'd be interested in helping host/distribute. I think they'd agree that such data is important and should be archived.
    - pilimi_anna 1y ago
      Yes, we'd happily host and torrent this.
  - treyd 1y ago
    I was imagining something like a month-by-month historical export so only recent data would need to be regularly exported. SQLite dumps would be awesome to work with though.
ivape 1y ago
Finding good Discord servers has been a great thing for me. I was getting super disconnected and isolated, so different Discord servers has made me feel human again.
- searchcord 1y ago
  I hope Searchcord helps you! <3
pabs3 1y ago
There are some Discord archives on archive.org too btw. https://archive.org/search?query=subject%3A%22DiscordChatExporter%22 https://archive.org/search?query=subject%3A%22DiscordChatExp... https://archive.org/search?query=subject%3A%22archiveteam_discord%22 https://archive.org/search?query=subject%3A%22archiveteam_di... https://wiki.archiveteam.org/index.php/Discord https://wiki.archiveteam.org/index.php/Discord
- searchcord 1y ago
  Hey, This is interesting, I somehow missed this. Unfortunately, those are not full text searchable. Maybe I will download them and import them into Searchcord, with proper credit of course. Thanks for this!
johnQdeveloper 1y ago
> This is my first large scale project, so I'd love to hear your feedback! > I have placed restrictions on searching directly by user ID to prevent doxing. I also made the opt out process one click, for those who do not want to be archived. 1) I'd suggest anonymizing the usernames / author ids to something more privacy friendly such as how some image sites were generating 3-4 random words as a human readable unique id. This removes a lot of the reason people would opt out (i.e. posts being tracked down years later) 2) You not seem to have a clear rate limit documentation. If you are asking people to pay for commercial use, I'd suggest making it clear what the rough original limits are as well as the rough price range of what you'd offer. 3) Tbh, the only real thing I want from this project is basically narrative / roleplay / writing content for LLM reasons as I'm trying to build a rules-oriented system that narrates via LLM. If you don't want people using this data for this purpose, I'd suggest making that clear.
- searchcord 1y ago
  Hey, Thanks for your suggestions. > 1) I'd suggest anonymizing the usernames / author ids to something more privacy friendly such as how some image sites were generating 3-4 random words as a human readable unique id. This removes a lot of the reason people would opt out (i.e. posts being tracked down years later) In the original iteration of Searchcord, it used to work similarly to that. The username was `sha256(userid+guildid)`, truncated to the first 8 characters. Unfortunately, it was pretty hard to follow chats. I will try your idea and see how it works, though. > 2) You not seem to have a clear rate limit documentation. This is a good idea. The rate limit varies by endpoint, and I haven't gotten around to documenting each one. > If you are asking people to pay for commercial use, I'd suggest making it clear what the rough original limits are as well as the rough price range of what you'd offer. I have absolutely zero idea what industry would be interested in this, in what form, and if anyone would even pay. > 3) Tbh, the only real thing I want from this project is basically narrative / roleplay / writing content for LLM reasons as I'm trying to build a rules-oriented system that narrates via LLM. If you don't want people using this data for this purpose, I'd suggest making that clear. I really don't care what people do with the data, as long as they are not spamming requests or using the data for commercial purposes without permission.
cinntaile 1y ago
All discord servers require an invitation link as far as I know, do you consider a link you find online as a public server?
- searchcord 1y ago
  Some very large servers are eligible for what Discord calls "discovery". This makes their data visible without joining the server. You can find a list of those on Discord's site here: https://discord.com/servers https://discord.com/servers
  - nottorp 1y ago
    Suggestion: a bot for smaller servers that do want to be archived like a public forum. Their admins could install the bot themselves and perhaps specify what channels they want archived.
    - pzmarzly 1y ago
      Check out Linen https://www.linen.dev/ https://www.linen.dev/
    - searchcord 1y ago
      This is something I have already completed but have not finished bug testing. The bot also includes functionality to recover any server in case it was nuked/wiped and Searchcord has a backup of it. It uses webhooks to resend the messages so you have an approximation of what the channels used to be.
Stagnant 1y ago
Incredible work! Truly eye-opening to see how some rarer keywords in my native language return pages of relevant results. Meanwhile google gives 0 results or just AI/ad spam.
IceWreck 1y ago
Do you plan to handle servers where you need to do some action (like send a message) to join all channels ? I was scrolling through the home page and came across afew where the only channels you're allowed to access are the verify-yourself or welcome channels.
- searchcord 1y ago
  Probably not. Discord will aggressively captcha you and every server has a different implementation of verification. It might be possible with a captcha solver and then some LLM to figure out the next steps.
bstsb 1y ago
nice project. how are you going to handle the issues involved with breaking Discord's TOS? > "scraping our services without our written consent" additionally, are these pages indexable? i know of other projects (opt-in) that create pages made from user discussion.
- searchcord 1y ago
  > how are you going to handle the issues involved with breaking Discord's TOS? Not sure. I will solve that problem if and when Discord takes issue with Searchcord. > additionally, are these pages indexable? Yes, I would actually like for search engines to index it as their search is much more contextually aware than mine.
  - woofer21 1y ago
    > Not sure. I will solve that problem if and when Discord takes issue with Searchcord. You are in for such a rude awakening once discord gets to you
EZ-Cheeze 1y ago
I'm in more than a hundred Discord servers. I've been wanting to scrape the members of each of them to discover the people with whom I share the most servers but we're not yet friends. Someone with 10+ would highly likely be a new friend since we'd have a lot of shared niche interests
- searchcord 1y ago
  This is something I have been trying to make as a way to learn about graph theory. If I can find a way to make it work efficiently, I will definitely add this.
  - EZ-Cheeze 1y ago
    Yeah that would be awesome! You could build a whole new social network on top of Discord. Whether in this way or others, I believe we'll all be finding an increasing number of hypercompatible people as technology advances.
3abiton 1y ago
This is an amazing project, I always wonder how much information is lost in those chat apps, not only Discord, but also Telegram. The latter has hude dev community specifically around Android Rom Development, which migrated from forum based XDA to more flexible chat/support platform like Telegram. I wish that also can be searchable without having their client.
- searchcord 1y ago
  Telegram is already heavily monitored and scraped due to the large volume of illegal or extremely controversial activity that happens there. This is something I will look into though, my XDA threads rarely get any replies anymore. Thanks for the suggestion!
jonasdoesthings 1y ago
Maybe also exclude messages by bots (e.g. "username has joined the server") from the index to decrease the stalking-potential of your site (99.9% of these bot messages have no informational-value for the index anyways). Currently you can still search for an username and get a subset of servers that the username is in (even if not active) by finding these bot messages.
- searchcord 1y ago
  This is something I have looked into. Unfortunately, every server uses a different format/bot. It might be possible to develop some sort of machine learning classifier to determine if it is a welcome message.
elpocko 1y ago
Any chance for a style option that doesn't have anime girls?
- searchcord 1y ago
  lol. It's only on the homepage, not on any content pages.
id44037319 1y ago
This is great, I'll be returning to this tool often. Thanks. A few suggestions and ideas for futher projects. -allow for "keyword", -negate operators and "mult word string" searches, [Pubmed](https://pubmed.ncbi.nlm.nih.gov/advanced https://pubmed.ncbi.nlm.nih.gov/advanced) is what I'd consider an Ideal search interface -allow for regex, or direct sql lookups with limited query time ratelimited by POW. for example, if the server is under load, require a token from something like [anubis](https://anubis.techaro.lol/ https://anubis.techaro.lol/) and lower the maximum DB query time -Index the title of all discussion/forum type posts with a VectorDB for semantic search. And add an option to sort by replies. (Like [answer overflow](https://github.com/AnswerOverflow/AnswerOverflow https://github.com/AnswerOverflow/AnswerOverflow))This would make it possible to find relevant discussions among ~60B messages. ScyllaDB doesn't support vector search, so I'd suggest something like [usearch](https://github.com/unum-cloud/usearch https://github.com/unum-cloud/usearch) for a detached index. Embedding models are faster and smaller than most people realize, pick whatever's on top of the [mteb leaderboard](https://huggingface.co/spaces/mteb/leaderboard https://huggingface.co/spaces/mteb/leaderboard) after deciding on size. -calculate the jaccard similarity (user overlap) between discord server members, this would allow for searching in "similar" severs, and potentially, mapping discord. [github](https://anvaka.github.io/map-of-github https://anvaka.github.io/map-of-github) [reddit](https://anvaka.github.io/map-of-reddit https://anvaka.github.io/map-of-reddit) -fix doxing. Searching by <@userid> is currently possible. -expect the alternative to the cloudflare captcha to be abused, it's too simple for modern solvers. -open source the stack? I'm interested in the scraper.
belst 1y ago
This clearly breaks discord TOS. I did consent for discord to have my data, I did NOT consent to you having my data. The discord TOS clearly state: > Our services might also provide you with access to other people’s content. You may not use this content without that person’s consent, or as allowed by law. As I was not informed of the usage BEFORE it was taken, I could neither opt in nor opt out. GDPR clearly states, even in the case of "legitimate interest" I have to be informed. I only found this randomly, but if I hadn't, I had no idea of the data validation happening, so I couldn't opt out. Technically, cool project. Legally, not so cool.
- xk_id 1y ago
  Unfortunately TOS consist of words and words cannot constrain technical affordances. There has been a black market of scraped discord data for years (it was even sold on the public web at spy.pet). Stuff like this is probably the only way people will wake up to the realities of digital privacy.
  - squigz 1y ago
    Hopefully this will also wake people up to the issues with putting so much information (announcements, support, documentation, etc) behind a closed platform, thus making efforts like this invaluable for the future.
belst 1y ago
https://support.discord.com/hc/en-us/articles/360039598252-Protecting-Your-Data https://support.discord.com/hc/en-us/articles/360039598252-P... > 3 years ago 100% on their radar. surely
Abishek_Muthian 1y ago
Congratulations and all the best. Could Searchcord API be useful for discord servers which want to archive their chats to their own website? e.g. I have discord server for my product and I want to copy the Q&A threads to FAQ section of my product website will Searchcord be useful for that or are there better solutions?
ternera 1y ago
How does this tool handle it when a user deletes or edits their message? Is the message still logged on your site?
- searchcord 1y ago
  This all depends on when the message is scraped. The scraper does not go back and check for edits/deletions. This is something that while could be added, would be incredibly inefficient.
deleted 1y ago
[deleted]
sadeshmukh 1y ago
This is very much against TOS (selfbots are explicitly against it) and the only way you should be doing this is through the normal bot interface with community consent. Even then you could get in trouble with consent issues.
- woofer21 1y ago
  Even then that is probably against the developer policy & terms > 20. Do not mine or scrape any data, content, or information available on or through Discord services (as defined in our Terms of Service). https://support-dev.discord.com/hc/en-us/articles/8563934450327-Discord-Developer-Policy#h_01HZPPR8CNCG66RDYHK6CCFBDZ https://support-dev.discord.com/hc/en-us/articles/8563934450... (point 20)
rakinar2 1y ago
So basically Spy.pet v2? Hmm. I do see how this might be useful for certain use cases, but I don't like the fact that someone is scraping my messages without my consent. You might tell me oh well I could just opt-out: but what about those who don't even know about this thing? I can already see Discord coming after this. Good luck fighting the legal battle with them.
outrage 1y ago
[dead]
helloanxiety 1y ago
I own one of the servers that you scraped and uploaded without consent. This violates Discord TOS and a whole host of privacy laws and I will be taking action.
- mudkipdev 1y ago
  Did you make an account just to post this
- areyourllySorry 1y ago
  cry lol. discord is a company that doesn't care about you
- DaSHacka 1y ago
  Wait until you find out about all the three-letter agencies and private scrapers doing the same. Probably best to not let it keep you up at night, especially on public servers you yourself explicitly decided to *opt-in* to Discord's 'Discovery' feature, but who am I to say.
someonetwoten 1y ago
[flagged]
QueryThyself 1y ago
This project was (I assume) now taken down after widespread backlash from Discord users who never consented to having their messages scraped, indexed, and published. The opt-out system was not "one click"—it required navigating a server where users were mocked and dismissed for objecting to their inclusion. While the developer claimed to be protecting privacy, the system displayed usernames and full message content from community spaces that were never intended for public indexing. The burden was placed on users to remove themselves from something they never opted into. Discord’s own Developer Policy explicitly forbids scraping or mass data harvesting. This wasn’t innovation—it was exploitation dressed in pseudo-academic language. If you can still search by channel + user ID, that’s traceable content. That's not “privacy preserving,” that’s thinly veiled exposure. The developer's status message (extremely concerning and very inappropriate) and behaviour during take-down tickets further emphasized the lack of empathy behind the project. This wasn’t a public service—it was a boundary violation, and the shutdown was well-earned. I also want to challenge the idea that this tool “solves the problem of not being able to easily search Discord servers.” That’s not a problem—that’s a design choice. Discord isn’t built for global indexing on purpose. Private communities, support groups, fandoms, and sensitive spaces rely on that separation to feel safe. Treating the lack of global search as a bug instead of a boundary shows a complete disregard for how real people use the platform.
- chavez 1y ago
  The "community spaces" opted-in to the discovery platform. It not only requires confirmation from the server owners but also that the server has specific metrics to even be elegible to enter. If the users don't want their messages to be public they should complain to the server owners instead. The problem of "not being able to easily search Discord server" is a real problem. Not too long ago I was unable to find any information about UE4 modding and had to dig deep until I found a few discord servers centered about it. The only place of information aside from two small docs sites.
- trumad 1y ago
  What was the developer's status message? There's no easy way to find it now. Can you share what it was? Thanks!
UbdU 1y ago
[dead]
arvyy 1y ago
hopefully a legal action is taken against such abhorrent project blatantly violating users' privacy
- DaSHacka 1y ago
  and in it's place, rather than a service that was censoring usernames in the interest of privacy being the leading service holding the spotlight, in the future we'll surely get another spy.pet created for the explicit purpose of doxxing getting all the attention and revenue. Round of applause all around, great job everyone! Next we should take down the internet archive, another "abhorrent project blatantly violating users' privacy" for all the forums it's archived over the years. Who knows, maybe there's a post by a gasp 12-year-old in there somewhere! Maybe even a European!
heythere112233 1y ago
I for one appreciate your efforts, and hope that you don't cave to the negative feedback you've gotten. There is no reasonable expectation of privacy in public chatrooms available to anyone, it's odd to hear so many people getting their pants in a bunch over it
basedffl 1y ago
I personally like the idea of this. It affords opportunity like the farms where peoples reprehensible behavior can be documented. I bet the roblox condo bros are really going to be harping on the "I didn't consent to you documenting my public communications!"
godwhite 1y ago
How can i talk to you via dms?
harmlesskid 1y ago
How much storage did you had to use? also are you shutting down already? its a shame that I couldn't find like minded people like me..
pankag13 1y ago
[flagged]
pankag13 1y ago
[flagged]

70 points by searchcord 1y ago

hofrogs 1y ago

This is really cool and actually useful for peeking behind those annoying login walls. What software do you use to store/index/search in so much data? How did you get the data in the first place? Discord isn't exactly known for letting its data be available easily. Have the administrators of the guilds asked you for this? Have you contacted them and made them aware after the fact?

searchcord 1y ago
Hey, Thanks for your feedback. For software, I use ScyllaDB and Elasticsearch. It's split across 6 physical nodes (8 including the CDN). Data collection is handled using standard user accounts, accessing only public, discoverable servers. I plan to write a blog post about the technical aspect of how this was done soon. Admins of these servers weren't contacted, as the content indexed is already publicly accessible, comparable to a forum like this or public subreddit. That said, I understand the sensitivity around data visibility, and I've made it very simple for any user to opt out of indexing at any time. Private or invite-only servers are, of course, completely excluded.
- hofrogs 1y ago
  That's a lot of compute, how much does it cost to keep it running? I don't see how that project would generate any income on its own
  - searchcord 1y ago
    I already own the hardware, so I only pay for colocation and transit. It's probably a lot less than you think. I hope to find some way to monetize it, but it is cheap enough that I can keep it running for quite a long time without any income.
    - hofrogs 1y ago
      Thanks for this. Well good luck with keeping it up, it's a really useful service.
    - uniqueuid 1y ago
      Wow, that must be quite expensive! You said the files alone are a few PB. So at least 2PB / 8 servers ~= 250TB per server, which would probably put each server at > 20k $ (unless you’re putting it together with duct tape and scraps, but even then the disks will cost a ton).
      - searchcord 1y ago
        Hey, Not exactly. Attachments are only fetched from Discord as the user requests them. This means that the vast majority of attachments are never stored on my server. Right now, I only have about 280TB of attachments locally on my own infrastructure. You can see more stats here: https://searchcord.io/about https://searchcord.io/about Thanks for your question!
- klntsky 1y ago
  I suggest you to remove the opt-out functionality and let it scrape private servers that it discovers via publicly posted invite links. You don't owe anyone posting on a public forum any privacy. Moreover, the most valuable data to search for is probably somewhat obscured.

legionof7 1y ago

I've been looking for something like this for so long, thanks for making! There's so much stuff locked in Discord now that forums have fallen in popularity, think this sort of thing really helps unlock that knowledge again.

searchcord 1y ago
Thanks for your feedback! <3

orph 1y ago

Can I download all the messages & attachments?

searchcord 1y ago
Sure, but there's a few petabytes of attachments and over 63 billion messages. Feel free to use the API.

treyd 1y ago

Would you consider making regular dumps of the database available in sharded torrents like Anna's Archive does so that users can back up the data themselves for preservation purposes? This would complicate retroactively removing users' activity, but that data could already be scraped. And related, I'd like to be able to run this locally for exports of guilds that I'm on myself. Is that even possible with the architect you've built?

searchcord 1y ago
Hey, This is absolutely something I want to do, but at the guild level. The database itself is over 13TB which is much to large to create regular exports of. I will probably provide a SQLite export of each guild, regenerated each week/month. Anyone is free to download whatever they want in real time from the API. Thanks for your question!
- squigz 1y ago
  Big +1 for dumps You might try reaching out to Anna's Archive and see if this would be a dataset they'd be interested in helping host/distribute. I think they'd agree that such data is important and should be archived.
  - pilimi_anna 1y ago
    Yes, we'd happily host and torrent this.
- treyd 1y ago
  I was imagining something like a month-by-month historical export so only recent data would need to be regularly exported. SQLite dumps would be awesome to work with though.

ivape 1y ago

Finding good Discord servers has been a great thing for me. I was getting super disconnected and isolated, so different Discord servers has made me feel human again.

searchcord 1y ago
I hope Searchcord helps you! <3

pabs3 1y ago

There are some Discord archives on archive.org too btw. https://archive.org/search?query=subject%3A%22DiscordChatExporter%22 https://archive.org/search?query=subject%3A%22DiscordChatExp... https://archive.org/search?query=subject%3A%22archiveteam_discord%22 https://archive.org/search?query=subject%3A%22archiveteam_di... https://wiki.archiveteam.org/index.php/Discord https://wiki.archiveteam.org/index.php/Discord

searchcord 1y ago
Hey, This is interesting, I somehow missed this. Unfortunately, those are not full text searchable. Maybe I will download them and import them into Searchcord, with proper credit of course. Thanks for this!

johnQdeveloper 1y ago

> This is my first large scale project, so I'd love to hear your feedback! > I have placed restrictions on searching directly by user ID to prevent doxing. I also made the opt out process one click, for those who do not want to be archived. 1) I'd suggest anonymizing the usernames / author ids to something more privacy friendly such as how some image sites were generating 3-4 random words as a human readable unique id. This removes a lot of the reason people would opt out (i.e. posts being tracked down years later) 2) You not seem to have a clear rate limit documentation. If you are asking people to pay for commercial use, I'd suggest making it clear what the rough original limits are as well as the rough price range of what you'd offer. 3) Tbh, the only real thing I want from this project is basically narrative / roleplay / writing content for LLM reasons as I'm trying to build a rules-oriented system that narrates via LLM. If you don't want people using this data for this purpose, I'd suggest making that clear.

searchcord 1y ago
Hey, Thanks for your suggestions. > 1) I'd suggest anonymizing the usernames / author ids to something more privacy friendly such as how some image sites were generating 3-4 random words as a human readable unique id. This removes a lot of the reason people would opt out (i.e. posts being tracked down years later) In the original iteration of Searchcord, it used to work similarly to that. The username was `sha256(userid+guildid)`, truncated to the first 8 characters. Unfortunately, it was pretty hard to follow chats. I will try your idea and see how it works, though. > 2) You not seem to have a clear rate limit documentation. This is a good idea. The rate limit varies by endpoint, and I haven't gotten around to documenting each one. > If you are asking people to pay for commercial use, I'd suggest making it clear what the rough original limits are as well as the rough price range of what you'd offer. I have absolutely zero idea what industry would be interested in this, in what form, and if anyone would even pay. > 3) Tbh, the only real thing I want from this project is basically narrative / roleplay / writing content for LLM reasons as I'm trying to build a rules-oriented system that narrates via LLM. If you don't want people using this data for this purpose, I'd suggest making that clear. I really don't care what people do with the data, as long as they are not spamming requests or using the data for commercial purposes without permission.

cinntaile 1y ago

All discord servers require an invitation link as far as I know, do you consider a link you find online as a public server?

searchcord 1y ago
Some very large servers are eligible for what Discord calls "discovery". This makes their data visible without joining the server. You can find a list of those on Discord's site here: https://discord.com/servers https://discord.com/servers
- nottorp 1y ago
  Suggestion: a bot for smaller servers that do want to be archived like a public forum. Their admins could install the bot themselves and perhaps specify what channels they want archived.
  - pzmarzly 1y ago
    Check out Linen https://www.linen.dev/ https://www.linen.dev/
  - searchcord 1y ago
    This is something I have already completed but have not finished bug testing. The bot also includes functionality to recover any server in case it was nuked/wiped and Searchcord has a backup of it. It uses webhooks to resend the messages so you have an approximation of what the channels used to be.

Stagnant 1y ago

Incredible work! Truly eye-opening to see how some rarer keywords in my native language return pages of relevant results. Meanwhile google gives 0 results or just AI/ad spam.

IceWreck 1y ago

Do you plan to handle servers where you need to do some action (like send a message) to join all channels ? I was scrolling through the home page and came across afew where the only channels you're allowed to access are the verify-yourself or welcome channels.

searchcord 1y ago
Probably not. Discord will aggressively captcha you and every server has a different implementation of verification. It might be possible with a captcha solver and then some LLM to figure out the next steps.

bstsb 1y ago

nice project. how are you going to handle the issues involved with breaking Discord's TOS? > "scraping our services without our written consent" additionally, are these pages indexable? i know of other projects (opt-in) that create pages made from user discussion.

searchcord 1y ago
> how are you going to handle the issues involved with breaking Discord's TOS? Not sure. I will solve that problem if and when Discord takes issue with Searchcord. > additionally, are these pages indexable? Yes, I would actually like for search engines to index it as their search is much more contextually aware than mine.
- woofer21 1y ago
  > Not sure. I will solve that problem if and when Discord takes issue with Searchcord. You are in for such a rude awakening once discord gets to you

EZ-Cheeze 1y ago

I'm in more than a hundred Discord servers. I've been wanting to scrape the members of each of them to discover the people with whom I share the most servers but we're not yet friends. Someone with 10+ would highly likely be a new friend since we'd have a lot of shared niche interests

searchcord 1y ago
This is something I have been trying to make as a way to learn about graph theory. If I can find a way to make it work efficiently, I will definitely add this.
- EZ-Cheeze 1y ago
  Yeah that would be awesome! You could build a whole new social network on top of Discord. Whether in this way or others, I believe we'll all be finding an increasing number of hypercompatible people as technology advances.

3abiton 1y ago

This is an amazing project, I always wonder how much information is lost in those chat apps, not only Discord, but also Telegram. The latter has hude dev community specifically around Android Rom Development, which migrated from forum based XDA to more flexible chat/support platform like Telegram. I wish that also can be searchable without having their client.

searchcord 1y ago
Telegram is already heavily monitored and scraped due to the large volume of illegal or extremely controversial activity that happens there. This is something I will look into though, my XDA threads rarely get any replies anymore. Thanks for the suggestion!

jonasdoesthings 1y ago

Maybe also exclude messages by bots (e.g. "username has joined the server") from the index to decrease the stalking-potential of your site (99.9% of these bot messages have no informational-value for the index anyways). Currently you can still search for an username and get a subset of servers that the username is in (even if not active) by finding these bot messages.

searchcord 1y ago
This is something I have looked into. Unfortunately, every server uses a different format/bot. It might be possible to develop some sort of machine learning classifier to determine if it is a welcome message.

elpocko 1y ago

Any chance for a style option that doesn't have anime girls?

searchcord 1y ago
lol. It's only on the homepage, not on any content pages.

id44037319 1y ago

This is great, I'll be returning to this tool often. Thanks. A few suggestions and ideas for futher projects. -allow for "keyword", -negate operators and "mult word string" searches, [Pubmed](https://pubmed.ncbi.nlm.nih.gov/advanced https://pubmed.ncbi.nlm.nih.gov/advanced) is what I'd consider an Ideal search interface -allow for regex, or direct sql lookups with limited query time ratelimited by POW. for example, if the server is under load, require a token from something like [anubis](https://anubis.techaro.lol/ https://anubis.techaro.lol/) and lower the maximum DB query time -Index the title of all discussion/forum type posts with a VectorDB for semantic search. And add an option to sort by replies. (Like [answer overflow](https://github.com/AnswerOverflow/AnswerOverflow https://github.com/AnswerOverflow/AnswerOverflow))This would make it possible to find relevant discussions among ~60B messages. ScyllaDB doesn't support vector search, so I'd suggest something like [usearch](https://github.com/unum-cloud/usearch https://github.com/unum-cloud/usearch) for a detached index. Embedding models are faster and smaller than most people realize, pick whatever's on top of the [mteb leaderboard](https://huggingface.co/spaces/mteb/leaderboard https://huggingface.co/spaces/mteb/leaderboard) after deciding on size. -calculate the jaccard similarity (user overlap) between discord server members, this would allow for searching in "similar" severs, and potentially, mapping discord. [github](https://anvaka.github.io/map-of-github https://anvaka.github.io/map-of-github) [reddit](https://anvaka.github.io/map-of-reddit https://anvaka.github.io/map-of-reddit) -fix doxing. Searching by <@userid> is currently possible. -expect the alternative to the cloudflare captcha to be abused, it's too simple for modern solvers. -open source the stack? I'm interested in the scraper.

belst 1y ago

This clearly breaks discord TOS. I did consent for discord to have my data, I did NOT consent to you having my data. The discord TOS clearly state: > Our services might also provide you with access to other people’s content. You may not use this content without that person’s consent, or as allowed by law. As I was not informed of the usage BEFORE it was taken, I could neither opt in nor opt out. GDPR clearly states, even in the case of "legitimate interest" I have to be informed. I only found this randomly, but if I hadn't, I had no idea of the data validation happening, so I couldn't opt out. Technically, cool project. Legally, not so cool.

xk_id 1y ago
Unfortunately TOS consist of words and words cannot constrain technical affordances. There has been a black market of scraped discord data for years (it was even sold on the public web at spy.pet). Stuff like this is probably the only way people will wake up to the realities of digital privacy.
- squigz 1y ago
  Hopefully this will also wake people up to the issues with putting so much information (announcements, support, documentation, etc) behind a closed platform, thus making efforts like this invaluable for the future.

belst 1y ago

https://support.discord.com/hc/en-us/articles/360039598252-Protecting-Your-Data https://support.discord.com/hc/en-us/articles/360039598252-P... > 3 years ago 100% on their radar. surely

Abishek_Muthian 1y ago

Congratulations and all the best. Could Searchcord API be useful for discord servers which want to archive their chats to their own website? e.g. I have discord server for my product and I want to copy the Q&A threads to FAQ section of my product website will Searchcord be useful for that or are there better solutions?

ternera 1y ago

How does this tool handle it when a user deletes or edits their message? Is the message still logged on your site?

searchcord 1y ago
This all depends on when the message is scraped. The scraper does not go back and check for edits/deletions. This is something that while could be added, would be incredibly inefficient.

deleted 1y ago

[deleted]

sadeshmukh 1y ago

This is very much against TOS (selfbots are explicitly against it) and the only way you should be doing this is through the normal bot interface with community consent. Even then you could get in trouble with consent issues.

woofer21 1y ago
Even then that is probably against the developer policy & terms > 20. Do not mine or scrape any data, content, or information available on or through Discord services (as defined in our Terms of Service). https://support-dev.discord.com/hc/en-us/articles/8563934450327-Discord-Developer-Policy#h_01HZPPR8CNCG66RDYHK6CCFBDZ https://support-dev.discord.com/hc/en-us/articles/8563934450... (point 20)

rakinar2 1y ago

So basically Spy.pet v2? Hmm. I do see how this might be useful for certain use cases, but I don't like the fact that someone is scraping my messages without my consent. You might tell me oh well I could just opt-out: but what about those who don't even know about this thing? I can already see Discord coming after this. Good luck fighting the legal battle with them.

outrage 1y ago

[dead]

helloanxiety 1y ago

I own one of the servers that you scraped and uploaded without consent. This violates Discord TOS and a whole host of privacy laws and I will be taking action.

mudkipdev 1y ago
Did you make an account just to post this
areyourllySorry 1y ago
cry lol. discord is a company that doesn't care about you
DaSHacka 1y ago
Wait until you find out about all the three-letter agencies and private scrapers doing the same. Probably best to not let it keep you up at night, especially on public servers you yourself explicitly decided to *opt-in* to Discord's 'Discovery' feature, but who am I to say.

someonetwoten 1y ago

[flagged]

QueryThyself 1y ago

This project was (I assume) now taken down after widespread backlash from Discord users who never consented to having their messages scraped, indexed, and published. The opt-out system was not "one click"—it required navigating a server where users were mocked and dismissed for objecting to their inclusion. While the developer claimed to be protecting privacy, the system displayed usernames and full message content from community spaces that were never intended for public indexing. The burden was placed on users to remove themselves from something they never opted into. Discord’s own Developer Policy explicitly forbids scraping or mass data harvesting. This wasn’t innovation—it was exploitation dressed in pseudo-academic language. If you can still search by channel + user ID, that’s traceable content. That's not “privacy preserving,” that’s thinly veiled exposure. The developer's status message (extremely concerning and very inappropriate) and behaviour during take-down tickets further emphasized the lack of empathy behind the project. This wasn’t a public service—it was a boundary violation, and the shutdown was well-earned. I also want to challenge the idea that this tool “solves the problem of not being able to easily search Discord servers.” That’s not a problem—that’s a design choice. Discord isn’t built for global indexing on purpose. Private communities, support groups, fandoms, and sensitive spaces rely on that separation to feel safe. Treating the lack of global search as a bug instead of a boundary shows a complete disregard for how real people use the platform.

chavez 1y ago
The "community spaces" opted-in to the discovery platform. It not only requires confirmation from the server owners but also that the server has specific metrics to even be elegible to enter. If the users don't want their messages to be public they should complain to the server owners instead. The problem of "not being able to easily search Discord server" is a real problem. Not too long ago I was unable to find any information about UE4 modding and had to dig deep until I found a few discord servers centered about it. The only place of information aside from two small docs sites.
trumad 1y ago
What was the developer's status message? There's no easy way to find it now. Can you share what it was? Thanks!

UbdU 1y ago

[dead]

arvyy 1y ago

hopefully a legal action is taken against such abhorrent project blatantly violating users' privacy

DaSHacka 1y ago
and in it's place, rather than a service that was censoring usernames in the interest of privacy being the leading service holding the spotlight, in the future we'll surely get another spy.pet created for the explicit purpose of doxxing getting all the attention and revenue. Round of applause all around, great job everyone! Next we should take down the internet archive, another "abhorrent project blatantly violating users' privacy" for all the forums it's archived over the years. Who knows, maybe there's a post by a gasp 12-year-old in there somewhere! Maybe even a European!

heythere112233 1y ago

I for one appreciate your efforts, and hope that you don't cave to the negative feedback you've gotten. There is no reasonable expectation of privacy in public chatrooms available to anyone, it's odd to hear so many people getting their pants in a bunch over it

basedffl 1y ago

I personally like the idea of this. It affords opportunity like the farms where peoples reprehensible behavior can be documented. I bet the roblox condo bros are really going to be harping on the "I didn't consent to you documenting my public communications!"

godwhite 1y ago

How can i talk to you via dms?

harmlesskid 1y ago

How much storage did you had to use? also are you shutting down already? its a shame that I couldn't find like minded people like me..

pankag13 1y ago

[flagged]

pankag13 1y ago

[flagged]