4 ms·

> This is my first large scale project, so I'd love to hear your feedback! > I have placed restrictions on searching directly by user ID to prevent doxing. I a

> This is my first large scale project, so I'd love to hear your feedback! > I have placed restrictions on searching directly by user ID to prevent doxing. I also made the opt out process one click, for those who do not want to be archived. 1) I'd suggest anonymizing the usernames / author ids to something more privacy friendly such as how some image sites were generating 3-4 random words as a human readable unique id. This removes a lot of the reason people would opt out (i.e. posts being tracked down years later) 2) You not seem to have a clear rate limit documentation. If you are asking people to pay for commercial use, I'd suggest making it clear what the rough original limits are as well as the rough price range of what you'd offer. 3) Tbh, the only real thing I want from this project is basically narrative / roleplay / writing content for LLM reasons as I'm trying to build a rules-oriented system that narrates via LLM. If you don't want people using this data for this purpose, I'd suggest making that clear.

searchcord 1y ago
Hey, Thanks for your suggestions. > 1) I'd suggest anonymizing the usernames / author ids to something more privacy friendly such as how some image sites were generating 3-4 random words as a human readable unique id. This removes a lot of the reason people would opt out (i.e. posts being tracked down years later) In the original iteration of Searchcord, it used to work similarly to that. The username was `sha256(userid+guildid)`, truncated to the first 8 characters. Unfortunately, it was pretty hard to follow chats. I will try your idea and see how it works, though. > 2) You not seem to have a clear rate limit documentation. This is a good idea. The rate limit varies by endpoint, and I haven't gotten around to documenting each one. > If you are asking people to pay for commercial use, I'd suggest making it clear what the rough original limits are as well as the rough price range of what you'd offer. I have absolutely zero idea what industry would be interested in this, in what form, and if anyone would even pay. > 3) Tbh, the only real thing I want from this project is basically narrative / roleplay / writing content for LLM reasons as I'm trying to build a rules-oriented system that narrates via LLM. If you don't want people using this data for this purpose, I'd suggest making that clear. I really don't care what people do with the data, as long as they are not spamming requests or using the data for commercial purposes without permission.
- sReinwald 1y ago
  The sheer audacity here is quite something. You're stating people can't use your scraped data for commercial purposes "without permission," while your entire project is built on vacuuming up content from countless users without their permission, and in direct violation of Discord's ToS. That's not just a double standard; it's bordering on next-level cognitive dissonance. And "privacy preserving"? With a one-click opt-out, that 99.999% of the affected users will never even know exists because they have no idea their conversations are now part of your archive, and you want it indexed by search engines? That's not "privacy preserving" - that's a bad joke. If privacy was a genuine concern, this project wouldn't exist in its current form. What you're offering is an opt-out fig leaf for a mass data harvesting operation. Most people using Discord, even on "public, discoverable" servers, aren't posting with the expectation that their words will be systematically scraped, archived indefinitely, and made globally searchable outside the platform's context. It's a fundamental misunderstanding (or willful dismissal) of user expectations on what is essentially a semi-public, yet distinctly siloed, platform. This isn't an open-web forum where content is implicitly intended for broad public consumption and indexing. Look, I get the frustration that (likely) motivated this. Discord has become an information black hole for many communities, and the shift away from open, searchable forums for project support is a genuine problem I've been incredibly frustrated with myself. But this "solution" - creating a massive, non-consensual archive that tramples over user privacy (and platform terms) - creates far graver ethical and practical issues than the one it purports to solve.
  - deakam 1y ago
    Ridiculous take. If you're posting in a server that's intentionally open to the public and accessible to anyone with a link or even indexed by server discovery you shouldn't expect privacy. That's just the basic reality of the internet.
    - sReinwald 1y ago
      No, what's "ridiculous" is this simplistic, black-and-white framing that deliberately ignores any nuance, the concept of contextual integrity or reasonable user expectations. Of course, no one expects absolute secrecy in a public-facing Discord server. That's a straw man. The issue isn't about some naive belief that messages are invisible. It's about the scope, permanence, and method of access and archiving. People participating in public Discord spaces have reasonable contextual expectations about how their words will be accessed and by whom. They expect their messages to be seen by current and maybe future server members - not extracted, permanently archived, and made globally searchable by entirely unrelated third parties. This is similar to how conversations in a public park are technically "public," but most people would be rightfully disturbed if someone recorded everything, transcribed it, published it online with their names attached, and made it all searchable forever. Just because something isn't strictly private doesn't mean any and all forms of collection, republication, and indexing are ethically justified. If you can't see the distinction between "not perfectly private within this specific semi-public space" and "archived indefinitely, and globally searchable forever by anyone, anywhere, for any reason," then you're either arguing in bad faith or your understanding of these issues is so superficial that further engagement is pointless.

searchcord 1y ago

Hey, Thanks for your suggestions. > 1) I'd suggest anonymizing the usernames / author ids to something more privacy friendly such as how some image sites were generating 3-4 random words as a human readable unique id. This removes a lot of the reason people would opt out (i.e. posts being tracked down years later) In the original iteration of Searchcord, it used to work similarly to that. The username was `sha256(userid+guildid)`, truncated to the first 8 characters. Unfortunately, it was pretty hard to follow chats. I will try your idea and see how it works, though. > 2) You not seem to have a clear rate limit documentation. This is a good idea. The rate limit varies by endpoint, and I haven't gotten around to documenting each one. > If you are asking people to pay for commercial use, I'd suggest making it clear what the rough original limits are as well as the rough price range of what you'd offer. I have absolutely zero idea what industry would be interested in this, in what form, and if anyone would even pay. > 3) Tbh, the only real thing I want from this project is basically narrative / roleplay / writing content for LLM reasons as I'm trying to build a rules-oriented system that narrates via LLM. If you don't want people using this data for this purpose, I'd suggest making that clear. I really don't care what people do with the data, as long as they are not spamming requests or using the data for commercial purposes without permission.

sReinwald 1y ago
The sheer audacity here is quite something. You're stating people can't use your scraped data for commercial purposes "without permission," while your entire project is built on vacuuming up content from countless users without their permission, and in direct violation of Discord's ToS. That's not just a double standard; it's bordering on next-level cognitive dissonance. And "privacy preserving"? With a one-click opt-out, that 99.999% of the affected users will never even know exists because they have no idea their conversations are now part of your archive, and you want it indexed by search engines? That's not "privacy preserving" - that's a bad joke. If privacy was a genuine concern, this project wouldn't exist in its current form. What you're offering is an opt-out fig leaf for a mass data harvesting operation. Most people using Discord, even on "public, discoverable" servers, aren't posting with the expectation that their words will be systematically scraped, archived indefinitely, and made globally searchable outside the platform's context. It's a fundamental misunderstanding (or willful dismissal) of user expectations on what is essentially a semi-public, yet distinctly siloed, platform. This isn't an open-web forum where content is implicitly intended for broad public consumption and indexing. Look, I get the frustration that (likely) motivated this. Discord has become an information black hole for many communities, and the shift away from open, searchable forums for project support is a genuine problem I've been incredibly frustrated with myself. But this "solution" - creating a massive, non-consensual archive that tramples over user privacy (and platform terms) - creates far graver ethical and practical issues than the one it purports to solve.
- deakam 1y ago
  Ridiculous take. If you're posting in a server that's intentionally open to the public and accessible to anyone with a link or even indexed by server discovery you shouldn't expect privacy. That's just the basic reality of the internet.
  - sReinwald 1y ago
    No, what's "ridiculous" is this simplistic, black-and-white framing that deliberately ignores any nuance, the concept of contextual integrity or reasonable user expectations. Of course, no one expects absolute secrecy in a public-facing Discord server. That's a straw man. The issue isn't about some naive belief that messages are invisible. It's about the scope, permanence, and method of access and archiving. People participating in public Discord spaces have reasonable contextual expectations about how their words will be accessed and by whom. They expect their messages to be seen by current and maybe future server members - not extracted, permanently archived, and made globally searchable by entirely unrelated third parties. This is similar to how conversations in a public park are technically "public," but most people would be rightfully disturbed if someone recorded everything, transcribed it, published it online with their names attached, and made it all searchable forever. Just because something isn't strictly private doesn't mean any and all forms of collection, republication, and indexing are ethically justified. If you can't see the distinction between "not perfectly private within this specific semi-public space" and "archived indefinitely, and globally searchable forever by anyone, anywhere, for any reason," then you're either arguing in bad faith or your understanding of these issues is so superficial that further engagement is pointless.