4 ms·

Fantastic. I wonder how many random technical info is buried in these servers. I hate what it's done for game modding.

by msp26 1y ago

Davidzheng 1y ago
The algebraic topology server probably contains a huge number of treasures in modern research algebraic topology. I really really hope it's archived in full
- DaSHacka 1y ago
  Its not difficult to archive yourself, if you really care[0] I use a dedicated alt account to archive tons of various servers I'm in, and auto-download all attachments. It's nice having regex search capabilities on my local copy of the data too. [0] https://github.com/Tyrrrz/DiscordChatExporter https://github.com/Tyrrrz/DiscordChatExporter
  - judge2020 1y ago
    Using a user account to do this is still considered risky since any automated API usage by a non-bot user is against TOS, and they have heuristics (maybe now ML-based heuristics) for banning accounts for 'things that "don't look like what our official client does"'[0]. 0: https://news.ycombinator.com/item?id=25215415 https://news.ycombinator.com/item?id=25215415
    - DaSHacka 1y ago
      This is why I use a dedicated account to scrape servers, since I unfortunately need my main to interact with(/run) communities unavailable elsewhere. FWIW, I haven't exactly been careful with it (oftentimes scraping 2 servers at once, and downloading all attachments) and have never had an account get banned. The only time I got 'banned' in any capacity was when I hammered the internal JSON API to get information about server's invite links, and even then it was only an automated IP ban from Cloudflare for a couple days. Although, it was an unauthenticated API.
ldoughty 1y ago
I think the average server size here is in the ballpark of 1200 people. These are servers that asked to be advertised by Discord ("Discovery"). These are unlikely to be any kind of servers used for private or even semi-private discussions. You likely don't know most of the people on the server. Most likely, the 'hottest' kind of data you might find is someone accidentally leaking info akin to the World of Tanks forum post 'corrections'.
- giancarlostoro 1y ago
  A fair number of those servers have tens of thousands, if not hundreds of thousands of members. I admin two with over 50 thousand members, both listed in Discord's Server Explorer.
Macha 1y ago
It seems they identified servers via the discovery feature, which servers need to opt into (and I think be recognised as a "community server"? Though that might be out of date). I guess this is better than just scanning the web for invite links, but it does mean that probably most of those game modding servers were not included.
strogonoff 1y ago
Game modding is profitable and people doing it professionally (which they increasingly do) are quite attuned to the fact that making it too accessible would decimate their revenue. As a result, you either pay for the mod (early access, extra content, etc.), or you pay to join some Discord, but ideally you pay for something. Discord, which I generally dislike, is not necessarily the cause of it; if there was no Discord, people would probably use some other closed community platform instead. I expect this would become more widespread as more traditional jobs are subsumed by unregulated ML tech (which, incidentally, the encumbent job-holders are helping train) and more people turn to what used to be generally a hobby as their means of making a living (not that that would last for too long either).
- squigz 1y ago
  > Game modding is profitable It can be. As I understand it, it's sort of like streaming or other content creation - yes, it's possible, but difficult, as it's a saturated market. Most mod authors don't make much money. As a slight aside, I think people would be more inclined to support creators like mod authors if it were simply easier. Patreon and the like make it fairly easy, but I don't think many people want to subscribe to 20+ Patreons for $5 apiece, as much as they might like to support those authors. On the other hand, I think more people would be willing to pledge $X per month to be split among all of their subscriptions. Sure, most creators would only get a few cents per user, but they'd likely get many more people subscribing, and I think it would add up quick. I might be wrong, and I don't take credit for this idea by any means; I read it some time ago, and possibly Patreon even offered this system before?
  - Voxany 1y ago
    [dead]
  - strogonoff 1y ago
    > I don't think many people want to subscribe to 20+ Patreons for $5 apiece, as much as they might like to support those authors I don’t think any bundle will take off. People like getting direct support, and people like giving direct support and getting individual messages in return. (Furthermore, creators know that as soon they help this hypothetical platform get enough traction it will immediately turn and arbitrage against them by paying less and less per user and obscuring the metrics.) There is only a limited number of mods you can play with, and a number of creators you really want to support. Many people have no problem with that number. (Sometimes their parents have, but that’s another matter.) One thing I missed is that in addition to gamers paying for mods or community there is the good old “slap an ad on mod page”. You are right in that you are competing against your fellow creators and that it is a saturated market. That is exactly why if you make something millions of people install you really don’t want to make the knowledge too accessible and spend time on any activity that literally takes food off your table.
nixpulvis 1y ago
I learned programming back in the day on the Tukui (a wow addon) forums. I hate that it's all discord now. Not well searchable and buried info.
hiccuphippo 1y ago
I wonder if LLM companies don't have ways to scrape private Discord servers already. Creating accounts and pulling all the historic data doesn't sound impossible.
- chneu 1y ago
  They absolutely can and are. Multiple posts in here discuss how to do it. It's like back in the days of IRC. People just logged all of it.

Fantastic. I wonder how many random technical info is buried in these servers. I hate what it's done for game modding.

by msp26 1y ago

Fantastic. I wonder how many random technical info is buried in these servers. I hate what it's done for game modding.

The algebraic topology server probably contains a huge number of treasures in modern research algebraic topology. I really really hope it's archived in full

DaSHacka 1y ago
Its not difficult to archive yourself, if you really care[0] I use a dedicated alt account to archive tons of various servers I'm in, and auto-download all attachments. It's nice having regex search capabilities on my local copy of the data too. [0] https://github.com/Tyrrrz/DiscordChatExporter https://github.com/Tyrrrz/DiscordChatExporter
- judge2020 1y ago
  Using a user account to do this is still considered risky since any automated API usage by a non-bot user is against TOS, and they have heuristics (maybe now ML-based heuristics) for banning accounts for 'things that "don't look like what our official client does"'[0]. 0: https://news.ycombinator.com/item?id=25215415 https://news.ycombinator.com/item?id=25215415
  - DaSHacka 1y ago
    This is why I use a dedicated account to scrape servers, since I unfortunately need my main to interact with(/run) communities unavailable elsewhere. FWIW, I haven't exactly been careful with it (oftentimes scraping 2 servers at once, and downloading all attachments) and have never had an account get banned. The only time I got 'banned' in any capacity was when I hammered the internal JSON API to get information about server's invite links, and even then it was only an automated IP ban from Cloudflare for a couple days. Although, it was an unauthenticated API.

ldoughty 1y ago

I think the average server size here is in the ballpark of 1200 people. These are servers that asked to be advertised by Discord ("Discovery"). These are unlikely to be any kind of servers used for private or even semi-private discussions. You likely don't know most of the people on the server. Most likely, the 'hottest' kind of data you might find is someone accidentally leaking info akin to the World of Tanks forum post 'corrections'.

giancarlostoro 1y ago
A fair number of those servers have tens of thousands, if not hundreds of thousands of members. I admin two with over 50 thousand members, both listed in Discord's Server Explorer.

Macha 1y ago

It seems they identified servers via the discovery feature, which servers need to opt into (and I think be recognised as a "community server"? Though that might be out of date). I guess this is better than just scanning the web for invite links, but it does mean that probably most of those game modding servers were not included.

strogonoff 1y ago

Game modding is profitable and people doing it professionally (which they increasingly do) are quite attuned to the fact that making it too accessible would decimate their revenue. As a result, you either pay for the mod (early access, extra content, etc.), or you pay to join some Discord, but ideally you pay for something. Discord, which I generally dislike, is not necessarily the cause of it; if there was no Discord, people would probably use some other closed community platform instead. I expect this would become more widespread as more traditional jobs are subsumed by unregulated ML tech (which, incidentally, the encumbent job-holders are helping train) and more people turn to what used to be generally a hobby as their means of making a living (not that that would last for too long either).

squigz 1y ago
> Game modding is profitable It can be. As I understand it, it's sort of like streaming or other content creation - yes, it's possible, but difficult, as it's a saturated market. Most mod authors don't make much money. As a slight aside, I think people would be more inclined to support creators like mod authors if it were simply easier. Patreon and the like make it fairly easy, but I don't think many people want to subscribe to 20+ Patreons for $5 apiece, as much as they might like to support those authors. On the other hand, I think more people would be willing to pledge $X per month to be split among all of their subscriptions. Sure, most creators would only get a few cents per user, but they'd likely get many more people subscribing, and I think it would add up quick. I might be wrong, and I don't take credit for this idea by any means; I read it some time ago, and possibly Patreon even offered this system before?
- Voxany 1y ago
  [dead]
- strogonoff 1y ago
  > I don't think many people want to subscribe to 20+ Patreons for $5 apiece, as much as they might like to support those authors I don’t think any bundle will take off. People like getting direct support, and people like giving direct support and getting individual messages in return. (Furthermore, creators know that as soon they help this hypothetical platform get enough traction it will immediately turn and arbitrage against them by paying less and less per user and obscuring the metrics.) There is only a limited number of mods you can play with, and a number of creators you really want to support. Many people have no problem with that number. (Sometimes their parents have, but that’s another matter.) One thing I missed is that in addition to gamers paying for mods or community there is the good old “slap an ad on mod page”. You are right in that you are competing against your fellow creators and that it is a saturated market. That is exactly why if you make something millions of people install you really don’t want to make the knowledge too accessible and spend time on any activity that literally takes food off your table.

nixpulvis 1y ago

I learned programming back in the day on the Tukui (a wow addon) forums. I hate that it's all discord now. Not well searchable and buried info.

hiccuphippo 1y ago

I wonder if LLM companies don't have ways to scrape private Discord servers already. Creating accounts and pulling all the historic data doesn't sound impossible.

chneu 1y ago
They absolutely can and are. Multiple posts in here discuss how to do it. It's like back in the days of IRC. People just logged all of it.