11 ms·

Every time I see social media recreated in a decentralised fashion I realise why the successful ones are centralised. The loading times of this are dreadful and

by dom96 3y ago

Every time I see social media recreated in a decentralised fashion I realise why the successful ones are centralised. The loading times of this are dreadful and I don't see how you'll be able to solve that problem when you're relying on P2P. In any case, it sounds like we'll be getting many new Reddit clones so I may as well plug what I've built recently: a Reddit alternative API that is free. Check it out if you're a dev that is being caught out by the Reddit pricing changes[1]. 1 - https://api.reddiw.com https://api.reddiw.com

smt88 3y ago
> The loading times of this are dreadful and I don't see how you'll be able to solve that problem when you're relying on P2P. It's easy enough to centralize a caching layer, perhaps through a popular CDN.
- all2 3y ago
  Why not a bittorrent style peering mechanism? Popular content will always have pools of people who are consuming it. I've had an idea in the back of my head for what would make a decentralized video service work, and it plays with the idea of peer driven networks. So, there a few kinds of people on the network: advertisers (hang in there, I'll explain why), leechers (or consumer), content producers, and seeders. The prime relationship is between a consumer and a content producer. Every other part of the network is designed to support and benefit their interaction. First, a content producer creates a piece of content (or steals it, let's be realistic) and publishes it on the network. Attached to the content is a pay-per-view requirement; the producer has elected to require payment to view this content. This money can come from one of two places: first, an advertiser can bid on slots the producer has chosen, second, the consumer can pay the producer directly. The producer doesn't have a choice whether the consumer pays or not, only that payment is required. Requiring payment for the content is optional, but advisable. You see, the producer has a second order relationship with the seeders. The producer can't possibly feed a million viewers in the first hour. That would be a very difficult task. But what a producer can do is elect to provide a percentage of their required income to seeders. Now there's a financial benefit to seeding a popular video/song. The producer can set their payments to whatever they want. The market may equalize itself. Finally, we have advertisers, which are the monetary fuel for the fire. Most viewers will tolerate ads and wont want to pay money; so instead they pay time. An advertiser can bid on slots in as precise or general way as they want. If they want to bid on a specific creator's video slots, they can. If they want to bid on a broad category of creators or videos, they can. Etc. This won't ever be truly decentralized, and there is a whole ecosystem of services external to the network that would be needed for this to function well (payment processors, content collation and moderation -- kind of like how mastodon works, a bidding marketplace for advertisers to interface with the content network, a front end for seeders to select and re-seed, and so on). I don't have the technical hutzpah to build out something like this. I wish I did. Is there someone out there who wants to be a technical mentor? I'd be game. -- [edit] There should be more and less granularity in how different parties engage. Some people want to pay 10 bucks a month (Amazon Prime) and forget about the rest. Some people want to pay per view. Advertisers might want to target specific creator's channels or whole segments of created content. Creators might want to be able to never even think about ads and just check a box that says "put an ad in front of every video I upload" and then forget it. Or a creator might want to put ads in videos longer than a certain length. And so on. What I've specified above is loosely conceived and would need a lot of thought to turn it into something useful.
  - dvt 3y ago
    You're just describing YouTube with extra steps. Decentralization is a meme, the great majority of consumers don't care. People used Morpheus because it was an easy way to watch (free) movies, not because it was decentralized. Now they use Netflix because it's an easy (and relatively cheap) way to watch movies, not because it's centralized. Besides, Matrix already has p2p chat, IRC has always been a thing, and so on, heck even BitTorrent is technically still alive. (De)centralization is not a feature, nor is it a bug, it's simply irrelevant.
    - all2 3y ago
      Hopefully, for the consumer it would be no different than YouTube or Netflix. Unless you don't want ads, then you pay your dues. I have no idea how the experience would vary for the advertisers. For the producer it should offer more flexibility in how they get paid. This is the major selling point I think. The ability to put out a high effort video -- say you made a full length movie -- and be able to charge theater prices (17.50USD, or close to it) and make sure the ads hit at the appropriate times? It might be worth it for larger creators to swap over.
      - warkdarrior 3y ago
        Wow, as a consumer I am really sold on this model! You tell me I will pay theater prices and get ad interruptions throughout the show? This is fantastic, sign me up. /s
        dustymcp 3y ago
        haha this is what im thinking, where is the incentive to actually use it.
        all2 3y ago
        I apologize. I guess I miscommunicated something. I mean that a creator can opt to charge what they want and also turn off ads if they choose. Or front load the ads, like theaters do. Really, it is up to the creator how they want to format their content.
- cortesoft 3y ago
  Caching is not easy for something like Reddit, where everyone has a personalized feed.
  - 101011 3y ago
    Couldn't you say the same of Mastodon?
    - Maxion 3y ago
      Mastodon is not P2P
  - hanniabu 3y ago
    Why not just cache sub feeds?
thefourthchime 3y ago
How does your non official API work? Is it using the official API and caching, redirecting to the website, and transforming it into json? I'm just curious.
- dom96 3y ago
  It basically scrapes reddit.com's HTML. Doesn't use the official API at all (that would defeat the purpose).
  - programmarchy 3y ago
    Impressive feat! Does reddit have rate limiting, or other hurdles in place similar to the hoops youtube-dl has to jump through? Curious what your thoughts are about maintaining a project like that.
    - sam0x17 3y ago
      As history has shown, you can only do so much to stop this. If you perfectly mimic the GoogleBot and use google IP ranges by hosting on google cloud, they either take an SEO hit or let you bot them at the end of the day. GoogleBot looks like a DDoS attack a lot of the time too You can also go the route of looking like a pool of users, then it's just a game of cat and mouse and one providers don't really have time to play
      - thih9 3y ago
        > As history has shown, you can only do so much to stop this. History has shown you can stop this well enough. Try accessing e.g. instagram; bibliogram attempted this, the project is now discontinued.
        sam0x17 3y ago
        This is true in the case of content that doesn't care about SEO. Reddit cares very very much about SEO, so it can never truly block bots.
ranger_danger 3y ago
How will you deal with the huge onslaught of abuse?
bmcclure 3y ago
Seems like scraping and providing an alternative API would violate TOS and will receive a takedown notice in due time.
- realusername 3y ago
  You don't need to agree to TOS to scrape a public website, what they write in it is moot.
  - midoridensha 3y ago
    Wasn't there a recent SCOTUS case that upheld the need to follow TOS?
  - deleted 3y ago
    [deleted]
  - Gasp0de 3y ago
    They can still ban your service, e.g. block your IPs. Sure, you could then play a cat and mouse game with them, but your API clone going down every few days will make it unusable for anyone with a real purpose. The companies that reddit (officially) wants to target, e.g. OpenAI et al., have no problem scraping.
    - realusername 3y ago
      You can just host on Gcloud and report as GoogleBot, good luck with that. There's also plenty of proxy services. > The companies that reddit (officially) wants to target, e.g. OpenAI et al., have no problem scraping. Indeed, on their case they just need the datasets anyways
  - thih9 3y ago
    Reddit may not be a public website for long. They push their app and ask for logging in so often, it wouldn’t surprise me if they shut off parts of the site for not logged in users.
shakna 3y ago
I don't really find Mastodon that slow. I do think it's got a lot, lot of sharp edges that need smoothing out before it's really viable. But speed isn't a problem, and it's decentralised.
- warkdarrior 3y ago
  Mastodon is not P2P/decentralized, it's federated.
  - dumpsterlid 3y ago
    [dead]
  - bheadmaster 3y ago
    > Mastodon is not P2P/decentralized Mastodon is decentralized, because there is no central server. However, it is not distributed.
    - deleted 3y ago
      [deleted]
    - nine_k 3y ago
      It is both decentralized and distributed. It's not P2P though, it's a hub-and-spokes architecture with several interconnected (federated) hubs. Hubs can have decent uptime and thick internet connections. Hubs are relatively few so they replicate state quickly between themselves, and then more locally between spokes when they come online. Most natural structures are more tree-form, like that.
      - bheadmaster 3y ago
        That just depends on how you define "distributed". If it means "running on multiple machines", then even centralized protocols are distributed, since a part of the computation is running on your computer. In context of protocols like Mastodon, if the end-user devices aren't primary holders of data, then I don't call it distributed. It's just decentralized. I guess that way, "distributed" necessarily implies peer-to-peer.
WatchDog 3y ago
Is the reddiw code open source? I imagine if your project sees any substantial usage, it won't be too long before you receive a cease and desist.
- dom96 3y ago
  Not at the minute. I might open source it in the future, especially if I get a cease and desist. Though hopefully even if it comes to that someone like the EFF will help me fight it.
  - drbscl 3y ago
    Not a lawyer, but it's probably a bad idea to open-source something _after_ getting a C&D for it
    - dom96 3y ago
      Yeah, true. I'd definitely ask for legal advice before doing that.
Semaphor 3y ago
So like libreddit (sp?) and teddit, another readonly scraper, the real API supports write access.
- dom96 3y ago
  Yep, very similar. My guess is that ~80% of API usage is read-only, so covering this should reduce API costs significantly for developers.
hanniabu 3y ago
For the GET /r/:subreddit/comments/:article endpoint, is there a limit to the amount of comments returned? If yes, how is it determined which get returned? Sorted by top? Best?
- dom96 3y ago
  Yeah, currently the limit is 50 comments. I plan to implement the ability to choose a sort and limit, assuming the API actually gets some usage. Right now the sort is whatever the new.reddit.com default is.
unlog 3y ago
When visiting the link, chrome just suggested I may mean to visit "reddit", never seen that before. I have safe browsing disabled, so I'm at a lost from where this comes from. https://i.imgur.com/l57osTi.png https://i.imgur.com/l57osTi.png
- dsab 3y ago
  clearly an anti phishing filter
darkstar_16 3y ago
So are you scraping reddit behind this ?
- dom96 3y ago
  yep
magpi3 3y ago
This may be a very, very naive question but: is it possible to have a decentralized reddit that has centralized servers that cache the data?
moneywoes 3y ago
Is this open source? I’d love to help out
- interestom 3y ago
  github.com/plebbit Lots of repos, everything is open-source

by dom96 3y ago

smt88 3y ago

> The loading times of this are dreadful and I don't see how you'll be able to solve that problem when you're relying on P2P. It's easy enough to centralize a caching layer, perhaps through a popular CDN.

all2 3y ago
Why not a bittorrent style peering mechanism? Popular content will always have pools of people who are consuming it. I've had an idea in the back of my head for what would make a decentralized video service work, and it plays with the idea of peer driven networks. So, there a few kinds of people on the network: advertisers (hang in there, I'll explain why), leechers (or consumer), content producers, and seeders. The prime relationship is between a consumer and a content producer. Every other part of the network is designed to support and benefit their interaction. First, a content producer creates a piece of content (or steals it, let's be realistic) and publishes it on the network. Attached to the content is a pay-per-view requirement; the producer has elected to require payment to view this content. This money can come from one of two places: first, an advertiser can bid on slots the producer has chosen, second, the consumer can pay the producer directly. The producer doesn't have a choice whether the consumer pays or not, only that payment is required. Requiring payment for the content is optional, but advisable. You see, the producer has a second order relationship with the seeders. The producer can't possibly feed a million viewers in the first hour. That would be a very difficult task. But what a producer can do is elect to provide a percentage of their required income to seeders. Now there's a financial benefit to seeding a popular video/song. The producer can set their payments to whatever they want. The market may equalize itself. Finally, we have advertisers, which are the monetary fuel for the fire. Most viewers will tolerate ads and wont want to pay money; so instead they pay time. An advertiser can bid on slots in as precise or general way as they want. If they want to bid on a specific creator's video slots, they can. If they want to bid on a broad category of creators or videos, they can. Etc. This won't ever be truly decentralized, and there is a whole ecosystem of services external to the network that would be needed for this to function well (payment processors, content collation and moderation -- kind of like how mastodon works, a bidding marketplace for advertisers to interface with the content network, a front end for seeders to select and re-seed, and so on). I don't have the technical hutzpah to build out something like this. I wish I did. Is there someone out there who wants to be a technical mentor? I'd be game. -- [edit] There should be more and less granularity in how different parties engage. Some people want to pay 10 bucks a month (Amazon Prime) and forget about the rest. Some people want to pay per view. Advertisers might want to target specific creator's channels or whole segments of created content. Creators might want to be able to never even think about ads and just check a box that says "put an ad in front of every video I upload" and then forget it. Or a creator might want to put ads in videos longer than a certain length. And so on. What I've specified above is loosely conceived and would need a lot of thought to turn it into something useful.
- dvt 3y ago
  You're just describing YouTube with extra steps. Decentralization is a meme, the great majority of consumers don't care. People used Morpheus because it was an easy way to watch (free) movies, not because it was decentralized. Now they use Netflix because it's an easy (and relatively cheap) way to watch movies, not because it's centralized. Besides, Matrix already has p2p chat, IRC has always been a thing, and so on, heck even BitTorrent is technically still alive. (De)centralization is not a feature, nor is it a bug, it's simply irrelevant.
  - all2 3y ago
    Hopefully, for the consumer it would be no different than YouTube or Netflix. Unless you don't want ads, then you pay your dues. I have no idea how the experience would vary for the advertisers. For the producer it should offer more flexibility in how they get paid. This is the major selling point I think. The ability to put out a high effort video -- say you made a full length movie -- and be able to charge theater prices (17.50USD, or close to it) and make sure the ads hit at the appropriate times? It might be worth it for larger creators to swap over.
    - warkdarrior 3y ago
      Wow, as a consumer I am really sold on this model! You tell me I will pay theater prices and get ad interruptions throughout the show? This is fantastic, sign me up. /s
      - dustymcp 3y ago
        haha this is what im thinking, where is the incentive to actually use it.
      - all2 3y ago
        I apologize. I guess I miscommunicated something. I mean that a creator can opt to charge what they want and also turn off ads if they choose. Or front load the ads, like theaters do. Really, it is up to the creator how they want to format their content.
cortesoft 3y ago
Caching is not easy for something like Reddit, where everyone has a personalized feed.
- 101011 3y ago
  Couldn't you say the same of Mastodon?
  - Maxion 3y ago
    Mastodon is not P2P
- hanniabu 3y ago
  Why not just cache sub feeds?

thefourthchime 3y ago

How does your non official API work? Is it using the official API and caching, redirecting to the website, and transforming it into json? I'm just curious.

dom96 3y ago
It basically scrapes reddit.com's HTML. Doesn't use the official API at all (that would defeat the purpose).
- programmarchy 3y ago
  Impressive feat! Does reddit have rate limiting, or other hurdles in place similar to the hoops youtube-dl has to jump through? Curious what your thoughts are about maintaining a project like that.
  - sam0x17 3y ago
    As history has shown, you can only do so much to stop this. If you perfectly mimic the GoogleBot and use google IP ranges by hosting on google cloud, they either take an SEO hit or let you bot them at the end of the day. GoogleBot looks like a DDoS attack a lot of the time too You can also go the route of looking like a pool of users, then it's just a game of cat and mouse and one providers don't really have time to play
    - thih9 3y ago
      > As history has shown, you can only do so much to stop this. History has shown you can stop this well enough. Try accessing e.g. instagram; bibliogram attempted this, the project is now discontinued.
      - sam0x17 3y ago
        This is true in the case of content that doesn't care about SEO. Reddit cares very very much about SEO, so it can never truly block bots.

ranger_danger 3y ago

How will you deal with the huge onslaught of abuse?

bmcclure 3y ago

Seems like scraping and providing an alternative API would violate TOS and will receive a takedown notice in due time.

realusername 3y ago
You don't need to agree to TOS to scrape a public website, what they write in it is moot.
- midoridensha 3y ago
  Wasn't there a recent SCOTUS case that upheld the need to follow TOS?
- deleted 3y ago
  [deleted]
- Gasp0de 3y ago
  They can still ban your service, e.g. block your IPs. Sure, you could then play a cat and mouse game with them, but your API clone going down every few days will make it unusable for anyone with a real purpose. The companies that reddit (officially) wants to target, e.g. OpenAI et al., have no problem scraping.
  - realusername 3y ago
    You can just host on Gcloud and report as GoogleBot, good luck with that. There's also plenty of proxy services. > The companies that reddit (officially) wants to target, e.g. OpenAI et al., have no problem scraping. Indeed, on their case they just need the datasets anyways
- thih9 3y ago
  Reddit may not be a public website for long. They push their app and ask for logging in so often, it wouldn’t surprise me if they shut off parts of the site for not logged in users.

shakna 3y ago

I don't really find Mastodon that slow. I do think it's got a lot, lot of sharp edges that need smoothing out before it's really viable. But speed isn't a problem, and it's decentralised.

warkdarrior 3y ago
Mastodon is not P2P/decentralized, it's federated.
- dumpsterlid 3y ago
  [dead]
- bheadmaster 3y ago
  > Mastodon is not P2P/decentralized Mastodon is decentralized, because there is no central server. However, it is not distributed.
  - deleted 3y ago
    [deleted]
  - nine_k 3y ago
    It is both decentralized and distributed. It's not P2P though, it's a hub-and-spokes architecture with several interconnected (federated) hubs. Hubs can have decent uptime and thick internet connections. Hubs are relatively few so they replicate state quickly between themselves, and then more locally between spokes when they come online. Most natural structures are more tree-form, like that.
    - bheadmaster 3y ago
      That just depends on how you define "distributed". If it means "running on multiple machines", then even centralized protocols are distributed, since a part of the computation is running on your computer. In context of protocols like Mastodon, if the end-user devices aren't primary holders of data, then I don't call it distributed. It's just decentralized. I guess that way, "distributed" necessarily implies peer-to-peer.

WatchDog 3y ago

Is the reddiw code open source? I imagine if your project sees any substantial usage, it won't be too long before you receive a cease and desist.

dom96 3y ago
Not at the minute. I might open source it in the future, especially if I get a cease and desist. Though hopefully even if it comes to that someone like the EFF will help me fight it.
- drbscl 3y ago
  Not a lawyer, but it's probably a bad idea to open-source something _after_ getting a C&D for it
  - dom96 3y ago
    Yeah, true. I'd definitely ask for legal advice before doing that.

Semaphor 3y ago

So like libreddit (sp?) and teddit, another readonly scraper, the real API supports write access.

dom96 3y ago
Yep, very similar. My guess is that ~80% of API usage is read-only, so covering this should reduce API costs significantly for developers.

hanniabu 3y ago

For the GET /r/:subreddit/comments/:article endpoint, is there a limit to the amount of comments returned? If yes, how is it determined which get returned? Sorted by top? Best?

dom96 3y ago
Yeah, currently the limit is 50 comments. I plan to implement the ability to choose a sort and limit, assuming the API actually gets some usage. Right now the sort is whatever the new.reddit.com default is.

unlog 3y ago

When visiting the link, chrome just suggested I may mean to visit "reddit", never seen that before. I have safe browsing disabled, so I'm at a lost from where this comes from. https://i.imgur.com/l57osTi.png https://i.imgur.com/l57osTi.png

dsab 3y ago
clearly an anti phishing filter

darkstar_16 3y ago

So are you scraping reddit behind this ?

dom96 3y ago
yep

magpi3 3y ago

This may be a very, very naive question but: is it possible to have a decentralized reddit that has centralized servers that cache the data?

moneywoes 3y ago

Is this open source? I’d love to help out

interestom 3y ago
github.com/plebbit Lots of repos, everything is open-source