3 ms·

I find it baffling that the difference in cost of serving 41GB/day vs 232GB/day is worth spending any dev time on. We're talking about a whopping 21.4Mbps on a

by bcrl 2y ago

I find it baffling that the difference in cost of serving 41GB/day vs 232GB/day is worth spending any dev time on. We're talking about a whopping 21.4Mbps on average, which costs me roughly CAD$3.76/month in transit (and my transit costs are about to be cut in half for 2 x 10Gbps links thanks to contracts being up and the market being very competitive). 1 hour of dev time is upwards of 2 years of bandwidth usage at that rate.

ericvolp12 2y ago
The current relay firehose has more than 250 subscribers. It's served more than 8.5Gbps in real-world peak traffic sustained for ~12 hours a day. That being said, Jetstream is a lot more friendly for devs to get started with consuming than the full protocol firehose, and helps grow the ecosystem of cool projects people build on the open network. Also, this was a fun thing I built mostly in my free time :)
- tecleandor 2y ago
  Also it's just not those concrete 190GBs a day. It's the 6x traffic you can fit in the same pipe :D
  - ericvolp12 2y ago
    Yeah exactly! The longer we can make it without having to shard the firehose, the better. It's a lot less complex to consume as a single stream.
    - tecleandor 2y ago
      Also, being some sort of streaming firehose with the same data for everyone (if I understood what this is in a quick read) I guess if it makes any sense to do any kind of P2P/distributed transfer of it to ease the load...
wkat4242 2y ago
It's a benefit on the receiving side too. And it has ecological benefits. Nothing to sneeze at.
steveklabnik 2y ago
A thing I have admired about the BlueSky development team is that they're always thinking about the future, not just the present. One area in which this is true is the system design: they explicitly considered how they would scale. Sure, at the current cost, this may not be worth it, but as BlueSky continues to grow, work like this will be more and more important. Also, it's just a good look towards the community. Remember, ideally not all of ATProto's infrastructure is run by BlueSky themselves, but by a diverse set of folks who want to be able to control their own data. These are more likely to be individuals or small groups, not capitalized startups. While the protocol itself is designed to be able to do this, "what happens when BlueSky is huge and running your own infra is too expensive" is a current question that some folks are reasonable skeptical about: it's no good being a federated protocol if nobody can afford to run a node. By doing stuff like this, the BlueSky team is signaling that they're aware of and responsive to these concerns, so don't look at it as trying to save money on bandwidth: look at it as a savvy way to generate some good marketing. That's my take anyway. EDIT: scrolling down, you can see this kind of sentiment here: https://news.ycombinator.com/item?id=41637307 https://news.ycombinator.com/item?id=41637307

by bcrl 2y ago

ericvolp12 2y ago

The current relay firehose has more than 250 subscribers. It's served more than 8.5Gbps in real-world peak traffic sustained for ~12 hours a day. That being said, Jetstream is a lot more friendly for devs to get started with consuming than the full protocol firehose, and helps grow the ecosystem of cool projects people build on the open network. Also, this was a fun thing I built mostly in my free time :)

tecleandor 2y ago
Also it's just not those concrete 190GBs a day. It's the 6x traffic you can fit in the same pipe :D
- ericvolp12 2y ago
  Yeah exactly! The longer we can make it without having to shard the firehose, the better. It's a lot less complex to consume as a single stream.
  - tecleandor 2y ago
    Also, being some sort of streaming firehose with the same data for everyone (if I understood what this is in a quick read) I guess if it makes any sense to do any kind of P2P/distributed transfer of it to ease the load...

wkat4242 2y ago

It's a benefit on the receiving side too. And it has ecological benefits. Nothing to sneeze at.

steveklabnik 2y ago

A thing I have admired about the BlueSky development team is that they're always thinking about the future, not just the present. One area in which this is true is the system design: they explicitly considered how they would scale. Sure, at the current cost, this may not be worth it, but as BlueSky continues to grow, work like this will be more and more important. Also, it's just a good look towards the community. Remember, ideally not all of ATProto's infrastructure is run by BlueSky themselves, but by a diverse set of folks who want to be able to control their own data. These are more likely to be individuals or small groups, not capitalized startups. While the protocol itself is designed to be able to do this, "what happens when BlueSky is huge and running your own infra is too expensive" is a current question that some folks are reasonable skeptical about: it's no good being a federated protocol if nobody can afford to run a node. By doing stuff like this, the BlueSky team is signaling that they're aware of and responsive to these concerns, so don't look at it as trying to save money on bandwidth: look at it as a savvy way to generate some good marketing. That's my take anyway. EDIT: scrolling down, you can see this kind of sentiment here: https://news.ycombinator.com/item?id=41637307 https://news.ycombinator.com/item?id=41637307