6 ms·

Just did the calc and 600GB/day is about 55Mbit/s. That's really not a lot and if there's not too much computation server-side you could serve this from a raspb

Just did the calc and 600GB/day is about 55Mbit/s. That's really not a lot and if there's not too much computation server-side you could serve this from a raspberry pi at home (provided you have good uplink). But that's assuming you keep the CloudFlare cache of course, or as author mentioned himself, advertising only torrents for the multi-gig files. I really don't understand the cloud craze. Everything is more complex to debug, more expensive, and more shitty in all the possible ways you can imagine. I mean i was not exactly a fan of the VPS craze 10-15 years ago, but at least it wouldn't automatically ruin your bank account whenever you got a little traffic. Kudos to the author for having so much money (thousands in one month?!) to waste. I wish i did too :)

SkipperCat 5y ago
The cloud is great for scaling. The lead time for new servers deployed in a data center is weeks compared to seconds in the cloud. Plus there's no sunk cost in the cloud - you can turn it off when done and it evaporates. Also, the cloud offers managed software as a service. You don't have to manage your own HA DB cluster or PubSub. It's all just there and it works. That can save you a lot on technical labor costs. But yes, I do agree with your point. If you don't know what you're doing, you can nuke your budget super quick.
- immibis 5y ago
  The technical ability to scale is a bit meaningless if you can't afford it.
  - alexpotato 5y ago
    Then that becomes part of your business + technology planning conversations: "This is the cost of scaling, this is the cost of owning our own infra, how does that fit into our budgeting and requirements?"
  - eropple 5y ago
    "If you can't afford it" is doing a lot of assuming and a lot of heavy lifting in that statement. Whether or not you can afford it depends strongly on your scaling bounds (how much you need to scale) and how you've chosen to implement it. There are plenty of tools and systems that can present a sufficiently linear cost relationship to load and usage that, should your COGS versus revenue make sense, the marginal cost of increased cloud resources a no-brainer--especially versus always-paid-for hardware. If you don't have such a linear relationship you're as much in the position of deciding whether the project is viable as you are anything else.
- moreira 5y ago
  > If you don't know what you're doing, you can nuke your budget super quick. And even if you do, which I think you’ll agree Troy Hunt does.
- olavgg 5y ago
  The cloud is great for scaling indeed, but a cheap Intel V4 server with 44 cores from Ebay for $2000 can handle a shit ton of traffic too. If I were building a new business, I would use both cloud and colo. But I do understand that everyone don't have that luxury.
- carlivar 5y ago
  If you have a large environment to build in a certain region, the cloud lead time is months also. We have to give our cloud provider months notice before building in a region. But we have a pretty serious and profitable workload. Your statement is correct for the 90% of companies with relatively small infrastructure needs.
- Nextgrid 5y ago
  > The lead time for new servers deployed in a data center is weeks compared to seconds in the cloud. The lead time for Hetzner or OVH is measured in minutes and is appropriate for the majority of use cases. Old-school providers like these used to run the entire internet before the cloud craze started. Colocation & own datacenter is not a good choice for most startups. Seems like a lot of people here miss a step and only go from one extreme to the other. There's a middle-ground of renting bare-metal that gives you amazing price/performance with none of the drawbacks of colocation or running your own DC.
Retric 5y ago
Most people that use cloud computing aren’t stuck with the bills the companies they work for are. As to difficulty, they “solve” organizational problems by avoiding sticker shock when someone wants 100+k in equip that’s often a huge number of hoops to jump through and possibly months of delays, a giant bill every month and nobody a complains about the electric bill etc.
- chasd00 5y ago
  > Most people that use cloud computing aren’t stuck with the bills the companies they work for are. you can rest assured that even the largest company will come looking for the person responsible for increasing an expense by that large of a percentage. So maybe it doesn't come out of your personal checking account but you will certainly pay for it.
  - Retric 5y ago
    That’s assuming it’s a large increase in the bill for no reason. It’s easy to justify having a larger bill with more traffic. “A retail store isn’t going to complain that they need to buy more stock after selling more stuff that’s just a cost of doing business.” Meanwhile it can be hard to justify a capital expenditure just because traffic increased.
Spooky23 5y ago
When you are growing, it’s a no brainer. When you are at steady state it depends. As a case in point, I worked in standing up a critical system in a large enterprise a few years ago. We spent about $12M on compute, storage, networking, etc. At operational state, it was about 40% cheaper than AWS. The problem is, it all sat there for 6-18 months filling up before we fully hit that state. With a cloud provider, you pay a high unit cost but if you engineer intelligently your costs should move with utilization. Except for government, most entities generally want to see opex move with revenue and prefer to minimize capex where possible.
- Symbiote 5y ago
  You're an order of magnitude larger than what I work on, but on our last big project we purchased and installed half in the first year, then the remaining half 18 months later.
  - Spooky23 5y ago
    Keep in mind that size tends to lower intelligence! ;)
brodouevencode 5y ago
> Everything is more complex to debug, more expensive, and more shitty in all the possible ways you can imagine. Coming from traditional infrastructure and development methods, you're mostly right. Part of the expectation of the cloud is that you do things _their way_. And even then each cloud provider does things a little differently. However, if you're willing to subscribe to the <insert provider> way of doing things it (and you'll have to trust me here) makes many things easier. Here's a short list: * networking setup is free/cheap/doesn't require a Cisco cert. you can trust a developer to set things up. * object storage is so much easier than any file hosting scheme you can come up with * the path from container-on-a-host to container-in-a-cluster to container-in-{serverless,k8s} is extremely straightforward * I turn all my dev/test servers off at night and they don't cost me a thing * consumption based compute will result in a much cheaper solution than a VPS or colo (admittedly there are many assumptions baked into this) * some core services (like sqs, sns on Amazon) are extremely cheap and have provably reduced development time because you're not having to build these abstractions yourself. This all being said I'm not advocating an all-in approach without thinking it through, but to do so where it's easy and makes sense. EDIT: clarity
- Nextgrid 5y ago
  > I turn all my dev/test servers off at night and they don't cost me a thing I bet it still costs you more than my Hetzner one despite me not having to care about turning it on or off. I mean it's great that the cloud gives you this flexibility but you wouldn't need it to begin with if it wasn't so expensive.
- api 5y ago
  > networking setup is free/cheap/doesn't require a Cisco cert. you can trust a developer to set things up. Bare metal hosts set up the network for you. You may need to know how to configure a local network interface. Even if you actually rack and stack many colos will give you a drop with network set up. You don't need to do what you describe unless you are building your own DC. > object storage is so much easier than any file hosting scheme you can come up with That matters if your data volume is truly massive. Only a small percentage have this problem. Also AWS inbound is free so you could upload big data to AWS and warehouse it there if you wanted. Not using big cloud for everything doesn't mean you can't use it for anything. > the path from container-on-a-host to container-in-a-cluster to container-in-{serverless,k8s} is extremely straightforward This is the one spot where admittedly you will have to spend more in administration. You'll need to either run your own k8s or Nomad or adopt a different configuration, and you may have to think about it a bit more. > I turn all my dev/test servers off at night and they don't cost me a thing You could still do this. Just host live somewhere else. You could also test on a local VM, which is what we do. Obviously that depends on how big your app is. > consumption based compute will result in a much cheaper solution than a VPS or colo (admittedly there are many assumptions baked into this) You only see the savings if they are passed onto you. What we've seen is that Moore's Law savings have not been passed on by cloud providers. Look at what you can get at a bare metal host compared to how much the same compute costs in cloud. Years ago the difference would not have been so large. Bandwidth costs in cloud are insane, and most use asymmetric pricing where inbound bandwidth is free. This is known as "roach motel pricing" for a reason. Data goes in, but it doesn't come out. > some core services (like sqs, sns on Amazon) are extremely cheap and have provably reduced development time because you're not having to build these abstractions yourself. Fair, but they make their money back elsewhere. Those are lures to get you locked in so you now have to pay their crazy compute and bandwidth egress charges. Here's an example. There are more. https://www.datapacket.com https://www.datapacket.com
oblio 5y ago
You're not the target audience. Startups growing fast are the secondary audience. The primary audience is large enterprises where their internal IT costs <<more>> than the cloud costs. Plus internal IT provides those resources after 6 months...
sockpuppet69 5y ago
> 600GB/day is about 55Mbit/s In what universe? This frictionless perfect vacuum where traffic comes in a wholly predictable consistent continuum?
- southerntofu 5y ago
  Good point, it's just an average. And to be fair i checked the numbers in the article it seemed closer to 3.2TB/day which is closer to 300Mbit/s. But what i meant is a home fiber connection can deal with that. Although consumer ISPs don't have good bandwidth over all routes (it's good to Youtube/Amazon but ridiculously slow to some other consumer ISPs). If you don't want to serve from home, i'm sure many entities would be happy to donate disk space and bandwidth to help a project like this, setting up a mirror list like we have for distro repositories. Also, we may be taking the problem the wrong way around: do these multigig files need to be accessed by everyone from a web browser? No, it's a dump file used by specific people in specific circumstances. Then why are we using HTTP for this in the first place? In this case, only publishing over Bittorrent/IPFS makes sense and many people will happily seed, pushing costs toward 0 for the publisher (and very close to 0 if you only push to a first circle of mirrors who can then push to more mirrors, some of which can be declared webseeds in your torrent).
rr808 5y ago
> 600GB/day is about 55Mbit/s. not really it was minimal traffic then sudden bursts of gigabytes. Of course throttling the big spikes would actually have been a good idea in hindsight to give an early warning.
jollybean 5y ago
"I really don't understand the cloud craze" The opposite, I don't understand why anyone would ever put up a server if they didn't have to. It's not 'processing power' that's going to be the 'big cost' for most projects. It's headcount and salary. If you can materially improve the operating ability of your company, then a few $K in cloud fees is dirt cheap. I used to work at a 'tech company' that made a physical product and our IT was abysmal. We had to wait weeks for our sysadmins to order blades, get things set up, there were outages etc.. If a project is definitely going to be 'a few linux servers and never more' - even then it would be cheaper and more reasonable to use virtual instances. The time to 'roll your own' is when the infra. operating costs are a material part of your business. For example, 'Dropbox' invariably had to roll their own infra, that was inevitable. Similarly others. That said - as this article indicates, it's easy to 'over do it' and end up in ridiculous amounts of complexity. The Amazon IAM security model has always been bizarre and confusing, and the number of AWS services is mind-boggling. But the core case of EC2+S3 +Networking, and then maybe a couple of other enhanced services for special case works fine. I also object to what I think is a vast overuse of Cloudflare, I just don't believe that in most scenarios needing to have content at the edge really changes the experience that much.
- Nextgrid 5y ago
  > It's headcount and salary. This only really applies to fully-managed services such as Heroku. Every other cloud still needs a DevOps person according to my experience in many companies.
  - jollybean 5y ago
    Yes, but one cloud devops can do what 10x what sysadmins, hardware and network engineers can do. Just security alone, in terms of managing access to all of those resources, various forms of backup, across regions ... it's just out of reach for most organizations.
TheIronMark 5y ago
> but at least it wouldn't automatically ruin your bank account whenever you got a little traffic. This only happens when consumers fail to set budget alerts. Troy could have saved himself $10k with 15min worth of work.

SkipperCat 5y ago

The cloud is great for scaling. The lead time for new servers deployed in a data center is weeks compared to seconds in the cloud. Plus there's no sunk cost in the cloud - you can turn it off when done and it evaporates. Also, the cloud offers managed software as a service. You don't have to manage your own HA DB cluster or PubSub. It's all just there and it works. That can save you a lot on technical labor costs. But yes, I do agree with your point. If you don't know what you're doing, you can nuke your budget super quick.

immibis 5y ago
The technical ability to scale is a bit meaningless if you can't afford it.
- alexpotato 5y ago
  Then that becomes part of your business + technology planning conversations: "This is the cost of scaling, this is the cost of owning our own infra, how does that fit into our budgeting and requirements?"
- eropple 5y ago
  "If you can't afford it" is doing a lot of assuming and a lot of heavy lifting in that statement. Whether or not you can afford it depends strongly on your scaling bounds (how much you need to scale) and how you've chosen to implement it. There are plenty of tools and systems that can present a sufficiently linear cost relationship to load and usage that, should your COGS versus revenue make sense, the marginal cost of increased cloud resources a no-brainer--especially versus always-paid-for hardware. If you don't have such a linear relationship you're as much in the position of deciding whether the project is viable as you are anything else.
moreira 5y ago
> If you don't know what you're doing, you can nuke your budget super quick. And even if you do, which I think you’ll agree Troy Hunt does.
olavgg 5y ago
The cloud is great for scaling indeed, but a cheap Intel V4 server with 44 cores from Ebay for $2000 can handle a shit ton of traffic too. If I were building a new business, I would use both cloud and colo. But I do understand that everyone don't have that luxury.
carlivar 5y ago
If you have a large environment to build in a certain region, the cloud lead time is months also. We have to give our cloud provider months notice before building in a region. But we have a pretty serious and profitable workload. Your statement is correct for the 90% of companies with relatively small infrastructure needs.
Nextgrid 5y ago
> The lead time for new servers deployed in a data center is weeks compared to seconds in the cloud. The lead time for Hetzner or OVH is measured in minutes and is appropriate for the majority of use cases. Old-school providers like these used to run the entire internet before the cloud craze started. Colocation & own datacenter is not a good choice for most startups. Seems like a lot of people here miss a step and only go from one extreme to the other. There's a middle-ground of renting bare-metal that gives you amazing price/performance with none of the drawbacks of colocation or running your own DC.

Retric 5y ago

Most people that use cloud computing aren’t stuck with the bills the companies they work for are. As to difficulty, they “solve” organizational problems by avoiding sticker shock when someone wants 100+k in equip that’s often a huge number of hoops to jump through and possibly months of delays, a giant bill every month and nobody a complains about the electric bill etc.

chasd00 5y ago
> Most people that use cloud computing aren’t stuck with the bills the companies they work for are. you can rest assured that even the largest company will come looking for the person responsible for increasing an expense by that large of a percentage. So maybe it doesn't come out of your personal checking account but you will certainly pay for it.
- Retric 5y ago
  That’s assuming it’s a large increase in the bill for no reason. It’s easy to justify having a larger bill with more traffic. “A retail store isn’t going to complain that they need to buy more stock after selling more stuff that’s just a cost of doing business.” Meanwhile it can be hard to justify a capital expenditure just because traffic increased.

Spooky23 5y ago

When you are growing, it’s a no brainer. When you are at steady state it depends. As a case in point, I worked in standing up a critical system in a large enterprise a few years ago. We spent about $12M on compute, storage, networking, etc. At operational state, it was about 40% cheaper than AWS. The problem is, it all sat there for 6-18 months filling up before we fully hit that state. With a cloud provider, you pay a high unit cost but if you engineer intelligently your costs should move with utilization. Except for government, most entities generally want to see opex move with revenue and prefer to minimize capex where possible.

Symbiote 5y ago
You're an order of magnitude larger than what I work on, but on our last big project we purchased and installed half in the first year, then the remaining half 18 months later.
- Spooky23 5y ago
  Keep in mind that size tends to lower intelligence! ;)

brodouevencode 5y ago

> Everything is more complex to debug, more expensive, and more shitty in all the possible ways you can imagine. Coming from traditional infrastructure and development methods, you're mostly right. Part of the expectation of the cloud is that you do things _their way_. And even then each cloud provider does things a little differently. However, if you're willing to subscribe to the <insert provider> way of doing things it (and you'll have to trust me here) makes many things easier. Here's a short list: * networking setup is free/cheap/doesn't require a Cisco cert. you can trust a developer to set things up. * object storage is so much easier than any file hosting scheme you can come up with * the path from container-on-a-host to container-in-a-cluster to container-in-{serverless,k8s} is extremely straightforward * I turn all my dev/test servers off at night and they don't cost me a thing * consumption based compute will result in a much cheaper solution than a VPS or colo (admittedly there are many assumptions baked into this) * some core services (like sqs, sns on Amazon) are extremely cheap and have provably reduced development time because you're not having to build these abstractions yourself. This all being said I'm not advocating an all-in approach without thinking it through, but to do so where it's easy and makes sense. EDIT: clarity

Nextgrid 5y ago
> I turn all my dev/test servers off at night and they don't cost me a thing I bet it still costs you more than my Hetzner one despite me not having to care about turning it on or off. I mean it's great that the cloud gives you this flexibility but you wouldn't need it to begin with if it wasn't so expensive.
api 5y ago
> networking setup is free/cheap/doesn't require a Cisco cert. you can trust a developer to set things up. Bare metal hosts set up the network for you. You may need to know how to configure a local network interface. Even if you actually rack and stack many colos will give you a drop with network set up. You don't need to do what you describe unless you are building your own DC. > object storage is so much easier than any file hosting scheme you can come up with That matters if your data volume is truly massive. Only a small percentage have this problem. Also AWS inbound is free so you could upload big data to AWS and warehouse it there if you wanted. Not using big cloud for everything doesn't mean you can't use it for anything. > the path from container-on-a-host to container-in-a-cluster to container-in-{serverless,k8s} is extremely straightforward This is the one spot where admittedly you will have to spend more in administration. You'll need to either run your own k8s or Nomad or adopt a different configuration, and you may have to think about it a bit more. > I turn all my dev/test servers off at night and they don't cost me a thing You could still do this. Just host live somewhere else. You could also test on a local VM, which is what we do. Obviously that depends on how big your app is. > consumption based compute will result in a much cheaper solution than a VPS or colo (admittedly there are many assumptions baked into this) You only see the savings if they are passed onto you. What we've seen is that Moore's Law savings have not been passed on by cloud providers. Look at what you can get at a bare metal host compared to how much the same compute costs in cloud. Years ago the difference would not have been so large. Bandwidth costs in cloud are insane, and most use asymmetric pricing where inbound bandwidth is free. This is known as "roach motel pricing" for a reason. Data goes in, but it doesn't come out. > some core services (like sqs, sns on Amazon) are extremely cheap and have provably reduced development time because you're not having to build these abstractions yourself. Fair, but they make their money back elsewhere. Those are lures to get you locked in so you now have to pay their crazy compute and bandwidth egress charges. Here's an example. There are more. https://www.datapacket.com https://www.datapacket.com

oblio 5y ago

You're not the target audience. Startups growing fast are the secondary audience. The primary audience is large enterprises where their internal IT costs <<more>> than the cloud costs. Plus internal IT provides those resources after 6 months...

sockpuppet69 5y ago

> 600GB/day is about 55Mbit/s In what universe? This frictionless perfect vacuum where traffic comes in a wholly predictable consistent continuum?

southerntofu 5y ago
Good point, it's just an average. And to be fair i checked the numbers in the article it seemed closer to 3.2TB/day which is closer to 300Mbit/s. But what i meant is a home fiber connection can deal with that. Although consumer ISPs don't have good bandwidth over all routes (it's good to Youtube/Amazon but ridiculously slow to some other consumer ISPs). If you don't want to serve from home, i'm sure many entities would be happy to donate disk space and bandwidth to help a project like this, setting up a mirror list like we have for distro repositories. Also, we may be taking the problem the wrong way around: do these multigig files need to be accessed by everyone from a web browser? No, it's a dump file used by specific people in specific circumstances. Then why are we using HTTP for this in the first place? In this case, only publishing over Bittorrent/IPFS makes sense and many people will happily seed, pushing costs toward 0 for the publisher (and very close to 0 if you only push to a first circle of mirrors who can then push to more mirrors, some of which can be declared webseeds in your torrent).

rr808 5y ago

> 600GB/day is about 55Mbit/s. not really it was minimal traffic then sudden bursts of gigabytes. Of course throttling the big spikes would actually have been a good idea in hindsight to give an early warning.

jollybean 5y ago

"I really don't understand the cloud craze" The opposite, I don't understand why anyone would ever put up a server if they didn't have to. It's not 'processing power' that's going to be the 'big cost' for most projects. It's headcount and salary. If you can materially improve the operating ability of your company, then a few $K in cloud fees is dirt cheap. I used to work at a 'tech company' that made a physical product and our IT was abysmal. We had to wait weeks for our sysadmins to order blades, get things set up, there were outages etc.. If a project is definitely going to be 'a few linux servers and never more' - even then it would be cheaper and more reasonable to use virtual instances. The time to 'roll your own' is when the infra. operating costs are a material part of your business. For example, 'Dropbox' invariably had to roll their own infra, that was inevitable. Similarly others. That said - as this article indicates, it's easy to 'over do it' and end up in ridiculous amounts of complexity. The Amazon IAM security model has always been bizarre and confusing, and the number of AWS services is mind-boggling. But the core case of EC2+S3 +Networking, and then maybe a couple of other enhanced services for special case works fine. I also object to what I think is a vast overuse of Cloudflare, I just don't believe that in most scenarios needing to have content at the edge really changes the experience that much.

Nextgrid 5y ago
> It's headcount and salary. This only really applies to fully-managed services such as Heroku. Every other cloud still needs a DevOps person according to my experience in many companies.
- jollybean 5y ago
  Yes, but one cloud devops can do what 10x what sysadmins, hardware and network engineers can do. Just security alone, in terms of managing access to all of those resources, various forms of backup, across regions ... it's just out of reach for most organizations.

TheIronMark 5y ago

> but at least it wouldn't automatically ruin your bank account whenever you got a little traffic. This only happens when consumers fail to set budget alerts. Troy could have saved himself $10k with 15min worth of work.