5 ms·

Ok, enough AWS outages to say I'm tired of hearing about low end stuff being flaky.

by throwaway81523 5y ago

henriquez 5y ago
Heroku isn’t “low end,” it’s a PaaS built on top of AWS. So you’re really just hearing about another AWS outage lol
- christophilus 5y ago
  They're not saying Heroku is low end. They're saying, "I'm tired of hearing that it's irresponsible to run your own servers." At least, that's what I understood.
  - ryanbrunner 5y ago
    Any place I've worked at that managed their own servers (to be fair, the last time I worked at a place like that was 2010) definitely had more protracted downtimes than AWS - it just felt not as bad because we were in control of the situation, but at the end of the day that didn't get us up any faster. Another side benefit of being with AWS is when you do have an outage, a lot of other people have outages, and so you sort of blend in with the noise. It's not great to be down, but if you're down and also "big service X" who's also an AWS customer is down, it makes your downtime look less like a lack of competence and more like an unavoidable force of nature.
    - dijit 5y ago
      I guess it's extremely dependent on an org to org basis. I worked at a company that's bread and butter was online services (e-commerce SaaS platform, similar to Netsuite) and we had significantly fewer outages than AWS had. But we had redundancies built in to most things, I'm not saying it was perfect but it worked. The major difference might be that almost nobody is willing to spend 20% of what they spend on AWS/GCP to have a self-hosted solution. The reason "cloud is so expensive" is because they're essentially telling you what the price will be and even if they only spend 40% of that on actual hardware and operations: it's more than most companies would invest in themselves. This is absurd, of course, but it's absolutely true.
    - deleted 5y ago
      [deleted]
  - deleted 5y ago
    [deleted]
- mijoharas 5y ago
  This comment doesn't say anything about heroku?
api 5y ago
Nobody ever got fired for using AWS.
- flatiron 5y ago
  If you rely solely on east 1 maybe?
  - omh2 5y ago
    AWS doesn't follow their own advice about hosting multi-regional. When us-east-1 is sufficiently borked the management API and IAM services in all regions tend to go down with it. Static infrastructures usually avoid the fallout, but anyone dependent on the API or otherwise dynamically created resources often get caught in the blast regardless of region
    - jeremyjh 5y ago
      I didn't hear any reports of that happening in the most recent outage. The console was inoperable but you could work around using regional console host names.
      - throwanem 5y ago
        It wasn't reliable. I heard of many more who weren't able to get in that way than who were, and was in the former category myself. We didn't take any downtime, but if anything had gone wrong there would have been nothing we could do about it until IAM came back up.
      - tyingq 5y ago
        There are some services that do have hard US-EAST-1 dependencies. Cloudfront, because of certificates. Route53. The control API for IAM (adding/removing roles, etc). And there's also the notion of "global endpoints" like https://sts.amazonaws.com https://sts.amazonaws.com... it's not clear why that exists, because it fails when us-east-1 does. It would be better to only have regional endpoints if the "global" ones are region-specific in reality. The endpoint thing is documented, but it's still confusing to people. The dependency chains can bite you too. During the us-east-1 outage, a Lambda run by cron-like schedules via EventBridge was itself in an okay state, but the EventBridge events that kick it off were stuck in a queue that was released when the problem was fixed. So if your Lambda wasn't idempotent, and you ran it in another region during the outage, you ended up with problems.
jacob019 5y ago
Right. I've had an excellent experince with Vultr for the last couple years, for about 1/10th the cost of AWS. I use other small VPS providers as well. I run my own small business and I need to keep costs down to stay competitive. I used to use AWS more but the bill always creeps up to inappropriate levels. AWS billing is insulting, oh you forgot to renew your reserved instance? That's going to be double this month. I still use cloudfront, route 53, and a few of the smallest instances for mail servers and asterisk though. It's foolish to go all in with AWS, or with anything really.
BiteCode_dev 5y ago
"Don't use a self hosted monolithe, it's not reliable! You need a cloud FS with a load balancer under observability and your data in a db that scales horizontally, all orchestrated by kubs." Meanwhile, I currently have a gig to work on a video service which features a never updated centos 6, an unsupported python 2 blob website, and a push to prod deployment procedure, running a single postgres db serving streaming for 4 millions users a month. And it's got years of up time, cost 1/100th of AWS, and can be maintained by one dev. Not saying "cloud is bad", but we got to stop screaming old techs are no good either.
- osrec 5y ago
  Purely out of interest, I'd like to know more about your streaming architecture. I assume postgres just holds the meta data, and the actual video content is stored elsewhere? What strategies have you employed to scale the streaming part of your service? I imagine 4 million users a month is quite a significant amount of traffic!
  - BiteCode_dev 5y ago
    1 - For the last 10 years, servers have been beasts. You have a lot of cores, plenty of HD and RAM. Servers are less expensives than devs. Scaling vertically can go VERY far. 2 - Caching is life. We have 3 layers of caching: cloudflare, varnish, and redis. Most things don't need to be real time. A lot of things can be a month old and the user doesn't care. User need immediate feedback to be happy, but not necessary fresh data. 3 - if you compile nginx manually, you get to use a lot of plugins that can do stuff super fast, including serving videos. You can script stuff in lua that will just skip the backend completly. 4 - mind your encoding. We carefully chose how we encode videos. The ffmpeg parameters are pretty insane, but the space / quality ratio is amazing, espacially on mobile. It takes a lot of time to experiment with those, nobody share them :) 5 - we offload everything we can to cron tasks or task queues. Including, obviously, encoding, screenshooting, etc. 6 - don't hold data you can't lose. E.G: billing. This way you can have a relaxed attitute toward data. If we ever loose a day of business, users will be in a bad mood for a week, but that won't be the end of the world. We don't need a bullet proof system if bullets can't kill us. 7 - give money to ffmpeg and opencv, because damn those things are fast. And good. 8 - servers are hosted accross 2 providers. This way, if one goes down, or decide to stop doing business with us Google style, we have a second one. Happened recently with leaseweb: they shutdown a whole room without offering an alternative. E.G: votes. They don't hit the backend on write. We pile them from nginx to redis, then once a day, we aggregate and store on postgres, which the backends will consumme. We just store each vote on localstorage as well so that the user feels like it's real time when they vote, but in reality it's updated once a day. But votes don't affect the money side of our business, so if we lose them one day, it does not mean death. P.S: yes, posgres/redis/elasticsearch only hold metadata. Videos are stored on disk. There is no docker images, no mircoservices, FS is ext4. Which means with a lot of RAM, the OS FS cache will have most popular videos already loaded and ready to be streamed. Everything is raid 0, so if we get one disk corrupted, you lose the server. But we upload each videos on severeal servers, so when a disk get corrupted, we just replace the whole server. In fact, anything goes wrong on a server, we replace it. It's not worth it to find the root cause, unless 2 servers die in the same way successively.

Ok, enough AWS outages to say I'm tired of hearing about low end stuff being flaky.

henriquez 5y ago

Heroku isn’t “low end,” it’s a PaaS built on top of AWS. So you’re really just hearing about another AWS outage lol

christophilus 5y ago
They're not saying Heroku is low end. They're saying, "I'm tired of hearing that it's irresponsible to run your own servers." At least, that's what I understood.
- ryanbrunner 5y ago
  Any place I've worked at that managed their own servers (to be fair, the last time I worked at a place like that was 2010) definitely had more protracted downtimes than AWS - it just felt not as bad because we were in control of the situation, but at the end of the day that didn't get us up any faster. Another side benefit of being with AWS is when you do have an outage, a lot of other people have outages, and so you sort of blend in with the noise. It's not great to be down, but if you're down and also "big service X" who's also an AWS customer is down, it makes your downtime look less like a lack of competence and more like an unavoidable force of nature.
  - dijit 5y ago
    I guess it's extremely dependent on an org to org basis. I worked at a company that's bread and butter was online services (e-commerce SaaS platform, similar to Netsuite) and we had significantly fewer outages than AWS had. But we had redundancies built in to most things, I'm not saying it was perfect but it worked. The major difference might be that almost nobody is willing to spend 20% of what they spend on AWS/GCP to have a self-hosted solution. The reason "cloud is so expensive" is because they're essentially telling you what the price will be and even if they only spend 40% of that on actual hardware and operations: it's more than most companies would invest in themselves. This is absurd, of course, but it's absolutely true.
  - deleted 5y ago
    [deleted]
- deleted 5y ago
  [deleted]
mijoharas 5y ago
This comment doesn't say anything about heroku?

api 5y ago

Nobody ever got fired for using AWS.

flatiron 5y ago
If you rely solely on east 1 maybe?
- omh2 5y ago
  AWS doesn't follow their own advice about hosting multi-regional. When us-east-1 is sufficiently borked the management API and IAM services in all regions tend to go down with it. Static infrastructures usually avoid the fallout, but anyone dependent on the API or otherwise dynamically created resources often get caught in the blast regardless of region
  - jeremyjh 5y ago
    I didn't hear any reports of that happening in the most recent outage. The console was inoperable but you could work around using regional console host names.
    - throwanem 5y ago
      It wasn't reliable. I heard of many more who weren't able to get in that way than who were, and was in the former category myself. We didn't take any downtime, but if anything had gone wrong there would have been nothing we could do about it until IAM came back up.
    - tyingq 5y ago
      There are some services that do have hard US-EAST-1 dependencies. Cloudfront, because of certificates. Route53. The control API for IAM (adding/removing roles, etc). And there's also the notion of "global endpoints" like https://sts.amazonaws.com https://sts.amazonaws.com... it's not clear why that exists, because it fails when us-east-1 does. It would be better to only have regional endpoints if the "global" ones are region-specific in reality. The endpoint thing is documented, but it's still confusing to people. The dependency chains can bite you too. During the us-east-1 outage, a Lambda run by cron-like schedules via EventBridge was itself in an okay state, but the EventBridge events that kick it off were stuck in a queue that was released when the problem was fixed. So if your Lambda wasn't idempotent, and you ran it in another region during the outage, you ended up with problems.

jacob019 5y ago

Right. I've had an excellent experince with Vultr for the last couple years, for about 1/10th the cost of AWS. I use other small VPS providers as well. I run my own small business and I need to keep costs down to stay competitive. I used to use AWS more but the bill always creeps up to inappropriate levels. AWS billing is insulting, oh you forgot to renew your reserved instance? That's going to be double this month. I still use cloudfront, route 53, and a few of the smallest instances for mail servers and asterisk though. It's foolish to go all in with AWS, or with anything really.

BiteCode_dev 5y ago

"Don't use a self hosted monolithe, it's not reliable! You need a cloud FS with a load balancer under observability and your data in a db that scales horizontally, all orchestrated by kubs." Meanwhile, I currently have a gig to work on a video service which features a never updated centos 6, an unsupported python 2 blob website, and a push to prod deployment procedure, running a single postgres db serving streaming for 4 millions users a month. And it's got years of up time, cost 1/100th of AWS, and can be maintained by one dev. Not saying "cloud is bad", but we got to stop screaming old techs are no good either.

osrec 5y ago
Purely out of interest, I'd like to know more about your streaming architecture. I assume postgres just holds the meta data, and the actual video content is stored elsewhere? What strategies have you employed to scale the streaming part of your service? I imagine 4 million users a month is quite a significant amount of traffic!
- BiteCode_dev 5y ago
  1 - For the last 10 years, servers have been beasts. You have a lot of cores, plenty of HD and RAM. Servers are less expensives than devs. Scaling vertically can go VERY far. 2 - Caching is life. We have 3 layers of caching: cloudflare, varnish, and redis. Most things don't need to be real time. A lot of things can be a month old and the user doesn't care. User need immediate feedback to be happy, but not necessary fresh data. 3 - if you compile nginx manually, you get to use a lot of plugins that can do stuff super fast, including serving videos. You can script stuff in lua that will just skip the backend completly. 4 - mind your encoding. We carefully chose how we encode videos. The ffmpeg parameters are pretty insane, but the space / quality ratio is amazing, espacially on mobile. It takes a lot of time to experiment with those, nobody share them :) 5 - we offload everything we can to cron tasks or task queues. Including, obviously, encoding, screenshooting, etc. 6 - don't hold data you can't lose. E.G: billing. This way you can have a relaxed attitute toward data. If we ever loose a day of business, users will be in a bad mood for a week, but that won't be the end of the world. We don't need a bullet proof system if bullets can't kill us. 7 - give money to ffmpeg and opencv, because damn those things are fast. And good. 8 - servers are hosted accross 2 providers. This way, if one goes down, or decide to stop doing business with us Google style, we have a second one. Happened recently with leaseweb: they shutdown a whole room without offering an alternative. E.G: votes. They don't hit the backend on write. We pile them from nginx to redis, then once a day, we aggregate and store on postgres, which the backends will consumme. We just store each vote on localstorage as well so that the user feels like it's real time when they vote, but in reality it's updated once a day. But votes don't affect the money side of our business, so if we lose them one day, it does not mean death. P.S: yes, posgres/redis/elasticsearch only hold metadata. Videos are stored on disk. There is no docker images, no mircoservices, FS is ext4. Which means with a lot of RAM, the OS FS cache will have most popular videos already loaded and ready to be streamed. Everything is raid 0, so if we get one disk corrupted, you lose the server. But we upload each videos on severeal servers, so when a disk get corrupted, we just replace the whole server. In fact, anything goes wrong on a server, we replace it. It's not worth it to find the root cause, unless 2 servers die in the same way successively.