5 ms·

Interestingly, I took a ~5 year hiatus from writing code right smack in the beginning of the "cloud phenomenon," and upon returning, realized how much things ha

Interestingly, I took a ~5 year hiatus from writing code right smack in the beginning of the "cloud phenomenon," and upon returning, realized how much things had changed, with both skill levels and specialization. Before my hiatus, a dev could/would single-handedly run a site for a small/medium business. Handling everything from server procurement to FTPing the files to the server and even server maintenance. When I got back, this was "a big no-no" and mostly frowned upon. Everything was deployed by "git push" (fancy FTP?), problem with a server? just deploy a new one, and if an AWS region goes down ... it's ok, everyone else is down too and we have no idea when it will be back. That last sentence was something you never said to a boss back in '07-08. You would always know what was wrong and how bad it was within a few minutes of the issue, at least 90%+ of the time anyway. You could probably even guess how long it would be down for within ~10-15 minutes of discovering the issue. FWIW, I think small/medium size businesses were/are probably the most affected by cloudification since those had the "most to gain" in the beginning. Then again, those were the types of businesses that I had the most interaction with until 2014-ish, I don't know what it looked like from a big-business perspective.

Moissanite 4y ago
The key to this is: which is worse for middle management; a 15 minute outage which is your fault, or a 2 hour outage you can blame on AWS? Cloud is to the IT department as big consulting companies are to HR.
- gw99 4y ago
  I learned recently that the customers kick you in the balls once per minute either way. Loudly on Twitter.
  - nikau 4y ago
    Your average person doesn't listen to twitter, they listen to the mainstream news. And mainstream news will be either "company X offline for hours" if its just you, or "Amazon cloud outage takes multiple companies off line for hours" - the latter is better.
    - gw99 4y ago
      That’s not true in our sector. We’re very industry specific and everyone shitposts on twitter and follows everyone else.
      - nikau 4y ago
        Your sector != average person. Average person hears on the news that company X or cloud Y had an outage, at most they have some anger towards the company for a day or so until the next thing in the news cycle comes along.
        Moissanite 4y ago
        In my experience, an average person seeing that one of their commonly-used tools is down due to a big tech company failure is just going to proclaim "snow day!" and not really mind at all.
        gw99 4y ago
        Depends if it gets in the way of them making commission or money.
- booi 4y ago
  Except realistically AWS is down maybe 2 hours a year and if you blow an on-prem switch with no configuration backup or a database server hard crashes while syncing its backup, you might be down days.
  - jsiepkes 4y ago
    > if you blow an on-prem switch with no configuration backup or a database server hard crashes while syncing its backup, you might be down days. Switches are easy and cheap to put in a fail-over config. DB's are also running atleast in a primary / standby config everywhere. Besides if your running a single DB instance in a cloud VM the same thing can happen. And don't get me started on the trouble you can get in with for example GCP SaaS PostgreSQL...
  - Corrado 4y ago
    In a former life (before clouds) we had a SAN go down because of a backplane problem. It was down for days while being diagnosed/fixed and then in read-only mode for another couple of days while the disks were in repair mode. I know people say things like "you should have a backup device" or "switches are cheap" but sometimes it IS the backup system that goes down and sometimes that system is VERY expensive.
PragmaticPulp 4y ago
> Handling everything from server procurement to FTPing the files to the server and even server maintenance. When I got back, this was "a big no-no" and mostly frowned upon. Everything was deployed by "git push" (fancy FTP?), While I do look back with nostalgia on the days of editing files in Notepad++, FTPing them to a live server, then quickly switching to my browser to refresh the page and make sure it works... Those days are in the past for a reason. No, git is not "fancy FTP". Yes, version control really does add much-needed structure to collaborative projects and change tracking. You can still run a small business web server on a colocated server and FTP files to it if you want. You just won't find any modern engineers who want to touch that with a 10ft pole. The world has moved forward since then, and for good reason.
- withinboredom 4y ago
  > No, git is not "fancy FTP" I was specifically referencing an old fad (might still be a thing but I haven't run into it in years) where production is a remote and a dev pushes their main branch to it to deploy. Yes, version control is amazing and it was something I used a bunch before my hiatus, but back then, it was usually SVN IIRC.
  - PragmaticPulp 4y ago
    Using source control tools as part of the deploy process really is a valuable innovation. It's not an "old fad". I think you're clinging a bit too heavily to the "good old days". It's ironic how this entire thread has turned into a mix of people either 1) complaining that modern technology is too complex and complicated or 2) complaining that modern job applicants have lost the skills necessary to run things
    - withinboredom 4y ago
      That's an interesting take. But nah, I like some things the way they are now vs. back then. I love having cheap/easy CI that deploys to prod for me. I love having IDEs that are actually geared towards whatever language I'm writing in this year. I love having access to cheaper coloc space because most people aren't using it. I love having skills that most people don't even bother learning anymore when digging into performance issues. ... There is much I love about today and I look forward to the future. I was just stating my observations. And yes, `git deploy prod` is a fancy FTP, but probably even worse than FTP if the build actually happens on the production server (which is something I saw more than once) and causes a production outage (also saw more than once).
pixl97 4y ago
AWS going down is in no way dissimilar to L3 going down back in the day. You were still down and there was not much you could do about it.
phpisthebest 4y ago
>>That last sentence was something you never said to a boss back in '07-08. Modern IT is swapped with CYA / No Accountablity Cloud is not the only area this presents itself, everyone is more concerns about not being blamed for an issue than they are with resolving the issue. The more vendors you can point the blame to the better in their eyes...
- sekh60 4y ago
  Not really IT's fault. Mainly management always wanting a neck to choke.
fragmede 4y ago
> it's ok, everyone else is down too and we have no idea when it will be back. AWS has never gone down across all regions at once. So we can do something about going down. You build the system to be multi-region and be able to evacuate traffic from one region to another, so when us-east-1 goes down again, you simply route around the damage and use resources in us-west2 until us-east-1 comes back up. People don't appreciate how much backend engineering goes into keeping something like Twitter up at all times. Put it another way. Little guys go down when us-east-1 goes down. The big boys stay up no matter what.
- jjav 4y ago
  > Little guys go down when us-east-1 goes down. The big boys stay up no matter what. Look back at all the big AWS outages of recent memory. Tons of the so-called big boys were down for the day along with AWS.
  - fragmede 4y ago
    Yeah, that was embarrassing. Point is, AWS being down isn't cause to sit around and do nothing but instead, work on how to make things more nesilient.
- simplotek 4y ago
  > AWS has never gone down across all regions at once. I still recall last year's AWS LSE when AWS went down globally, due to us-east-1 https://www.datacenterdynamics.com/en/news/aws-us-east-1-outage-brings-down-services-around-the-world/ https://www.datacenterdynamics.com/en/news/aws-us-east-1-out...

Moissanite 4y ago

The key to this is: which is worse for middle management; a 15 minute outage which is your fault, or a 2 hour outage you can blame on AWS? Cloud is to the IT department as big consulting companies are to HR.

gw99 4y ago
I learned recently that the customers kick you in the balls once per minute either way. Loudly on Twitter.
- nikau 4y ago
  Your average person doesn't listen to twitter, they listen to the mainstream news. And mainstream news will be either "company X offline for hours" if its just you, or "Amazon cloud outage takes multiple companies off line for hours" - the latter is better.
  - gw99 4y ago
    That’s not true in our sector. We’re very industry specific and everyone shitposts on twitter and follows everyone else.
    - nikau 4y ago
      Your sector != average person. Average person hears on the news that company X or cloud Y had an outage, at most they have some anger towards the company for a day or so until the next thing in the news cycle comes along.
      - Moissanite 4y ago
        In my experience, an average person seeing that one of their commonly-used tools is down due to a big tech company failure is just going to proclaim "snow day!" and not really mind at all.
        gw99 4y ago
        Depends if it gets in the way of them making commission or money.
booi 4y ago
Except realistically AWS is down maybe 2 hours a year and if you blow an on-prem switch with no configuration backup or a database server hard crashes while syncing its backup, you might be down days.
- jsiepkes 4y ago
  > if you blow an on-prem switch with no configuration backup or a database server hard crashes while syncing its backup, you might be down days. Switches are easy and cheap to put in a fail-over config. DB's are also running atleast in a primary / standby config everywhere. Besides if your running a single DB instance in a cloud VM the same thing can happen. And don't get me started on the trouble you can get in with for example GCP SaaS PostgreSQL...
- Corrado 4y ago
  In a former life (before clouds) we had a SAN go down because of a backplane problem. It was down for days while being diagnosed/fixed and then in read-only mode for another couple of days while the disks were in repair mode. I know people say things like "you should have a backup device" or "switches are cheap" but sometimes it IS the backup system that goes down and sometimes that system is VERY expensive.

PragmaticPulp 4y ago

> Handling everything from server procurement to FTPing the files to the server and even server maintenance. When I got back, this was "a big no-no" and mostly frowned upon. Everything was deployed by "git push" (fancy FTP?), While I do look back with nostalgia on the days of editing files in Notepad++, FTPing them to a live server, then quickly switching to my browser to refresh the page and make sure it works... Those days are in the past for a reason. No, git is not "fancy FTP". Yes, version control really does add much-needed structure to collaborative projects and change tracking. You can still run a small business web server on a colocated server and FTP files to it if you want. You just won't find any modern engineers who want to touch that with a 10ft pole. The world has moved forward since then, and for good reason.

withinboredom 4y ago
> No, git is not "fancy FTP" I was specifically referencing an old fad (might still be a thing but I haven't run into it in years) where production is a remote and a dev pushes their main branch to it to deploy. Yes, version control is amazing and it was something I used a bunch before my hiatus, but back then, it was usually SVN IIRC.
- PragmaticPulp 4y ago
  Using source control tools as part of the deploy process really is a valuable innovation. It's not an "old fad". I think you're clinging a bit too heavily to the "good old days". It's ironic how this entire thread has turned into a mix of people either 1) complaining that modern technology is too complex and complicated or 2) complaining that modern job applicants have lost the skills necessary to run things
  - withinboredom 4y ago
    That's an interesting take. But nah, I like some things the way they are now vs. back then. I love having cheap/easy CI that deploys to prod for me. I love having IDEs that are actually geared towards whatever language I'm writing in this year. I love having access to cheaper coloc space because most people aren't using it. I love having skills that most people don't even bother learning anymore when digging into performance issues. ... There is much I love about today and I look forward to the future. I was just stating my observations. And yes, `git deploy prod` is a fancy FTP, but probably even worse than FTP if the build actually happens on the production server (which is something I saw more than once) and causes a production outage (also saw more than once).

pixl97 4y ago

AWS going down is in no way dissimilar to L3 going down back in the day. You were still down and there was not much you could do about it.

phpisthebest 4y ago

>>That last sentence was something you never said to a boss back in '07-08. Modern IT is swapped with CYA / No Accountablity Cloud is not the only area this presents itself, everyone is more concerns about not being blamed for an issue than they are with resolving the issue. The more vendors you can point the blame to the better in their eyes...

sekh60 4y ago
Not really IT's fault. Mainly management always wanting a neck to choke.

fragmede 4y ago

> it's ok, everyone else is down too and we have no idea when it will be back. AWS has never gone down across all regions at once. So we can do something about going down. You build the system to be multi-region and be able to evacuate traffic from one region to another, so when us-east-1 goes down again, you simply route around the damage and use resources in us-west2 until us-east-1 comes back up. People don't appreciate how much backend engineering goes into keeping something like Twitter up at all times. Put it another way. Little guys go down when us-east-1 goes down. The big boys stay up no matter what.

jjav 4y ago
> Little guys go down when us-east-1 goes down. The big boys stay up no matter what. Look back at all the big AWS outages of recent memory. Tons of the so-called big boys were down for the day along with AWS.
- fragmede 4y ago
  Yeah, that was embarrassing. Point is, AWS being down isn't cause to sit around and do nothing but instead, work on how to make things more nesilient.
simplotek 4y ago
> AWS has never gone down across all regions at once. I still recall last year's AWS LSE when AWS went down globally, due to us-east-1 https://www.datacenterdynamics.com/en/news/aws-us-east-1-outage-brings-down-services-around-the-world/ https://www.datacenterdynamics.com/en/news/aws-us-east-1-out...