11 ms·

Servers can also be liability. You need to document, implement and maintaing hardening, have a process for regularly patching os and apps, monitor logs, have ba

Servers can also be liability. You need to document, implement and maintaing hardening, have a process for regularly patching os and apps, monitor logs, have backup and disaster recovery procedures, regularly test the procedures, figure how to implement data encryption at rest, implement high-availability and so on. Good platform-as-service can solve many things for you and let you focus on the core thing you are providing. Obviously not everybody needs to worry so much about the stuff mentioned above. If you are providing SaaS solution, there’s a good chance some customer will start asking these questions as part of their procurement process.

jsiepkes 4y ago
I'd rather worry about (and fix) those technical problems then having to deal with possible billing-pocalyse.
- jpalomaki 4y ago
  Take for example OS and app upgrades. Quite often you have few requirements, for example 1)all updates should be first tested in test env 2)updates need to be installed in timely manner 3)critical updates need to be installed quickly (30 days is too slow) When you start thinking these, they are not so easy. (1) means you can’t just run “apt upgrade” on every server - you need to manage the updates to make sure they get tested first. (2) is kind of ok, but requires some work on regular basis (at least checking things) (3) means you need to monitor the updates for your stack and classify them. You can get feeds for example for Ubuntu, but does that cover whole stack. And checking these weekly (or daily) actually gets boring. All this stuff can be done, but IMHO it is time consuming and takes time from more important things. The weekly/monthly JIRA tickets for checking x, y and z get quite annoying when you also have n other things to finish. Then you start slacking on them and feel the pain when trying to collect evidence for the next procurement process check. If you have tens or hundreds of servers and can have a separate infra team with professional sysadmins then this is all fine. My rant is mainly for small teams, where same guys are supposed to develop and run things.
  - wooque 4y ago
    Small teams should just configure apt unattended upgrades and call it a day. I do this with my personal server and haven't had any issues for years.
    - remram 4y ago
      You should probably reboot it then.
- peterangular 4y ago
  Also the cost of those services vs. a few well-configured basic VPS hosts is nuts. I was shocked to learn that to get MySQL/Postgres in GCP it's often >$40/month - even for the insanely tiny instance sizes. We're talking for like 20G storage... In contrast I've ran some pretty hot-and-heavy MySQL/Postgres databases on basic DO/Linode VPS's for half of that cost with much success. I get these cloud tools give a ton of "out of box" features - but you are paying for it at the end of the day. Anecdotally I've noticed a huge shift away from general devs/engineers having general CLI/Linux/sysadmin knowledge.
  - turtlebits 4y ago
    If you had no knowledge on standing up a DB server (installing, configuring, monitoring, logging, access control, and backups, restricting network access), how much time would that take you to learn? That and your VPS requires setup for access control, patches, updates, monitoring, logging.
    - peterangular 4y ago
      Point taken, upvote given, but for a lot of people standing up a standard redundant DB config for MySQL and Postgres isn't that big of a deal. I say this as an engineer who has a ton of relational DB experience. Before cloud providers w/DB options were so plentiful devs/engineers/sysadmins would have to set this up by hand. --- > If you had no knowledge on standing up a DB server My point is, "but yea what if you do?" There's actual value in that I've found, especially when I'm looking to save pennies at the early-stage of a startup while in the customer aq phase. Many folks have expertise in those areas without needing a cloud provider. I do get the "but all of these things!" argument (access control, patches, updates, monitoring, logging) but often those are easily solved/solvable problems even without leveraging cloud offerings. I can get very far along with an ELK/TIG stack + uptime robot and basic VPS networking features... all for a very affordable monthly bill on the infra.
    - fartcannon 4y ago
      Starting a db on a web-server and running apt update/upgrade are fundamental parts of working on the web. If you or any of your developers don't know how to do that, you/they should seriously seriously spend the 40 minutes it takes to learn.
- throwaway894345 4y ago
  An occasional $4K charge is negligible for a business (almost everyone is better off dealing with this sort of overrun than troubleshooting servers, all else equal), and anyway this is a particularity of AWS CloudFront's billing protections rather than a fundamental flaw with serverless.
  - ggktk 4y ago
    In most countries in the world, $4K charge is absolutely not negligible.
    - throwaway894345 4y ago
      I doubt this is true. If you're operating a tech company anywhere in the world, I'm guessing $4K is pretty close to negligible. In whatever case, indexing by country is misleading, "tech companies" in general is more interesting, if there are a handful of tech companies in the third world for which this doesn't hold, it doesn't invalidate the generalization.
      - jsiepkes 4y ago
        In SMB companies around the world "an occasional $4K charge" is definitely not negliable.
        throwaway894345 4y ago
        Again, we’re not talking about SMBs, we’re talking about tech companies generally. (What share of SMBs are there doing serverless or with any cloud footprint at all?)
- ImPostingOnHN 4y ago
  I'd rather have to to plan and budget than have to deal with patching (after testing the patch) every piece of software on a system the moment a patch comes out to avoid an undetectable rootkit being installed during that gap
cft 4y ago
I ran a website with 37 million users and 3.6Gbps peak bandwidth (JavaScript+thumbnails, no video) from my own two racks of Linux servers thay i have not systematically updated for years. The OSes were beyond the lts support winows. I manually compiled my own updates, but very rarely and only those that i deemed critical. Granted, the site stack was completely custom so the standard automated hacks didn't work. In 15 years there were no accidents.
- booi 4y ago
  I dont think this should be standard. It sounds like you’re saying you ran a service with 37 million people’s information on a software stack that was so old that not even the vendor is supporting it anymore and could be riddled with security issues that you wouldn’t even know much less be able to detect? It may work but certainly not going to get any security certifications this way..
  - cft 4y ago
    I hardly stored any PII. Also, this was a high profile site that you know: if it was hacked, they would probably try to deny access and extort: it would make a lot of monetary sense. We would also have lost our merchant accounts very quickly (although CC numbers were not stored, only MD5s, but i suppose they could have been captured from the application's memory after TLS decryption but before MD5 hashing, although this would have been a difficult task even on a rooted server). I am 99.999% confident by observing and recording loads, traffic, logs and other parameters of all servers that there were no intrusions. I understand that there's market size and appeal for bureaucracy and certifications, but i have my own data.
    - Fnoord 4y ago
      Do you save logs? If yes, there's your PII.
      - cft 4y ago
        Thankfully, the servers were not in Europe! I said "hardly". It's genuinely amusing how people downvote facts, simply because they don't fit the zeitgeist.
deleted 4y ago
[deleted]
asddubs 4y ago
you can get servers with managed hosting, still at a fixed rate each month
davidfstr 4y ago
> Servers can also be liability. You need to document, implement and maintaing hardening, have a process for regularly patching os and apps, […], have backup and disaster recovery procedures, regularly test the procedures, […] and so on. Much of this complexity I eliminate by using the Immutable Server pattern: Specifically I deploy my app as a Docker container hosted on a virtual machine cluster managed by AWS ElasticBeanstalk. OS upgrades and patches (outside the container) are done by AWS. If anything goes wrong with a VM I just terminate it and a different fresh VM spins up to take its place.
- huksley 4y ago
  This is that I was thinking doing also - switch to using Elastic Beanstalk. What do you do if you need to launch another instance of the app? For staging or testing.
rollcat 4y ago
I agree with you in general, that it's best not to solve problems you don't have, but it's worth considering what exactly do the alternatives offer, as you might end up trading one kind of complexity or maintenance burden for another. Fixed workload, shared dev boxes, all kinds of small-scale operations fit very well onto a bare metal / VPS box in someone else's rack. The maintenance work often boils down to setting up cron with unattended upgrades and borgbackup, Docker with Traefik+ACME as the web frontend, etc. PaaS solutions will often pull you in the direction of getting "addons" or using extra paid services for every component you might need: a relational database, a cache, a queue, a load balancer... Which are usually a very good call if you need to scale up, but which could've all been entries in your docker-compose.yml if that's all you needed. Most importantly, and regardless of your hosting strategy, you can't just dismiss security, disaster recovery, or capacity planning. Clouds have outages and bottlenecks too, and no product can save you from human error.
masukomi 4y ago
> You need to document, implement and maintaing hardening, have a process for regularly patching os and apps, monitor logs, have backup and disaster recovery procedures, regularly test the procedures... There are many hosts out there who will take care of _all_ of that for you. You just need to install your stuff on the box, and they take care of the rest. Logs and monitoring... you'll prolly want something custom anyway rather than whatever anyone else offers. > figure how to implement data encryption at rest, you have to do that anyway > implement high-availability and so on again, many hosts who provide easy ways to manage that for you.
drdaeman 4y ago
> You need to document, implement and maintaing hardening, have a process for regularly patching os and apps, monitor logs, have backup and disaster recovery procedures, regularly test the procedures, figure how to implement data encryption at rest, implement high-availability and so on. Hey, that's not exactly a correct comparison. You don't simply get half of those in the cloud either. Log monitoring and disaster recovery is something you have to figure out yourself, the best clouds have are some foundations to build upon, and possibly - some cookiecutter template that might fit your use case (if you're really lucky it'll even be decent). And you can get same stuff on traditional servers, just with different pre-baked solutions (which also may or may not fit a particular use case and may vary from perfectly good to quite crappy). People love to brag about all the features (most not needed for your casual website), but somehow no one tells the fact that those features just won't be there when you'll start to use the cloud - because you have to be actively aware that you need them, explicitly enable some, and explicitly spend time learning, setting up and testing others. Unless we're talking about PaaS (and not a "classic" cloud like AWS, GCP or Azure), you still need someone with some sysadmin experience - except that this person must wear a different kind of sweater (with $Cloud logo rather than Tux or Beastie). All you get is some hardening an OS + managed software (like LB servers and databases) patching. Which is something that's not that hard to do on a self-managed server (well, the software updates part; hardening is a rabbit hole). But not application patching, mind you - that's your responsibility to maintain your app, the very best it can do is to run a security audit (which you can get as a service separately). And even though managed databases are tuned (still a lot of manual tuning to do if you want the engine to truly purr) and maintained they aren't all that fun and peaches the marketing materials say - sometimes you just have to e.g. spin up your own self-hosted PostgreSQL to perform the tricky migration, then replicate it back to a managed solution.
icedchai 4y ago
You'd be surprised. I've worked at many startups and mid-sized companies. You don't actually "need" to do most of those things. I've did a consulting job for a company with 100's of millions in revenue that didn't patch its servers for years. One had a 1400 day uptime!
- drdaeman 4y ago
  > didn't patch its servers for years. One had a 1400 day uptime Well, this isn't exactly the best practice to follow - but, yes, this is very common, and it works (until someone finds that there's a vulnerability to exploit). So, from a purely practical perspective - software updates are certainly not a hard requirement, one can do quite well without them (just be aware about the risks). Anyway, there are many fairly age-tested solutions for updating software on the fly, live kernel patching and live network sockets included. And a lot of cloud software updates (e.g. managed databases) incur noticeable downtime so you need a failover (and appropriate software design that can survive it). But then you can do the very same thing with two servers, no cloud magic necessary.
  - manishsharan 4y ago
    Hardening servers isn't as hard as it used to be. Ansible does a pretty good job and there are several examples of using it to harden your server. If you are using a VPS .. you could use packer and start with a fleet of servers based on your custom image.
    - drdaeman 4y ago
      Oh, I don't disagree, I just meant it can be as complex as you want it to be. Hardening is just a rabbit hole that can go on almost forever - it's never really final until you declare that enough is enough, typically there's always something more one can do, if they have time and capacity. Also, out-of-box distributions those days can be said to be already "hardened" compared to what we had 15 years ago.
tete 4y ago
> Servers can also be liability. You need to document, implement and maintaing hardening, have a process for regularly patching os and apps, monitor logs, have backup and disaster recovery procedures, regularly test the procedures, figure how to implement data encryption at rest, implement high-availability and so on. That is true. > Good platform-as-service can solve many things for you and let you focus on the core thing you are providing. I don't see how with platform-as-a-service solutions you can skip documentation, maintenance, log monitoring, disaster recovery, backups, test procedures, or encryption at rest.

jsiepkes 4y ago

I'd rather worry about (and fix) those technical problems then having to deal with possible billing-pocalyse.

jpalomaki 4y ago
Take for example OS and app upgrades. Quite often you have few requirements, for example 1)all updates should be first tested in test env 2)updates need to be installed in timely manner 3)critical updates need to be installed quickly (30 days is too slow) When you start thinking these, they are not so easy. (1) means you can’t just run “apt upgrade” on every server - you need to manage the updates to make sure they get tested first. (2) is kind of ok, but requires some work on regular basis (at least checking things) (3) means you need to monitor the updates for your stack and classify them. You can get feeds for example for Ubuntu, but does that cover whole stack. And checking these weekly (or daily) actually gets boring. All this stuff can be done, but IMHO it is time consuming and takes time from more important things. The weekly/monthly JIRA tickets for checking x, y and z get quite annoying when you also have n other things to finish. Then you start slacking on them and feel the pain when trying to collect evidence for the next procurement process check. If you have tens or hundreds of servers and can have a separate infra team with professional sysadmins then this is all fine. My rant is mainly for small teams, where same guys are supposed to develop and run things.
- wooque 4y ago
  Small teams should just configure apt unattended upgrades and call it a day. I do this with my personal server and haven't had any issues for years.
  - remram 4y ago
    You should probably reboot it then.
peterangular 4y ago
Also the cost of those services vs. a few well-configured basic VPS hosts is nuts. I was shocked to learn that to get MySQL/Postgres in GCP it's often >$40/month - even for the insanely tiny instance sizes. We're talking for like 20G storage... In contrast I've ran some pretty hot-and-heavy MySQL/Postgres databases on basic DO/Linode VPS's for half of that cost with much success. I get these cloud tools give a ton of "out of box" features - but you are paying for it at the end of the day. Anecdotally I've noticed a huge shift away from general devs/engineers having general CLI/Linux/sysadmin knowledge.
- turtlebits 4y ago
  If you had no knowledge on standing up a DB server (installing, configuring, monitoring, logging, access control, and backups, restricting network access), how much time would that take you to learn? That and your VPS requires setup for access control, patches, updates, monitoring, logging.
  - peterangular 4y ago
    Point taken, upvote given, but for a lot of people standing up a standard redundant DB config for MySQL and Postgres isn't that big of a deal. I say this as an engineer who has a ton of relational DB experience. Before cloud providers w/DB options were so plentiful devs/engineers/sysadmins would have to set this up by hand. --- > If you had no knowledge on standing up a DB server My point is, "but yea what if you do?" There's actual value in that I've found, especially when I'm looking to save pennies at the early-stage of a startup while in the customer aq phase. Many folks have expertise in those areas without needing a cloud provider. I do get the "but all of these things!" argument (access control, patches, updates, monitoring, logging) but often those are easily solved/solvable problems even without leveraging cloud offerings. I can get very far along with an ELK/TIG stack + uptime robot and basic VPS networking features... all for a very affordable monthly bill on the infra.
  - fartcannon 4y ago
    Starting a db on a web-server and running apt update/upgrade are fundamental parts of working on the web. If you or any of your developers don't know how to do that, you/they should seriously seriously spend the 40 minutes it takes to learn.
throwaway894345 4y ago
An occasional $4K charge is negligible for a business (almost everyone is better off dealing with this sort of overrun than troubleshooting servers, all else equal), and anyway this is a particularity of AWS CloudFront's billing protections rather than a fundamental flaw with serverless.
- ggktk 4y ago
  In most countries in the world, $4K charge is absolutely not negligible.
  - throwaway894345 4y ago
    I doubt this is true. If you're operating a tech company anywhere in the world, I'm guessing $4K is pretty close to negligible. In whatever case, indexing by country is misleading, "tech companies" in general is more interesting, if there are a handful of tech companies in the third world for which this doesn't hold, it doesn't invalidate the generalization.
    - jsiepkes 4y ago
      In SMB companies around the world "an occasional $4K charge" is definitely not negliable.
      - throwaway894345 4y ago
        Again, we’re not talking about SMBs, we’re talking about tech companies generally. (What share of SMBs are there doing serverless or with any cloud footprint at all?)
ImPostingOnHN 4y ago
I'd rather have to to plan and budget than have to deal with patching (after testing the patch) every piece of software on a system the moment a patch comes out to avoid an undetectable rootkit being installed during that gap

cft 4y ago

I ran a website with 37 million users and 3.6Gbps peak bandwidth (JavaScript+thumbnails, no video) from my own two racks of Linux servers thay i have not systematically updated for years. The OSes were beyond the lts support winows. I manually compiled my own updates, but very rarely and only those that i deemed critical. Granted, the site stack was completely custom so the standard automated hacks didn't work. In 15 years there were no accidents.

booi 4y ago
I dont think this should be standard. It sounds like you’re saying you ran a service with 37 million people’s information on a software stack that was so old that not even the vendor is supporting it anymore and could be riddled with security issues that you wouldn’t even know much less be able to detect? It may work but certainly not going to get any security certifications this way..
- cft 4y ago
  I hardly stored any PII. Also, this was a high profile site that you know: if it was hacked, they would probably try to deny access and extort: it would make a lot of monetary sense. We would also have lost our merchant accounts very quickly (although CC numbers were not stored, only MD5s, but i suppose they could have been captured from the application's memory after TLS decryption but before MD5 hashing, although this would have been a difficult task even on a rooted server). I am 99.999% confident by observing and recording loads, traffic, logs and other parameters of all servers that there were no intrusions. I understand that there's market size and appeal for bureaucracy and certifications, but i have my own data.
  - Fnoord 4y ago
    Do you save logs? If yes, there's your PII.
    - cft 4y ago
      Thankfully, the servers were not in Europe! I said "hardly". It's genuinely amusing how people downvote facts, simply because they don't fit the zeitgeist.

deleted 4y ago

[deleted]

asddubs 4y ago

you can get servers with managed hosting, still at a fixed rate each month

davidfstr 4y ago

> Servers can also be liability. You need to document, implement and maintaing hardening, have a process for regularly patching os and apps, […], have backup and disaster recovery procedures, regularly test the procedures, […] and so on. Much of this complexity I eliminate by using the Immutable Server pattern: Specifically I deploy my app as a Docker container hosted on a virtual machine cluster managed by AWS ElasticBeanstalk. OS upgrades and patches (outside the container) are done by AWS. If anything goes wrong with a VM I just terminate it and a different fresh VM spins up to take its place.

huksley 4y ago
This is that I was thinking doing also - switch to using Elastic Beanstalk. What do you do if you need to launch another instance of the app? For staging or testing.

rollcat 4y ago

I agree with you in general, that it's best not to solve problems you don't have, but it's worth considering what exactly do the alternatives offer, as you might end up trading one kind of complexity or maintenance burden for another. Fixed workload, shared dev boxes, all kinds of small-scale operations fit very well onto a bare metal / VPS box in someone else's rack. The maintenance work often boils down to setting up cron with unattended upgrades and borgbackup, Docker with Traefik+ACME as the web frontend, etc. PaaS solutions will often pull you in the direction of getting "addons" or using extra paid services for every component you might need: a relational database, a cache, a queue, a load balancer... Which are usually a very good call if you need to scale up, but which could've all been entries in your docker-compose.yml if that's all you needed. Most importantly, and regardless of your hosting strategy, you can't just dismiss security, disaster recovery, or capacity planning. Clouds have outages and bottlenecks too, and no product can save you from human error.

masukomi 4y ago

> You need to document, implement and maintaing hardening, have a process for regularly patching os and apps, monitor logs, have backup and disaster recovery procedures, regularly test the procedures... There are many hosts out there who will take care of _all_ of that for you. You just need to install your stuff on the box, and they take care of the rest. Logs and monitoring... you'll prolly want something custom anyway rather than whatever anyone else offers. > figure how to implement data encryption at rest, you have to do that anyway > implement high-availability and so on again, many hosts who provide easy ways to manage that for you.

drdaeman 4y ago

> You need to document, implement and maintaing hardening, have a process for regularly patching os and apps, monitor logs, have backup and disaster recovery procedures, regularly test the procedures, figure how to implement data encryption at rest, implement high-availability and so on. Hey, that's not exactly a correct comparison. You don't simply get half of those in the cloud either. Log monitoring and disaster recovery is something you have to figure out yourself, the best clouds have are some foundations to build upon, and possibly - some cookiecutter template that might fit your use case (if you're really lucky it'll even be decent). And you can get same stuff on traditional servers, just with different pre-baked solutions (which also may or may not fit a particular use case and may vary from perfectly good to quite crappy). People love to brag about all the features (most not needed for your casual website), but somehow no one tells the fact that those features just won't be there when you'll start to use the cloud - because you have to be actively aware that you need them, explicitly enable some, and explicitly spend time learning, setting up and testing others. Unless we're talking about PaaS (and not a "classic" cloud like AWS, GCP or Azure), you still need someone with some sysadmin experience - except that this person must wear a different kind of sweater (with $Cloud logo rather than Tux or Beastie). All you get is some hardening an OS + managed software (like LB servers and databases) patching. Which is something that's not that hard to do on a self-managed server (well, the software updates part; hardening is a rabbit hole). But not application patching, mind you - that's your responsibility to maintain your app, the very best it can do is to run a security audit (which you can get as a service separately). And even though managed databases are tuned (still a lot of manual tuning to do if you want the engine to truly purr) and maintained they aren't all that fun and peaches the marketing materials say - sometimes you just have to e.g. spin up your own self-hosted PostgreSQL to perform the tricky migration, then replicate it back to a managed solution.

icedchai 4y ago

You'd be surprised. I've worked at many startups and mid-sized companies. You don't actually "need" to do most of those things. I've did a consulting job for a company with 100's of millions in revenue that didn't patch its servers for years. One had a 1400 day uptime!

drdaeman 4y ago
> didn't patch its servers for years. One had a 1400 day uptime Well, this isn't exactly the best practice to follow - but, yes, this is very common, and it works (until someone finds that there's a vulnerability to exploit). So, from a purely practical perspective - software updates are certainly not a hard requirement, one can do quite well without them (just be aware about the risks). Anyway, there are many fairly age-tested solutions for updating software on the fly, live kernel patching and live network sockets included. And a lot of cloud software updates (e.g. managed databases) incur noticeable downtime so you need a failover (and appropriate software design that can survive it). But then you can do the very same thing with two servers, no cloud magic necessary.
- manishsharan 4y ago
  Hardening servers isn't as hard as it used to be. Ansible does a pretty good job and there are several examples of using it to harden your server. If you are using a VPS .. you could use packer and start with a fleet of servers based on your custom image.
  - drdaeman 4y ago
    Oh, I don't disagree, I just meant it can be as complex as you want it to be. Hardening is just a rabbit hole that can go on almost forever - it's never really final until you declare that enough is enough, typically there's always something more one can do, if they have time and capacity. Also, out-of-box distributions those days can be said to be already "hardened" compared to what we had 15 years ago.

tete 4y ago

> Servers can also be liability. You need to document, implement and maintaing hardening, have a process for regularly patching os and apps, monitor logs, have backup and disaster recovery procedures, regularly test the procedures, figure how to implement data encryption at rest, implement high-availability and so on. That is true. > Good platform-as-service can solve many things for you and let you focus on the core thing you are providing. I don't see how with platform-as-a-service solutions you can skip documentation, maintenance, log monitoring, disaster recovery, backups, test procedures, or encryption at rest.