6 ms·

Ask HN: Hacking concerns: does it make a difference which OS I run my SAAS on?

38 points by hoodoof 12y ago

So I'm launching a new SAAS startup. Does it make any difference from a security perspective if I use Linux or FreeBSD or something else?

patio11 12y ago
Not much. Use an LTS version of a distribution which you understand well. You're more likely to have appsec problems caused by yourself rather than being compromised by problems in the OS. Make sure you're able to do firedrills like e.g. upgrading the version of Nginx or OpenSSL which you use in response to a vulnerability in them getting discovered. We've been doing a lot of that these last two years, and it probably won't get any less common.
- WestCoastJustin 12y ago
  Going to echo patio11's comment. But add a little too. Do not run Wordpress / Wordpress Plugins, or Drupal, on the same system you are doing anything with user data. These continually have security issues and you will almost certainly be owned [1, 2]. Out source blog hosting to one of the many hosting providers that maintains the software install for you. The chances of actually getting owned running stock Apache, or Nginx, are extremely low. You really need to watch what third party apps you install, like Wordpress, Drupal, or an unsecured phpMyAdmin. ps. Make sure you have source code and database backups happening on a regular basis. If the server dies tonight, are you able to restore it to a point in time? What about getting the application stack installed quickly? Think about these things. [1] https://wordpress.org/news/category/security/ https://wordpress.org/news/category/security/ [2] https://www.drupal.org/security https://www.drupal.org/security
  - cat9 12y ago
    For any X, getting rooted is still often a matter of someone with 1000 botnet nodes to burn running a for loop against an IP range. At which point, your odds of getting rooted are the product of how desirable that target is to write for loops against and how long it has been since you last applied security updates. The reason WordPress tends to be root city is that there are a great many installs out there, of which many have never had step one done to harden it, and were last given security updates ~ when they were installed. I would still outsource WordPress, but that's mostly in the vein of "the cost of outsourcing is less than the cost of me having to think about it one hour a month, while 1 hr/mo is a reasonable floor for the time cost function, but the actual value will probably exceed that at least once in a given year." A WPEngine subscription costs $30 to "I don't care, why are you wasting my time with numbers this small." Developer time costs between $75/hr and "everything is on fire and you can only put out one fire at a time." Ergo it makes economic sense to configure nginx / DNS once, then outsource further complications to paid external support.
    - meowface 12y ago
      >For any X, getting rooted is still often a matter of someone with 1000 botnet nodes to burn running a for loop against an IP range. At which point, your odds of getting rooted are the product of how desirable that target is to write for loops against and how long it has been since you last applied security updates. If you're looking at all Internet-facing servers in aggregate, yes, but if you know what you're doing then the odds of getting owned by a random botnet are very slim. Assuming you have a good security posture, your risks are dedicated and intelligent attackers and 0-days, not botnets scanning for low hanging fruit.
      - JoachimSchipper 12y ago
        Careful, there - it's easy to forget to patch one site out of some hundred-odd. Big organizations are hard. (But yes, a single competent admin who doesn't take holidays will usually have pretty decent security.)
        meowface 12y ago
        Oh yes, once you have more than a few employees you become much more at risk of that. I was assuming a scenario of one sys admin managing a small set of servers.
  - collyw 12y ago
    I know that Wordpress has a bit of a bad reputation, but this is the first have heard that Drupal has similar problems. Is there anywhere you can check how good or bad a particular CMS or framework is regarding security?
    - spoiler 12y ago
      I work for a hosting company, and we get to see quite a few hacked CMSs. The most hacked—of the popular CMSs—would be Joomla, closely followed by WordPress, and then Drupal. However the margin between Drupal and WordPress is quite big. I can never fathom why Joomla is so popular. It's horrendous to work with. Also, the attention given to security by the majority of the plugin development community, and even Joomla's core dev team is minuscule.
    - cpach 12y ago
      I’m afraid I have no good links for you. And if you really want to to know, you should probably have a security research firm do a survey for you. But that is probably not within the budget of a startup.
    - mechanical_fish 12y ago
      I work for a very large Drupal hosting company. We host important web sites on Drupal for corporations big and small. Web service software is not secure unless you patch it rapidly. There is no silver bullet. Last year there was a big security hole in Drupal, which we patched. Much like the ones that routinely turn up in Wordpress, which they patched. Much like the ones that happened, what, three or four times to Rails in the last few years? Which are like the ones they found in OpenSSL, which are like the ones they found in bash - bash, which is not supposed to be a web service! But which is nonetheless connected to the web in lo these many ways. You have to watch for patches and apply the patches. Ubuntu gets six of them every week. There are tools that apply them automatically, or with one line of typing (which rapidly ends up in the autocomplete buffer, let me tell you), and server reboots are only needed once a week or so, but you have to do it. You have to spot the patches and you have to apply them. It is like brushing your teeth, with the occasional excitement of an urgent tooth-brushing emergency. It's mostly boring. You must also have a plan to recover if a patch doesn't get applied in time, because these patches can be hard to write and can take several tries, and because you won't always be perfectly fast at getting your chores done. So you need to keep your family jewels away from the Apache server process, preferably in an alternate universe. If you are a company, this stuff graduates from boring to "hard", because how do you know that the employee who swore to do the upgrades every week didn't forget for six critical days? But mostly it is just about patience and routine. Routine turns out to be surprisingly hard.
- michaelbuckbee 12y ago
  The OP also asked about a difference between distros (DIY) and "something else" - and I do think that there is a large difference in the security + maintenance surface area that you have to contend with on a Platform As A Service like Heroku versus your own box. What's more I think you hit on exactly the reasons why: I'm much more likely to screw something up by misconfiguration or lack of attention than a team of security and maintenance people at a company whose sole job is to make sure that I don't get burnt by these things. OP is trying to get a SAAS off the ground and not (presumably) doing something with handling distro security and server maintenance - for the minimal cost difference in getting started it's hard to see how the time+effort difference in being your own sysadmin is worth it.
  - patio11 12y ago
    FWIW: this is imminently sensible advice, too.
  - logn 12y ago
    Heroku can be expensive. You're looking at $240/yr just for SSL (vs. $0 - $30/yr for non-wildcard SSL). For a 1GB VPS, a dyno costs ~3x as much as Digital Ocean. For protection from cyber crime, you might be better off paying an insurance company than Heroku prices.
    - michaelbuckbee 12y ago
      Heroku is expensive from a hosting perspective but cheap from a sysadmin cost perspective. And it's this total cost of running a service (or at the very least the opportunity cost of not working on things essential to your business and the risk of your site getting screwed by somebody malicious) that I'm trying to convey. For a business, $240/year to get a SSL terminating load balancer maintained by a dedicated security sysadmin team to sit in front of their app looks like a heckuva deal. There is just so much work that is involved in keeping a system properly up and secure that unless you're valuing your time at near zero dollars an hour, or you have so much traffic and cash coming in that you can get a dedicated person to handle it Heroku makes monetary sense.
- stock_toaster 12y ago
  In addition to what patio11 said with regard to firedrills for updates, make sure you use some type of repeatable automation. chef, salt, puppet, ansible -- just pick one and use it... _religiously_
yeukhon 12y ago
I don't mean to offend you, but I guess offense well taken. If you are asking this question, it's really important you hire someone with really good system knowledge and security knowledge before you launch your SaaS production, besides having someone doing a proper security testing. If you are running on AWS then make sure you have a proper IAM policy and proper security group settings. If you are running applications then containers are probably a nice addition. Basic WAF is essential. Even enable fail2ban even if your servers are in private subnet. No password-based SSH. UPDATE UPDATE UPDATE. These will get you enough before someone finds a remote shell exploit take over your server or someone leaks secret key in github repo or before you get XSS pwned.
- todd3834 12y ago
  I disagree, unless your SaaS is storing highly sensitive data.
  - TallGuyShort 12y ago
    I wouldn't use a SaaS if I wasn't convinced you considered everything about me highly sensitive.
    - todd3834 12y ago
      I respect your opinion and clearly the person asking the question is trying to learn how to secure the server. I do not feel like they need to hire someone to handle the security for them. However there most certainly is a difference between storing some web analytics and personal finance information.
  - yeukhon 12y ago
    Sorry, but if your server is hacked, you lose trust, even if all you do is keeping my five years old cousin's drawing. deb.haskell.org didn't store anything sensitive, but as a server storing the .deb it was hacked and has impact on possibly ten of thousands of users. Just an example.
    - 12y ago
todd3834 12y ago
I suggest you pick an LTS version of an OS that is extremely popular like Ubuntu. That way you will have lots of people paying attention and patching security flaws ASAP. Then make sure you update regularly/automatically. Put it behind a strong firewall, AWS makes this really easy but you can do everything you need to with iptables. Disable password based auth in ssh and force key auth. I disagree with yeukhon. You don't need to hire someone just because you asked that question. Unless you are storing highly sensitive data, then I'll leave that up to you. Just remember to constantly update your server.
philips 12y ago
It generally comes down to picking something you understand how to update and manage. Most of the base OS security is a practice in staying up to date and applying patches very regularly. There are two types of Linux OSes in general: versioned release and rolling release, which I will explain below. When choosing a versioned release distro you need to pick something that you feel comfortable you can update on a regular basis without breaking your application dependencies AND plan for the eventual migration to the next versioned release of the distro. Any release you choose will be supported for several years with security updates and then fazed out. When that end of life date hits you must move your application to that next version. Examples of these types of distros are CentOS, Ubuntu LTS, etc. An alternative is building your application with containers and choosing a rolling update distro. The advantage for security is two fold: 1) It makes it easier to roll out new versions of your application on a regular basis because it is possible for two versions of your app to live side-by-side. And for a new web based project most of your security problems will likely be from your web frameworks, custom code, etc not from the underlying OS. So, having a system in place to get those updates out will be critical. 2) The Linux OS can be updated on a much tighter cadence because your application isn't relying on much from the OS besides the container runtime. Rolling release server distros include Arch Linux, CoreOS, Gentoo, etc. CoreOS even fully-automates the OS update process to remove the burden of remembering to audit and apply updates. Using a rolling release distro without containers is an option too but can be bumpier since you will regularly be going through migrations like /usr/bin/python being 2.7 and then migrating to 3.0 without you noticing. The final thing you might consider, particularly if operations aren't your forte, is using something like Google App Engine, Heroku, etc. These systems are ideal for building a prototype and focusing on the initial minimal viable product. disclosure: I work at CoreOS
jmbwell 12y ago
What is your exit strategy? What will your buyer value? What is best for your product? What do your engineers understand?
Galacticus 12y ago
Yes, it should be able to not only Read but Write data.
nikanj 12y ago
Don't go for analysis paralysis
zhte415 12y ago
My 2p: Use something you're familiar with, and that you have a good source of information flow, by which I mean, subscribe to security updates (including for whatever stack you put on your server), and certainly have a test server simply for playing with things, getting used to functionality. I have a shell app on my phone, which I love, so I can log in at a moment's notice, wherever I am, just to check things out or apply a change. The Linode and DO tutorials are pretty nice, and cover the vast amount of eventualities. If you're unsure which to choose, go with the one that has more documentation for your use case. DevOps is not cloak-and-dagger mysticism; having control and understanding of your stack is, imho, far more important than choosing between a distro to run it on. Just make sure your server is locked down and that you regularly apply updates (that you've tested on your test server).
PeterWhittaker 12y ago
If you are not using someone else's proven platform, then yours will be hacked, possibly several times, at least until you've learned how to secure it and have kept it secure for some time. With that in mind, you must, must, must do a few things: 1. Get really, really good at rollback and recovery. Make sure you can completely rebuild your operation from a clean system in as short a time as possible, with no data loss. Certain types of compromise will require this, it will be the cleanest, safest way to recover. 2. #1 requires you to be really good at backups. Unless you have experience here, they're harder than you think. 3. Notice that #1 starts with "rollback": You will occasionally be your own worst enemy, at least until your release processes get really, really good. Every now and again you will apply something that breaks something else. Being able to get back to a known good state quickly and easily will save you countless hours and much stress. 4. Find an experienced penetration tester that you trust implicitly. Have them review your operational architecture; they will find flaws (that's just on paper). Have them review your hardware and software configurations; they will find flaws. Listen to their proposals for risk mitigation. Once you have a system that passes their paper attack, have them actually kick at it, from three different places: one, from the outside, from the Internet everyone comes from; two, from just inside, from within your first tier, between outside interfaces to the world and app logic; three, from deep within, behind the app tier (using test data, of course). The three reports should be very different, with increasing levels of access and vulnerability as they go (because you are letting them past the mitigations). This will give you considerable information about how at risk you are. (I know quite a good one, does this sort of thing professionally for government, private sector, the military, police forces, etc. He'll break your stuff. And tell you how to make it better.) 5. #4 implies, strongly implies, that you have a restricted access test environment that matches your operational environment as closely as possible, with two exceptions: one, no production data; two, it may be a patch or two or a major update ahead of production, since it's where you will soak changes to make sure they work properly and break nothing. 6. Managing #5, #3, and #1 means you must, must, must understand and get good at change management. And risk assessment and risk management. IT Security is mostly about governance and management. If you focus on technology first, you've already lost.

38 points by hoodoof 12y ago

So I'm launching a new SAAS startup. Does it make any difference from a security perspective if I use Linux or FreeBSD or something else?

patio11 12y ago

Not much. Use an LTS version of a distribution which you understand well. You're more likely to have appsec problems caused by yourself rather than being compromised by problems in the OS. Make sure you're able to do firedrills like e.g. upgrading the version of Nginx or OpenSSL which you use in response to a vulnerability in them getting discovered. We've been doing a lot of that these last two years, and it probably won't get any less common.

WestCoastJustin 12y ago
Going to echo patio11's comment. But add a little too. Do not run Wordpress / Wordpress Plugins, or Drupal, on the same system you are doing anything with user data. These continually have security issues and you will almost certainly be owned [1, 2]. Out source blog hosting to one of the many hosting providers that maintains the software install for you. The chances of actually getting owned running stock Apache, or Nginx, are extremely low. You really need to watch what third party apps you install, like Wordpress, Drupal, or an unsecured phpMyAdmin. ps. Make sure you have source code and database backups happening on a regular basis. If the server dies tonight, are you able to restore it to a point in time? What about getting the application stack installed quickly? Think about these things. [1] https://wordpress.org/news/category/security/ https://wordpress.org/news/category/security/ [2] https://www.drupal.org/security https://www.drupal.org/security
- cat9 12y ago
  For any X, getting rooted is still often a matter of someone with 1000 botnet nodes to burn running a for loop against an IP range. At which point, your odds of getting rooted are the product of how desirable that target is to write for loops against and how long it has been since you last applied security updates. The reason WordPress tends to be root city is that there are a great many installs out there, of which many have never had step one done to harden it, and were last given security updates ~ when they were installed. I would still outsource WordPress, but that's mostly in the vein of "the cost of outsourcing is less than the cost of me having to think about it one hour a month, while 1 hr/mo is a reasonable floor for the time cost function, but the actual value will probably exceed that at least once in a given year." A WPEngine subscription costs $30 to "I don't care, why are you wasting my time with numbers this small." Developer time costs between $75/hr and "everything is on fire and you can only put out one fire at a time." Ergo it makes economic sense to configure nginx / DNS once, then outsource further complications to paid external support.
  - meowface 12y ago
    >For any X, getting rooted is still often a matter of someone with 1000 botnet nodes to burn running a for loop against an IP range. At which point, your odds of getting rooted are the product of how desirable that target is to write for loops against and how long it has been since you last applied security updates. If you're looking at all Internet-facing servers in aggregate, yes, but if you know what you're doing then the odds of getting owned by a random botnet are very slim. Assuming you have a good security posture, your risks are dedicated and intelligent attackers and 0-days, not botnets scanning for low hanging fruit.
    - JoachimSchipper 12y ago
      Careful, there - it's easy to forget to patch one site out of some hundred-odd. Big organizations are hard. (But yes, a single competent admin who doesn't take holidays will usually have pretty decent security.)
      - meowface 12y ago
        Oh yes, once you have more than a few employees you become much more at risk of that. I was assuming a scenario of one sys admin managing a small set of servers.
- collyw 12y ago
  I know that Wordpress has a bit of a bad reputation, but this is the first have heard that Drupal has similar problems. Is there anywhere you can check how good or bad a particular CMS or framework is regarding security?
  - spoiler 12y ago
    I work for a hosting company, and we get to see quite a few hacked CMSs. The most hacked—of the popular CMSs—would be Joomla, closely followed by WordPress, and then Drupal. However the margin between Drupal and WordPress is quite big. I can never fathom why Joomla is so popular. It's horrendous to work with. Also, the attention given to security by the majority of the plugin development community, and even Joomla's core dev team is minuscule.
  - cpach 12y ago
    I’m afraid I have no good links for you. And if you really want to to know, you should probably have a security research firm do a survey for you. But that is probably not within the budget of a startup.
  - mechanical_fish 12y ago
    I work for a very large Drupal hosting company. We host important web sites on Drupal for corporations big and small. Web service software is not secure unless you patch it rapidly. There is no silver bullet. Last year there was a big security hole in Drupal, which we patched. Much like the ones that routinely turn up in Wordpress, which they patched. Much like the ones that happened, what, three or four times to Rails in the last few years? Which are like the ones they found in OpenSSL, which are like the ones they found in bash - bash, which is not supposed to be a web service! But which is nonetheless connected to the web in lo these many ways. You have to watch for patches and apply the patches. Ubuntu gets six of them every week. There are tools that apply them automatically, or with one line of typing (which rapidly ends up in the autocomplete buffer, let me tell you), and server reboots are only needed once a week or so, but you have to do it. You have to spot the patches and you have to apply them. It is like brushing your teeth, with the occasional excitement of an urgent tooth-brushing emergency. It's mostly boring. You must also have a plan to recover if a patch doesn't get applied in time, because these patches can be hard to write and can take several tries, and because you won't always be perfectly fast at getting your chores done. So you need to keep your family jewels away from the Apache server process, preferably in an alternate universe. If you are a company, this stuff graduates from boring to "hard", because how do you know that the employee who swore to do the upgrades every week didn't forget for six critical days? But mostly it is just about patience and routine. Routine turns out to be surprisingly hard.
michaelbuckbee 12y ago
The OP also asked about a difference between distros (DIY) and "something else" - and I do think that there is a large difference in the security + maintenance surface area that you have to contend with on a Platform As A Service like Heroku versus your own box. What's more I think you hit on exactly the reasons why: I'm much more likely to screw something up by misconfiguration or lack of attention than a team of security and maintenance people at a company whose sole job is to make sure that I don't get burnt by these things. OP is trying to get a SAAS off the ground and not (presumably) doing something with handling distro security and server maintenance - for the minimal cost difference in getting started it's hard to see how the time+effort difference in being your own sysadmin is worth it.
- patio11 12y ago
  FWIW: this is imminently sensible advice, too.
- logn 12y ago
  Heroku can be expensive. You're looking at $240/yr just for SSL (vs. $0 - $30/yr for non-wildcard SSL). For a 1GB VPS, a dyno costs ~3x as much as Digital Ocean. For protection from cyber crime, you might be better off paying an insurance company than Heroku prices.
  - michaelbuckbee 12y ago
    Heroku is expensive from a hosting perspective but cheap from a sysadmin cost perspective. And it's this total cost of running a service (or at the very least the opportunity cost of not working on things essential to your business and the risk of your site getting screwed by somebody malicious) that I'm trying to convey. For a business, $240/year to get a SSL terminating load balancer maintained by a dedicated security sysadmin team to sit in front of their app looks like a heckuva deal. There is just so much work that is involved in keeping a system properly up and secure that unless you're valuing your time at near zero dollars an hour, or you have so much traffic and cash coming in that you can get a dedicated person to handle it Heroku makes monetary sense.
stock_toaster 12y ago
In addition to what patio11 said with regard to firedrills for updates, make sure you use some type of repeatable automation. chef, salt, puppet, ansible -- just pick one and use it... _religiously_

yeukhon 12y ago

I don't mean to offend you, but I guess offense well taken. If you are asking this question, it's really important you hire someone with really good system knowledge and security knowledge before you launch your SaaS production, besides having someone doing a proper security testing. If you are running on AWS then make sure you have a proper IAM policy and proper security group settings. If you are running applications then containers are probably a nice addition. Basic WAF is essential. Even enable fail2ban even if your servers are in private subnet. No password-based SSH. UPDATE UPDATE UPDATE. These will get you enough before someone finds a remote shell exploit take over your server or someone leaks secret key in github repo or before you get XSS pwned.

todd3834 12y ago
I disagree, unless your SaaS is storing highly sensitive data.
- TallGuyShort 12y ago
  I wouldn't use a SaaS if I wasn't convinced you considered everything about me highly sensitive.
  - todd3834 12y ago
    I respect your opinion and clearly the person asking the question is trying to learn how to secure the server. I do not feel like they need to hire someone to handle the security for them. However there most certainly is a difference between storing some web analytics and personal finance information.
- yeukhon 12y ago
  Sorry, but if your server is hacked, you lose trust, even if all you do is keeping my five years old cousin's drawing. deb.haskell.org didn't store anything sensitive, but as a server storing the .deb it was hacked and has impact on possibly ten of thousands of users. Just an example.
  - 12y ago

todd3834 12y ago

I suggest you pick an LTS version of an OS that is extremely popular like Ubuntu. That way you will have lots of people paying attention and patching security flaws ASAP. Then make sure you update regularly/automatically. Put it behind a strong firewall, AWS makes this really easy but you can do everything you need to with iptables. Disable password based auth in ssh and force key auth. I disagree with yeukhon. You don't need to hire someone just because you asked that question. Unless you are storing highly sensitive data, then I'll leave that up to you. Just remember to constantly update your server.

philips 12y ago

It generally comes down to picking something you understand how to update and manage. Most of the base OS security is a practice in staying up to date and applying patches very regularly. There are two types of Linux OSes in general: versioned release and rolling release, which I will explain below. When choosing a versioned release distro you need to pick something that you feel comfortable you can update on a regular basis without breaking your application dependencies AND plan for the eventual migration to the next versioned release of the distro. Any release you choose will be supported for several years with security updates and then fazed out. When that end of life date hits you must move your application to that next version. Examples of these types of distros are CentOS, Ubuntu LTS, etc. An alternative is building your application with containers and choosing a rolling update distro. The advantage for security is two fold: 1) It makes it easier to roll out new versions of your application on a regular basis because it is possible for two versions of your app to live side-by-side. And for a new web based project most of your security problems will likely be from your web frameworks, custom code, etc not from the underlying OS. So, having a system in place to get those updates out will be critical. 2) The Linux OS can be updated on a much tighter cadence because your application isn't relying on much from the OS besides the container runtime. Rolling release server distros include Arch Linux, CoreOS, Gentoo, etc. CoreOS even fully-automates the OS update process to remove the burden of remembering to audit and apply updates. Using a rolling release distro without containers is an option too but can be bumpier since you will regularly be going through migrations like /usr/bin/python being 2.7 and then migrating to 3.0 without you noticing. The final thing you might consider, particularly if operations aren't your forte, is using something like Google App Engine, Heroku, etc. These systems are ideal for building a prototype and focusing on the initial minimal viable product. disclosure: I work at CoreOS

jmbwell 12y ago

What is your exit strategy? What will your buyer value? What is best for your product? What do your engineers understand?

Galacticus 12y ago

Yes, it should be able to not only Read but Write data.

nikanj 12y ago

Don't go for analysis paralysis

zhte415 12y ago

My 2p: Use something you're familiar with, and that you have a good source of information flow, by which I mean, subscribe to security updates (including for whatever stack you put on your server), and certainly have a test server simply for playing with things, getting used to functionality. I have a shell app on my phone, which I love, so I can log in at a moment's notice, wherever I am, just to check things out or apply a change. The Linode and DO tutorials are pretty nice, and cover the vast amount of eventualities. If you're unsure which to choose, go with the one that has more documentation for your use case. DevOps is not cloak-and-dagger mysticism; having control and understanding of your stack is, imho, far more important than choosing between a distro to run it on. Just make sure your server is locked down and that you regularly apply updates (that you've tested on your test server).

PeterWhittaker 12y ago

If you are not using someone else's proven platform, then yours will be hacked, possibly several times, at least until you've learned how to secure it and have kept it secure for some time. With that in mind, you must, must, must do a few things: 1. Get really, really good at rollback and recovery. Make sure you can completely rebuild your operation from a clean system in as short a time as possible, with no data loss. Certain types of compromise will require this, it will be the cleanest, safest way to recover. 2. #1 requires you to be really good at backups. Unless you have experience here, they're harder than you think. 3. Notice that #1 starts with "rollback": You will occasionally be your own worst enemy, at least until your release processes get really, really good. Every now and again you will apply something that breaks something else. Being able to get back to a known good state quickly and easily will save you countless hours and much stress. 4. Find an experienced penetration tester that you trust implicitly. Have them review your operational architecture; they will find flaws (that's just on paper). Have them review your hardware and software configurations; they will find flaws. Listen to their proposals for risk mitigation. Once you have a system that passes their paper attack, have them actually kick at it, from three different places: one, from the outside, from the Internet everyone comes from; two, from just inside, from within your first tier, between outside interfaces to the world and app logic; three, from deep within, behind the app tier (using test data, of course). The three reports should be very different, with increasing levels of access and vulnerability as they go (because you are letting them past the mitigations). This will give you considerable information about how at risk you are. (I know quite a good one, does this sort of thing professionally for government, private sector, the military, police forces, etc. He'll break your stuff. And tell you how to make it better.) 5. #4 implies, strongly implies, that you have a restricted access test environment that matches your operational environment as closely as possible, with two exceptions: one, no production data; two, it may be a patch or two or a major update ahead of production, since it's where you will soak changes to make sure they work properly and break nothing. 6. Managing #5, #3, and #1 means you must, must, must understand and get good at change management. And risk assessment and risk management. IT Security is mostly about governance and management. If you focus on technology first, you've already lost.