3 ms·

Building once and deploying the binaries everywhere is standard practice everywhere I know of. If you have subtle differences between libraries on different ma

by amock 15y ago

Building once and deploying the binaries everywhere is standard practice everywhere I know of. If you have subtle differences between libraries on different machines you should fix that rather than recompiling your code on each one. Removing the compiler isn't less secure, and since they are the ones making the rules you have to follow them. It should also dramatically speed up your deployments when you don't have to rebuild on each machine. I would recommend that you don't do things just because that's how you were taught 15 years ago. There are probably some things that haven't changed since then, but so many things are different. With virtualization you can run the exact same image on all of your machines so that there are no library issues. Almost all Linux distributions provide binaries rather than source code if you use their package management systems and they work very well.

tingletech 15y ago
Well, we are paying them, so we can ask for the rules to be changed. We just need to have a business justification and negotiate the change. I have no power to fix the fact that stage is in a different datacenter with different admins who use different repositories for SLES packages. The two data centers will not not accept vm images from each other. SLES is not like most linuxes in terms of package support. And the production datacenter does not even let us use full list of SLES packages, only the ones in their local repo. Also, my build script works on solaris and linux so the environment is pretty close between those two by building my own packages. I don't need to change python/apache etc. that often, it is not like I need to recompile for every code deploy. I have it pretty well automated on solaris. If it is not more secure to remove the compiler in production, then I can make a strong argument for them to change their rule.
- amock 15y ago
  Even if you can make an argument that it is not less secure I would still recommend being able to run the same binaries in production as you do in test. Having a staging system that is as close as feasible to your production system is wonderful for debugging. If I were you I'd recommend trying to get the two data centers to use the same images. If that's not possible and all the prod machines run the same image I'd build a package on one prod machine and install that on all of them. If not even all the prod machines are the same then building on each one might be the best you can do unless you want to statically link everything. Getting back to your original question, I don't see how having compilers in production is bad for security except if the compilers have malicious code in them. However, my team's policies do not allow anything on our production boxes that didn't go through our build and deployment systems. If an attacker can run a compiler he can probably run arbitrary code anyway.
  - tingletech 15y ago
    > If I were you I'd recommend trying to get the two data centers to use the same images. Yes... we have tried... this is not going to happen. The battle I'm picking is to get a compiler on production. In SUSE dev and stage started as clones and production is very close to dev/stage but built by a different group; but in solaris land things are more divergent between dev stage and production because the openCSW packages were all installed on different days from the main public repository which is consistently being updated. Plus, since -dev is shared with ~30 developers and stage is shared with about ~10 developers and folks ask the dev/stage admins for different packages things start to diverge more as dependencies get upgraded. Where I have problems is trying to compile things like GD and openquicktime that are used by some legacy processes that seem to be very sensitive to certain libraries being at the right version -- maybe this is not going to be an issue on SLES but I've been bitten in the past so many times by subtle library version issues and I find I have the best luck if I always rebuild my entire stack after any OS update or package changes. I know it was another age (when in one day I might have to work on old SunOS, new Solaris, HP/UX, AIX, BSDi, and IRIX boxes -- and dev and stage machines were unheard of), but the grey beards (these dudes literally had awesome grey beards) I learned this trick from had a lot of experience and I still believe that building locally is the best bet unless you can guarantee the systems are identical in every detail (which I will never be able to do because I'm not a sysadmin anymore). > If that's not possible and all the prod machines run the same image I'd build a package on one prod machine and install that on all of them. I will only have one production vm for my app. The main application I run has ~6k unique visitors a day and ~130k unique visitors a month with an average of 4 pages per visit -- not super high load so I don't need multiple back end servers -- at least on my shared solaris host. I haven't done load testing on the vms yet -- but if I do need multiple production vms they will be able to clone the vms for me in production.

by amock 15y ago

tingletech 15y ago

Well, we are paying them, so we can ask for the rules to be changed. We just need to have a business justification and negotiate the change. I have no power to fix the fact that stage is in a different datacenter with different admins who use different repositories for SLES packages. The two data centers will not not accept vm images from each other. SLES is not like most linuxes in terms of package support. And the production datacenter does not even let us use full list of SLES packages, only the ones in their local repo. Also, my build script works on solaris and linux so the environment is pretty close between those two by building my own packages. I don't need to change python/apache etc. that often, it is not like I need to recompile for every code deploy. I have it pretty well automated on solaris. If it is not more secure to remove the compiler in production, then I can make a strong argument for them to change their rule.

amock 15y ago
Even if you can make an argument that it is not less secure I would still recommend being able to run the same binaries in production as you do in test. Having a staging system that is as close as feasible to your production system is wonderful for debugging. If I were you I'd recommend trying to get the two data centers to use the same images. If that's not possible and all the prod machines run the same image I'd build a package on one prod machine and install that on all of them. If not even all the prod machines are the same then building on each one might be the best you can do unless you want to statically link everything. Getting back to your original question, I don't see how having compilers in production is bad for security except if the compilers have malicious code in them. However, my team's policies do not allow anything on our production boxes that didn't go through our build and deployment systems. If an attacker can run a compiler he can probably run arbitrary code anyway.
- tingletech 15y ago
  > If I were you I'd recommend trying to get the two data centers to use the same images. Yes... we have tried... this is not going to happen. The battle I'm picking is to get a compiler on production. In SUSE dev and stage started as clones and production is very close to dev/stage but built by a different group; but in solaris land things are more divergent between dev stage and production because the openCSW packages were all installed on different days from the main public repository which is consistently being updated. Plus, since -dev is shared with ~30 developers and stage is shared with about ~10 developers and folks ask the dev/stage admins for different packages things start to diverge more as dependencies get upgraded. Where I have problems is trying to compile things like GD and openquicktime that are used by some legacy processes that seem to be very sensitive to certain libraries being at the right version -- maybe this is not going to be an issue on SLES but I've been bitten in the past so many times by subtle library version issues and I find I have the best luck if I always rebuild my entire stack after any OS update or package changes. I know it was another age (when in one day I might have to work on old SunOS, new Solaris, HP/UX, AIX, BSDi, and IRIX boxes -- and dev and stage machines were unheard of), but the grey beards (these dudes literally had awesome grey beards) I learned this trick from had a lot of experience and I still believe that building locally is the best bet unless you can guarantee the systems are identical in every detail (which I will never be able to do because I'm not a sysadmin anymore). > If that's not possible and all the prod machines run the same image I'd build a package on one prod machine and install that on all of them. I will only have one production vm for my app. The main application I run has ~6k unique visitors a day and ~130k unique visitors a month with an average of 4 pages per visit -- not super high load so I don't need multiple back end servers -- at least on my shared solaris host. I haven't done load testing on the vms yet -- but if I do need multiple production vms they will be able to clone the vms for me in production.