9 ms·

I agree that many of these convenient setups are embarrassingly sloppy, but it's the sysadmin's responsibility to insist on production deployments being far mor

I agree that many of these convenient setups are embarrassingly sloppy, but it's the sysadmin's responsibility to insist on production deployments being far more rigorous. No one can tell you how to build hadoop? Well, figure it out. Random Docker containers being downloaded? Use a local Docker repo with vetted containers and Dockerfiles only. I don't even allow vendor installers to run on my production systems. My employer buys some software that is distributed as binary installers. So I've written a script that will run that installer in a VM, and repackage the resulting files into something I'm comfortable working with to deploy to production. If a sysadmin is unable to insist on good deployment practices, it's a failure of the company or organization or of his own communication skills. If a sysadmin allows sloppy developer-created deployments and doesn't make constant noise about it, then they aren't doing their job properly.

Nursie 12y ago
>> No one can tell you how to build hadoop? Well, figure it out. I get the impression that several people working on debian couldn't work this one out!
- azinman2 12y ago
  Surely at minimum Hadoop developers could tell you!
  - marcosdumay 12y ago
    Have you even been in a project where the developers didn't know how to build it? It's a strange situation, with huge environments being passed from one computer to another, and treasured with more care than the code itself.
    - StillBored 12y ago
      This happened to me about a decade ago. A very smart sysadmin in the company created an acronis image for machine deployments. They very carefully documented everything they changed, and how to recreate it. Then someone else created an image from one of the imaged machines without documenting what they changed. This happened a couple dozen or so times until the image pretty much was a mess of hand installed binaries, configuration hacks, etc. It literally took another person 6 months to untwist what was actually on the machine by md5suming the crap out of everything guessing at versions until they found a match, and documenting it. That sounds like the state of a lot of docker images.
      - EC1 12y ago
        Well fuck me. I just spent two weeks fiddling with Vagrant and Docker and finally got everything up and humming only to come into this thread. Going to refrain from slapping the SysAdmin title on myself for now.
        odonnellryan 12y ago
        Docker is awesome, but you shouldn't be using blind base images. Use Dockerfiles, they're self-documenting.
        tracker1 12y ago
        Unless you build your own base images... odds are you will be using something someone else built. Even the host OS probably wasn't compiled by you. In general, my base images are often debian:wheezy, ubuntu:trusty or alpine:latest ... From here, a number of times I've tracked down the dockerfiles (usually in github) for a given image... for the most part, if the image is a default image, I've got a fair amount of trust in that (the build system is pretty sane in that regard)... though some bits aren't always as straight forward. I learned a lot just from reading/tracing through the dockerfiles for iojs and mono ... What is interesting is often the dockerfile simply adds a repository, and installs package X using the base os's package manager. I'm not certain it's nearly as big of a problem as people make it out to be (with exception to hadoop/java projects, which tend to be far more complicated than they should be). golang's onbuild containers are really interesting. I've also been playing with building in one node container with build tools, then deploying the resulting node_modules + app into another more barebones container base.
        odonnellryan 12y ago
        Well, you have to trust something somewhere. Unless you're always compiling from source (which you can do with Docker), and you've read the source, etc.. but even then, you have to trust the compiler and the hardware. Anyway, yes, you can make your own base images. But, images `should` be light enough where you can build them each iteration. I've done dev stacks where literally each `save/commit/run of a test` built the docker container from the dockerfile in the background! With the caching docker does it really doesn't add any overhead to the process. > What is interesting is often the dockerfile simply adds a repository, and installs package X using the base os's package manager. Yup! Pretty much. Other than some config stuff for very specific use cases (VPN, whatever.)
    - pnathan 12y ago
      A legend at one company about 5 years ago is that the company's next world-shaking product was being built partially with a single computer that was shipped around from office to office, because no one knew how to build the build environment again. Again this was circa 2010. :-)
- api 12y ago
  Hadoop is insane. The elephant is fitting. Is it really the best choice, or has someone done something cleaner in golang or c++11?
  - Alupis 12y ago
    > Is it really the best choice, or has someone done something cleaner in golang or c++11? What does the language have to do with the program? Hadoop is what it is because it's a complex problem with a fittingly complex solution. Simply re-writing it in your pet language won't somehow make it "better".
    - getsat 12y ago
      Go and modern C++ are both quite a bit more terse than Java. They also produce binaries which don't necessarily require a runtime to be available on every server (just ABI compatibility). (I have no horse in this race, I am just writing what I think the grandparent comment was referring to)
      - pjmlp 12y ago
        > They also produce binaries which don't necessarily require a runtime to be available on every server Just like Java[0]. It is just a matter of choosing the right compiler for the use case at hand. [0] - http://www.excelsiorjet.com/ http://www.excelsiorjet.com/ (one from many vendors)
        Alupis 12y ago
        There's also things like exec4j which bundles everything including a JVM into an executable which one can just run... and things like AdvancedInstaller and Install4j will also allow one to bundle a JVM. So producing a binary which doesn't require a separate runtime really isn't a problem.
        pjmlp 12y ago
        Since you mention it, Java 8 brings bundling and installers support into the reference JDK.
- bobbyi_settv 12y ago
  I think most people who use debian would tend to install things using debian packages, which in this case usually means adding cloudera to your apt sources list and using apt-get. It is a pretty straightforward process: http://www.cloudera.com/content/cloudera/en/documentation/cdh5/v5-0-0/CDH5-Installation-Guide/cdh5ig_cdh5_install.html http://www.cloudera.com/content/cloudera/en/documentation/cd...
  - growse 12y ago
    I think the complaint was that it's difficult figuring out how to build Hadoop from source. That page you linked is how to install pre-built binaries, which you rightly point out is fairly trivial.
  - Nursie 12y ago
    Sure, I agree that debian people would want to install debian packages. What debian users/hackers/amateur admins like me really want is packages that are first class citizens, that the debian guys have picked up, sanitised, analysed and made part of the system. I'll take software from the debian repos every time if I can. And it's pretty damning if people who are familiar with build systems and package creation can't figure it out!
ksml 12y ago
I think more disconcerting is the rise of "sysadmins" who think they're qualified sysadmins because they know how to bash and docker.
- tedivm 12y ago
  This is hardly a new problem- and in many ways, I'm not sure it's a problem at all compared to the company cultural issues brought up by skywhopper. Whether it's programming or system administration, you're always going to have new people getting excited about the sudden power they've learned. Being able to make computers do things opens up this whole new world, and when people find themselves in that world they may end up overestimating their skills and underestimating how much they need to grow. What they fail at understanding they make up with in enthusiasm, and with experience they become more knowledgable about what they don't know. If we waited until they were "qualified" for jobs they would never get the experience to become qualified. At the same time there is more than enough room in the current job market to support people of lower skillsets, and for some companies that's considered an investment (junior people tend to turn to senior people over time). This is where it becomes a company culture issue. If a company is smart they'll have a few senior people making sure things are held to the right standard, and a few junior people who can get things done but need some guidance and direction. However, lots of companies (especially the smaller ones who may be more constrained by budget) go for the cheaper route and would rather hire someone junior as their main support. The problem isn't that the sysadmins aren't qualified sysadmins, it's that they're junior system admins who have been hired for the wrong job. Companies that fail to value experience tend to suffer as a result.
  - dba7dba 12y ago
    Amen to that. When I see some of the job desc in job postings for DevOps/Sysadmin, I wonder. Is there really someone out there will all the skills that are asked for?
    - Aloha 12y ago
      I'm reasonably certain there isnt - not for the payband offered.
jetpks 12y ago
> it's the sysadmin's responsibility to insist on production deployments What decade are you from? No startups are hiring sysadmins to do any kind of work anymore. They're hiring "dev-ops" people, which seems to mean "Amateur $popularLanguage developer that deployed on AWS this one time." That's the whole problem with the dev-ops ecosystem. None of these dev-ops people seem to have any ops experience.
- thaumaturgy 12y ago
  > No startups are hiring sysadmins to do any kind of work anymore. Then maybe people should be willing to work for more grown-up businesses. HN tends to get a distorted view of what's important in the tech industry. The tech industry is way, way, way bigger than startups, and there are still plenty of companies that recognize the value of good sysadmins. Let the startups learn their lesson in their own time.
  - dikaiosune 12y ago
    The alternative is that many of the startups don't learn this in their own time, and they go on to become bigger, more successful companies who can set the tone and shift the market. Of course, if they're actually able to succeed by doing so, then that says something too. Although the trend of many data breaches certainly wouldn't decline in that case.
    - rudolf0 12y ago
      >Although the trend of many data breaches certainly wouldn't decline in that case. Exactly. Successful and profitable are not mutually exclusive with "secure" or "well-architected". At least until those last two come to bite you later and start eating into your profits.
      - Lord_Zero 12y ago
        Sony is a great example of this.
ommunist 12y ago
@skywhopper "it's a failure of the company or organization or of his own communication skills" <~ Oh man, ever had a rant from The Management like "we pay you to do what we say"? No one usually cares about communication skills of sysadmin. Yes, its a failure of organisation. Sad truth is - most organisations are failed. Sysadmin today is a marginal job at a small company, where people respect you, or a job in the medium or large company where he or she are just peons.

Nursie 12y ago

>> No one can tell you how to build hadoop? Well, figure it out. I get the impression that several people working on debian couldn't work this one out!

azinman2 12y ago
Surely at minimum Hadoop developers could tell you!
- marcosdumay 12y ago
  Have you even been in a project where the developers didn't know how to build it? It's a strange situation, with huge environments being passed from one computer to another, and treasured with more care than the code itself.
  - StillBored 12y ago
    This happened to me about a decade ago. A very smart sysadmin in the company created an acronis image for machine deployments. They very carefully documented everything they changed, and how to recreate it. Then someone else created an image from one of the imaged machines without documenting what they changed. This happened a couple dozen or so times until the image pretty much was a mess of hand installed binaries, configuration hacks, etc. It literally took another person 6 months to untwist what was actually on the machine by md5suming the crap out of everything guessing at versions until they found a match, and documenting it. That sounds like the state of a lot of docker images.
    - EC1 12y ago
      Well fuck me. I just spent two weeks fiddling with Vagrant and Docker and finally got everything up and humming only to come into this thread. Going to refrain from slapping the SysAdmin title on myself for now.
      - odonnellryan 12y ago
        Docker is awesome, but you shouldn't be using blind base images. Use Dockerfiles, they're self-documenting.
        tracker1 12y ago
        Unless you build your own base images... odds are you will be using something someone else built. Even the host OS probably wasn't compiled by you. In general, my base images are often debian:wheezy, ubuntu:trusty or alpine:latest ... From here, a number of times I've tracked down the dockerfiles (usually in github) for a given image... for the most part, if the image is a default image, I've got a fair amount of trust in that (the build system is pretty sane in that regard)... though some bits aren't always as straight forward. I learned a lot just from reading/tracing through the dockerfiles for iojs and mono ... What is interesting is often the dockerfile simply adds a repository, and installs package X using the base os's package manager. I'm not certain it's nearly as big of a problem as people make it out to be (with exception to hadoop/java projects, which tend to be far more complicated than they should be). golang's onbuild containers are really interesting. I've also been playing with building in one node container with build tools, then deploying the resulting node_modules + app into another more barebones container base.
        odonnellryan 12y ago
        Well, you have to trust something somewhere. Unless you're always compiling from source (which you can do with Docker), and you've read the source, etc.. but even then, you have to trust the compiler and the hardware. Anyway, yes, you can make your own base images. But, images `should` be light enough where you can build them each iteration. I've done dev stacks where literally each `save/commit/run of a test` built the docker container from the dockerfile in the background! With the caching docker does it really doesn't add any overhead to the process. > What is interesting is often the dockerfile simply adds a repository, and installs package X using the base os's package manager. Yup! Pretty much. Other than some config stuff for very specific use cases (VPN, whatever.)
  - pnathan 12y ago
    A legend at one company about 5 years ago is that the company's next world-shaking product was being built partially with a single computer that was shipped around from office to office, because no one knew how to build the build environment again. Again this was circa 2010. :-)
api 12y ago
Hadoop is insane. The elephant is fitting. Is it really the best choice, or has someone done something cleaner in golang or c++11?
- Alupis 12y ago
  > Is it really the best choice, or has someone done something cleaner in golang or c++11? What does the language have to do with the program? Hadoop is what it is because it's a complex problem with a fittingly complex solution. Simply re-writing it in your pet language won't somehow make it "better".
  - getsat 12y ago
    Go and modern C++ are both quite a bit more terse than Java. They also produce binaries which don't necessarily require a runtime to be available on every server (just ABI compatibility). (I have no horse in this race, I am just writing what I think the grandparent comment was referring to)
    - pjmlp 12y ago
      > They also produce binaries which don't necessarily require a runtime to be available on every server Just like Java[0]. It is just a matter of choosing the right compiler for the use case at hand. [0] - http://www.excelsiorjet.com/ http://www.excelsiorjet.com/ (one from many vendors)
      - Alupis 12y ago
        There's also things like exec4j which bundles everything including a JVM into an executable which one can just run... and things like AdvancedInstaller and Install4j will also allow one to bundle a JVM. So producing a binary which doesn't require a separate runtime really isn't a problem.
        pjmlp 12y ago
        Since you mention it, Java 8 brings bundling and installers support into the reference JDK.
bobbyi_settv 12y ago
I think most people who use debian would tend to install things using debian packages, which in this case usually means adding cloudera to your apt sources list and using apt-get. It is a pretty straightforward process: http://www.cloudera.com/content/cloudera/en/documentation/cdh5/v5-0-0/CDH5-Installation-Guide/cdh5ig_cdh5_install.html http://www.cloudera.com/content/cloudera/en/documentation/cd...
- growse 12y ago
  I think the complaint was that it's difficult figuring out how to build Hadoop from source. That page you linked is how to install pre-built binaries, which you rightly point out is fairly trivial.
- Nursie 12y ago
  Sure, I agree that debian people would want to install debian packages. What debian users/hackers/amateur admins like me really want is packages that are first class citizens, that the debian guys have picked up, sanitised, analysed and made part of the system. I'll take software from the debian repos every time if I can. And it's pretty damning if people who are familiar with build systems and package creation can't figure it out!

ksml 12y ago

I think more disconcerting is the rise of "sysadmins" who think they're qualified sysadmins because they know how to bash and docker.

tedivm 12y ago
This is hardly a new problem- and in many ways, I'm not sure it's a problem at all compared to the company cultural issues brought up by skywhopper. Whether it's programming or system administration, you're always going to have new people getting excited about the sudden power they've learned. Being able to make computers do things opens up this whole new world, and when people find themselves in that world they may end up overestimating their skills and underestimating how much they need to grow. What they fail at understanding they make up with in enthusiasm, and with experience they become more knowledgable about what they don't know. If we waited until they were "qualified" for jobs they would never get the experience to become qualified. At the same time there is more than enough room in the current job market to support people of lower skillsets, and for some companies that's considered an investment (junior people tend to turn to senior people over time). This is where it becomes a company culture issue. If a company is smart they'll have a few senior people making sure things are held to the right standard, and a few junior people who can get things done but need some guidance and direction. However, lots of companies (especially the smaller ones who may be more constrained by budget) go for the cheaper route and would rather hire someone junior as their main support. The problem isn't that the sysadmins aren't qualified sysadmins, it's that they're junior system admins who have been hired for the wrong job. Companies that fail to value experience tend to suffer as a result.
- dba7dba 12y ago
  Amen to that. When I see some of the job desc in job postings for DevOps/Sysadmin, I wonder. Is there really someone out there will all the skills that are asked for?
  - Aloha 12y ago
    I'm reasonably certain there isnt - not for the payband offered.

jetpks 12y ago

> it's the sysadmin's responsibility to insist on production deployments What decade are you from? No startups are hiring sysadmins to do any kind of work anymore. They're hiring "dev-ops" people, which seems to mean "Amateur $popularLanguage developer that deployed on AWS this one time." That's the whole problem with the dev-ops ecosystem. None of these dev-ops people seem to have any ops experience.

thaumaturgy 12y ago
> No startups are hiring sysadmins to do any kind of work anymore. Then maybe people should be willing to work for more grown-up businesses. HN tends to get a distorted view of what's important in the tech industry. The tech industry is way, way, way bigger than startups, and there are still plenty of companies that recognize the value of good sysadmins. Let the startups learn their lesson in their own time.
- dikaiosune 12y ago
  The alternative is that many of the startups don't learn this in their own time, and they go on to become bigger, more successful companies who can set the tone and shift the market. Of course, if they're actually able to succeed by doing so, then that says something too. Although the trend of many data breaches certainly wouldn't decline in that case.
  - rudolf0 12y ago
    >Although the trend of many data breaches certainly wouldn't decline in that case. Exactly. Successful and profitable are not mutually exclusive with "secure" or "well-architected". At least until those last two come to bite you later and start eating into your profits.
    - Lord_Zero 12y ago
      Sony is a great example of this.

ommunist 12y ago

@skywhopper "it's a failure of the company or organization or of his own communication skills" <~ Oh man, ever had a rant from The Management like "we pay you to do what we say"? No one usually cares about communication skills of sysadmin. Yes, its a failure of organisation. Sad truth is - most organisations are failed. Sysadmin today is a marginal job at a small company, where people respect you, or a job in the medium or large company where he or she are just peons.