9 ms·

I've been hearing the "boring" comment a lot lately, and I totally agree. I don't want the exciting filesystem (btrfs which just took me for a ride), or the exc

I've been hearing the "boring" comment a lot lately, and I totally agree. I don't want the exciting filesystem (btrfs which just took me for a ride), or the exciting programming language/library that will strand me in 6 months with multiple man-years of work. No, what I want is boring. That means everything works as I expect it too, and it doesn't create drama every few months when the cool kids decide that the old way of doing things isn't cool and rewrite a library i've based my application on or change the syntax of the compiler so my code doesn't work. A few years ago, I started challenging people at work when they were trying to pull in the latest language/library/toolkit, with a simple question. "How does that help our customers". Sure, I got plenty of the 2/3rd order arguments about how making the dev team happy/more productive would result in better feature development/etc. But I've seen these kinds of changes enough to know that usually these 2nd order affects are crushed by the unseen problems that _ALWAYS_ seem to crop up after your already committed to that new technology. Its truly rare to have a new technology that is so much better than the last that the cost of switching pays for itself. That doesn't mean it doesn't happen, it just means that the immature crap everyone is yammering about is likely not that technology. Give it a couple years and if it turns out to still be around (and people are still actively switching to it) then its probably good. This mindset has produced a long list of "successful" products that pay the bills and make people happy without making a lot of noise and crashing/burning regularly. Bottom line, I will take the old "cruddy" technology with the long list of known problems, to the new cool one with the long list of unknown problems. At least part of that seems to be what RH provides for the opensource community. The old guys that actually fix the bugs, rather than switching to some cool new product.

SkyMarshal 9y ago
Agreed. I would add a second question to your first: What are the risks of implementing this into production? Get them thinking as a matter of habit about both of those in tandem - what's the customer benefit, and what are the risks of doing it, and does the former justify the latter? Also, all programmers have a burning desire to build stuff from scratch, which is part of what's driving yours to incorporate cool new stuff like this at a risk to your revenue. No can do. But it may be worth exploring side or 20% projects specifically dedicated to building stuff from scratch. Get their itch scratched on non-revenue projects that serve as potential useful on-the-job training and tech evaluation, while possibly getting something of value out of it.
- watwut 9y ago
  The burning desire to build from scratch is simply because it is easier and because it tend to be more verbally valued by management then working on older product which tend to be snarked at as merely maintennance. The moment management praise also people working on less new and gives them appropriate salaries and autonomy, burning desire changes - except in few people who are truly naturally like that. It is win win, because the remaining people tend to be trully better in cutting edge tech problems.
  - hinkley 9y ago
    I'm starting to have doubts about this line of reasoning. It just doesn't explain enough of the risky behavior I see. I'm starting to think it might be the same thing that gets young developers to put up with 10+ hour days. That feeling you get when you lose yourself in a problem for hours and hours. Being in a Flow state is seductive. Nobody gets that feeling when looking through the bug database or knowledge base, or hunting through the source of an OSS module trying to figure out how to ask it the right questions. But they do get that feeling when trying to rewrite it themselves. All that open space to run as fast as you can... in the wrong direction. It's thrill seeking. BASE jumping for programmers.
    - harryf 9y ago
      Programming is based on hero worship. Anyone who makes even half an effort to better themselves as a software developer will encounter famous names from the past - those that "made a difference" by creating languages, operating systems and algorithms. Keyword here is "create". This is not an industry that celebrates maintence programmers. There's no glory for the guy that spent 20 years keeping the system that your banking transactions depend upon. So naturally young programmers will see to be the next Richie, Torvalds or Knuth and to do that you gotta create something new. Gotta change the world
      - snaky 9y ago
        There are some wannabe-heroes, but they aren't so widespread. The vast majority is just being pragmatic. Trendy toys and 'designed and implemented new X' in resume looks so much better for career than 'maintained something old so customers were happy'. Of course, there are some technologies that helps to make pretty good money even being ancient, like COBOL, APL, AS/400 and IMS. But for how long? So it seems rational to move forward instead.
raverbashing 9y ago
Also people don't remember how the old proprietary Unixes were very boring Few things changed between versions and they would tell you exactly what would change between versions so you could be prepared. (however I have much less trust in a distro running SystemD, it is not stable or reliable)
- deleted 9y ago
  [deleted]
- hyperbovine 9y ago
  I thought the same thing. RedHat has become the open source Solaris. I guess that's not such a bad thing. Solaris had some truly astounding engineering -- dtrace and ZFS felt like using software products that had been shipped to me from the future.
digi_owl 9y ago
Amen. Sadly it seems in a sense RH is funding both sides at the same time, as many of their employees are the very source of the "exciting" developments happening in the Linux ecosystem these days. Thing is though that in a sense the Linux community has painted itself into a corner. Thanks to the likes of RHEL, Ubuntu LTS and Debian Stable, there is little reason for the upstream to be anything but "exciting".
- mac01021 9y ago
  Out of curiosity, what are the exciting developments happening in the Linux ecosystem these days?
  - cyphar 9y ago
    Checkpoint-restore of processes (CRIU) is a fairly cool technology that is quite unique to Linux, and has gotten a lot of work in recent years. eBPF (both for seccomp and tracing) is also quite exciting in terms of its applications. User namespaces are also allowing for developments such as rootless containers. There's also some interesting stuff for trusted computing that are happening with IMA and TPM developments in general. In user-space there's even more interesting developments, you just have to look a little closer. ;)
    - snuxoll 9y ago
      CRIU looks amazing, live-migration of LXC containers :)
      - jacques_chester 9y ago
        There are rough edges to be worked out before we arrive at solidly productionised systems, but I believe Red Hat and Docker are both heading in that direction. Mind you, ask yourself: why do you need it? If it's for apps, isn't the point of these platforms to not need to care about individual instances? If it's for data, don't you have an existing way to manage HA or parallelism?
Avshalom 9y ago
I've said it before but basically every other post on HN makes me want to learn Ada out of spite.
cyphar 9y ago
The thing that makes distributions like Red Hat and SUSE "boring" is not that we only do maintenance work, it's that we've effectively solved the problem of maintenance and release engineering -- though of course there's still improvements to be made (and on the SUSE side, openSUSE Leap and the plans for SLE15 are quite interesting developments). We do a lot of work on exciting and interesting projects, some of which crash and burn as well. But part of the benefit of working for an enterprise distribution is that generally if some project is not ready to ship yet, we don't ship it. Having an out-dated version of some package is fairly common, and as long as we can do the maintenance work on it, it doesn't really matter. So we have a lot of time to improve upstreams to be stable enough to ship. [I work for SUSE.]
- jacques_chester 9y ago
  I am super interested to see how the SUSE stemcell and rootfs work turns out, now that SUSE is contributing to Cloud Foundry.
- exikyut 9y ago
  > The thing that makes distributions like Red Hat and SUSE "boring" is ...that we've effectively solved the problem of maintenance and release engineering Hmm. There are a lot of developers that could probably really do with some exposure to this kind of environment. I could definitely see myself hugely appreciating, say, a 6 month stint doing... something relating to the nitty-gritty of keeping a distribution going, learning what the pitfalls are, what the unintuitive stuff is, etc. Basically being some key point-person's assistant (the one who deals with major disasters), or something. It would be even better if some of this work could involve direct enterprise support: learning what customer requirements are, getting a sense of scale, etc. I can see this kind of information (enterprise support) being the most valuable - actual bespoke software would likely be a major focus of such training, and this would tech would-be developers about the disasters that can happen 5 years or 10 years on from "I have an awesome idea" "wow it's so elegant" etc etc. (Continuing from the previous point, perhaps there could also be an opportunity to be the assistant to the on-call person who sometimes wakes up at 3AM.) You could position this as something for people looking for DevOps jobs, and maybe make some kind of cert for it as well (this would likely help with/create traction for/be needed for enterprise-level interaction). I personally would have zero issues with signing a small book of NDAs to get this type of training myself. The NDAs would just cover how I learned, not what I learned, and in this context the how would be largely irrelevant (but at the same time incredibly invaluable). FWIW, I'd personally greatly benefit from a good debrief at the end of the course to help me find all the little ways I might accidentally leak something (eg a few too many words fall out before I stop myself etc) and learn to catch myself quickly.
pjmlp 9y ago
> A few years ago, I started challenging people at work when they were trying to pull in the latest language/library/toolkit, with a simple question. "How does that help our customers". I take a similar approach. Anyone trying to win my vote has to prove what is the business value of adopting technology X.
yesiamyourdad 9y ago
Preach it! Current job shows all the scars of this, every 6 months people hate the current system and want to rewrite something using some framework/lang/library that will really make everything better. The one positive aspect is that my resume now has every single buzzword. The downside is that the current team inherited 5+ years of failure to focus and address the problem at hand and instead try to run away from their problems.

SkyMarshal 9y ago

Agreed. I would add a second question to your first: What are the risks of implementing this into production? Get them thinking as a matter of habit about both of those in tandem - what's the customer benefit, and what are the risks of doing it, and does the former justify the latter? Also, all programmers have a burning desire to build stuff from scratch, which is part of what's driving yours to incorporate cool new stuff like this at a risk to your revenue. No can do. But it may be worth exploring side or 20% projects specifically dedicated to building stuff from scratch. Get their itch scratched on non-revenue projects that serve as potential useful on-the-job training and tech evaluation, while possibly getting something of value out of it.

watwut 9y ago
The burning desire to build from scratch is simply because it is easier and because it tend to be more verbally valued by management then working on older product which tend to be snarked at as merely maintennance. The moment management praise also people working on less new and gives them appropriate salaries and autonomy, burning desire changes - except in few people who are truly naturally like that. It is win win, because the remaining people tend to be trully better in cutting edge tech problems.
- hinkley 9y ago
  I'm starting to have doubts about this line of reasoning. It just doesn't explain enough of the risky behavior I see. I'm starting to think it might be the same thing that gets young developers to put up with 10+ hour days. That feeling you get when you lose yourself in a problem for hours and hours. Being in a Flow state is seductive. Nobody gets that feeling when looking through the bug database or knowledge base, or hunting through the source of an OSS module trying to figure out how to ask it the right questions. But they do get that feeling when trying to rewrite it themselves. All that open space to run as fast as you can... in the wrong direction. It's thrill seeking. BASE jumping for programmers.
  - harryf 9y ago
    Programming is based on hero worship. Anyone who makes even half an effort to better themselves as a software developer will encounter famous names from the past - those that "made a difference" by creating languages, operating systems and algorithms. Keyword here is "create". This is not an industry that celebrates maintence programmers. There's no glory for the guy that spent 20 years keeping the system that your banking transactions depend upon. So naturally young programmers will see to be the next Richie, Torvalds or Knuth and to do that you gotta create something new. Gotta change the world
    - snaky 9y ago
      There are some wannabe-heroes, but they aren't so widespread. The vast majority is just being pragmatic. Trendy toys and 'designed and implemented new X' in resume looks so much better for career than 'maintained something old so customers were happy'. Of course, there are some technologies that helps to make pretty good money even being ancient, like COBOL, APL, AS/400 and IMS. But for how long? So it seems rational to move forward instead.

raverbashing 9y ago

Also people don't remember how the old proprietary Unixes were very boring Few things changed between versions and they would tell you exactly what would change between versions so you could be prepared. (however I have much less trust in a distro running SystemD, it is not stable or reliable)

deleted 9y ago
[deleted]
hyperbovine 9y ago
I thought the same thing. RedHat has become the open source Solaris. I guess that's not such a bad thing. Solaris had some truly astounding engineering -- dtrace and ZFS felt like using software products that had been shipped to me from the future.

digi_owl 9y ago

Amen. Sadly it seems in a sense RH is funding both sides at the same time, as many of their employees are the very source of the "exciting" developments happening in the Linux ecosystem these days. Thing is though that in a sense the Linux community has painted itself into a corner. Thanks to the likes of RHEL, Ubuntu LTS and Debian Stable, there is little reason for the upstream to be anything but "exciting".

mac01021 9y ago
Out of curiosity, what are the exciting developments happening in the Linux ecosystem these days?
- cyphar 9y ago
  Checkpoint-restore of processes (CRIU) is a fairly cool technology that is quite unique to Linux, and has gotten a lot of work in recent years. eBPF (both for seccomp and tracing) is also quite exciting in terms of its applications. User namespaces are also allowing for developments such as rootless containers. There's also some interesting stuff for trusted computing that are happening with IMA and TPM developments in general. In user-space there's even more interesting developments, you just have to look a little closer. ;)
  - snuxoll 9y ago
    CRIU looks amazing, live-migration of LXC containers :)
    - jacques_chester 9y ago
      There are rough edges to be worked out before we arrive at solidly productionised systems, but I believe Red Hat and Docker are both heading in that direction. Mind you, ask yourself: why do you need it? If it's for apps, isn't the point of these platforms to not need to care about individual instances? If it's for data, don't you have an existing way to manage HA or parallelism?

Avshalom 9y ago

I've said it before but basically every other post on HN makes me want to learn Ada out of spite.

cyphar 9y ago

The thing that makes distributions like Red Hat and SUSE "boring" is not that we only do maintenance work, it's that we've effectively solved the problem of maintenance and release engineering -- though of course there's still improvements to be made (and on the SUSE side, openSUSE Leap and the plans for SLE15 are quite interesting developments). We do a lot of work on exciting and interesting projects, some of which crash and burn as well. But part of the benefit of working for an enterprise distribution is that generally if some project is not ready to ship yet, we don't ship it. Having an out-dated version of some package is fairly common, and as long as we can do the maintenance work on it, it doesn't really matter. So we have a lot of time to improve upstreams to be stable enough to ship. [I work for SUSE.]

jacques_chester 9y ago
I am super interested to see how the SUSE stemcell and rootfs work turns out, now that SUSE is contributing to Cloud Foundry.
exikyut 9y ago
> The thing that makes distributions like Red Hat and SUSE "boring" is ...that we've effectively solved the problem of maintenance and release engineering Hmm. There are a lot of developers that could probably really do with some exposure to this kind of environment. I could definitely see myself hugely appreciating, say, a 6 month stint doing... something relating to the nitty-gritty of keeping a distribution going, learning what the pitfalls are, what the unintuitive stuff is, etc. Basically being some key point-person's assistant (the one who deals with major disasters), or something. It would be even better if some of this work could involve direct enterprise support: learning what customer requirements are, getting a sense of scale, etc. I can see this kind of information (enterprise support) being the most valuable - actual bespoke software would likely be a major focus of such training, and this would tech would-be developers about the disasters that can happen 5 years or 10 years on from "I have an awesome idea" "wow it's so elegant" etc etc. (Continuing from the previous point, perhaps there could also be an opportunity to be the assistant to the on-call person who sometimes wakes up at 3AM.) You could position this as something for people looking for DevOps jobs, and maybe make some kind of cert for it as well (this would likely help with/create traction for/be needed for enterprise-level interaction). I personally would have zero issues with signing a small book of NDAs to get this type of training myself. The NDAs would just cover how I learned, not what I learned, and in this context the how would be largely irrelevant (but at the same time incredibly invaluable). FWIW, I'd personally greatly benefit from a good debrief at the end of the course to help me find all the little ways I might accidentally leak something (eg a few too many words fall out before I stop myself etc) and learn to catch myself quickly.

pjmlp 9y ago

> A few years ago, I started challenging people at work when they were trying to pull in the latest language/library/toolkit, with a simple question. "How does that help our customers". I take a similar approach. Anyone trying to win my vote has to prove what is the business value of adopting technology X.

yesiamyourdad 9y ago

Preach it! Current job shows all the scars of this, every 6 months people hate the current system and want to rewrite something using some framework/lang/library that will really make everything better. The one positive aspect is that my resume now has every single buzzword. The downside is that the current team inherited 5+ years of failure to focus and address the problem at hand and instead try to run away from their problems.