10 ms·

You've heard of “yes men”. Now learn about “no men” (2011)

188 points by derwiki 7y ago

scarejunba 7y ago
The funny thing is we're all making trade-offs at various times. So there will be people who will point at someone else and say "He's a no-man" and at other points they will be the "no-man". I remember saying "no" a lot at some points of my life, and I think it was because I hadn't yet learned to say "the cost of that is this and in my opinion it isn't worth it" but there are still times I am speaking to people where they are unable to truly grok the cost given my current skill of explaining and then it probably looks like I'm a no-man. Besides, come on, who even has dogmatic rules like "No Swap In Production". That's just short-hand for a concept. When it's in dispute you have to trace the reasoning again. There's no alternative. I cannot imagine doing something only because someone said so. I certainly recall a time when someone said "Person X isn't going to like this" and saying "Well, then, person X is going to have to find another way to solve your problem". I certainly hope that if I'm person X in someone else's story, they take the same approach. And if you have to trace the reasoning more than once, it's probably worth writing down your argument. Not as a big RFC-style thing necessarily, but even just a Slack thread you can point to.
- jacquesm 7y ago
  Have Slack threads risen to the level of documentation now?
  - NateEag 7y ago
    When trying to start a culture of documentation, permalinks to Slack conversations where people made decisions are often helpful. Think of them as real-time meeting minutes and it may be easier to swallow.
  - scarejunba 7y ago
    A principle I have is that I lower the barrier for others to contribute the things I want to see more of. Rather than holding a Slack thread up as the apotheosis of documentation, I recognize that I am more likely to receive wisdom if I allow for a Slack thread to be sufficient. It's just a permanent link to a conversation. This is reinforced by the additional fact that the people with the most I have to learn from are the people with the least time to teach me. So I minimize the effort I require of them to give me their best. For instance, if I had a chance to ask Feynman stuff, I would happily accept answers in txtspeak with emojis and broken grammar. Sort of a personal Postel principle.
    - Aloha 7y ago
      I'm a big believer in apply postel is interpersonal communications. Particularly in the emotional realm.
CamouflagedKiwi 7y ago
I've heard the "kernel extension guy" story at my last job; it's fairly well-known there although it wasn't a "completely obvious change", it was heavily obfuscated and possibly done over several PRs in order to sneak it in. Hadn't heard any subsequent stories though, including the swap story of the rest of the article.
raldi 7y ago
I've found there's something even worse: Policies that aren't actually attached to any known person. At least with the swap example, there was someone you could talk to, or ask your manager to talk to. But what if it was just an Onion in the Varnish and nobody remembered who said no swap in prod, it was just "That's the [company name] Way"?
- cortesoft 7y ago
  Happened at my company, where we had some policy in response to an incident, put into place by a couple of senior leaders... the policy was overly broad, but kinda made sense for the problem. After a while, those two leaders left, and we started seeing issues that the policy was causing. However, people still stuck to the policy. When I would argue that the policy wasn't really accomplishing what it was originally intended to do anymore, I was repeatedly told it didn't matter, it was the policy. There was no one to even argue against anymore, because they were gone, but the policy remained.
  - Aeolun 7y ago
    It’s like the monkeys being shocked every time one of them goes for the banana.
  - fanf2 7y ago
    It is super important that policies are mutable, that they can change with changes in circumstances and knowledge. They should be as easy to change as the org chart.
zeroxfe 7y ago
Wow, I had multiple very similar experiences with the same person she's talking about.
fragsworth 7y ago
I'll take a stab at explaining what happened here: "No swap in production" probably stemmed from someone who worked on the servers, and that person discovered they were swapping too much, which would eventually lead to hard drive decay. And, maybe for budget reasons or whatever, they decided to make it a policy to disable swap, rather than increase RAM or change the underlying software that caused the servers to use too much RAM. And now, later, on different servers with different software and different specs, the policy still exists, because nobody really paid much attention to why it was created in the first place.
- pjc50 7y ago
  Swap causes not disk decay but performance decay; the system gets unamanageably slow and would have been better off with a clean crash and failover.
  - 8ytecoder 7y ago
    Only if there's thrashing. It’s prohibitively more expensive to build a machine that has enough memory for way more than peak usage than to have some swap that can handle some level of overflow pages. Like everything swap has to be monitored and there’s no need To disable it entirely
    - nailer 7y ago
      It depends. Sudden peaks of latency can get comparatively more expensive on a trading floor than buying an absurd amount of RAM.
- devonkim 7y ago
  This reminds me of a previous company that paid consulting fees to have their Postgres configuration tuned. When the hardware was upgraded nobody knew to review the memory settings. After a series of production outages we realized that the new Postgres servers were utilizing 16 GB of RAM when we had 128 GB available.
- tabtab 7y ago
legitster 7y ago
There are any number of these type of people at any large organization, and I often flip flop on their usefulness. Usually, they are in the form of protectors of the "process". This wasn't vetted by so and so, or didn't go through the proper channels, or doesn't use the right technology. And 51/100 times, they may be right - they keep unnecessary scope creep, bad ideas, technical debt, overhead out of the process. Literally the only reason people bother following a process is because of them. But at the same time they are a huge drain on time and resources, and can enable bad or inefficient processes by manually keeping things in line. They add nothing to innovation - and it's a huge bummer to work with/around them.
- ryandrake 7y ago
  As one of the dreaded "process enforcers" at my company, it's a tough line to walk. Any organization with more than a handful of people needs some written and agreed upon process or it's just chaos. But you can't just blindly follow a process for its own sake. You need to be able to articulate why we follow it with examples from the past of what happens without it. And, you need to be willing to constantly look for ways to improve it. Anything I can do to automate processes or simplify them makes the developers happier and more productive and me less of the bad guy. An inefficient process where people are always waiting on something to happen is demotivating to everyone. Most process is scar tissue from something that went wrong in the past. We have mandatory code reviews because when we didn't we let a severe security bug get deployed to production. We have legal review all changes to the terms of service so we don't get successfully sued over it again. We have a schedule with code freeze N weeks before release because when we didn't we could never release anything because...just...one...more...change. I was hired at a small company once without any process for anything. No release process or schedule. No bug tracker. No testing plan. No code reviews. Nobody used source control. Builds from different workstations produced wildly different binaries. It was a total madhouse and it was amazing that anything got done at all. The CEO simply thought the engineers just weren't competent enough, but they were--it was just a circus because nobody stepped up to formalize anything. Once we got a bug tracker and SVN set up, things turned around pretty quickly.
shados 7y ago
I definitely felt this decades ago when I started in the industry. It's 2019 now though. only 8 years later, but a lot of things changed. The industry is growing fast, faster than knowledge can be shared even with all the power of the internet. It's very bottom up, with newer engineers severely outnumbering those that have been around. People are making fat salaries and companies are catering to their whims. They/we're more spoiled than they've ever been. People go to work hoping they'll be find a reason today to try the latest pattern they saw on Twitter or try the trendy new tech or library their friend just talked about. If you don't have people saying no, it quickly (very quickly) get out of control. As organizations grow, you need to start saying no. A lot. Else within 0.2 seconds of the new intern joining or the new grad coming in, or the lateral hire who was the king of their 5 people startup will be pushing some stuff that isn't mature enough or doesn't follow established patterns. More often, they'll be reintroducing something problematic that you just finished getting rid of 6 months ago. It sucks. Sure, there are some power hungry people who get off on power trip by being gate keepers. But overall, it's just people desperately trying to keep things in one piece and the company going, and that means making hard decisions and enforcing them. It just doesn't feel good to say no, arguing all day with people who have way more energy than you do, but it has to be done.
- james_in_the_uk 7y ago
  I have to say "no" a lot in my role. Thank you for this post.
- deleted_account 7y ago
  "Chesterton's fence".
  - hinkley 7y ago
    Chesterton's fence is about taking away things we don't understand. There's a big gap between that and fighting cargo culting.
    - mark-r 7y ago
rangewookie 7y ago
This doesn't have much to do with saying "no" and everything to do with not understanding the fundamentals of a policy. If they guy who said "no swap" had a reason, then you could have discussed and potentially changed the policy.
SubiculumCode 7y ago
One of the easiest strategies to protect your reputation in life and business is to never venture to say yes (because if it fails you are on the hook), and to let your discussion always revolve around an idea's failings, never its strengths. It gives an appearance of expertise and command, but risks little, and thus often leads to promotion. But I personally find it poisonous to the enterprise's progress: Even the best plans have weaknesses.
- dijit 7y ago
  Er. This is awful advice. You’re also on the hook for impeding the organisation. I’m a “no” guy. My boss is a pure yes man. I don’t say no to impede progress. I say no to avoid overworking my team which would reduce the quality of our work. I say no, because it changes the scope of work in progress, and that requires sufficient justification because it means either throwing work away (which is both demoralising and wasteful), potentially increasing time to delivery or creates technical debt in droves. I say “no” so that people don’t make their own standards (like not following logging patterns, or metric naming conventions) which would cause all kinds of gunk in the infrastructure. It’s not about following a strict process. Although that can be important for streamlining workflow through an understaffed team like mine, it’s about ensuring that the quality of what we do is not at risk. Quality is our only currency, without it everything suffers.
  - SubiculumCode 7y ago
    What you say is true, which makes the topic so very subtle because it depends on the context. Saying no to protect your people/process is a good thing. Saying no because it is personally profitable to be disdainful of other's ideas while offering up none of your own, irrespective of what is good for the organization or your team, is not. If one needs a mental picture, think of the quintessential hipster: https://sabotagetimes.com/life/hipster-irony-is-destroying-our-idea-of-sincerity-and-creativity https://sabotagetimes.com/life/hipster-irony-is-destroying-o...
  - Aloha 7y ago
    Change for the sake of change is almost always bad, making decisions without considering externalities is worse.
OJFord 7y ago
Isn't no swap in prod fairly standard? Kubernetes pre-flight checks check it, for example. If enabling swap 'fixes' it, isn't the proper fix to provision more memory?
- toast0 7y ago
  > If enabling swap 'fixes' it, isn't the proper fix to provision more memory? Mr. No says all servers must match one of the existing configurations, none of which have very much ram. If you're on the border where a little bit of swap gets you zero pages, and no swap gets you stuck machines, you probably need to figure out how to scale within your constraints, but a little bit of swap should be ok, in the neighborhood of 1GB gives you a little bit of breathing room without prolonging the inevitable for very long.
  - OJFord 7y ago
    1. The article states that the older machines had swap enabled, so they weren't homogenous anyway; 2. At some point you do have to change configuration, and if you have insufficient memory on one you either have insufficient memory on all machines, or a badly distributed workload.
- sitkack 7y ago
  Wait what? How do machines running K8s exec processes when they are near their physical memory limit? Swap does more than just provide a place for pages to get evicted to, it also increases the virtual address space.
  - umanwizard 7y ago
    I don't think that's true, at least on Linux. This program uses 1 TiB of virtual address space, and runs fine on a machine with no swap (and much less than 1 TiB of physical RAM). #include <sys/mman.h> #include <stdlib.h> int main() { size_t const len = 1024 * 1024 * 1024; // 1 GB for (int i = 0; i < 1024; ++i) { char *p = (char *)mmap(0, len, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0); if (p == -1) exit(-1); } } The space won't be associated with any physical object (whether RAM or swap) until you actually touch it.
WalterSear 7y ago
I'm coming to the conclusion that many technical problems are actually morale problems. This especially goes for flamebait, where some people come down heavily on one side of the argument or another. If you don't care, it's much easier to default to someone else's directive, or complain about a toolset you haven't bothered to learn.
draw_down 7y ago
Obviously such a person can exist but my Spidey senses always tingle when I’m reading about what an asshole someone else is.
blueyes 7y ago
I think of most in-house corporate lawyers as playing the role of "no men". There job is to highlight risks and stop things from happening. It's a rare lawyer who understands business goals and can balance them against risks.
- isostatic 7y ago
  That's not the lawyer's job - the lawyers' job is to present the risks, not to make the decisions as to whether the risk is worthwhile
  - Aeolun 7y ago
    In theory. In practice it often turns out that the decision makers think that any risk at all is unacceptable, giving the lawyers de-facto control.
    - isostatic 7y ago
      In which case you have poor decision makers, not poor lawyers. The benefit of domain aware lawyers is they can explain the specific risks and help the decision makers apply those risks to the business.
- mobjack 7y ago
  Good lawyers figure out how to say yes while minimizing legal risks.
hrktb 7y ago
I don’t get it. 1 - there is a Mr.No guy but, the author only knows him by reputation 2 - there is a technical problem that seems to revolve around memory 3 - the author has an idea of a solution, she actually tries as people don’t believe her 4 - her solution works, but she is told the “No” guy said “no swap” once a long time ago Her reaction: “Did I care what he supposedly said? No.” Why doesn’t she go talk to him ? The “No” guy could have had another solution than raising swap. He could have recognized being wrong, allow swap and learn from it. He could continue say “No” but now the author has an actual experience of it, and not just hearsay and reputation. It seems like nobody communicates correctly in that company, the author included, and it must hurt in so many places. Edit: fixed author’s gender, thanks
- Shoop 7y ago
  Just an FYI, the poster is female.
- jedberg 7y ago
  For the record, the author is a "she".
- wfbarks 7y ago
  These where my thoughts, "No" guy could have a good reason for no swap. no swap might have been a fix for some other issue OP is unaware of, which OP has now reverted. Don't know until you ask...
  - romwell 7y ago
    >no swap might have been a fix The chances of that are astronomically low. From the article: >I later found out that all of our older machines had swap enabled. It was just the ones which had been replaced or reinstalled which had been popping up without it, perhaps as part of a policy change. That means it was perfectly valid for us to run with swap on, since most of the machines we owned were using it.
- vxNsr 7y ago
  you caught like half the male pronouns, there are still a bunch left, it actually makes it more confusing.
beiller 7y ago
Trust but verify. I'd bet if the author went to the original source, the "no man" himself, they may find that there is actually no such policy! Just a mishearing of a conversation parroted by "wrong people".
- hrktb 7y ago
  “Trust but verify” litteraly meant “Don’t trust” (as in “don’t trust the Russians” in context). Was it the nuance you intended ?
  - yawaramin 7y ago
    That's not what it meant. https://en.wikipedia.org/wiki/Trust,_but_verify https://en.wikipedia.org/wiki/Trust,_but_verify
    - hrktb 7y ago
      does it ? > Reagan used the phrase to emphasize "the extensive verification procedures that would enable both sides to monitor compliance with the treaty"
      - yawaramin 7y ago
        But you have to look at why he used that particular phrasing–a clever bit of wordplay to quickly get the idea across to Russians in familiar terms. It's a Russian figure of speech in the first place.
        hrktb 7y ago
        “Wordplay” is too generous IMO. The original russian proverb is also double speak, as we would say now. It’s obvious when applied to trivial situations: “I trust my wife, but I take the time to verify her phone every night and check the private eye report of her day” We wouldn’t call that “trust”, pretty much the opposite
deleted 7y ago
[deleted]
incomplete 7y ago
i was at this company when this happened, and oh boy, EVERYONE who knew the person in question and the 'new' kernel feature got some amazing, long-lasting and well deserved laughs. btw, it was /proc/<idiot's username>, not /dev :)
- d2mw 7y ago
  I didn't recognize the story until you corrected the path. https://news.ycombinator.com/item?id=1242831 https://news.ycombinator.com/item?id=1242831
msla 7y ago
The extreme example of Chesterton's Fence. Is this, then, Chesterton's Berlin Wall? That is, a completely idiotic waste based on idiot politics?
- elihu 7y ago
  I suppose it's also worth noting that Chesterton's Fence is an (imagined) real physical thing which required time and resources to construct. Thus the odds are pretty good that it was made for some real purpose, as opposed to a policy which is ephemeral and does not intrinsically require a significant expense of time or effort to create.
sriram_sun 7y ago
Interesting combination of the "yes men" parroting the "no men"! Great example!
domnomnom 7y ago
May as well start off with an introduction to toxic masculinity.
- Aloha 7y ago
  How so? The author is a woman.
drewg123 7y ago
A powerful "no man" can be a valuable asset sometimes. I was involved in the design of an ASIC. This was going to be used by people from all different areas of the company, and they all had various demands. Most had never worked on a chip before, and didn't realize the cost of what they were asking for. The easy thing to do would have been to ensure that every requested feature was added But that would have burned area, power, and huge amounts of design and verification time. Luckily we had a senior guy who had the clout to say "no". Over and over again, he would tell people how much things would cost in terms of money and time, and ask if the features they wanted were worth that much. Or he'd point out that what they wanted could be done with firmware plus existing features. Overall, he probably saved millions of dollars and caused the project to be delivered on time.
- freehunter 7y ago
  I have an unofficial board of advisors for my small company, and one of them is a "No Man". I knew that when I asked him to join. Every time the company has an idea, this guy will shut it down immediately. He forces us to defend every single decision. Do we always listen to him? No. Does he always win? No. Is he always right? Absolutely not. But I keep him around because he forces everyone in the company to defend every idea. If you still feel good about the idea after defending it from him, then go ahead. If you've lost faith, then I'm glad we didn't waste any time on it.
  - daotoad 7y ago
    Every company really, really, really....really needs someone like that. Imagine how much better the Star Wars prequels could have been if only Lucasfilm had a strong No Person to stop them from doing stupid shit.
    - wincy 7y ago
      George Lucas’ ex-wife was the no-woman on the original trilogy. She was his editor!
      - 7y ago
quickthrower2 7y ago
I still have an internal no-man that says "Don't use OR's in where clauses" in my head from years ago, then I have to stop myself and think "OK lets actually test the performance of this particular query rather than rely on a no-man policy in another company from years ago"!
yawaramin 7y ago
Sounds like that workplace is a no man's land.
dredmorbius 7y ago
This seems far less about "just say no" than it is about not understanding, or insisting on, long-obsoleted policies. I've encountered something of a converse of this situation: a set of prod servers running a Java workload on which performance was frequently abysmal, which turned out to be due to a very poorly tuned set of JVM GC parameters. The feature(s) were poorly documented (lots of detail, little useful guidance), so it took some tuning and ongoing performance monitoring to see that 1) the problem had been solved and 2) that no new problems were introduced. The latter being of particular concern as this was a long-lived project that had gone through generations of development and ops support (I was latter), few of whom were on current speaking terms with the shop in question. The management feedback I got on my monitoring was (a rudimentary bit of instrumentation, and periodic checking) was that this seemed to reflect an excess of concern and/or attachment to the issue and/or solution. Given the numerous other years-outstanding issues which I'd identified and (eventually) largely remedied, I found this response tremendously perplexing.
m463 7y ago
Even saying "no" should have checks and balances.
the-rc 7y ago
I know this "no man", had to work on an approval from him once, and I actually saw the infamous code change that implemented this. I don't remember him reviewing it. IIRC it was sent to a mailing list, to which he might have or have not subscribed.
stevenjohns 7y ago
I'm a bit of a "no man", and at a company I worked at in the past my (pseudo-technical) manager told me to spend hours creating some sort of extremely purpose-specific test environment to aid in resolving a ticket that would only take me 30 minutes to resolve directly - so I just resolved the ticket instead and closed it, and got on with the rest of my work. Eventually I was dismissed from this company a couple of days later and this situation was raised as one of the points. I was told "you can't refuse a mission given to you by your manager, regardless of the circumstances, that's ridiculous" and when I replied "I have different views to you on this" it was considered to be rude. The person who dismissed me is more or less the management boxing bag - a certified "yes man" who nobody respected. Of course this hasn't changed my attitude, though. Still a proud "no man" and wouldn't go with the flow unless I'm sure everyone completely understands the consequences. At some point people have to realise that their loyalties don't go towards their manager and instead should go towards the shareholders and clients. There is a bit of give on this, of course, but eventually you have to do the right thing by the people that actually stand to gain and lose - not just being another cog in the machine that exists only to support other cogs in that machine.
- ilovecaching 7y ago
  Sounds like a toxic place. Instead of compromising or arguing a technical point and making the discussion an engineer effort, your manager ruled by feudal law. This is the antithesis of good engineer, in which good technical arguments and trade offs are made with everyone equal at the table. Why your manager is making technical decisions is beyond me as well. Managers are supposed to be managing people, not doing engineering work.
pipingdog 7y ago
The most important part of a fast car is good brakes.
- AstralStorm 7y ago
  As opposed to stuck brakes, or ones that activate at random.
mr_tristan 7y ago
I don't think this is necessarily about people who are critical, it's that you can have someone who is just as stupid as a "yes man": they just say "no" just because instead of, you know, actually thinking. I don't believe this "no man" is someone who is managing their attention or saying no for an actual reason. They're just contrary. And yeah, there's a lot of them around. What I actually find even more frequently are "parrots". These are people who "learn by runbook". Instead of asking why something is done a particular way, they want to learn how to parrot some procedure. It's another variation of things: instead of saying "yes", or "no", they just parrot "process X". I think this is really where the "no swap in production" example came from. I've seen _lots_ of stupid similar ideas in software development.
kev009 7y ago
Honestly saying no is one of the most important things either a startup or large company can have. Saying no gracefully, respectfully, and having the receiving person understand the rational is the hard part. This is one of the function of "fellows", "Senior Members of the Technical Staff", "Principal Architects" or whatever your org calls it and the people selected for that should be excellent at it. I guess the problem the author points out is that is the only thing a person might do and that is not good. Folks in this position should also know how to say yes and build buyin from business people/exec/program management. Most organizations fail horribly at saying no. Instead they try to silo things to cope. This works to some extent, and you will always have silos as a company grows up, but you really should never end up with 12 programming languages, 5 RDBMS, 4 NoSQL DBs, 3 Javascript frameworks, 3 CIs, 3 cluster schedulers and then all the aforementioned also duplexed across organization units. That level of sprawl reduces margins, increases onboarding time, stresses even the most experienced staff, and makes adaption and business decision making murky at best.
bcrosby95 7y ago
Most developers probably view sysadmnis as "no men". You want to do WHAT in production? flipoff
beardedwizard 7y ago
You ran with swap on because you saw a bunch of other machines with swap on and decided the policy was invalid. Thanks for making real security jobs hell, this is the #1 developer excuse for avoiding security requirements.
jcrites 7y ago
I agree with the author's overall premise, which is that other people who don't own and who are not responsible for your system should not have authority over how you build and operate it. One of the things that I like about the current company where I work is that owners always have the final say about their systems, not any external technical authority. There is no such authority who could say no where I work (with rare and limited exceptions), and I'm glad for that; service owners make the judgment call about trade-offs. > "so-and-so says no swap in production". That said, while I understand the point of the article is something different, I broadly agree with this advice: generally no swap in production. If you need swap, then that's probably a sign that the workload isn't matched well to the resources of the machine (unless the workload was specifically designed to employ swap, which most are not). The problem is that: maybe most of the time the application's working set will fit within memory, and so it will perform well, but some of the time it will begin to swap and performance will degrade. The result of adding swap to a system is non-deterministic performance. If you're building high-availability, high-performance production systems, then you don't want the potentiality of arbitrary performance degradation or non-deterministic behavior. Instead, you would prefer to solve the root cause -- such as by increasing the amount of memory available to the process or system, so that swap is no longer required. An engineer adding swap to a system is like a doctor treating a patient's symptoms instead of curing their disease. Both are useful but one is preferable. If you encounter a situation where you could solve an application problem by adding swap, then an even better solution is to address the root cause -- ensure the application doesn't run out of physical memory. Swap is a band-aid that masks a risk better addressed directly. Hence: no swap in production. (On the other hand, it might be fine to employ swap in production for services that aren't performance sensitive, and where it's preferable to accept the performance hit from using swap rather than pay more for better hardware. That's a fine trade-off if consciously made. In my experience, variations in performance are usually much more problematic to a business than changes to cost.)
- _visgean 7y ago
  But not having swap will cause the machine to misbehave (https://utcc.utoronto.ca/~cks/space/blog/unix/NoSwapConsequence https://utcc.utoronto.ca/~cks/space/blog/unix/NoSwapConseque...). Swap in this situation acts like a backup, surely you would prefer temporal degradation then total system shutdown. The solution here is continuous monitoring of free memory (we use datadog) + swap.
papagoat 7y ago
I used to have a “don’t know/can’t remember” man in my team. Can you recall what this code you wrote previously do? “I can’t remember.” Do you know where the files are placed? “I don’t know.” Where are the documentations for xxx. “I don’t know.” Do you know the difference between xxx in testing in production server? “I can’t remember.” It was frustrating. 9 out of 10 questions were “I don’t knows” and “I can’t remembers”. I’d much rather he said yes or no.
- taejo 7y ago
  Of course it would be great if everyone remembered everything, but isn't an "I don't know" better than an 80% guess? The longer I do this job, the more likely I am to say I don't know, when I don't know. > Can you recall what this code you wrote previously do? What it actually does might be different from what I remember. It might have been changed since then -- maybe by me, maybe by someone else. I might have made a mistake in the first place, and it doesn't do what I think. "I'm not sure, let's look at it."
slewis 7y ago
I know exactly who she's talking about! When I was at the unnamed company I worked on a project that he retained control over and said no to a lot. We found that if we sent him a very well reasoned, documented and measured change he'd say yes. I was there when the no swap policy was under lots of active discussion. It was there to ensure that application authors could have strong expectations on how long a memory access would take. Imagine trying to provide evidence that removing the "no swap" policy wouldn't affect any of the applications that were running across a very large infrastructure. Her point really resounds though, this person was referred to all the time in conversation, "well, we probably couldn't get this past X". He was mythical. In general that was often a good thing. Making changes that could have such a broad impact should be well reasoned. But it's also quite annoying, having a nagging voice inside your head that just says "no", seemingly disconnected from the actual source.
- nailer 7y ago
  What the app something that required deterministic latency? There are situations where your rather hit OOM and kill your app and go buy some RAM rather than have it operate slowly - executing trades too late, or pointing at targets that have moved, etc.
unsined 7y ago
One time my app started crashing when I added a large async routine. I ssh'd in and topd and noticed memory and swap were maxed out. Doubled the swap and it runs just like it did before. Gotta love simple fixes.
kjgkjhfkjf 7y ago
The issue here isn't with "Mr No", it's with those who mindlessly parrot his policies. I've met "Mr No" at Google, and he is in fact quite a reasonable and funny guy.
lazylizard 7y ago
Why not just buy more ram?
m0zg 7y ago
Microsoft used to be like this in 00's (I don't know about now). You had to run everything by Legal, and the default answer from Legal was always "no", no matter what you're asking. Want to use zlib in your product? Even though Windows is using it in like 3 different places already, the answer is still "no". JQuery? Out of the question. Anything more exotic, might not even ask. I'm pretty sure Microsoft wasted hundreds of millions of dollars reinventing a wheel, and doing it badly, over the past few decades.
crimsonalucard 7y ago
There's also the Maybe guy. All this guy ever says are things like "Maybe that's possible," "in theory that maybe could work," "I'm not sure if that will work." The thing that the maybe guy differs from the Yes or the No guy is that he's always 100% correct. He's never wrong ever.

188 points by derwiki 7y ago

The funny thing is we're all making trade-offs at various times. So there will be people who will point at someone else and say "He's a no-man" and at other points they will be the "no-man". I remember saying "no" a lot at some points of my life, and I think it was because I hadn't yet learned to say "the cost of that is this and in my opinion it isn't worth it" but there are still times I am speaking to people where they are unable to truly grok the cost given my current skill of explaining and then it probably looks like I'm a no-man. Besides, come on, who even has dogmatic rules like "No Swap In Production". That's just short-hand for a concept. When it's in dispute you have to trace the reasoning again. There's no alternative. I cannot imagine doing something only because someone said so. I certainly recall a time when someone said "Person X isn't going to like this" and saying "Well, then, person X is going to have to find another way to solve your problem". I certainly hope that if I'm person X in someone else's story, they take the same approach. And if you have to trace the reasoning more than once, it's probably worth writing down your argument. Not as a big RFC-style thing necessarily, but even just a Slack thread you can point to.

jacquesm 7y ago
Have Slack threads risen to the level of documentation now?
- NateEag 7y ago
  When trying to start a culture of documentation, permalinks to Slack conversations where people made decisions are often helpful. Think of them as real-time meeting minutes and it may be easier to swallow.
- scarejunba 7y ago
  A principle I have is that I lower the barrier for others to contribute the things I want to see more of. Rather than holding a Slack thread up as the apotheosis of documentation, I recognize that I am more likely to receive wisdom if I allow for a Slack thread to be sufficient. It's just a permanent link to a conversation. This is reinforced by the additional fact that the people with the most I have to learn from are the people with the least time to teach me. So I minimize the effort I require of them to give me their best. For instance, if I had a chance to ask Feynman stuff, I would happily accept answers in txtspeak with emojis and broken grammar. Sort of a personal Postel principle.
  - Aloha 7y ago
    I'm a big believer in apply postel is interpersonal communications. Particularly in the emotional realm.

CamouflagedKiwi 7y ago

I've heard the "kernel extension guy" story at my last job; it's fairly well-known there although it wasn't a "completely obvious change", it was heavily obfuscated and possibly done over several PRs in order to sneak it in. Hadn't heard any subsequent stories though, including the swap story of the rest of the article.

raldi 7y ago

I've found there's something even worse: Policies that aren't actually attached to any known person. At least with the swap example, there was someone you could talk to, or ask your manager to talk to. But what if it was just an Onion in the Varnish and nobody remembered who said no swap in prod, it was just "That's the [company name] Way"?

cortesoft 7y ago
Happened at my company, where we had some policy in response to an incident, put into place by a couple of senior leaders... the policy was overly broad, but kinda made sense for the problem. After a while, those two leaders left, and we started seeing issues that the policy was causing. However, people still stuck to the policy. When I would argue that the policy wasn't really accomplishing what it was originally intended to do anymore, I was repeatedly told it didn't matter, it was the policy. There was no one to even argue against anymore, because they were gone, but the policy remained.
- Aeolun 7y ago
  It’s like the monkeys being shocked every time one of them goes for the banana.
- fanf2 7y ago
  It is super important that policies are mutable, that they can change with changes in circumstances and knowledge. They should be as easy to change as the org chart.

zeroxfe 7y ago

Wow, I had multiple very similar experiences with the same person she's talking about.

fragsworth 7y ago

I'll take a stab at explaining what happened here: "No swap in production" probably stemmed from someone who worked on the servers, and that person discovered they were swapping too much, which would eventually lead to hard drive decay. And, maybe for budget reasons or whatever, they decided to make it a policy to disable swap, rather than increase RAM or change the underlying software that caused the servers to use too much RAM. And now, later, on different servers with different software and different specs, the policy still exists, because nobody really paid much attention to why it was created in the first place.

pjc50 7y ago
Swap causes not disk decay but performance decay; the system gets unamanageably slow and would have been better off with a clean crash and failover.
- 8ytecoder 7y ago
  Only if there's thrashing. It’s prohibitively more expensive to build a machine that has enough memory for way more than peak usage than to have some swap that can handle some level of overflow pages. Like everything swap has to be monitored and there’s no need To disable it entirely
  - nailer 7y ago
    It depends. Sudden peaks of latency can get comparatively more expensive on a trading floor than buying an absurd amount of RAM.
devonkim 7y ago
This reminds me of a previous company that paid consulting fees to have their Postgres configuration tuned. When the hardware was upgraded nobody knew to review the memory settings. After a series of production outages we realized that the new Postgres servers were utilizing 16 GB of RAM when we had 128 GB available.
tabtab 7y ago

legitster 7y ago

There are any number of these type of people at any large organization, and I often flip flop on their usefulness. Usually, they are in the form of protectors of the "process". This wasn't vetted by so and so, or didn't go through the proper channels, or doesn't use the right technology. And 51/100 times, they may be right - they keep unnecessary scope creep, bad ideas, technical debt, overhead out of the process. Literally the only reason people bother following a process is because of them. But at the same time they are a huge drain on time and resources, and can enable bad or inefficient processes by manually keeping things in line. They add nothing to innovation - and it's a huge bummer to work with/around them.

ryandrake 7y ago
As one of the dreaded "process enforcers" at my company, it's a tough line to walk. Any organization with more than a handful of people needs some written and agreed upon process or it's just chaos. But you can't just blindly follow a process for its own sake. You need to be able to articulate why we follow it with examples from the past of what happens without it. And, you need to be willing to constantly look for ways to improve it. Anything I can do to automate processes or simplify them makes the developers happier and more productive and me less of the bad guy. An inefficient process where people are always waiting on something to happen is demotivating to everyone. Most process is scar tissue from something that went wrong in the past. We have mandatory code reviews because when we didn't we let a severe security bug get deployed to production. We have legal review all changes to the terms of service so we don't get successfully sued over it again. We have a schedule with code freeze N weeks before release because when we didn't we could never release anything because...just...one...more...change. I was hired at a small company once without any process for anything. No release process or schedule. No bug tracker. No testing plan. No code reviews. Nobody used source control. Builds from different workstations produced wildly different binaries. It was a total madhouse and it was amazing that anything got done at all. The CEO simply thought the engineers just weren't competent enough, but they were--it was just a circus because nobody stepped up to formalize anything. Once we got a bug tracker and SVN set up, things turned around pretty quickly.

shados 7y ago

I definitely felt this decades ago when I started in the industry. It's 2019 now though. only 8 years later, but a lot of things changed. The industry is growing fast, faster than knowledge can be shared even with all the power of the internet. It's very bottom up, with newer engineers severely outnumbering those that have been around. People are making fat salaries and companies are catering to their whims. They/we're more spoiled than they've ever been. People go to work hoping they'll be find a reason today to try the latest pattern they saw on Twitter or try the trendy new tech or library their friend just talked about. If you don't have people saying no, it quickly (very quickly) get out of control. As organizations grow, you need to start saying no. A lot. Else within 0.2 seconds of the new intern joining or the new grad coming in, or the lateral hire who was the king of their 5 people startup will be pushing some stuff that isn't mature enough or doesn't follow established patterns. More often, they'll be reintroducing something problematic that you just finished getting rid of 6 months ago. It sucks. Sure, there are some power hungry people who get off on power trip by being gate keepers. But overall, it's just people desperately trying to keep things in one piece and the company going, and that means making hard decisions and enforcing them. It just doesn't feel good to say no, arguing all day with people who have way more energy than you do, but it has to be done.

james_in_the_uk 7y ago
I have to say "no" a lot in my role. Thank you for this post.
deleted_account 7y ago
"Chesterton's fence".
- hinkley 7y ago
  Chesterton's fence is about taking away things we don't understand. There's a big gap between that and fighting cargo culting.
  - mark-r 7y ago

rangewookie 7y ago

This doesn't have much to do with saying "no" and everything to do with not understanding the fundamentals of a policy. If they guy who said "no swap" had a reason, then you could have discussed and potentially changed the policy.

SubiculumCode 7y ago

One of the easiest strategies to protect your reputation in life and business is to never venture to say yes (because if it fails you are on the hook), and to let your discussion always revolve around an idea's failings, never its strengths. It gives an appearance of expertise and command, but risks little, and thus often leads to promotion. But I personally find it poisonous to the enterprise's progress: Even the best plans have weaknesses.

dijit 7y ago
Er. This is awful advice. You’re also on the hook for impeding the organisation. I’m a “no” guy. My boss is a pure yes man. I don’t say no to impede progress. I say no to avoid overworking my team which would reduce the quality of our work. I say no, because it changes the scope of work in progress, and that requires sufficient justification because it means either throwing work away (which is both demoralising and wasteful), potentially increasing time to delivery or creates technical debt in droves. I say “no” so that people don’t make their own standards (like not following logging patterns, or metric naming conventions) which would cause all kinds of gunk in the infrastructure. It’s not about following a strict process. Although that can be important for streamlining workflow through an understaffed team like mine, it’s about ensuring that the quality of what we do is not at risk. Quality is our only currency, without it everything suffers.
- SubiculumCode 7y ago
  What you say is true, which makes the topic so very subtle because it depends on the context. Saying no to protect your people/process is a good thing. Saying no because it is personally profitable to be disdainful of other's ideas while offering up none of your own, irrespective of what is good for the organization or your team, is not. If one needs a mental picture, think of the quintessential hipster: https://sabotagetimes.com/life/hipster-irony-is-destroying-our-idea-of-sincerity-and-creativity https://sabotagetimes.com/life/hipster-irony-is-destroying-o...
- Aloha 7y ago
  Change for the sake of change is almost always bad, making decisions without considering externalities is worse.

OJFord 7y ago

Isn't no swap in prod fairly standard? Kubernetes pre-flight checks check it, for example. If enabling swap 'fixes' it, isn't the proper fix to provision more memory?

toast0 7y ago
> If enabling swap 'fixes' it, isn't the proper fix to provision more memory? Mr. No says all servers must match one of the existing configurations, none of which have very much ram. If you're on the border where a little bit of swap gets you zero pages, and no swap gets you stuck machines, you probably need to figure out how to scale within your constraints, but a little bit of swap should be ok, in the neighborhood of 1GB gives you a little bit of breathing room without prolonging the inevitable for very long.
- OJFord 7y ago
  1. The article states that the older machines had swap enabled, so they weren't homogenous anyway; 2. At some point you do have to change configuration, and if you have insufficient memory on one you either have insufficient memory on all machines, or a badly distributed workload.
sitkack 7y ago
Wait what? How do machines running K8s exec processes when they are near their physical memory limit? Swap does more than just provide a place for pages to get evicted to, it also increases the virtual address space.
- umanwizard 7y ago
  I don't think that's true, at least on Linux. This program uses 1 TiB of virtual address space, and runs fine on a machine with no swap (and much less than 1 TiB of physical RAM). #include <sys/mman.h> #include <stdlib.h> int main() { size_t const len = 1024 * 1024 * 1024; // 1 GB for (int i = 0; i < 1024; ++i) { char *p = (char *)mmap(0, len, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0); if (p == -1) exit(-1); } } The space won't be associated with any physical object (whether RAM or swap) until you actually touch it.

WalterSear 7y ago

I'm coming to the conclusion that many technical problems are actually morale problems. This especially goes for flamebait, where some people come down heavily on one side of the argument or another. If you don't care, it's much easier to default to someone else's directive, or complain about a toolset you haven't bothered to learn.

draw_down 7y ago

Obviously such a person can exist but my Spidey senses always tingle when I’m reading about what an asshole someone else is.

blueyes 7y ago

I think of most in-house corporate lawyers as playing the role of "no men". There job is to highlight risks and stop things from happening. It's a rare lawyer who understands business goals and can balance them against risks.

isostatic 7y ago
That's not the lawyer's job - the lawyers' job is to present the risks, not to make the decisions as to whether the risk is worthwhile
- Aeolun 7y ago
  In theory. In practice it often turns out that the decision makers think that any risk at all is unacceptable, giving the lawyers de-facto control.
  - isostatic 7y ago
    In which case you have poor decision makers, not poor lawyers. The benefit of domain aware lawyers is they can explain the specific risks and help the decision makers apply those risks to the business.
mobjack 7y ago
Good lawyers figure out how to say yes while minimizing legal risks.

hrktb 7y ago

I don’t get it. 1 - there is a Mr.No guy but, the author only knows him by reputation 2 - there is a technical problem that seems to revolve around memory 3 - the author has an idea of a solution, she actually tries as people don’t believe her 4 - her solution works, but she is told the “No” guy said “no swap” once a long time ago Her reaction: “Did I care what he supposedly said? No.” Why doesn’t she go talk to him ? The “No” guy could have had another solution than raising swap. He could have recognized being wrong, allow swap and learn from it. He could continue say “No” but now the author has an actual experience of it, and not just hearsay and reputation. It seems like nobody communicates correctly in that company, the author included, and it must hurt in so many places. Edit: fixed author’s gender, thanks

Shoop 7y ago
Just an FYI, the poster is female.
jedberg 7y ago
For the record, the author is a "she".
wfbarks 7y ago
These where my thoughts, "No" guy could have a good reason for no swap. no swap might have been a fix for some other issue OP is unaware of, which OP has now reverted. Don't know until you ask...
- romwell 7y ago
  >no swap might have been a fix The chances of that are astronomically low. From the article: >I later found out that all of our older machines had swap enabled. It was just the ones which had been replaced or reinstalled which had been popping up without it, perhaps as part of a policy change. That means it was perfectly valid for us to run with swap on, since most of the machines we owned were using it.
vxNsr 7y ago
you caught like half the male pronouns, there are still a bunch left, it actually makes it more confusing.

beiller 7y ago

Trust but verify. I'd bet if the author went to the original source, the "no man" himself, they may find that there is actually no such policy! Just a mishearing of a conversation parroted by "wrong people".

hrktb 7y ago
“Trust but verify” litteraly meant “Don’t trust” (as in “don’t trust the Russians” in context). Was it the nuance you intended ?
- yawaramin 7y ago
  That's not what it meant. https://en.wikipedia.org/wiki/Trust,_but_verify https://en.wikipedia.org/wiki/Trust,_but_verify
  - hrktb 7y ago
    does it ? > Reagan used the phrase to emphasize "the extensive verification procedures that would enable both sides to monitor compliance with the treaty"
    - yawaramin 7y ago
      But you have to look at why he used that particular phrasing–a clever bit of wordplay to quickly get the idea across to Russians in familiar terms. It's a Russian figure of speech in the first place.
      - hrktb 7y ago
        “Wordplay” is too generous IMO. The original russian proverb is also double speak, as we would say now. It’s obvious when applied to trivial situations: “I trust my wife, but I take the time to verify her phone every night and check the private eye report of her day” We wouldn’t call that “trust”, pretty much the opposite

deleted 7y ago

[deleted]

incomplete 7y ago

i was at this company when this happened, and oh boy, EVERYONE who knew the person in question and the 'new' kernel feature got some amazing, long-lasting and well deserved laughs. btw, it was /proc/<idiot's username>, not /dev :)

d2mw 7y ago
I didn't recognize the story until you corrected the path. https://news.ycombinator.com/item?id=1242831 https://news.ycombinator.com/item?id=1242831

msla 7y ago

The extreme example of Chesterton's Fence. Is this, then, Chesterton's Berlin Wall? That is, a completely idiotic waste based on idiot politics?

elihu 7y ago
I suppose it's also worth noting that Chesterton's Fence is an (imagined) real physical thing which required time and resources to construct. Thus the odds are pretty good that it was made for some real purpose, as opposed to a policy which is ephemeral and does not intrinsically require a significant expense of time or effort to create.

sriram_sun 7y ago

Interesting combination of the "yes men" parroting the "no men"! Great example!

domnomnom 7y ago

May as well start off with an introduction to toxic masculinity.

Aloha 7y ago
How so? The author is a woman.

drewg123 7y ago

A powerful "no man" can be a valuable asset sometimes. I was involved in the design of an ASIC. This was going to be used by people from all different areas of the company, and they all had various demands. Most had never worked on a chip before, and didn't realize the cost of what they were asking for. The easy thing to do would have been to ensure that every requested feature was added But that would have burned area, power, and huge amounts of design and verification time. Luckily we had a senior guy who had the clout to say "no". Over and over again, he would tell people how much things would cost in terms of money and time, and ask if the features they wanted were worth that much. Or he'd point out that what they wanted could be done with firmware plus existing features. Overall, he probably saved millions of dollars and caused the project to be delivered on time.

freehunter 7y ago
I have an unofficial board of advisors for my small company, and one of them is a "No Man". I knew that when I asked him to join. Every time the company has an idea, this guy will shut it down immediately. He forces us to defend every single decision. Do we always listen to him? No. Does he always win? No. Is he always right? Absolutely not. But I keep him around because he forces everyone in the company to defend every idea. If you still feel good about the idea after defending it from him, then go ahead. If you've lost faith, then I'm glad we didn't waste any time on it.
- daotoad 7y ago
  Every company really, really, really....really needs someone like that. Imagine how much better the Star Wars prequels could have been if only Lucasfilm had a strong No Person to stop them from doing stupid shit.
  - wincy 7y ago
    George Lucas’ ex-wife was the no-woman on the original trilogy. She was his editor!
    - 7y ago

quickthrower2 7y ago

I still have an internal no-man that says "Don't use OR's in where clauses" in my head from years ago, then I have to stop myself and think "OK lets actually test the performance of this particular query rather than rely on a no-man policy in another company from years ago"!

yawaramin 7y ago

Sounds like that workplace is a no man's land.

dredmorbius 7y ago

This seems far less about "just say no" than it is about not understanding, or insisting on, long-obsoleted policies. I've encountered something of a converse of this situation: a set of prod servers running a Java workload on which performance was frequently abysmal, which turned out to be due to a very poorly tuned set of JVM GC parameters. The feature(s) were poorly documented (lots of detail, little useful guidance), so it took some tuning and ongoing performance monitoring to see that 1) the problem had been solved and 2) that no new problems were introduced. The latter being of particular concern as this was a long-lived project that had gone through generations of development and ops support (I was latter), few of whom were on current speaking terms with the shop in question. The management feedback I got on my monitoring was (a rudimentary bit of instrumentation, and periodic checking) was that this seemed to reflect an excess of concern and/or attachment to the issue and/or solution. Given the numerous other years-outstanding issues which I'd identified and (eventually) largely remedied, I found this response tremendously perplexing.

m463 7y ago

Even saying "no" should have checks and balances.

the-rc 7y ago

I know this "no man", had to work on an approval from him once, and I actually saw the infamous code change that implemented this. I don't remember him reviewing it. IIRC it was sent to a mailing list, to which he might have or have not subscribed.

stevenjohns 7y ago

I'm a bit of a "no man", and at a company I worked at in the past my (pseudo-technical) manager told me to spend hours creating some sort of extremely purpose-specific test environment to aid in resolving a ticket that would only take me 30 minutes to resolve directly - so I just resolved the ticket instead and closed it, and got on with the rest of my work. Eventually I was dismissed from this company a couple of days later and this situation was raised as one of the points. I was told "you can't refuse a mission given to you by your manager, regardless of the circumstances, that's ridiculous" and when I replied "I have different views to you on this" it was considered to be rude. The person who dismissed me is more or less the management boxing bag - a certified "yes man" who nobody respected. Of course this hasn't changed my attitude, though. Still a proud "no man" and wouldn't go with the flow unless I'm sure everyone completely understands the consequences. At some point people have to realise that their loyalties don't go towards their manager and instead should go towards the shareholders and clients. There is a bit of give on this, of course, but eventually you have to do the right thing by the people that actually stand to gain and lose - not just being another cog in the machine that exists only to support other cogs in that machine.

ilovecaching 7y ago
Sounds like a toxic place. Instead of compromising or arguing a technical point and making the discussion an engineer effort, your manager ruled by feudal law. This is the antithesis of good engineer, in which good technical arguments and trade offs are made with everyone equal at the table. Why your manager is making technical decisions is beyond me as well. Managers are supposed to be managing people, not doing engineering work.

pipingdog 7y ago

The most important part of a fast car is good brakes.

AstralStorm 7y ago
As opposed to stuck brakes, or ones that activate at random.

mr_tristan 7y ago

I don't think this is necessarily about people who are critical, it's that you can have someone who is just as stupid as a "yes man": they just say "no" just because instead of, you know, actually thinking. I don't believe this "no man" is someone who is managing their attention or saying no for an actual reason. They're just contrary. And yeah, there's a lot of them around. What I actually find even more frequently are "parrots". These are people who "learn by runbook". Instead of asking why something is done a particular way, they want to learn how to parrot some procedure. It's another variation of things: instead of saying "yes", or "no", they just parrot "process X". I think this is really where the "no swap in production" example came from. I've seen _lots_ of stupid similar ideas in software development.

kev009 7y ago

Honestly saying no is one of the most important things either a startup or large company can have. Saying no gracefully, respectfully, and having the receiving person understand the rational is the hard part. This is one of the function of "fellows", "Senior Members of the Technical Staff", "Principal Architects" or whatever your org calls it and the people selected for that should be excellent at it. I guess the problem the author points out is that is the only thing a person might do and that is not good. Folks in this position should also know how to say yes and build buyin from business people/exec/program management. Most organizations fail horribly at saying no. Instead they try to silo things to cope. This works to some extent, and you will always have silos as a company grows up, but you really should never end up with 12 programming languages, 5 RDBMS, 4 NoSQL DBs, 3 Javascript frameworks, 3 CIs, 3 cluster schedulers and then all the aforementioned also duplexed across organization units. That level of sprawl reduces margins, increases onboarding time, stresses even the most experienced staff, and makes adaption and business decision making murky at best.

bcrosby95 7y ago

Most developers probably view sysadmnis as "no men". You want to do WHAT in production? flipoff

beardedwizard 7y ago

You ran with swap on because you saw a bunch of other machines with swap on and decided the policy was invalid. Thanks for making real security jobs hell, this is the #1 developer excuse for avoiding security requirements.

jcrites 7y ago

I agree with the author's overall premise, which is that other people who don't own and who are not responsible for your system should not have authority over how you build and operate it. One of the things that I like about the current company where I work is that owners always have the final say about their systems, not any external technical authority. There is no such authority who could say no where I work (with rare and limited exceptions), and I'm glad for that; service owners make the judgment call about trade-offs. > "so-and-so says no swap in production". That said, while I understand the point of the article is something different, I broadly agree with this advice: generally no swap in production. If you need swap, then that's probably a sign that the workload isn't matched well to the resources of the machine (unless the workload was specifically designed to employ swap, which most are not). The problem is that: maybe most of the time the application's working set will fit within memory, and so it will perform well, but some of the time it will begin to swap and performance will degrade. The result of adding swap to a system is non-deterministic performance. If you're building high-availability, high-performance production systems, then you don't want the potentiality of arbitrary performance degradation or non-deterministic behavior. Instead, you would prefer to solve the root cause -- such as by increasing the amount of memory available to the process or system, so that swap is no longer required. An engineer adding swap to a system is like a doctor treating a patient's symptoms instead of curing their disease. Both are useful but one is preferable. If you encounter a situation where you could solve an application problem by adding swap, then an even better solution is to address the root cause -- ensure the application doesn't run out of physical memory. Swap is a band-aid that masks a risk better addressed directly. Hence: no swap in production. (On the other hand, it might be fine to employ swap in production for services that aren't performance sensitive, and where it's preferable to accept the performance hit from using swap rather than pay more for better hardware. That's a fine trade-off if consciously made. In my experience, variations in performance are usually much more problematic to a business than changes to cost.)

_visgean 7y ago
But not having swap will cause the machine to misbehave (https://utcc.utoronto.ca/~cks/space/blog/unix/NoSwapConsequence https://utcc.utoronto.ca/~cks/space/blog/unix/NoSwapConseque...). Swap in this situation acts like a backup, surely you would prefer temporal degradation then total system shutdown. The solution here is continuous monitoring of free memory (we use datadog) + swap.

papagoat 7y ago

I used to have a “don’t know/can’t remember” man in my team. Can you recall what this code you wrote previously do? “I can’t remember.” Do you know where the files are placed? “I don’t know.” Where are the documentations for xxx. “I don’t know.” Do you know the difference between xxx in testing in production server? “I can’t remember.” It was frustrating. 9 out of 10 questions were “I don’t knows” and “I can’t remembers”. I’d much rather he said yes or no.

taejo 7y ago
Of course it would be great if everyone remembered everything, but isn't an "I don't know" better than an 80% guess? The longer I do this job, the more likely I am to say I don't know, when I don't know. > Can you recall what this code you wrote previously do? What it actually does might be different from what I remember. It might have been changed since then -- maybe by me, maybe by someone else. I might have made a mistake in the first place, and it doesn't do what I think. "I'm not sure, let's look at it."

slewis 7y ago

I know exactly who she's talking about! When I was at the unnamed company I worked on a project that he retained control over and said no to a lot. We found that if we sent him a very well reasoned, documented and measured change he'd say yes. I was there when the no swap policy was under lots of active discussion. It was there to ensure that application authors could have strong expectations on how long a memory access would take. Imagine trying to provide evidence that removing the "no swap" policy wouldn't affect any of the applications that were running across a very large infrastructure. Her point really resounds though, this person was referred to all the time in conversation, "well, we probably couldn't get this past X". He was mythical. In general that was often a good thing. Making changes that could have such a broad impact should be well reasoned. But it's also quite annoying, having a nagging voice inside your head that just says "no", seemingly disconnected from the actual source.

nailer 7y ago
What the app something that required deterministic latency? There are situations where your rather hit OOM and kill your app and go buy some RAM rather than have it operate slowly - executing trades too late, or pointing at targets that have moved, etc.

unsined 7y ago

One time my app started crashing when I added a large async routine. I ssh'd in and topd and noticed memory and swap were maxed out. Doubled the swap and it runs just like it did before. Gotta love simple fixes.

kjgkjhfkjf 7y ago

The issue here isn't with "Mr No", it's with those who mindlessly parrot his policies. I've met "Mr No" at Google, and he is in fact quite a reasonable and funny guy.

lazylizard 7y ago

Why not just buy more ram?

m0zg 7y ago

Microsoft used to be like this in 00's (I don't know about now). You had to run everything by Legal, and the default answer from Legal was always "no", no matter what you're asking. Want to use zlib in your product? Even though Windows is using it in like 3 different places already, the answer is still "no". JQuery? Out of the question. Anything more exotic, might not even ask. I'm pretty sure Microsoft wasted hundreds of millions of dollars reinventing a wheel, and doing it badly, over the past few decades.

crimsonalucard 7y ago

There's also the Maybe guy. All this guy ever says are things like "Maybe that's possible," "in theory that maybe could work," "I'm not sure if that will work." The thing that the maybe guy differs from the Yes or the No guy is that he's always 100% correct. He's never wrong ever.