8 ms·
On the flip side, when something DOESN'T need HA, sub-millisecond query response time, or lynx and IE Edge compatability, people need to know when "good enough"
by unethical_ban 9y ago
On the flip side, when something DOESN'T need HA, sub-millisecond query response time, or lynx and IE Edge compatability, people need to know when "good enough" is good enough.
On my ops team, we've gotten some flak for not building robust enough of a request queue for some tasks. But it's been down several hours in the past year. The server almost never needs maintenance. None of the workload is real-time. App restarts are acceptable if memory leaks occur.
If we did everything by the enterprise book, we'd still be 70% of the way to a deployed product, instead of 15 months into its completion.
- mistermann 9y agoExactly, this is the substantial other side of the argument that almost always seems to be left out of the discussion. "Good enough, done in a week" is a perfectly viable option that is rarely offered by timesheet-padding developers.
- mcbruiser3 9y agoseems often developers tend to over engineer or focus on edge cases that inflate costs, and not given enough attention to ROI
- dsp1234 9y agoConversation from two days ago: manager: Why did you allow this thing to happen. me: It was an edge case that I pointed out 7 months ago. I've put it on the backlog 3 times, and you've removed it each time citing that it's an edge case that would never happen, and other newer features needed to be implemented. manager: That's an engineer's response. Just fix it! me: welp, shrug
- majormajor 9y agoI've had luck redirecting this type of question into "why did the process let this happen" where it becomes less my fault vs your fault and more "i did what you said, you did what the business person told you to prioritize, something broke" from a blame-agnostic point of view. In certain cases I've also overridden the "ignore that for now" in favor of at least some minimal fix because part of my job is to give you what you don't know you need to ask for, not just exactly what you ask for. And there's the value add because compared to people who don't consider that part of their job, my stuff / my team's stuff works better out the door. Another similar question that I've noticed others not always asking are things like "how often would this be used again, is it just a temporary one-off" which, of course, has an answer that's subject to change, but at least guides the initial design. And when it changes it gets revisited.
- taneq 9y ago> In certain cases I've also overridden the "ignore that for now" in favor of at least some minimal fix because part of my job is to give you what you don't know you need to ask for, not just exactly what you ask for. This is a two-edged sword, though. I agree it's best practice from a technical point of view, but enabling wilful ignorance on the part of someone above you in the chain will eventually lead to a showdown when you really do need a week to fix it and it really is critical, and they tell you not to, and they'll stick to their guns because (as far as they know) they've been right every time.
- majormajor 9y ago> This is a two-edged sword, though. I agree it's best practice from a technical point of view, but enabling wilful ignorance on the part of someone above you in the chain will eventually lead to a showdown when you really do need a week to fix it and it really is critical, and they tell you not to, and they'll stick to their guns because (as far as they know) they've been right every time. Yeah, pushing that too much requires either (a) sufficiently-padded early estimates so it's not a slippage and a nasty surprise (a good practice anyway, but hard), or (b) a sufficient level of don't-give-a-fuck, or (c) really good judgment around "when this is necessary to give them what they need vs what they want" and when not to e.g. truly critical and immediately messagable stuff. A healthy dose of don't-give-a-fuck/willing-to-say-no is often a good thing anyway, though, if you're able to do it politically/smoothly enough :)
- JustSomeNobody 9y agoThis is so not uncommon. :)
- defined 9y ago> manager: That's an engineer's response. Just fix it! Translation: I didn't actually require any answer other than "I am sorry, sir/ma'am, it will never happen again." Facts are irrelevant. I may not always be right, but I am never wrong, code monkey.
- TheSpiceIsLife 9y agoThat's right. I have often spoken rapidly while saying the real reason(s) I believe lead to failure and then slow down and say "but never mind that, let's work out how to fix this and prevent it from happening, I'll get right on it", or something to that affect.
- hermitdev 9y agoIt's not necessarily over engineering. It can often be about competing priorities. Multiple pressures on your time. I worked at a hedge fund for 9 years in back office IT developing firm-critical software. We traded pretty much anything you can trade, so we had a lot of trading desks requesting changes all the time. So, while making a change for one desk, at least one other is bitching about why their changes is taking so long. Yeah, the change might only take 5 minutes, but it might be 4 weeks before I get to it. We traded for all but about 2 hours of the week. Means you had a narrow deployment window. Even once the change has been made and tested, you've got to wait for the deployment window. One weekend of each month was off-limits to system changes to options expiration. So, even the smallest, simplest change might take at least a week or two to get into production. Quite longer big or breaking changes. Toss in regulatory and compliance issues and you've got a lot of paperwork and sign-offs to do a deployment. You've got to track those people down, explain the changes to the managing director and the risk with making the change. Emergencies were fun. Either getting called or having to call a manager at 2am in the morning to get approval... I once had an emergency at 11pm. I got a call from my director about 15 minutes after I'd popped sleeping pills (I was having insomnia at the time). I went through every source of caffeine I had in my apartment to get through that; I dropped off the conference call at 4am. I recall hearing the call went on for several hours afterwards. TL;DR It's not always over-engineering, it's more often misunderstood or invisible (to the business side) pressures on the engineer's time.
- hysan 9y ago> and not given enough attention to ROI Which itself could be a result of management often viewing engineering as a cost center with no perceivable ROI. This can be reflected in pay not being commensurate with company performance and not stacking up to other employees (sales, management, etc.) and gives the engineers no reason to pay attention to ROI in their decisions. I would bet that in companies where engineering is very much involved with cost, ROI, and paid based on how those numbers turn out, the engineers would focus less on edge cases and know when to draw the line.
- kls 9y agoGuess who's ass gets chewed when the system craps out in the middle of the night, when Asia your emerging market is just coming on line? Funny this story reminded me of a recent event where my team was working on some reg-ex expressions for a language processor. Not overly complex stuff but not simple by any definition (they had been working on the set for 2 days), they had a side line manager from another department, that knew enough to be dangerous, so he decides he is going to run in his office and whip up some reg-ex, after reading the docs. So we threw it in the test cases, after tons of failures he got the picture that edge cases count, as he got a good visualization of how multiple edge cases increase the the odds of failure by orders or magnitude.
- sbuttgereit 9y agoI've been on both sides and I understand both points of view. When my responsibility is managerial in nature, I'll tend to prioritize the dates and the ROI, then weigh the risks more optimistically because risk mitigation usually makes it harder to achieve what I am being judged on. However, when I'm the one actually doing the work... I tend to over-engineer (assuming "bullet-proof" is over-engineering for a circumstance), test, and the like. While I'm certainly rewarded for on-time deliveries, I have a professional interest in seeing a solid piece of work; and I tend to face more negative consequence if the work isn't of sufficient quality than I do if it's late. What's hard is finding the right balance. The proverbial "SQL in 5 minutes" moment from the blog post may not fully appreciate the developer's job, but it can serve as a reality check to self-question if you're over-baking it.
- s73ver 9y agoDevelopers are also the ones that get yelled at, called on the weekends, and even fired if those cases happen. If you're willing to come in on the weekend to fix your query, be my guest. But I like being able to spend my weekend relaxing.
- grogenaut 9y agoHuh, where I work the managers are the ones who get fired for bad ops or oversights. The engineers are just penalized by dealing with shitty decisions so they dream about quitting. The smart ones just fulton out of shitty managers to good teams.
- salesguy222 9y agoYou most likely work for a company that hasn't yet been overrun by the corporate world's version of "career politicians" who are there to leech a paycheck at all costs. They often blame others while finding very sneaky and specific ways to, despite all odds, rise to a relatively insulated level without any modicum of technical or managerial prowess congrats!!!
- grogenaut 9y agoIt's likely the ship, drive more revenue or get fired culture that keeps people from settling in.
- WhitneyLand 9y agoInteresting observation. I assume you have the skills and experience as a software developer to properly judge the technical aspect of these decisions, not just what appears to be clear from the business aspect?
- borplk 9y agoThe problem is the businesses want to have their cake and eat it too. Damned if you do damned if you don't. If you don't focus on edge cases then later it's your fault for being sloppy. If you focus on edge cases you are "over engineering". Most of the time "over engineering" just means "I wish this was cheaper".
- dragonwriter 9y ago> "Good enough, done in a week" is a perfectly viable option that is rarely offered by timesheet-padding developers. I see it all the time from timesheet-padding developers who also are in a position that they'll benefit from the higher long-term support costs that that approach generates. It's just a matter of whether they want to pad timesheets in the short term or the long term.
- WhitneyLand 9y agoAre you aware that your observation does not generalize very well to developers overall? Or are you referring to some sort of government bureaucracy dealing with loosely coupled contractors bidding on jobs? Back on earth I can tell you full time devs at big 4 tech companies padding estimates would be a very difficult thing to pull off, for a variety of realities. Same situation for many other developers.
- dragonwriter 9y ago> Are you aware that your observation does not generalize very well to developers overall? I was responding to (as should be clear, since I quoted it) an upthread generalization about what “timesheet-padding developers” would or would not do. It's expressly not a statement about developers generally. > Back on earth I can tell you full time devs at big 4 tech companies padding estimates would be a very difficult thing And pointless, since they aren't paid hourly. Timesheet-padding is only incentivized when developers are paid (or work for a company that is paid) based on timesheets, that is, hourly. But, anyway, I wasn't talking about padding estimates, that was the poster I was responding to. My observations was that developers with incentive to pad timesheets often will do the opposite of padding estimates, going for quick and dirty development solutions which create more maintenance and support hours (of course, that only occurs as a result of structural incentives where the contractors building the system or feature also expect to be supporting it, but that's often the case.)
- s73ver 9y agoWhy are your developers feeling they have to pad their timesheets?
- WhitneyLand 9y agoHe wouldn't have a clue if they were or were not padding their timesheets. I don't blame him for that part - it doesn't take much complexity before any manager is incapable of understanding every detail. The scary part is he doesn't seem to understand the fact that this implies the most successful teams will utilize some degree of trust and efficient communication to deal with this reality.
- WhitneyLand 9y agoWhat experience has led you to make a comment that is insulting and I'm guessing, without foundation, that there exists a phenomenon whereby developers request more time than is actually needed to complete a project?
- manigandham 9y agoWhat? Do you think every single developer is some noble professional only focused on the project? 15 years with software dev experience and every single dev team at midsize and larger companies has always asked for far more time than necessary. Same with almost all enterprise consultants.
- WhitneyLand 9y agoHow do you know it was more time than necessary? Did you give them the time they asked for and notice at the end of the project people were sitting around bored? Or did you not give them the time they asked for and declare yourself correct after squeezing in a deadline? And if so how do you know what corners were cut or what technical debt was accrued to make a deadline the team did not sign off on? Whether every single person is noble is irrelevant. People are making broad generalizations here against the reputation of developers with what I see to be weak and anecdotal evidence. If someone has made serious inquiry into the subject and controlled for complexities like I suggested above then by all means let the chips where they may.
- manigandham 9y ago15+ years of experience gives me enough info to know what timelines should normally be. These people/teams took far longer and produced less (overall and per day). This is human nature, people will look out for themselves. Many, if not most, are looking to do the least amount of work in return for the largest reward, and it's very easy at bigger companies to pad hours, compounded by bureaucracy and poor management. This is seen in many other professions amongst all types of people, nothing specific to software.
- arethuza 9y agoI once had an infrastructure guy who worked in the same company quote me £70K for new hardware to host a single static HTML web page. £70K of new hardware! I stuck it on an existing server and nobody noticed though I might have got into trouble with the Change Prevention Board for not using the right form or something... Edit: He did have a carefully worked out explanation of where the money had to be spent.... which I ignored after hearing "£70K".
- mabbo 9y agoTsk, poor choice on your part. You should have started a side consultancy (get a spouse to own it maybe?) and say that you can do it for a mere £40K. Then dump the static file on S3, and let the interest from the £40K in your bank account pay for it.
- WrtCdEvrydy 9y agoOkay, that sounds pretty cool. S3 can host static HTML files?
- venning 9y agoYes. Works quite well and can handle quite a bit of load. Some native support within Route 53 helps make setting it up easy. http://docs.aws.amazon.com/AmazonS3/latest/dev/WebsiteHosting.html http://docs.aws.amazon.com/AmazonS3/latest/dev/WebsiteHostin... http://docs.aws.amazon.com/AmazonS3/latest/user-guide/static-website-hosting.html http://docs.aws.amazon.com/AmazonS3/latest/user-guide/static...
- Luuseens 9y agoYes, I host my blog that way: - Generate static files using Pelican - Dump the files to S3 - Set up Route53 to point a domain at the bucket - Set up CloudFront (Amazon's CDN) to sit in front of the S3 bucket, since I found pure S3 to be too slow Works incredibly well.
- RonInDune 9y ago
- deleted 9y ago[deleted]
- kafkaesq 9y agoYeah, you have to read the piece as a kind of parable -- emphasizing the most extreme cases of what can go wrong when the manager says "I don't understand, we had this other guy who said it could be done in 10 minutes / 1 hour / 1 day" or whatever. (And who hasn't been through that drill, like, a thousand times?) But as you said, knowing how and and when to negotiate between said extremes is exactly where the "art" lies.
- ktRolster 9y agoBut it's been down several hours in the past year. So has AWS, so you can kind of feel good.
- mullen 9y agoThe good news is that you can blame someone else! PS> Joking, but not really.
- still_grokking 9y ago>The server almost never needs maintenance. None of the workload is real-time. App restarts are acceptable if memory leaks occur. Ah! Classic duct tape programming! Something like this is fine as long as the requirements don't change. But we all know that at a point the requirements will change. And then you have to tell your customer that the "little new requirement" can't be accomplished with a cheap little change but requires instead a complete rewrite that will cost even more then the previous system did. The only question is whether you are a honest company that told the customer before that that will most likely happen in the future or your planed for that inevitable outcome in the first place...
- taneq 9y agoThe "new little requirement" which you couldn't have anticipated is probably no easier accommodated by your Hadoop cluster with GPU processing stack than it would be by a PHP script on a VM.
- paulddraper 9y agoProbably harder, I dare say.
- blackoil 9y agoLot of times, it is easier to rebuild, then try to an anticipate scale and requirements years in advance. Also in most cases a well written simple solution is easy to refactor, then a complex over engineered architecture.
- technion 9y agoHow much is IE Edge hurting the build? I ask because, last project I was on, I "introduced" Edge compat by removing the section of JavaScript that sniffed the UA and kicked people out.
- jessaustin 9y agoI appreciate the sentiment, but without rigorous testing this exercise could be a shot in the dark...
- mancerayder 9y agoOn my ops team, we've gotten some flak for not building robust enough of a request queue for some tasks. But it's been down several hours in the past year. The server almost never needs maintenance. None of the workload is real-time. App restarts are acceptable if memory leaks occur. If we did everything by the enterprise book, we'd still be 70% of the way to a deployed product, instead of 15 months into its completion. Sure. Except if this Ops team is like most Ops teams, they'd prefer to do it the (cue dark, brooding orchestral dum-dum-dum musical interlude) 'enterprise' way rather than be woken up by PagerDuty in the middle of the night because a new release push lead to the CPU spiking. Sorry, but I've almost never seen situations where people have skimped on HA and someone else didn't get excoriated during an outage. Maybe other people's experiences are different, but as a DevOps guy, even in a relatively new environment, my priority is stability. Not faith-based-computing based on someone's positive seat-of-their-pants past experience.
- toomuchtodo 9y agoDevOps guy here! Completely agree. Reliable infrastructure at the cost of velocity isn't just good practice, it's self-preservation if you're ops.
- mancerayder 9y agoYeah. I think that's a huge part of the reason why the so-called 'full stack developers' will probably never obviate pure 'DevOps' roles. A lot of people can't wrap their heads around why setting up infrastructure takes so much time, and focus attention on speed of raw product (code) delivery.
- unethical_ban 9y agoAh, see, we're talking about different things. Our little request queue, though prod-necessary, only needs 98-99% uptime. Same with much of our automation. Once you ask the simple question "Does this cause an interruption to our core business, or to security?" and you can say "no", HA requirements go down, down, down.
- 9y ago
- alfalfasprout 9y agoOverengineering. It's either the young engineer that wants to use the latest trendy technology or the old "architect astronaut" that believes only "enterprise level architecture" can solve all your problems. Even when I worked in electronic trading and we were dealing with 10 million events/second on combined feeds, we kept our architecture simple. Why? Experienced engineers realized that the more moving parts, there's more things that can go wrong (and it makes it that much harder to pinpoint). Just time your queries, get a P95, and stream data to an analytics database if you don't want your prod to be exposed to added latency. No need to create some fancy distributed consistent system with caching layers and enormous test harnesses if the analytics workload might change next week. I'm a huge fan of just creating a separate analytics database you stream prod data to and letting those with SQL knowledge play around there. Surprisingly, they rarely break anything. And if they do, it's not going to take down everything else with it