10 ms·
You're forgetting the cost of fighting IT in a bureaucratic corporation to get them to let you buy/run non-standard hardware Much easier to spend huge amounts
by dagw 8y ago
You're forgetting the cost of fighting IT in a bureaucratic corporation to get them to let you buy/run non-standard hardware
Much easier to spend huge amounts of money of Azure/AWS and politely tell them it's their own fucking fault when they complain about the costs. (what me? no I'm not bitter, why do you ask?)
- pjc50 8y agoThis is absolutely the right answer as to how AWS got so big in the first place. Capex and IT are huge pain points. Starting something up on the free tier isn't. Once something's running and providing value, spending money on it becomes a "necessity" and the obstructionism goes away.
- mlthoughts2018 8y agoIn most companies, AWS just becomes a new front-end to the same old IT bureaucracy, and dev teams are still disallowed from creating their own instances or EMR clusters or setting up new products like their own Redshift instance or ECR deployment solution.
- ethbro 8y agoRespectfully, those companies' cloud architects suck. If someone goes to the trouble to migrate onto cloud, and then replicates pre-devops workflows... wow.
- askmike 8y agoThis has less to do with cloud vs own hardware and more about how the company is structured. I've worked in companies before with an ops department: a few people responsible for managing all the cloud servers. All the devs (like me at the time) work locally and have access to an isolated dev/uat environment provided by that team. That team had most of the show automated, they weren't really provisioning any machines by hand. If something bad slipped into prod there was a process in place of how a dev would work together with someone from that team to fix it. > pre-devops workflows I guess this is what you are calling a pre-devops workflow? In a lot of fields not all devs are allowed to see/touch the complete production environment. Not everyone can go the netflix way of "everyone pushes to production and we'll just fix it when it breaks".
- meddlepal 8y ago> I guess this is what you are calling a pre-devops workflow? In a lot of fields not all devs are allowed to see/touch the complete production environment. Not everyone can go the netflix way of "everyone pushes to production and we'll just fix it when it breaks" Unwillingness shouldn't be confused with ability. Most companies can do this if they're not handling PII/PHI. It takes investment in smart people and time but most companies besides pure software companies see tech as a cost center and avoid investing in better infrastructure and platform systems.
- walshemj 8y agoVery much so when I was arguably doing devops for BT using PR1ME super minis' back in the day. A mate who was in operations for an IBM shop was horrified that our team were allowed to write our own JCL.
- ethbro 8y agoOne of my pet peeves is that there's no mandatory "History of ideas in software development" course in most CS curriculums.
- tetha 8y agoI dunno, that sounds like the a good way to do it. I kinda shake my head if people call for devs to handle all the infrastructure and everything in prod. Why should a developer concern himself with details of scaling and tuning postgres, elasticsearch or loadbalancers? If you don't outsource that, that's the job for the ops team. However, if that's the responsibility of the ops team, there's no reason for dev to have access to these systems beyond the application layer. That just seems like a good way to split the work of running and scaling an application. Now if we're talking about applications, that's something different. In fact, it's the contrary. I am a big proponent of having developers manage and configure their own applications, including prod. It's fast and makes them develop better applications. However, doing this for critical systems in a responsible way requires automation to enforce best practices. At our place, devs don't have root access to productive application servers, but devs have permissions to configure and use the automation controlling the productive application servers. And it's safe and rather risk-free because the automation does the right thing. And it's also a different thing if we're talking about test. By all means, deploy a 3 container database cluster to poke around with it. I like experimentation and volatility in test setups and PoCs. Sometimes its just faster to solve the problem with 3 candidate technologies and go from there. Just don't expect things to go into production just like that. We'll have to think about availability, scaling and automating that properly first.
- solatic 8y agoA lot of those companies don't have cloud architects, just Amazon/Microsoft/Google sales reps talking into the ears of MBAs about turning capex into opex.
- badpun 8y agoIn my current company (a large bank), our team (~50 people) is trying to leave the on-premise infrastructure and move to the cloud because the on-perm stuff is managed in such a way that it's hard for us to acomplish anything. It will probably cost a lot more to use the cloud, but we're gladly willing to pay for it if we can shed the bureucracy this way. I've read that in medieval times, kings have sometimes abandoned their castles for new ones after too much fecies were accumulated in them (as people were shitting wherever back then). I feel a strong parallel in this story to our situation.
- supernovae 8y agoNot every company is a single page webapp and simple service portfolio... I work at a place with 3,000+ developers and over 700 applications - there is no way in hell our cloud portfolio would have any standards if we didn't have a robust operations/engineering team making it work. Sometimes, even when you adopt the cloud, you realize that your operations model is even more important and there is nothing wrong with that. The real problem is believing that there is a pure model that works for everyone.
- whatshisface 8y agoThe parent's point was that if the ops/deployment engineering team is unresponsive to the needs of developers, it may end up being better to run with no standards in cowboy-mode. If they ops team is extremely fast and highly skilled, they will be a boon full stop. If they are unskilled and politically obstructive, they will be a curse, full stop.
- supernovae 8y agoI got a more "you're doing it wrong" vibe...
- ethbro 8y agoI work at a Top25 US company by revenue. In no way am I responsible for our migration going well. But I do feel if a company this size can do it, then others failing to do so says more about their architecture teams than the endeavor's futility.
- deleted 8y ago[deleted]
- throwaway2048 8y agowhy not cut out the middle man and just say those companies suck, and the cloud isn't going to fix it for you.
- ethbro 8y agoBecause that seems overly reductive. There are things you can do in cloud (especially as an SMB) that weren't possible on-prem. To look at those new opportunities and say "No, how can we do things exactly the way we were?" seems like the real mistake.
- VHRanger 8y agoThey seriously can't buy a graphics card and slap it in the PCIe slot?
- rjzzleep 8y agoLast time I worked in big corporate I had to use their laptop even if I didn't plug mine into anything, because of security. At some point said laptop got stolen and I sent a buying request for a new one. Took a year without feedback and a heated discussion for my boss to accuse me for having been to passive on it. I was then using my own laptop. At that point you can't even call it shadow IT anymore. Any buying requests go through the direct supervisor, the financial department, the director of the company, then back to the supervisor for the final signature. And because it's Germany, it was on paper. When I say any buying request, I mean any. From any programming book, to any accounting book for finance or any HR book. You'd think people at the top have better things to do. You think that, until you hear the department heads discuss which excel format files should be saved in.
- dagw 8y agoThey seriously can't buy a graphics card and slap it in the PCIe slot? Well I mean they obviously can... First of all you're assuming IT will just let you have a decent machines with PCIe slots. It's all about laptops don't you know. Workstations are so 2012. Secondly while they might, after much begging, let me have one graphics card to put into this one old workstation I've scavenged, they certainly won't let me have a machine with top of line specs with several cards or, heaven forbid, several machines, (but they'll happily let me have a laptop that costs just as much for less than half the performance).
- Moter8 8y agoAhh yes, corporate laptops. At our enterprise, we have HP ultrabooks or however they are called. Windows 7, 32 Bit. You know, it has a decent i5 on it, but there is some specific bottleneck making it terrible. Every employee that sits in an office has this piece of crap. Takes 3 minutes to start IntelliJ. Since some years ~50 developers were able to get Macbooks, luckily I am one of them, else I'd just be pissed off every single work day.
- nabla9 8y agoThat's very passive aggressive way to deal with it. If that's your only option you are really a cubicle slave in corporate hell. In my opinion it's better to escalate upwards with proposals and not back down easily. You just have to frame it correctly and use right names and terms. * Usually big companies understand the concept of "a lab" that has infrastructure managed outside the corporate IT. Once you fight the hard fight, you get your own corner and are left alone to do your job and can gradually grow it for other things. * Asking forgiveness works even for large companies. Sometimes someone is not happy and you 'are in trouble' but not really. You just have to have a personality that does not care if somebody gets mad at you sometimes.
- speby 8y ago> You just have to have a personality that does not care if somebody gets mad at you sometimes. THIS
- scurvy 8y agoIn my experience, the blind drive to the cloud comes from above. It's someone's pet project and you're either on-board or not. I won't go as far as calling it yet another cyclical IT phase, but it has all the hallmarks of one.
- lotyrin 8y agoIt has to be, even with as irrational as the market can be I don't see people sustaining this kind of spend on public clouds -- the cultural barriers and perverse incentives that prevent effective use of private / hybrid cloud have to erode eventually.
- mmt 8y ago> have to erode eventually That's not borne out by history, especially in the face of "enterprise" hardware, software, and support, which, in many large companies, is being replaced by cloud (at lower cost!). For smaller companies, it may be a different story, especially at the next economic downturn, especially if VC money becomes scarce enough for long enough.
- ConcernedCoder 8y agoInnovation being sidelined in favor of status quo and 'cover-my-ass' risk management highlights exactly the reason I won't work for a bureaucratic corporation...
- jimmy1 8y agoMoney^, benefits^^, ample PTO^^^, and a 8-5 workday allowing plenty of time for the important things in life like my family are exactly the reasons I do. ^: You know, like real money, not 90k plus 2.5% of something that will probably not exist next year ^^: You know, like real benefits such as low-cost, low-deduct. health insurance, matching 401k, etc. ^^^: As in, real, actual PTO, not this "unlimited" bullshit some startups and now sadly companies are replicating. If you decide to quit and can't "cash out" your PTO, you don't have real PTO.
- tapatio 8y agoLow deductible healthcare? Where do you live? Europe?
- jimmy1 8y agoAmerica, surprisingly, in a southern state no less, where heart disease and diabetes rates are higher. Our work has a grandfathered non-ACA plan (we got to keep our old insurance but means we miss out on "free" stuff like contraceptives and a breast pump, and other banal things, but in exchange, we get much better coverage, and a lower deductible)
- w8rbt 8y agoYep. No one telling me no anymore and I can write Lambas to replace cron jobs, use RDS to replace DBs, use S3 and Glacier to replace Storage, etc. Fargate is awesome too. No gate keepers nor bureaucracy just code and git repos. That's why AWS is so awesome. And, I can show exactly how much everything costs. As well as work to reduce those costs. AWS added to a small, skilled dev team is a huge multiplier.
- benbristow 8y ago> use RDS to replace DBs RDS is just managed databases though...
- unbearded 8y agoUnfortunately, the fight goes even further than that when you go against the cloud. Last week I was in an event with the CTOs of many of the hottest startups in America. It was shocking how much money is wasted on the cloud because inefficiencies and they simply don't care how much it costs. I guess since they are not wasting their own money, they can always come up with the same excuse: developers are more expensive than infrastructure. Well... that argument starts to fall apart very quickly when a company spends six figures every month on AWS. I'm on the other extreme. I run my company stuff on ten $300 servers I bought on eBay in 2012 and put inside a soundproof rack in my office in NJ, with a 300 Mb FIOS connection using Cloudflare as a proxy / CDN. The servers run Proxmox for private cloud and CEPH for storage. They all have SSDs and some have Optane storage. In 6 years, there were only 3 outages that weren't my fault. All at the cost of office rent ($1000) + FIOS ($359) + Cloudflare costs and S3 for images and backups. With my infrastructure, I can run 6k requests per minute on the main Rails app (+ Scala backend) with a 40ms response time with plenty of resources to spare.
- spullara 8y agoOnly a Rails developer would think 6k requests per minute with 40ms latency is reasonable with all that hardware. If you rewrote it you probably only need 1 server but you will probably make an argument about how developer time is more valuable :)
- captain_crabs 8y agoIt's a matter of dosage. You might be talking about how another tech stack would perform better at this metric, but the price of that is the company would have been simply unable to ship anything in the time they could afford. Swinging the conversation beyond the dosages of either side doesn't produce interesting insight
- unbearded 8y agoExactly. I'm pretty sure if I developed the whole thing in C, it would be really fast. But I would be the slowest part of the development process.
- mandeepj 8y agoYou always have at least two options, and this case is not extreme - 1. You can blow up any amount if you like to. 2. Or, you can figure out what you are trying to do. Then, learn how to do it better. There is a cheaper way to run in the cloud too - https://twitter.com/troyhunt/status/968407559102058496 https://twitter.com/troyhunt/status/968407559102058496
- poulsbohemian 8y agoCould be very interesting when we have another economic downturn to see if attitudes on this change. It certainly seems more cost-effective to run ones own technical operations rather than offloading onto AWS / Google / Microsoft.
- mmt 8y agoIt will only matter if the infrastructure is a large part of their cost of technical operations (as opposed to labor) or, more importantly, overall cost of operations. In another thread on here (unfortunately I can't recall which), an executive shared a sentiment along the lines of "I don't care if it's 1% or 0.1% of my overall budget". Perhaps 0.9% would become more significant during a downturn, especially for startups if VC money dries up.
- kamaal 8y ago>>politely tell them it's their own ... fault when they complain about the costs. They don't care. Neither do their bosses. Not even the CFO or the CEO. The only people who will care are the investors. And sometimes not even them, they will likely just sell and walk. Only people who are likely to care are activist investors with large block of shares, those types have vested interests in these things. Part of the reason why there is so much waste every where is because organizations are not humans. And most people in authority have no real stake or long term consequences for decisions. This is everywhere, religious organizations, companies, governments etc. Everywhere.
- chaosbutters314 8y agoWe just got around them and argue after the fact. It is a pain no matter what. But we actually do both, interernal custom builds and cloud computing so no matter what we are set