8 ms·
Promotion based architecture is self fulfilling prophecy at least in BI/ data world. I see everybody around me moving to cloud, without really good explanation
by i_like_waiting 4y ago
Promotion based architecture is self fulfilling prophecy at least in BI/ data world.
I see everybody around me moving to cloud, without really good explanation why. Only reasonable thing I can see as an pattern is that cloud experience on top of data things gets paid 30% more. It made me consider cloud a lot.
I was considering switching to cloud, just so I can put in my CV "experience with migration to cloud".
For next person commenting that it makes sense: it doesn't with 200gb database and super predictable workload, growth and usage.
- morelisp 4y agoHell, I'm working with ~200TB and it still doesn't make sense if you work out the costs. As far as I can tell it only makes sense if you have a ton of data, and only ever use a fraction of it at a time, for one-off jobs, infrequently.
- recursive 4y agoThere are non-IT perspectives on this as well. My company offers two deployment scenarios: host it yourself and cloud hosted (SaaS). Many of our customers choose the latter because their internal IT systems require more process, and they just want to get something up and running.
- wil421 4y agoJust to add, at my last company it could take up to 6 months to get a VM provisioned on prem. We could provision what we needed in Azure on demand.
- garciasn 4y ago>For next person commenting that it makes sense: it doesn't with 200gb database and super predictable workload, growth and usage. Depends on the company. I've been working for marketing agencies for the last 15 years and they're generally staffed by, at most, 1 IT person who is in charge of a third-party vendor relationship that offers managed IT services. Those IT resources (internal or vendor) don't specialize in data and often don't know how to deal with it well, predictable workloads or not, and often offer up solutions which are not appropriate (cost or otherwise) whereas there are managed solutions from cloud vendors that do. BigQuery, for example, handles compute and storage for you and rolls it into one reasonable query price. No need to worry about any management of anything there and just sit Data Studio (included) or any other BI tool (Tableau, etc) on top and you're good to go. I get your skepticism and welcome it, but you're being a little rough. We're cloud native at my current company (I did a partial cloud migration at my last which was completed after I left) and it makes my life leading a Data Engineering and Data Science team MUCH easier without upfront hardware/software costs or long-term contracts, which were STAGGERING and left us with much more difficult to maintain, upgrade, etc hardware that took up most of my job as opposed to almost none of it today. YMMV.
- i_like_waiting 4y agoPartially depends but company with 1 IT person that is not data focused, there is low probability that they will have that amount of data. For anything above what we have I think it weirdly depends on country/ salaries. In US, 100k per year is no brainer for cloud as even 1 FTE would cost much more. In non-western Europe, 100k is deal killer as you can hire 2 senior DBAs and you still have enough money for quite a reasonable server.
- garciasn 4y agoWith extensive experience in this vertical, (marketing agency and analytics), 200GB is a _partial_ day for one datasource, let alone table or total DB size.
- i_like_waiting 4y agobut does team have 1 IT/ data person? Edit: sorry I see that you are the same person I was replying to. Then to understand it better, who is consumer of so vast amount of data?
- deleted 4y ago[deleted]
- garciasn 4y agoThe company has 1 IT person and uses a third-party vendor for IT support. We have several data engineers and scientists, but we are not specialists as DBA or cloud infrastructure; it's just one very small part of our work.
- i_like_waiting 4y agoInteresting, thats correct, its segment of the companies that I didn't take into consideration. Thanks
- funcDropShadow 4y ago
- reaperducer 4y agoI see everybody around me moving to cloud, without really good explanation why. People just buy into the "cloud" marketing. They don't have the ability to think and reason, and so don't understand that "cloud" just means "renting someone else's computer." I built a complex in-house medical system. Quick, reliable, and liked by the users. I was in the middle of adding a major new feature when all of the management in the IT department quit. The new people immediately decided that the whole thing had to be "in the cloud." I was removed from the project, and they hired three people full-time to rebuild it. That was three years ago. The new system is still not online. The users are still using my old system, and the feature I was working on never got added because my presence and input was not welcome because I "don't understand the cloud." So I got moved to other projects. People talk about "the cloud" with the same fervor and language as members of a cult. And you will be an outcast if you dare challenge their way of thinking.
- aurelianito 4y agoGiven what you explained, it seems to me that the cloud was an excuse given by the new overlords to just get you out off the way. Cloud is fashionable now, but any excuse would work for them. This seems to be a power grab.
- reaperducer 4y agoI don't think it's about power. Because of the way the organization is structured, I am isolated from the IT department. Moving me out brings no advantages to either part of the org. In fact, it's probably a disadvantage to IT because it has to burn three people from its allocated headcount. Where previously, it had a bonus person that didn't come out of the IT budget.
- guhidalg 4y agoIt's always about power and politics when bone-headed decisions like that are made. You correctly identified that _you_ did not report to IT, therefore _you_ were a threat to IT's power. Maybe you could have been re-orged under IT, but they probably saw you as a troublemaker who wouldn't kotow.
- WJW 4y ago> Only reasonable thing I can see as an pattern is that cloud experience on top of data things gets paid 30% more. You say that like a pay rise of 30% is not a good enough reason all by itself for many people. > For next person commenting that it makes sense: it doesn't with 200gb database and super predictable workload, growth and usage. Perhaps not for technical or business financial reasons, but those are not the only possible reasons someone might do something. As mentioned before it can make a lot of sense to migrate to cloud if it means you can get a 30% pay rise.
- tomc1985 4y agoThere's a finance angle behind cloud stuff too that's irrestible for the bean counters: cloud stuff is operations expense, on-prem is a capital expense. Unfortunately these folks are heavily incentivized to favor OpEx I'm not a bean counter, all I know is those guys at my last job would rattle off about it like zombies. IIRC its a tax thing
- jscheel 4y agoOpex is fully deductible each year the expense occurs, wheras capex requires amortized depreciation over the life of the purchased item. Makes it harder to calculate taxes.
- goodpoint 4y agoWrong. People has been leasing on-prem hardware for decades before the cloud existed. Depending on taxation rules you might want to buy or lease.
- tomc1985 4y agoAt this employer it seems most of the stuff in their datacenter was purchased.
- morelisp 4y agoIt's not a tax thing, long-term leases or colocation is a liability for acquisitions. (You could also build a revenue-positive company, but that's the loser way out, apparently.)
- oneplane 4y agoIt makes sense when you aren't a datacenter hosting company as your core business. Once you're done with your DRP and BCP it's highly unlikely that your server-under-the-desk (as it usually is) is worth the risk. By the way, moving 'to the cloud' never means a specific thing because people have made words (intentionally?) vague to the point where you have to explicitly specify all the factors you take into account with your work in order to figure out which 'cloud' they had in mind. Running a static workload doesn't require elasticity, but a 'cloud' isn't just elasticity. If you want "a program on a server with some storage that is always there" without having to deal with hardware, software, networking, storage, backup, maintenance, upgrades, service contracts etc. then an arbitrary virtual machine where someone else takes care of the rest makes total sense.
- AaronM 4y agoAnd its easy to compare the cost of the cloud running your hardware to the cost of physical hardware. However, its much more difficult to compare the indirect costs between the two, and that's where I think many people go sideways.
- dtech 4y agoIt's a bit dismissive to just focus on scaling as a cloud advantage. You also have lower maintenance, easier redundancy, lower chance of outages, easier backup and more such non-functionals. A 200GB DB costs about $50/month on AWS (other providers are less), that's not an interesting amount of money for most businesses.
- i_like_waiting 4y agoThats way off. With 64gb ram we have around $50 month on hetzner yes. But from asking around the other BU with very similar size and usage to ours, somehow they are paying around $2600-5000/ month on azure
- TheCoelacanth 4y agoAre they using Oracle or MSSQL where you're paying insane software licensing costs? With Postgres[1], you would need to be using an absolute beast of a server to be paying that much, even if you're paying the 2x premium to go month-to-month, and a low-powered machine could be around $50/month for a 200GB DB. [1] https://azure.microsoft.com/en-us/pricing/details/postgresql/server/ https://azure.microsoft.com/en-us/pricing/details/postgresql...
- i_like_waiting 4y agoYeah, its true that they are using MSSQL. Based on link you provided, I calculated that 40gb memory would be $290 and 80gb $570 per month (in UK). So price seems still quite steep, considering that same specs on-site would be ~$2000 one off. (8 months return on investment). Then for this server, I honestly don't believe its easier to do maintenance in cloud, as you need to do lots of initial configuration there (be it IAM, permissions, firewall etc. ) compared to have it just in local network
- arinlen 4y ago> With 64gb ram we have around $50 month on hetzner yes. A single box on Hetzner is by far not comparable to, say, using a managed service as AWS RDS. It's not even apples to oranges, and more of apples to freshly-squeezed orange juice served by a buttler. Think about it: do you get any form of fault tolerance with your single box?
- taylodl 4y agoNo one has given you a good explanation for why to move to the cloud? Let me fix that for you! :) I work for a Fortune 200 company who has their own data centers and has for decades. We just completed the construction of two new data centers about four years ago at a cost of $110 million dollars. Those new data centers are now at 70% of their capacity for power requirements. You should take a look at the specs on Intel's latest server-class processors. It's insane! We're talking half horse power and higher for each processor - and you want blades full of these things and then a rack of those blades and a row of those racks! Data centers now consume more power than industrial manufacturing! Most maddening of all - most of that power is going to go to heat, and guess what? You need giant chiller units to cool the place down. The power costs alone are insane. You also lose agility. Need a new server? Go to your favorite cloud console and provision it. Better yet, use a serverless architecture and don't even worry about servers! Your own data center? Right now there's a 4-6 month wait time for new servers and storage equipment, and an 8-12 month wait time for networking equipment. Not exactly agile, is it? You already know you need to staff to manage and patch your servers but you're also going to need staff to procure and manage your software licenses - and those licenses aren't cheap! Don't forget the ongoing tail - you pay for the license and then you get to pay 20% per annum forever after for that license. Those licenses also restrict your agility - you're pressured to use the software that's licensed in order to get better value. Try working with a procurement-based architecture! You think you're going to avoid the software licensing hell by using open source? Good luck with that! That means you are responsible for the packaging, distribution, and support of whatever it is you're utilizing. That's more staff you need. They will inevitably miss patching some critical vulnerability that's going to land you in the news! I can go on, and on, and on. But here's one thing I can say about applications that we've moved to the cloud: they cost 30% as much to host in the cloud as they do on-premise, and that's not even accounting for the entirety of all the costs I enumerated above! We have ultimate flexibility and agility, and we can quickly utilize managed open source platforms to solve business problems. On-prem? You lose all of that. Faster, cheaper, better - that's why people are moving to the cloud.
- ungamedplayer 4y agoOpen source clearly does not mean packaging your own software. This casts a shadow of doubt on the rest of your valid points.
- arinlen 4y ago>>I see everybody around me moving to cloud, without really good explanation why. In that case you certainly aren't looking very hard for that explanation. The rationale of moving to "the cloud" is crisp and very well-defined, to the point where even AWS makes it their point to explain in very clear terms in their Architecting With AWS courses. Basically it's all about being able to scale without bothering with procurement, not having to manage everything down to power bills and rentals, turning big capex into small opex, and being able to deploy globally and reliably with a click of a button. That, however, comes at a steep cost. The more serverless you get, the higher the price tag. Also, beyond a certain scale you're better off going back to managing your own infrastructure. > Only reasonable thing I can see as an pattern is that cloud experience on top of data things gets paid 30% more. It made me consider cloud a lot. People with cloud experience are paid more because they can deliver more value. A random guy with, say, experience in EC2 and CloudFormation and CloudWatch, can single-handedly put together a robust fault-tolerant webservice that's deployed globally and automatically scales to meet any demand fluctuation.
- raxxorraxor 4y agoThere are 10, probably more, cloud consultants per developer. Hard to argue against that. There is nothing to say against a cloud version of your office suit but otherwise it is quite underwhelming. Short lived applications with a 90% API because the developer want to keep the door open to imprison the users once the application has captured a critical mass of users. I host on AWS because I don't pay the fairly high invoices. Less domain and TLS maintenance for me. Personally I rent a server and host my personal stuff there including software repositories. edit: Guess that technically would also qualify as cloud, but I think a differentiation is necessary. Cloud mostly means product x or y instead of just another hosted server.
- deleted 4y ago[deleted]
- AveburySutton 4y agoHired onto a project as a specialist. Funders had hired a project manager also. Project manager wanted to use technology X ( which has long since faded ) instead of well entrenched and dev-hireable technology Y because technology Y had not yet entered the trough of disillusionment and anyone who could honestly say they had managed a technology X project could 10x their hourly. Money was ignorant about this level of detail. PM won, as they do. Whole thing a money pyre.