4 ms·
I find it fun how the cost of the cloud is forcing people to consider what absolutely must run in the cloud (presumably for stability and compliance reasons) an
by bob1029 7y ago
I find it fun how the cost of the cloud is forcing people to consider what absolutely must run in the cloud (presumably for stability and compliance reasons) and what can be brought back on-prem.
We don't train ML models, but we are in a similar boat regarding cloud compute costs. Building our solutions for our clients is a compute-heavy task which is getting expensive in the cloud. We are considering options such as building commodity threadripper rigs, throwing them in various developers' (home) offices, installing a VPN client on each and then attaching as build agents to our AWS-hosted jenkins instance. In this configuration we could drop down to a t3a.micro for Jenkins and still see much faster builds. The reduction in iteration time over a month would easily pay for the new hardware. An obvious next step up from this is to do proper colocation, but I am of a mindset that if I have to start racking servers I am bringing 100% of our infrastructure out of the cloud.
- m0zg 7y agoThis is not a new phenomenon. As early as in 2009 I worked for a company (ads, but not Google) which outgrew the typical "cloud" cost structure at the time, and moved everything to a more traditional datacenter, and saved substantial money even considering 3 more SREs they had to hire to absorb the increased support needs. AWS charges what the market will bear, and as such it was never designed to make sense for everyone. One needs to re-evaluate on the back of the napkin from time to time.
- buckminster 7y agoI once had a borrowed Sun blade server in my home office. The fan in it sounded like an industrial vacuum cleaner. It got moved to a different room and was powered on as little as possible. Your plan makes sense but be mindful of the acoustics or your devs may grow to hate you.
- m0zg 7y agoBTW, the only reason why consumer vacuum cleaners are so loud is because consumers associate loudness with suction power. "Backpack" style commercial vacuum cleaners have more suction, and are barely audible in comparison.
- buckminster 7y agoI stand corrected. It was much louder than a consumer vacuum but my analogy skills are weak.
- ggm 7y agoYour analogy skills were strong, because analogy is rooted in myth, not fact. Achilles did not actually have an Achilles tendon because Achilles did not exist There does not have to be an increadibly loud functional industrial vacuum cleaner, for figuratively everyone to get your analogy, because the Herculean reality of vacuum cleaners is that you cannot clean an augean stable of lego on the floor, without a lot of noise. If you get my analogy.
- ColanR 7y agoTo continue the pedantry, I don't think we know for sure that Achilles didn't exist. Troy certainly did, and we have the Iliad to thank for knowing to look for it.
- ggm 7y agoNo, we have Schliemann to thank, for digging through it in his lust for anachronistically Indy-Jonesing the archeology. Troy only exists because Schliemann stole the gold. Otherwise its just some hill in Turkey.
- ColanR 7y agoI know the story...Schliemann went looking for Troy because the Illiad told him it existed.
- ggm 7y agoDidn't work on Ararat. False in unum, false in omnibus. Anyway, he went to the wrong hill first. But overall yes, myth informs reality sometimes. Shame he didn't find the bones of the giant wooden platypus the Greek ice cream salesmen hid in.
- bob1029 7y agoExcellent point. If we are building these rigs by hand (which is a likely option considering the initial usage context), the cooling solution would probably be a Noctua NH-U14S or similar. I already have one of these in my office attached to a 2950X and it is dead silent. You can definitely hear it when every core is pegged, but it's hardly noticeable over any other arbitrary workstation. The sound is nowhere near as intrusive as something like a blower on a GPU (or god forbid a sun microsystems blade).
- maremp 7y agoWhat you’re saying sounds like a PC custom build enthusiast. I respect that, I like it too. Please be mindful of the fact that consumer products are not designed for the workstation/server type of load. It’s related to why the hardware is cheaper compared to the server hardware. Also, the consumer ISP connection is most likely not as reliable as that of a data center. I’m working remotely from home and I experienced this many times, bad performance in peak times or a half of a day of downtime can happen, without any warning. And account for maintenance, everyone on the team must be able to figure out a problem or deal with getting someone else in to fix it. I know I sound like a buzzkill. I am writing this with good intentions. Even if this works right now, it’s not a reliable long-term solution. Maybe instead of dumping a couple of grand on consumer PCs to handle server’s work, look into building a proper server. Or you could find a datacenter provider to rent their hardware, something that is not as shiny and full of features like AWS.
- linksnapzz 7y agoOnce upon a time, a company I worked for convinced Sun Microsystems that it would be cool to provide us with hardware at a discount. The discount hardware was six UltraSPARC E4000 servers, each with 8 cpus and 8GB of memory, and a creator3d card in the io mezzanine upa slot. This company was using them as desktop workstations, in an open office. One was used as a build host. Often, the shopvac wail of E4000 fans would be cut short by some poor dev going berserk and unplugging the thing when nobody was looking...
- blt 7y agoIf I worked from home and my employer asked me to install a server in my home, I would tell them to go fuck themselves. It's noisy, it takes up space, and presumably I'm on call to fix it if it breaks. You should pay them an extra 24x(PSU wattage)x(peak $/Wh in area) per day for the electricity too. I'm alarmed that someone in your company felt this idea was appropriate enough to propose.
- breischl 7y agoTangentially, I once joined a startup where a week after I started the CEO demanded that I run their crawler software on my own personal hardware, on my personal internet connection, 24x7. I (politely) told him to go fuck himself. I later learned that the CTO had to spend a couple hours talking him out of firing me. I probably should've just quit on the spot anyway.
- jiofih 7y agoYou can stop at “go f* themselves”, plenty of jobs to choose from.
- bob1029 7y agoWe would certainly compensate employees. I didn't feel it appropriate to disclose every last detail regarding the arrangement in a thread which is only tangentially-related to the OP. This was my idea. I am a developer in my company. We are a flat structure. We have a lot of respect for each other. I am on a standup with the CEO every day. We all believe in our product and would happily participate in whatever activity brings it to market more quickly. We do not hire or retain the kind of talent that would flatly refuse to participate in experimental projects like this. At least not without some sort of initial conversation about why it's not a good fit for a particular individual. I certainly see how someone might share your perspective. I used to work for a souless megacorp and I could have easily found myself telling my former employer to "go fuck themselves" if a proposal similar to this was imposed upon me.
- xemdetia 7y agoI think it was funny to presume that the employer wouldn't pay for it. At one of my old roles I had a second laptop I used solely for VMs just because I ran out of space on primary workstation. A simple fix orders of magnitude cheaper for a developer compared to more ESXi or cloud hardware.
- echelon 7y ago> I find it fun how the cost of the cloud is forcing people to consider what absolutely must run in the cloud Honestly why ever go to the cloud? It seems like a Larry Ellison boondoggle with the absurdly high costs and lock-in. (Ever look at moving your data?) Running your own metal is cheaper if you actually fund it.
- CoolGuySteve 7y agoIn my experience, teams will rack up thousands in monthly expenses just being parked in a shell on very large On-Demand or Reserved EC2 instances. Basically using them as development boxes without realizing how much they cost. I've saved a ton of money just giving them dedicated workstations to develop on and then having everyone use a shared EC2 instance to push jobs to a fleet of spot instances for large scale training.