7 ms·

Currently I work in a company that's very cost-conscious about hosting. They are generous in salaries but are mercilessly practical about IT costs and the abili

Currently I work in a company that's very cost-conscious about hosting. They are generous in salaries but are mercilessly practical about IT costs and the abilities of computers -- especially the latter part is something that most of the programmers today have almost completely forgot. I have 3 spare laptops lying around; weakest of them is with a Celeron J4155 and I have put a web app with no caching on it (it does have a persistent DB) and hammered it with my workstation until it finally started giving up at ~2500 req/s (Elixir/Phoenix stack). Again, that's a Celeron J4155 with a SATA III SSD in an M.2 factor (so disk speed caps at 550MB/s at best; usually 400-460) and 12GB RAM. Most programmers wouldn't touch such a machine. I imagine I can buy 2 more of these laptops and make a completely replicated 3-cluster of the entire stack of our company and the slowest requests (on admin UI where we have a lot of SQL JOINs) would likely never go above 200ms. That totals at about 600 EUR (yep, I bought the laptop second-hand for 200 EUR). Then 500 more EUR for a good UPS to plug the laptops and my routers to. Boom 1100 EUR and several weekends later I can likely charge my own employer for hosting at 100 EUR a month for their entire infrastructure and I would likely still be ripping them off even with that. The only real cost is human time and energy invested in making it work. But for most companies that's not a 24/7 fight so that cost is fairly low. You can do it twice a year and you're likely never going to have problems. So yep, I am completely with you here (if my rant didn't make it obvious). Infrastructure costs are already being heavily optimized by companies out there.

lotsofpulp 5y ago
> The only real cost is human time and energy invested in making it work. But for most companies that's not a 24/7 fight so that cost is fairly low. You can do it twice a year and you're likely never going to have problems. If this were true, what made the cloud providers popular in the first place?
- pdimitar 5y ago
  Outsourcing culture. Nobody wants to nurture talent -- that would also mean to invest in relations with your employees and not alienating them. Shareholders prefer to sacrifice a little more of their profit so they deal with less potential problems. And the pull of the idea that every human in the org must be an inter-changeable cog is too strong (even if that idea continues to be absurd, and always was). Hosting apps in the cloud was a fair exchange 10 years ago because operational tooling in general was more immature. Nowadays it's much easier to self-host many pieces of software though.
  - lotsofpulp 5y ago
    Interesting. I am not intimately familiar with the subject matter, so I will be curious then to see how the next 10 years play out.
  - AzzieElbab 5y ago
    What about scaling down? Once you bought hardware for self hosting you are stuck with it.
    - pdimitar 5y ago
      What about it? We're talking small businesses here, this will not be a warehouse full of racks worth of machines to try and sell. We're likely talking 3 to 10 PC-sized machines in a closet or storage room. You can sell those on 2nd hand market pretty quickly and even if you sell them at 30-40% loss you are extremely likely to have already paid off the other 60-70% during the machines' service time.
      - AzzieElbab 5y ago
        old laptop is fine then :)
    - naasking 5y ago
      The costs the OP described are pretty minimal.
      - JBlue42 5y ago
        You never know. While working in an ITAM role, I've had a manager point out a line item asking why we need 30 DisplayPort cables. This was in an org with multiple sites, 2000+ employees, using dual monitor setups both in house and remote. Whether physically, or in the cloud, companies whose primary business isn't tech-related see the IT side as an annoying cost center rather than a cost of doing business in the 21st century.
  - kazen44 5y ago
- rossmohax 5y ago
  Beancounters love OPEX and avoid CAPEX
  - lotsofpulp 5y ago
    Surely the beancounters can also see the effects of capex on the valuations of the cloud providers.
    - ethbr0 5y ago
      If they're publicly traded, no one's going to give Bob's Consolidated Widgets a 65x PE or 2x+ PEG ratio, even if they have the best IT strategy in their market.
  - kazen44 5y ago
    beancounters are not in charge of the direction of the company. In a lot of cases, it makes a ton of sense to go for capex vs opex. My old employer had a strategy which basically boiled down to owning everything inside this company expect for the coffee machines and cleaning crew. His reason being? This made it possible to run on very thight margins when the economy takes a downturn without having to scramble for money because of leased/loaned equipment etc. In the 2008 crisis, this is how he stayed afloat with his company, and even made a pretty profit during a time of crisis too.
  - abecode 5y ago
    Curious, can you explain why? In my experience (just one company where I had some interaction with accounting), there was a preference toward CAPEX. The costs of building a product, ie software development, was prefered over OPEX, in this case, analytics. Since I was doing both development and analytics, they preferred me to account part of my time as CAPEX. Not sure if this is normal but for context it was a non/pre-public company.
    - xunn0026 5y ago
      OPEX is just an expense in the current fiscal year. CAPEX depreciates over N years. This alone means that a one-time payment is much simpler on the books. I don't think your salary counts as CAPEX, it's a normal monthly expense for the company.
- nicoburns 5y ago
  That human time happens to be quite expensive.
  - pdimitar 5y ago
    Not in small companies.
    - rcpt 5y ago
      What if the cheap guy quits?
      - pdimitar 5y ago
        I'm aware of the risks. That's why I said a culture of long-term relationships must be fostered if you want to pull such operations off. Where I work right now people don't just quit two days later. They like the company and if they feel they want to go someplace else they're not being difficult about it. They cooperate in passing down their knowledge to colleagues before they go.
    - ghaff 5y ago
      It's most of the cost at small companies. A lot of people underestimate the total cost of having employees perform tasks at unpredictable scale with unpredictable problems versus just writing a check each month. It's true that cloud providers aren't reliably cheaper as some assumed at one point. But there's something to be said for cutting out the fully burdened costs of some number of employees (especially given everyone is saying tech talent is expensive and scarce at the moment) and just letting AWS deal with it even if you could theoretically do it cheaper yourself.
      - 13of40 5y ago
        Not only that, but if you have a business that actually makes, sells, and ships something, your AWS bill is probably a tiny fraction of your expenses. Switching to physical servers would be a little like getting solar installed on your house - great in theory, but if the most expense it will displace is $150/month, is it the right problem to focus on?
- mindslight 5y ago
  Infinite scale up (direct fan-out to mass market), high margins (able to overpay for commodities), and outsourcing production work to stable businesses are all facets of the Hollywood business model that Surveillance Valley adopted. Once the concept was socially proven, then nobody gets fired for buying IBM.
- Nextgrid 5y ago
  Keep in mind that most big clients of cloud providers are startups running on VC money. It's not their money and they don't care about wasting it, and VCs don't particularly care either. This has now created an entire ecosystem of developers and "devops engineers" (what we used to call sysadmins) who know little beyond the cloud and have to keep using it for career-related reasons. This in turn pushes companies to use the cloud as finding talent for old-school on-premises infrastructure is difficult.
_alex_ 5y ago
that works as long as there are enough of you for every team/app/service/whatever to build and operate all that around the clock and without bus risk. A lot of these threads end up talking about "scaling" from a pure hardware perspective. Once you start talking about medium-to-large companies, the real scaling headache is people, process, and organizations.
- pdimitar 5y ago
  Not sure if I made myself clear but I definitely meant my above as aimed at small businesses (which my current employer is). Obviously from one scale and on the cloud is very much worth it. IMO the discussion has to be shifted to "before which point you can easily get away without the cloud?".
  - makapuf 5y ago
    At some point, an internal cloud starts being interesting and gives a clear org boundary, using cloud-based software and interfaces.
    - pdimitar 5y ago
      That's true as well. It all depends on how much is the employer willing to invest in their employees in the end; if they foster longer-lasting relationships then many interesting internal projects become feasible or even desirable.
      - ghaff 5y ago
        Those invested-in employees will probably still leave, especially if they're very good and you can't match big tech salaries. And, in any case, you need redundancies for when they're on vacation, out sick, etc. Not to say you shouldn't develop in house competencies but you should do so deliberately because there are probably a lot more costs and risks than are obvious.
        5y ago
capitalsigma 5y ago
What happens when your local internet connection goes out and you can no longer serve traffic for your business? What if there is a power outage? Who will spend the time replacing disks as they fail? Where is the data backed up to, so that a failed disk doesn't lose business data?
xanaxagoras 5y ago
I like where you're going with this but you could take it a lot further with little expense. Amazon and MicroCenter have a huge stock of refurbished HP mini PCs with i5s, 16gb of ram and an M2 slot for about $350. Throw in a small m2 ssd in each (~$50) and keep your UPS, that's about $1700 you have a decent cluster for an efficient stack like phoenix.
pawelmurias 5y ago
How much money would your company save by hosting on some old laptops instead of in the cloud?
- pdimitar 5y ago
  5 digits of USD per month. Whereas normal on-prem hosting can be started with almost no dollar cost, tomorrow. Right now they/we can't do it though, the talent is there but that talent is super busy reworking software that's no longer well-adapted to their new requirements. We'll see if the topic becomes more relevant in the future.
  - thow-58d4e8b 5y ago
    Not to forget - there are more choices than big-cloud and host-everything-yourself. One can rent VMs from Hetzner at 1/3 the price of AWS (and 1/10 the data transfer fees). No managed services, like Dynamo or S3, but will full convenience of on-demand scale-up Or, for mid-sized corporations - buying a rack of computers, wiring them together, and installing OpenStack to make an internal cloud - that isn't some black magic wizardry either
    - pdimitar 5y ago
      Oh yeah, completely agreed. As mentioned in another comment we might just go Hetzner and buy a singular server in another provider for backup and failover. We'd likely be just fine with 6-7 of the smaller Hetzner servers, too.
  - pawelmurias 5y ago
    It seems crazy that you are paying that much for hosting something that can be self-hosted on an old laptop.
GabeIsko 5y ago
You are vastly overestimating the IT capabilities of most companies. The human time and energy cost that you handwaved away is a real killer because that is expert knowledge capability that is highly valued by the market. Maintaining an IT staff and infrastructure to support a large organization that has to perform any kind of reasonable SLA for services eventually puts you in direct competition with Major Cloud Providers at scale. A scenario where you can outcompete them with their large customer bases becomes very difficult to imagine under current technological and market conditions. It's not a matter of outsourcing culture. Cloud providers really do provide computing in a cost efficient matter since they practically provide IT infrastructure at wholesale, and it produces an anti-trust risk because, as you pointed out, the actual hardware that can make it go is not nearly as expensive.
- pdimitar 5y ago
  I am not handwaving anything away. We're a company that has less but very strong programmers (we don't even have DevOps / sysadmins, our CTO is doing it and I'll likely be taking part of his responsibilities at one point). And we don't have the requirements of most big Silicon Valley companies that you reference. It boggles my mind why so many commenters IMMEDIATELY assumed we're a hyper-growth startup or whatever. We're an expanding data provider and our growth is super predictable, and our set of requirements changes like once a year. So really, not sure why you and others keep repeating things I already agree with. The cloud wins from one point / scale and on, absolutely; otherwise it wouldn't ever take off. I am saying that people give up way too quickly and run to the cloud long before they need it -- that's all.
  - dash2 5y ago
    But you yourself admit that you have an unusual setup with a few very strong programmers. Few are like this even in Silicon Valley. As for other industries, they're not even close. So your company's setup won't transfer well to the rest of the world.
    - pdimitar 5y ago
      I would not be so sure of your last. The SV mold is the outlier but most USA programmers can't see it because they don't work for non-US companies. Everywhere I was all around the EU in my 20 years of career so far, the SV way of doing things has not been the norm. So it pays off to be wary of filter bubbles. The SV way of doing things assumes a ton of VC money to burn, something that, to this day, is still not a wide trend in the EU (even if it happens here and there). Plus that company's setup is not unusual in any way per se; the only "unusual" thing is "we don't run to the cloud at the first IT problem". A ton of other companies are like this, I'd bet even in the US.

lotsofpulp 5y ago

> The only real cost is human time and energy invested in making it work. But for most companies that's not a 24/7 fight so that cost is fairly low. You can do it twice a year and you're likely never going to have problems. If this were true, what made the cloud providers popular in the first place?

pdimitar 5y ago
Outsourcing culture. Nobody wants to nurture talent -- that would also mean to invest in relations with your employees and not alienating them. Shareholders prefer to sacrifice a little more of their profit so they deal with less potential problems. And the pull of the idea that every human in the org must be an inter-changeable cog is too strong (even if that idea continues to be absurd, and always was). Hosting apps in the cloud was a fair exchange 10 years ago because operational tooling in general was more immature. Nowadays it's much easier to self-host many pieces of software though.
- lotsofpulp 5y ago
  Interesting. I am not intimately familiar with the subject matter, so I will be curious then to see how the next 10 years play out.
- AzzieElbab 5y ago
  What about scaling down? Once you bought hardware for self hosting you are stuck with it.
  - pdimitar 5y ago
    What about it? We're talking small businesses here, this will not be a warehouse full of racks worth of machines to try and sell. We're likely talking 3 to 10 PC-sized machines in a closet or storage room. You can sell those on 2nd hand market pretty quickly and even if you sell them at 30-40% loss you are extremely likely to have already paid off the other 60-70% during the machines' service time.
    - AzzieElbab 5y ago
      old laptop is fine then :)
  - naasking 5y ago
    The costs the OP described are pretty minimal.
    - JBlue42 5y ago
      You never know. While working in an ITAM role, I've had a manager point out a line item asking why we need 30 DisplayPort cables. This was in an org with multiple sites, 2000+ employees, using dual monitor setups both in house and remote. Whether physically, or in the cloud, companies whose primary business isn't tech-related see the IT side as an annoying cost center rather than a cost of doing business in the 21st century.
- kazen44 5y ago
rossmohax 5y ago
Beancounters love OPEX and avoid CAPEX
- lotsofpulp 5y ago
  Surely the beancounters can also see the effects of capex on the valuations of the cloud providers.
  - ethbr0 5y ago
    If they're publicly traded, no one's going to give Bob's Consolidated Widgets a 65x PE or 2x+ PEG ratio, even if they have the best IT strategy in their market.
- kazen44 5y ago
  beancounters are not in charge of the direction of the company. In a lot of cases, it makes a ton of sense to go for capex vs opex. My old employer had a strategy which basically boiled down to owning everything inside this company expect for the coffee machines and cleaning crew. His reason being? This made it possible to run on very thight margins when the economy takes a downturn without having to scramble for money because of leased/loaned equipment etc. In the 2008 crisis, this is how he stayed afloat with his company, and even made a pretty profit during a time of crisis too.
- abecode 5y ago
  Curious, can you explain why? In my experience (just one company where I had some interaction with accounting), there was a preference toward CAPEX. The costs of building a product, ie software development, was prefered over OPEX, in this case, analytics. Since I was doing both development and analytics, they preferred me to account part of my time as CAPEX. Not sure if this is normal but for context it was a non/pre-public company.
  - xunn0026 5y ago
    OPEX is just an expense in the current fiscal year. CAPEX depreciates over N years. This alone means that a one-time payment is much simpler on the books. I don't think your salary counts as CAPEX, it's a normal monthly expense for the company.
nicoburns 5y ago
That human time happens to be quite expensive.
- pdimitar 5y ago
  Not in small companies.
  - rcpt 5y ago
    What if the cheap guy quits?
    - pdimitar 5y ago
      I'm aware of the risks. That's why I said a culture of long-term relationships must be fostered if you want to pull such operations off. Where I work right now people don't just quit two days later. They like the company and if they feel they want to go someplace else they're not being difficult about it. They cooperate in passing down their knowledge to colleagues before they go.
  - ghaff 5y ago
    It's most of the cost at small companies. A lot of people underestimate the total cost of having employees perform tasks at unpredictable scale with unpredictable problems versus just writing a check each month. It's true that cloud providers aren't reliably cheaper as some assumed at one point. But there's something to be said for cutting out the fully burdened costs of some number of employees (especially given everyone is saying tech talent is expensive and scarce at the moment) and just letting AWS deal with it even if you could theoretically do it cheaper yourself.
    - 13of40 5y ago
      Not only that, but if you have a business that actually makes, sells, and ships something, your AWS bill is probably a tiny fraction of your expenses. Switching to physical servers would be a little like getting solar installed on your house - great in theory, but if the most expense it will displace is $150/month, is it the right problem to focus on?
mindslight 5y ago
Infinite scale up (direct fan-out to mass market), high margins (able to overpay for commodities), and outsourcing production work to stable businesses are all facets of the Hollywood business model that Surveillance Valley adopted. Once the concept was socially proven, then nobody gets fired for buying IBM.
Nextgrid 5y ago
Keep in mind that most big clients of cloud providers are startups running on VC money. It's not their money and they don't care about wasting it, and VCs don't particularly care either. This has now created an entire ecosystem of developers and "devops engineers" (what we used to call sysadmins) who know little beyond the cloud and have to keep using it for career-related reasons. This in turn pushes companies to use the cloud as finding talent for old-school on-premises infrastructure is difficult.

_alex_ 5y ago

that works as long as there are enough of you for every team/app/service/whatever to build and operate all that around the clock and without bus risk. A lot of these threads end up talking about "scaling" from a pure hardware perspective. Once you start talking about medium-to-large companies, the real scaling headache is people, process, and organizations.

pdimitar 5y ago
Not sure if I made myself clear but I definitely meant my above as aimed at small businesses (which my current employer is). Obviously from one scale and on the cloud is very much worth it. IMO the discussion has to be shifted to "before which point you can easily get away without the cloud?".
- makapuf 5y ago
  At some point, an internal cloud starts being interesting and gives a clear org boundary, using cloud-based software and interfaces.
  - pdimitar 5y ago
    That's true as well. It all depends on how much is the employer willing to invest in their employees in the end; if they foster longer-lasting relationships then many interesting internal projects become feasible or even desirable.
    - ghaff 5y ago
      Those invested-in employees will probably still leave, especially if they're very good and you can't match big tech salaries. And, in any case, you need redundancies for when they're on vacation, out sick, etc. Not to say you shouldn't develop in house competencies but you should do so deliberately because there are probably a lot more costs and risks than are obvious.
      - 5y ago

capitalsigma 5y ago

What happens when your local internet connection goes out and you can no longer serve traffic for your business? What if there is a power outage? Who will spend the time replacing disks as they fail? Where is the data backed up to, so that a failed disk doesn't lose business data?

xanaxagoras 5y ago

I like where you're going with this but you could take it a lot further with little expense. Amazon and MicroCenter have a huge stock of refurbished HP mini PCs with i5s, 16gb of ram and an M2 slot for about $350. Throw in a small m2 ssd in each (~$50) and keep your UPS, that's about $1700 you have a decent cluster for an efficient stack like phoenix.

pawelmurias 5y ago

How much money would your company save by hosting on some old laptops instead of in the cloud?

pdimitar 5y ago
5 digits of USD per month. Whereas normal on-prem hosting can be started with almost no dollar cost, tomorrow. Right now they/we can't do it though, the talent is there but that talent is super busy reworking software that's no longer well-adapted to their new requirements. We'll see if the topic becomes more relevant in the future.
- thow-58d4e8b 5y ago
  Not to forget - there are more choices than big-cloud and host-everything-yourself. One can rent VMs from Hetzner at 1/3 the price of AWS (and 1/10 the data transfer fees). No managed services, like Dynamo or S3, but will full convenience of on-demand scale-up Or, for mid-sized corporations - buying a rack of computers, wiring them together, and installing OpenStack to make an internal cloud - that isn't some black magic wizardry either
  - pdimitar 5y ago
    Oh yeah, completely agreed. As mentioned in another comment we might just go Hetzner and buy a singular server in another provider for backup and failover. We'd likely be just fine with 6-7 of the smaller Hetzner servers, too.
- pawelmurias 5y ago
  It seems crazy that you are paying that much for hosting something that can be self-hosted on an old laptop.

GabeIsko 5y ago

You are vastly overestimating the IT capabilities of most companies. The human time and energy cost that you handwaved away is a real killer because that is expert knowledge capability that is highly valued by the market. Maintaining an IT staff and infrastructure to support a large organization that has to perform any kind of reasonable SLA for services eventually puts you in direct competition with Major Cloud Providers at scale. A scenario where you can outcompete them with their large customer bases becomes very difficult to imagine under current technological and market conditions. It's not a matter of outsourcing culture. Cloud providers really do provide computing in a cost efficient matter since they practically provide IT infrastructure at wholesale, and it produces an anti-trust risk because, as you pointed out, the actual hardware that can make it go is not nearly as expensive.

pdimitar 5y ago
I am not handwaving anything away. We're a company that has less but very strong programmers (we don't even have DevOps / sysadmins, our CTO is doing it and I'll likely be taking part of his responsibilities at one point). And we don't have the requirements of most big Silicon Valley companies that you reference. It boggles my mind why so many commenters IMMEDIATELY assumed we're a hyper-growth startup or whatever. We're an expanding data provider and our growth is super predictable, and our set of requirements changes like once a year. So really, not sure why you and others keep repeating things I already agree with. The cloud wins from one point / scale and on, absolutely; otherwise it wouldn't ever take off. I am saying that people give up way too quickly and run to the cloud long before they need it -- that's all.
- dash2 5y ago
  But you yourself admit that you have an unusual setup with a few very strong programmers. Few are like this even in Silicon Valley. As for other industries, they're not even close. So your company's setup won't transfer well to the rest of the world.
  - pdimitar 5y ago
    I would not be so sure of your last. The SV mold is the outlier but most USA programmers can't see it because they don't work for non-US companies. Everywhere I was all around the EU in my 20 years of career so far, the SV way of doing things has not been the norm. So it pays off to be wary of filter bubbles. The SV way of doing things assumes a ton of VC money to burn, something that, to this day, is still not a wide trend in the EU (even if it happens here and there). Plus that company's setup is not unusual in any way per se; the only "unusual" thing is "we don't run to the cloud at the first IT problem". A ton of other companies are like this, I'd bet even in the US.