3 ms·
I'd been wondering about the possibility of #2 when I made my other original reply a few hours ago. Thanks for clarifying. The first thing I'm reminded of is h
by exikyut 8y ago
I'd been wondering about the possibility of #2 when I made my other original reply a few hours ago. Thanks for clarifying.
The first thing I'm reminded of is https://www.backblaze.com/blog/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/ https://www.backblaze.com/blog/petabytes-on-a-budget-how-to-.... It's from 2009 but this image jumped out at me: https://www.backblaze.com/blog/wp-content/uploads/2009/08/cost-of-a-petabyte-chart.jpg https://www.backblaze.com/blog/wp-content/uploads/2009/08/co...
Let me guess... you're using kit from the vendor at the bottom of that list? :)
No need to Y/N; I'll assume Y because the sentiment's probably representatively/loosely correct.
The commentator your/this reply is to points out an interesting aspect:
> What likely would not be welcomed, and therefore might make it impossible to do, is the political upheaval, since you'd have to put the kibosh on buying expensive, brand-name gear, all but a few bare-bones service contracts, and the level of enterprise sales perks that the decision makers have come to expect.
This sort of jumped out at me a bit, it's a very reasonable assumption to make, particularly as you say you're an old-school operation, which suggests to me that the tech components are heavily politicized due to their management by non-technical types.
Well... the landscape is indeed changing, as clarified by the 9-year old article above.
So major shifting-around of vendor selection is going to ruffle (and maybe squash) a few feathers, but these "old world" tech enterprises cannot reasonably expect everything to keep going indefinitely because commodity-hardware based solutions have become so accessible that it's harder than ever to sugarcoat proprietary systems to even mildly-technical management, particularly with compute and storage, so the writing is on the wall, and you can use that in your favor.
(Well, except clouds. That seems to be where the old-world's shifted to. Heh.)
And the thing is, for a $B operation (for really any value of $B..$BBB), the cost-benefit absolutely goes in your favor if you build everything out and run it yourself. Plus, if you really are a $BBB operation, you can build your own servers and network kit. As in, you get to start answering questions like what features you want on the motherboards (no Ethernet since you're supplying your own 10G kit? sure thing... oh you don't want any onboard SATA either? okay), how you want the systems to boot (build your own secure boot implementation!), what BIOS/EFI features you want (you want the half-hour RAM test disableable for your staging boxes? fine), how long you'd like your enclosures to be (36" is really long but your racks can handle the length you'll be able to fit a remarkable number disks in 4U), what firmware optimization suggestions you have for your HDDs (a writeable vendor area in SMART config for how many seconds you want each disk to pause for before spinning up so systems come up slowly and don't kill their breakers? done)... and all of these things will directly translate to cost savings.
If there's one thing that differentiates "old-world" datacenters from shots of FAANG, Backblaze, etc, it's that all the big players' servers look really, realy boring, particularly the enclosures. (Well, okay, the blue LEDs make up for it. :P) And this is because there's almost nothing in those machines that isn't needed. Can you get away with removing VGA? (You almost certainly can with virtualization hosts, even if you're just doing KVM or Xen on top of Linux.) Would doing short 24V or 48V runs help? (Complex, but could lower power consumption.) All of these things serve to save money. How much exactly probably depends on what sorts of overheads you're currently dealing with; just switching to barebones systems that do have a few bells and whistles but are still very minimalist may get you an appreciable saving that all the obscure things I've listed here may not be necessary (ie may have negative/negligible/not-worth-it cost-benefit analyses).
I guess my question is at this point, is the decision scoped down to "what vendor are we selecting" or is building your own DC still on the table? I so hope it is, because IMO that's where the cost savings really is.
Hm, maybe you could go "I have one more idea I want to try before we commit to this, it'll require $$$,$$$" and then go and replace your most expensive storage array or something :P that could be pretty convincing...
> I would like to retrain to to end all manual configuration and have the current workforce take over all of this, which is a big deal for me personally.
I think it's cool you're really getting behind this and making your job description part of your identity :P hopefully this is respected and you don't get too heavily stepped on.
There are some really interesting comments in this thread, and a fair bit of reiteration that clouds are generally expensive solutions. I hold this sentiment myself, FWIW, despite admittedly having a LOT less experience with the practical side of things than everyone else here.
This was what I was getting at with the "the offer will be what the buying team wants to hear" - at this point it's almost like the vendor sales are actors in a real-time movie that's exactly what the buying team wants to watch. On top of that one vendor is really going to be as good as another (I've been mentioning GCE and AWS, I forgot Azure.)
Cannot keep writing as I must go, may continue later
- mmt 8y ago> Plus, if you really are a $BBB operation, you can build your own servers and network kit. As in, you get to start answering questions like what features you want on the motherboards I suspect it's this kind of over-reach that ends up scaring decision-makers off from any in-sourcing (for lack of a better term) solution. To me, the key is in sticking with commodity solutions. Even putting aside that the computer tech part of the business isn't their core competency, custom hardware is hard. In the worst case, you create interdepedent vendor lock-in and indefinite delays. Is there an example of any large company, other than Google, who has mastered this? I found it interesting that you would mention Backblaze since, even back then, I questioned (I think even here on HN) their decision to go with a custom chassis and the non-standard, low-performance technology of SATA port multipliers. Well over a year before that blog post, there was enough competition in the SAS-expander backplane-in-a-chassis market (e.g. Chenbro, SuperMicro) that I was using them at a cash-strapped startup instead of used Dell MD1000s. What we don't know, however, is if those backplanes were actually a commodity yet, at their scale. Maybe Backblaze couldn't actually get enough cheap SuperMicro chasses soon enough. I do recall some resellers charging what seemed like comically high markups, but maybe they were banking on a scarcity I was blissfully unaware of, at my low volume. Still, the fact that the option is conspiciously absent from their bar chart tends to make me suspect their summaries and conclusions, even if I always appreciate their raw data (e.g. disk stats). To tie this all back to the original topic, sticking with commodity solutions, be they hardware or cloud, and re-training your staff on them will help them avoid the potential career dead-end of over-specialization.
- exikyut 8y ago> Even putting aside that the computer tech part of the business isn't their core competency, custom hardware is hard. Hmmm. Good point. (Maybe I scared OP away, heh.) > Is there an example of any large company, other than Google, who has mastered this? Well, I certainly don't know. I've heard Cloudflare gets their systems made, for some definition of "from scratch", by Quanta. Presumably there's a bit of a decision-making process in there, and presumably Quanta handles everything related to the "why is the devkit sample broken again #51383" side of things and ships a more or less finished product (maybe even with finished enclosures). My reference point for this knowledge is https://www.youtube.com/watch?v=LA-gNoxSLCE&t=19m31s https://www.youtube.com/watch?v=LA-gNoxSLCE&t=19m31s (2015). Regarding SAS, maybe past a certain ordering quantity (or anything similar that would reveal a massive operation) the market would expect certain behavior like requesting value-adds, ordering with only one supplier, or other things that would generate lock-in. I of course have absolutely no idea [if this sort of thinking is reasonable]. Your points about commodity at scale sound quite reasonable. (You clearly have more experience than I do, which is kind of zero, heh.) As for suspicion about summaries and conclusions, well, (tangential example) the radial graph on https://about.gitlab.com/ https://about.gitlab.com/ (the one from Forrester) doesn't show GitHub :D Yeah, building own hardware is sadly still within the realm of over-specialization at this point. I guess this is because 4+-layer PCB printing is still eye-watering so the amount of cheap iteration that can be done is restricted. There are probably similar cost-benefit disparities as well. I guess, for the sake of discussion, then... what's the best vector into the middle of "commodity solutions" that scale massively (which this operation sounds like it needs a bit of)? I mean in terms of rough vendor selection etc, what to look for, etc. Admittedly this question is based on my own complete naivete and OP probably has at least some info on this sort of thing available or accessible.