4 ms·

Dunno if this is what you are looking for.... but I've specified hundreds of high end servers (all HP) in very large banks onprem and in colo in 4 geos. - ofte

Dunno if this is what you are looking for.... but I've specified hundreds of high end servers (all HP) in very large banks onprem and in colo in 4 geos. - often you are limited to "off the shelf" approved subsets of hardware+rhel configurations that have been tested by a centralised team. These teams rarely have a clue about what they are doing, are highly inflexible and their specs very out of date. - with colo there has been more flexibility as these are bought for the bank by a 3rd party - very large number of NICs needed for bonding + seperate physical network for client connectivity/internal connectivity/exchange connectivity. - driven by business requirements from whatever trading area - their own technology team - then massive multilayer procurement sign off process that can take 6+ months - the available hardware might change or increment during this process requiring a reset if the appetite exists - usually not bought direct from manufacturer but a "pre approved" vendor who passes bank procurement onboarding procedures - servers all have iLo for mgmt/failure etc - usually requires interaction with an official systems team via internal bank ticketing/helpdesk system - colo you open tickets with e.g. Eqinix to receive the hardware and have it put in your cage or whatever pending weekend install by bank or 3rd party team who hook them up to switches as per agreed design with network+systems and business tech team - often when live there are 2/3 different teams monitoring the hosts 24/7 using various tools prob left out a load of stuff.

chomp 4y ago
This is pretty accurate from my time in a large company. My team chose the pre-approved SKUs because we had other teams going nuts buying extremely dense servers and then being shocked with "what do you mean our power commitments don't allow us to go this dense in a rack?". So, we wound up with 2U (2U Twin^2)/3U (microcloud)/4U variants, and if you needed less than that, then you get to snuggle up with other teams' VMs on the VM stack. By far, the most painful part of the process is the ends of the pipeline (procurement for the project, and the datacenter) not knowing anything about the opposite end. The team procuring the hardware would just assume it'd go fine into the datacenter however they dreamed it up, and the datacenter team would be flying blind with little details as to how the system needed to be set up. It was rare that a procuring team would know what they're doing with which nics need to go in which ports, and what VLANs would need to be spun up, and stuff like that.
- easytiger 4y ago
  > It was rare that a procuring team would know what they're doing with which nics need to go in which ports, and what VLANs would need to be spun up, and stuff like that. yea. many of the teams would be devs who wouldn't understand that kind of thing (which was a large part of my job, being in such teams and knowing, roughly, how things worked helped get things done from day 1).
  - balex 4y ago
    We came across this time and time again, even when the procuring team was "an ops team". Our solution was to automate the boilerplate stuff, like the NICs, PSUs, etc. So the order would focus on the compute and IO stuff (though they often struggled with RAID controllers too). We were thinking of switching to a "how much capacity and iops do you need?" method and breaking that down to spec automatically. Or even a "give me a baseline server and a performance multiplier per capacity, read/write, etc". In case they didn't know the actual numbers. Not sure if it would hold?
    - easytiger 4y ago
      Every business case is different (as in the software requirements of the software that supports the business). Alas tricky to determine without analysing how much software they want to run. Not everything is small pieces of scalable n instances s/w.
    - toast0 4y ago
      It really depends on your organization what you can do there. You can go a long way with a menu of servers, pretty limited customization, and an exception process for customization that's really needed. Some places don't want to do any customization, and that can work as long as the menu is reasonable, and you're OK with the consequences: when loads need more CPU, but not more RAM, they'll get more nodes and have underused ram; when they need more RAM, but not more CPU, they'll get more nodes and have underused CPU. Maybe you can build some sort of scheduling system and make use of those pockets of underuse; maybe you can just deal with it, because standardization saves enough effort (and maybe money) to pay for the underuse; maybe you have a pretty large menu and teams can find a good fit. Exceptions could be some specific use cases can customize, or maybe you can add something to the menu if it will save $X or be an order of N nodes. If you can measure existing loads and suggest hardware based on that, that's way more useful IMHO than having teams tell you their iops (cause teams are just going to make wild guesses about their loads anyway)
bradwood 4y ago
You forgot the 15 CAB calls and Change Requests, PTWs and the general wall of ITIL process drag that you need to get through to get the machine racked, stacked, powered and brought online
- easytiger 4y ago
  Yea days of paperwork. I was doing the Hollywood version
  - bradwood 4y ago
    And then the SPF module gets patched into the wrong port. Server guys blame network guys. Network guys blame DC guys. Switch goes apeshit because of a spanning tree problem. IPAM not up to date. Tempers and more CRs. Did that shit for waaay to long. Now I do terraform.
    - balex 4y ago
      We actually know how to solve that. What we haven't figured out yet is who's the buyer.
balex 4y ago
Yes, this is what we're looking for. We're wondering if automating this headache to some SaaS that offers BoM templating, compatibility validation, interchangeable parts and such, could be of interest to such an org. Essentially they'd be left with approvals and the rest would be stepping through an automated workflow. WDYT?

chomp 4y ago

This is pretty accurate from my time in a large company. My team chose the pre-approved SKUs because we had other teams going nuts buying extremely dense servers and then being shocked with "what do you mean our power commitments don't allow us to go this dense in a rack?". So, we wound up with 2U (2U Twin^2)/3U (microcloud)/4U variants, and if you needed less than that, then you get to snuggle up with other teams' VMs on the VM stack. By far, the most painful part of the process is the ends of the pipeline (procurement for the project, and the datacenter) not knowing anything about the opposite end. The team procuring the hardware would just assume it'd go fine into the datacenter however they dreamed it up, and the datacenter team would be flying blind with little details as to how the system needed to be set up. It was rare that a procuring team would know what they're doing with which nics need to go in which ports, and what VLANs would need to be spun up, and stuff like that.

easytiger 4y ago
> It was rare that a procuring team would know what they're doing with which nics need to go in which ports, and what VLANs would need to be spun up, and stuff like that. yea. many of the teams would be devs who wouldn't understand that kind of thing (which was a large part of my job, being in such teams and knowing, roughly, how things worked helped get things done from day 1).
- balex 4y ago
  We came across this time and time again, even when the procuring team was "an ops team". Our solution was to automate the boilerplate stuff, like the NICs, PSUs, etc. So the order would focus on the compute and IO stuff (though they often struggled with RAID controllers too). We were thinking of switching to a "how much capacity and iops do you need?" method and breaking that down to spec automatically. Or even a "give me a baseline server and a performance multiplier per capacity, read/write, etc". In case they didn't know the actual numbers. Not sure if it would hold?
  - easytiger 4y ago
    Every business case is different (as in the software requirements of the software that supports the business). Alas tricky to determine without analysing how much software they want to run. Not everything is small pieces of scalable n instances s/w.
  - toast0 4y ago
    It really depends on your organization what you can do there. You can go a long way with a menu of servers, pretty limited customization, and an exception process for customization that's really needed. Some places don't want to do any customization, and that can work as long as the menu is reasonable, and you're OK with the consequences: when loads need more CPU, but not more RAM, they'll get more nodes and have underused ram; when they need more RAM, but not more CPU, they'll get more nodes and have underused CPU. Maybe you can build some sort of scheduling system and make use of those pockets of underuse; maybe you can just deal with it, because standardization saves enough effort (and maybe money) to pay for the underuse; maybe you have a pretty large menu and teams can find a good fit. Exceptions could be some specific use cases can customize, or maybe you can add something to the menu if it will save $X or be an order of N nodes. If you can measure existing loads and suggest hardware based on that, that's way more useful IMHO than having teams tell you their iops (cause teams are just going to make wild guesses about their loads anyway)

bradwood 4y ago

You forgot the 15 CAB calls and Change Requests, PTWs and the general wall of ITIL process drag that you need to get through to get the machine racked, stacked, powered and brought online

easytiger 4y ago
Yea days of paperwork. I was doing the Hollywood version
- bradwood 4y ago
  And then the SPF module gets patched into the wrong port. Server guys blame network guys. Network guys blame DC guys. Switch goes apeshit because of a spanning tree problem. IPAM not up to date. Tempers and more CRs. Did that shit for waaay to long. Now I do terraform.
  - balex 4y ago
    We actually know how to solve that. What we haven't figured out yet is who's the buyer.

balex 4y ago

Yes, this is what we're looking for. We're wondering if automating this headache to some SaaS that offers BoM templating, compatibility validation, interchangeable parts and such, could be of interest to such an org. Essentially they'd be left with approvals and the rest would be stepping through an automated workflow. WDYT?