4 ms·

It's hard to believe for me how you would not start buying own hardware at this scale. In particular when the hyperscalers (at first glance) don't have anything

by ktpsns 2y ago

It's hard to believe for me how you would not start buying own hardware at this scale. In particular when the hyperscalers (at first glance) don't have anything to provide to match the needs.

rbanffy 2y ago
The biggest x86 machine I found tops out at 960 cores, but I'm not sure what exactly they need, if having more cores would solve their problems or would only make some other pipe burst. To figure that out, we'd need to look deep into what's happening in the machine, down to counting cache misses, memory bandwidth usage (per channel), QPI link usage (because NUMA), and, maybe, even go down to the usage stats of the CPU execution units. When they mention a lot of what was stored procedures has been moved to external web services, I get concerned they replaced memory and CPU occupancy with it waiting for network IO.
- cjbgkagh 2y ago
  I would hazard a guess that they're not really CPU bound. Assuming the poster Aussiepete80 is Australian I should point out that the much higher salaries in the US and the favorable E3 visa has largely brain drained Australia of their best and brightest. This guys army (dozens) of DBAs is likely the residual.
  - rbanffy 2y ago
    Not really a nice thing to say - not everyone is inclined to move to the US (for me, and I'm not that cool, it'd need to be more than 6 digits). I suspect there is plenty competent people wishing to stay in Australia (in fact, a lot of healthcare folk here in Ireland have decided to move there - weather is so much better, despite the giant spider and murderous fauna problem). Predators here in Ireland are terrifying. The other day I was almost adopted into a family of feral cats.
    - cjbgkagh 2y ago
      One competent person probably, 'some' possibly, 'plenty' defiantly not. The general advice for tech people in Australia is to move to the US ASAP - the more competent the tech person the stronger the advice. The insidious effect of brain drain is the effect is self reinforcing, i.e. the more it happens the stronger the pull. In tech it's important to work with other competent people. Historically Australians are unusually good programmers, so I am not saying Australians are unskilled, but I am saying their best and brightest leave Australia ASAP. Plus I would submit the reddit thread 'Biggest CPU for the Bad System' as evidence to my point. How could this person possibly think that 100s of CPU cores are the bottleneck and the solution is more CPU cores - how could the dozens of DBAs not figure it out - the amount of work that can be done in 100s of CPU cores is utterly insane and the fact that they didn't rule out obvious IOWait bottlenecks in their initial posts suggest that none of them know what they're doing. I guess they figured that Azure would take care of all of that. The initial step would be to bring that computer in house and design that computer around the problem - Azure is not going to be open and honest about their bottlenecks. Last I checked Azure was big on their Network Attached Storage and Managed Disks which add quite a lot of latency and throughput bottlenecks compared to a PCIe 5.0 Enterprise SSD.
      - rbanffy 2y ago
        I’d say the telltale sign is picking SQL Server in the first place. If you think there is a risk your database will grow to be enormous, it’s likely it’ll be the worst possible choice, with the possible exception of Oracle and DB2 on AIX (but, on the last two, at least you have fewer limits to scaling up). Could also be a B-player hires C-players situation, with borderline incompetence trickling down from the C-level.
        cjbgkagh 2y ago
        Certainly, if you're planning on outgrowing a SQL Server you should probably DIY tailor made indexes and query languages/engines. LLVM makes DIY languages much easier to make than they used to be and very with fast interpreters and AOT it's very performant. The reason for my brain drain statement is that people don't understand how pernicious the brain drain is in Australia. The Australian government believes that there isn't a net brain drain because more people with degrees immigrate than those emigrating. But people are not substitutable cogs and the very smart are very rare and it is the loss of those few very smart Australians to the US which is so devastating.
- bee_rider 2y ago
  If they need really 1600 cores, perhaps they should be looking at a cluster with an Infiniband connection? Infiniband isn’t cheap but they have to beat nearly a million dollars per month which gives them a lot of headroom.
  - rbanffy 2y ago
    They are also tied to MS SQL Server, it seems. Not sure it supports the fanciest gear out there. I've only seen Infiniband in HPC settings which are overwhelmingly non-Windows.
    - bee_rider 2y ago
      Another comment pointed out that the Reddit poster was probably just making the whole thing up, the account had made a bunch of different posts indicating they had a bunch of incompatible high-ranking jobs. The creative writing on that site is very odd.
hot_gril 2y ago
It might have uptime requirements that they can't provide on their own.

by ktpsns 2y ago

It's hard to believe for me how you would not start buying own hardware at this scale. In particular when the hyperscalers (at first glance) don't have anything to provide to match the needs.

rbanffy 2y ago

The biggest x86 machine I found tops out at 960 cores, but I'm not sure what exactly they need, if having more cores would solve their problems or would only make some other pipe burst. To figure that out, we'd need to look deep into what's happening in the machine, down to counting cache misses, memory bandwidth usage (per channel), QPI link usage (because NUMA), and, maybe, even go down to the usage stats of the CPU execution units. When they mention a lot of what was stored procedures has been moved to external web services, I get concerned they replaced memory and CPU occupancy with it waiting for network IO.

cjbgkagh 2y ago
I would hazard a guess that they're not really CPU bound. Assuming the poster Aussiepete80 is Australian I should point out that the much higher salaries in the US and the favorable E3 visa has largely brain drained Australia of their best and brightest. This guys army (dozens) of DBAs is likely the residual.
- rbanffy 2y ago
  Not really a nice thing to say - not everyone is inclined to move to the US (for me, and I'm not that cool, it'd need to be more than 6 digits). I suspect there is plenty competent people wishing to stay in Australia (in fact, a lot of healthcare folk here in Ireland have decided to move there - weather is so much better, despite the giant spider and murderous fauna problem). Predators here in Ireland are terrifying. The other day I was almost adopted into a family of feral cats.
  - cjbgkagh 2y ago
    One competent person probably, 'some' possibly, 'plenty' defiantly not. The general advice for tech people in Australia is to move to the US ASAP - the more competent the tech person the stronger the advice. The insidious effect of brain drain is the effect is self reinforcing, i.e. the more it happens the stronger the pull. In tech it's important to work with other competent people. Historically Australians are unusually good programmers, so I am not saying Australians are unskilled, but I am saying their best and brightest leave Australia ASAP. Plus I would submit the reddit thread 'Biggest CPU for the Bad System' as evidence to my point. How could this person possibly think that 100s of CPU cores are the bottleneck and the solution is more CPU cores - how could the dozens of DBAs not figure it out - the amount of work that can be done in 100s of CPU cores is utterly insane and the fact that they didn't rule out obvious IOWait bottlenecks in their initial posts suggest that none of them know what they're doing. I guess they figured that Azure would take care of all of that. The initial step would be to bring that computer in house and design that computer around the problem - Azure is not going to be open and honest about their bottlenecks. Last I checked Azure was big on their Network Attached Storage and Managed Disks which add quite a lot of latency and throughput bottlenecks compared to a PCIe 5.0 Enterprise SSD.
    - rbanffy 2y ago
      I’d say the telltale sign is picking SQL Server in the first place. If you think there is a risk your database will grow to be enormous, it’s likely it’ll be the worst possible choice, with the possible exception of Oracle and DB2 on AIX (but, on the last two, at least you have fewer limits to scaling up). Could also be a B-player hires C-players situation, with borderline incompetence trickling down from the C-level.
      - cjbgkagh 2y ago
        Certainly, if you're planning on outgrowing a SQL Server you should probably DIY tailor made indexes and query languages/engines. LLVM makes DIY languages much easier to make than they used to be and very with fast interpreters and AOT it's very performant. The reason for my brain drain statement is that people don't understand how pernicious the brain drain is in Australia. The Australian government believes that there isn't a net brain drain because more people with degrees immigrate than those emigrating. But people are not substitutable cogs and the very smart are very rare and it is the loss of those few very smart Australians to the US which is so devastating.
bee_rider 2y ago
If they need really 1600 cores, perhaps they should be looking at a cluster with an Infiniband connection? Infiniband isn’t cheap but they have to beat nearly a million dollars per month which gives them a lot of headroom.
- rbanffy 2y ago
  They are also tied to MS SQL Server, it seems. Not sure it supports the fanciest gear out there. I've only seen Infiniband in HPC settings which are overwhelmingly non-Windows.
  - bee_rider 2y ago
    Another comment pointed out that the Reddit poster was probably just making the whole thing up, the account had made a bunch of different posts indicating they had a bunch of incompatible high-ranking jobs. The creative writing on that site is very odd.

hot_gril 2y ago

It might have uptime requirements that they can't provide on their own.