8 ms·
My biggest concern in moving to AMD is support for profiling and system visibility. On Intel, there are many more performance counters exposed, and Intel thems
by drewg123 7y ago
My biggest concern in moving to AMD is support for profiling and system visibility. On Intel, there are many more performance counters exposed, and Intel themselves make great tools (Vtune, Intel PCM).
AMD has a profiler, but it is nowhere near as capable as Vtune.
AMD has ways to do some things that Intel's PCM tools do (like monitor memory bandwidth), but they are exceedingly awkward (eg, only option is producing a CSV, no live output). And there are lots of things which are just missing (like support for monitoring NUMA fabric bandwidth, PCIe bandwidth, power, etc). I gave a talk at AMD in Austin last fall where I beat them up about some of these things.
Oh, and I'm up to over 260Gb/s of 100% TLS traffic from an AMD Rome :)
- uluyol 7y agoCompared to both Intel on the CPU side and Nvidia on the graphics side, AMD has always been lacking in the software department. Hopefully they can fix that now that they're making some money.
- akiselev 7y agoAMD is a classical hardware manufacturer in that they don't respect software as an equal partner in modern hardware development, despite the fact that their core competency is making general purpose computers. It's not a capital issue, it's a cultural one. That said, AMD (and Intel) have reinvented themselves several times with pretty big cultural shifts so I'm hopeful they get someone like Jim Keller with the authority to shake things up.
- hitpointdrew 7y ago>so I'm hopeful they get someone like Jim Keller Who do you think designed the Zen architecture? He as since left AMD, went to Tesla, and is now at Intel.
- akiselev 7y agoWhy do you think I brought him up? AMD needs someone like Jim Keller but for software.
- DiabloD3 7y agoWho the hell is the Jim Keller of software? I don't think Dave Cutler is the right comparison, although they're both pretty fucking awesome.
- creddit 7y agoJeff Dean?
- baybal2 7y ago> Why do you think I brought him up? AMD needs someone like Jim Keller but for software. If there will be a JK for software, I'd say if he lands in a typical dotcom, half of the dev team with careers typical to Facebook/Amazon/Google will be relieved of their duties in a week.
- bsder 7y agoThey'd never hire him. Too old.
- zokula 7y agoJim Keller didn't design Zen.
- baybal2 7y ago> I'm hopeful they get someone like Jim Keller with the authority to shake things up. I's say it is even less likely if Jim will be on the board
- klingonopera 7y agoDespite this, I find it monumentally easier to get drivers and software of AMD's website, compared to Intel. And then there's also "Intel is Removing End of Life Drivers and BIOS Downloads"[0]. Great, if I didn't save them, I need to go to a 3rd party site, run the risk of getting malware, have to compare hashes, etc, etc, etc... This is, in my opinion, an absolute no-go as a vendor, and instantly makes me want to drop them. Which I did, all the PCs my clients get from me are AMD systems now. AMD is a serious vendor, you can even get the old ATI RAGE drivers off their website. [0]: https://www.bleepingcomputer.com/news/hardware/intel-is-removing-end-of-life-drivers-and-bios-downloads/ https://www.bleepingcomputer.com/news/hardware/intel-is-remo... EDIT: The linked tweet[1] in the article mirrors my sentiments on this topic almost perfectly: > "DEAR COMPANIES THAT MAKE HARDWARE: YOU'RE SUPPOSED TO HOST YOUR DRIVERS UNTIL THE SUN EXPLODES OR YOU GO OUT OF BUSINESS, NOT UNTIL YOU GET TIRED OF HOSTING 20 MEGABYTES" [1]: https://twitter.com/Foone/status/1172237142485078016 https://twitter.com/Foone/status/1172237142485078016
- dirtydroog 7y ago20mb? My latest nvidia 'driver' download was over 500mb and prompted me to log-in to something before installing.
- iopq 7y agoWhat? I expect to get drivers even after the sun explodes
- klingonopera 7y agoYES!
- AaronFriel 7y agoI think I see a comment to this effect on every post about AMD, that their software ecosystem isn't as advanced. But, isn't it a pretty narrow market that benefits from Intel's tooling here? Startups running their entire business on JITed runtimes on top of multiple layers of virtualization in cloud providers aren't using this. Enterprises with enough resources to revise their entire stack to work on ARM (Cloudflare, Microsoft, Google, Amazon) likely have the internal resources and skills already to build the missing tooling and profile statistically using massive fleets of servers. How large is the target market of Intel purchasers that are large enough and performance sensitive enough that it's profitable to micro-optimize assembly, and yet not large enough that it's a rounding error to build similar tooling for AMD x86-64 or ARM or otherwise?
- drewg123 7y agoIts not a question of just building tooling. It is a question of AMD having fewer performance counters. So even organizations like Google with tons of SWEs can't do anything if there are no performance counters, or if the performance counters are not exposed. And we (Netflix Open Connect) kind of fit that middle ground that you're talking about.
- baybal2 7y ago> Its not a question of just building tooling. It is a question of AMD having fewer performance counters. So even organizations like Google with tons of SWEs can't do anything if there are no performance counters, or if the performance counters are not exposed. Have you tried JTAG?
- mda 7y agoAny examples of missing counters that are essential? I am curious because I rarely see people care about things further than TLB, L1, L2 misses (which exist on AMD).
- hinkley 7y agoIf you have a large enough problem space that you have the capacity to develop a 'bench' of people to tackle deep analysis, you probably should do so, rather than relying entirely solely on tools to tell you what's going on. It's not enough to know how to do something, you have to have done it routinely enough that when it really matters you don't treat it like a rehearsal. > How large is the target market of Intel purchasers that are large enough and performance sensitive enough that it's profitable to micro-optimize assembly I'd like to think if a large customer picked up the phone and called Intel and asked why the new processors they got are slower than the old ones, that asking that they send someone to take a look at it would not be outside of the realm of reasonable requests. But not being in a position to buy millions of dollars worth of gear from Intel so I'll have to wait for someone to tell me what happened. If they bother to send someone out, that person would probably like to have - or even insist on - some decent tools. Reliable and informative, if not necessarily friendly enough for consumers. I wonder to what extent Intel's tools prevent such customer visits, versus being made to give small and medium sized customer a feeling that they have some secret weapon using Intel, rather than the reality of one. A pretty collar for the pig to wear to the Fair.
- ksec 7y ago>Oh, and I'm up to over 260Gb/s of 100% TLS traffic from an AMD Rome :) Hold on, is that 30% improvement from 3-6 months ago? I thought it was 200Gb/s last time and it was sort of pushed to the limit with memory bandwidth bottleneck? Do you think you can push it even further something like 300Gb/s? Seriously this is mind boggling.
- drewg123 7y agoThis is a motherboard designed for Rome, so it has 3200 memory, rather than 2933 (which was OC'ed). Plus a few improvements here and there in FreeBSD, minus the fact that we're using 2 Gen4 x16 NICs, so there is some cross-domain IO going on.
- foobiekr 7y agocan you expand on the TLS bit?
- drewg123 7y agoTransport Layer Security, which used to be known as SSL (eg, https). It uses up nearly half of our CPU time and our memory bandwidth to read data from memory into the CPU, encrypt it, and write the encrypted data back to memory. If we were serving plain http without TLS, we could probably reach 400Gb/s from this box (based purely on back of the envelope math)
- JoshTriplett 7y agoYou can't use any kind of TLS offloading accelerator?
- greglindahl 7y agoAre you asking what’s most cost-effective?
- JoshTriplett 7y agoI'm asking what best meets the requirements, which may be a balance of cost and performance.
- wmf 7y agoLooks like it's in progress: https://2019.eurobsdcon.org/slides/Kernel%20TLS%20and%20TLS%20hardware%20offload%20-%20Drew%20Gallatin.pdf https://2019.eurobsdcon.org/slides/Kernel%20TLS%20and%20TLS%...
- drewg123 7y agoThere are 2 classes of accelerators -- lookaside and inline. The lookaside accelerators (like QAT) sit on the PCIe bus and DMA plaintext into the accelerator, and DMA encrypted data out. This offloads the CPU, but it does not help with memory bandwidth. Since memory bandwidth is our biggest bottleneck, they don't help. Inline accelerators are "smart nics". The NIC DMAs a pre-formatted TLS record in plain text inside a (series of) TCP packet(s) from the host, encrypts it, and sends it in an almost stateless way. This is what we call "NIC TLS" in FreeSD. Sadly, the inline accelerators we've tried all have fatal flaws which prevent us from using them. I cannot get into those flaws for NDA reasons, but we're hopeful that there will be one that we can use soon.
- fulafel 7y agoAre you talking about a Linux version of Vtune or are you running your datacenters on Windows? I know Vtune for Linux exists but I've never heard much a bout it, everyone seems to be recommending "perf". Is there more to measuring memory bandwidth than multiplying last level cache misses with cache line size? I think where Intel does best is their open source drivers, tools and contributions, though AMD has improved a lot in that respect too.
- agoodthrowaway 7y agoHow is that possible? Raw Bw of PCIE is around 200Gb/s.
- shaklee3 7y agoPcie4 (of which rome is) can do about 220Gbps realistically per card. Multiple cards, which I'm sure Netflix does, can go higher.
- agoodthrowaway 7y agoTrue but usually it’s difficult to actually achieve that speed on the bus. Isn’t the NIC a bottleneck? Usually they aren’t capable of wirespeed due to shallow buffers.
- shaklee3 7y agoIt's doable. We do 100Gbps on pcie3, and AMD presented at the dpdk conference last year showing 200gbps: https://dpdkna2019.sched.com/event/WYAs/dpdk-and-pcie-gen4-benchmarking-amir-ancel-mellanox-keesang-song-amd&ved=2ahUKEwiW3bnh7IjoAhUKqZ4KHZndCWQQFjAAegQIAxAC&usg=AOvVaw3mHNUQcjzQhHwBB0drn1Py https://dpdkna2019.sched.com/event/WYAs/dpdk-and-pcie-gen4-b...
- gct 7y agoI'm curious, why encrypt Netflix video traffic at all?
- uluyol 7y ago(Not at Netflix) So that ISPs don't interfere with it by blocking, inserting ads, etc.
- shaklee3 7y agoI believe AMD has been contributing upstream to intel-cmt-cat, and Intel has accepted those patches.