4 ms·

> the manufacturer confirmed that the X21 offers 100 PCIe lanes, suggesting the presence of a PCIe switch. Almost like it's custom designed for a particular ap

> the manufacturer confirmed that the X21 offers 100 PCIe lanes, suggesting the presence of a PCIe switch. Almost like it's custom designed for a particular application where money's no concern... Perhaps someone in Utah needs big rainbow tables?

mikece 4y ago
This is No Such Agency who would buy as many of these as could be produced...
jeffbee 4y ago
People like to throw the innuendo around but the NSA's pathetic little datacenter is something that you would lose in a corner of a real datacenter operated by a real hyperscale system like Amazon or Google.
- xen2xen1 4y ago
  But the NSA one is dedicated to invading everyone's privac... Hey, wait!
- zamnos 4y ago
  Just like a cluster of Bitcoin miners will run absolute circles around a similarly sized corner of an AWS data center, and the supercomputer at Oak Ridge will run circles around a similar sized corner of AWS of GPU EC2 instances connected via gigabit Ethernet, the NSA's cluster's got a different use case than running web services for every SaaS company that wants to run in AWS. I imagine it's aimed at saving and analyzing/decrypting large amounts of data, and thus is architected and tuned towards that purpose, and thus runs circles around a similarly sized corner of AWS for that particular task. Unless you have experience with the NSA's cluster that you'd like to share with the rest of the class, that is.
  - jeffbee 4y ago
    Needs citation. I think the idea that the NSA has stronger data storage and analysis infrastructure than commercial operators is not even conjecture, it's something weaker, a fantasy. Commercial hyperscale operators claimed the ability to sort 50PB datasets at 600GB/s, eleven years ago. Storing and analyzing bulk data is the #1 thing these guys are good at.
rektide 4y ago
PCIe switches just shouldn't be so dammed expensive. A decade ago there was a lot more market competition but now there are, what, two companies with chips? It has gotten a good bit harder to build, especially with so many of the tricks & tight timings in PCIe 5 and 6, but the lack of market competition has made getting any kind of parts at all much much more expensive.
- bick_nyers 4y ago
  With the cost of PLX PCIE Switches (allows you to e.g. PCIE 4.0 x16 -> PCIE 3.0 x32) it is actually worth considering just buying a second desktop and throwing in some high-bandwidth NIC and forming your own HPC. Or instead of using desktop parts just going EPYC/Xeon/Threadripper. Of course it all comes down to the fact that if you need those PCIE lanes, there's a very good chance that it's for your job, meaning that businesses are the target market, not the enthusiast building a homelab for tinkering with LLM off the clock.
amluto 4y ago
This actually sounds like it could be a nice mid-range product. For lots of money, you can get a fancy motherboard and enclosure that routes a ton of CPU PCIe lanes direct to the NVMe drives. This ends up with a lot of performance per unit storage, which one might not want. With a card like this, one can get a ton of high-speed (much better than SATA but not as fast as direct NVMe) storage in a regular machine.
- bick_nyers 4y ago
  Could one hypothetically install an M.2 -> PCIE x16 riser and install a GPU?
  - amluto 4y ago
    I don’t see why not. OTOH, if you try this on an NVMe “hardware RAID” slot, you may get hilarious results.
    - bick_nyers 4y ago
      Stripe your GPU in a Raid 0 configuration for maximum performance, if your GPU doesn't have ECC VRAM, consider mirroring them :)
eqvinox 4y ago
> Almost like it's custom designed […] https://www.microchip.com/en-us/product/PM42100 https://www.microchip.com/en-us/product/PM42100 It's a standard COTS part. Coincidentally, the PCIe 5.0 variant is in "Samples available", i.e. not full production yet, which is very likely the reason for this card only being PCIe 4.0. https://www.microchip.com/en-us/product/PM50100 https://www.microchip.com/en-us/product/PM50100
- rasz 4y ago
  Doesnt help that PM42100 is $7.5K and out of stock.
  - eqvinox 4y ago
    That's the price of the development/evaluation kit. Those are produced in small numbers, have provisions for everything and debugging the kitchen sink, and thus always this expensive. With the PM40100 being $800 (single unit, no bulk pricing), the PM41100 / PM42100 are probably < $1500. (They do seem to have more features, not quite clear without proper datasheet sadly.)
deleted 4y ago
[deleted]

mikece 4y ago

This is No Such Agency who would buy as many of these as could be produced...

jeffbee 4y ago

People like to throw the innuendo around but the NSA's pathetic little datacenter is something that you would lose in a corner of a real datacenter operated by a real hyperscale system like Amazon or Google.

xen2xen1 4y ago
But the NSA one is dedicated to invading everyone's privac... Hey, wait!
zamnos 4y ago
Just like a cluster of Bitcoin miners will run absolute circles around a similarly sized corner of an AWS data center, and the supercomputer at Oak Ridge will run circles around a similar sized corner of AWS of GPU EC2 instances connected via gigabit Ethernet, the NSA's cluster's got a different use case than running web services for every SaaS company that wants to run in AWS. I imagine it's aimed at saving and analyzing/decrypting large amounts of data, and thus is architected and tuned towards that purpose, and thus runs circles around a similarly sized corner of AWS for that particular task. Unless you have experience with the NSA's cluster that you'd like to share with the rest of the class, that is.
- jeffbee 4y ago
  Needs citation. I think the idea that the NSA has stronger data storage and analysis infrastructure than commercial operators is not even conjecture, it's something weaker, a fantasy. Commercial hyperscale operators claimed the ability to sort 50PB datasets at 600GB/s, eleven years ago. Storing and analyzing bulk data is the #1 thing these guys are good at.

rektide 4y ago

PCIe switches just shouldn't be so dammed expensive. A decade ago there was a lot more market competition but now there are, what, two companies with chips? It has gotten a good bit harder to build, especially with so many of the tricks & tight timings in PCIe 5 and 6, but the lack of market competition has made getting any kind of parts at all much much more expensive.

bick_nyers 4y ago
With the cost of PLX PCIE Switches (allows you to e.g. PCIE 4.0 x16 -> PCIE 3.0 x32) it is actually worth considering just buying a second desktop and throwing in some high-bandwidth NIC and forming your own HPC. Or instead of using desktop parts just going EPYC/Xeon/Threadripper. Of course it all comes down to the fact that if you need those PCIE lanes, there's a very good chance that it's for your job, meaning that businesses are the target market, not the enthusiast building a homelab for tinkering with LLM off the clock.

amluto 4y ago

This actually sounds like it could be a nice mid-range product. For lots of money, you can get a fancy motherboard and enclosure that routes a ton of CPU PCIe lanes direct to the NVMe drives. This ends up with a lot of performance per unit storage, which one might not want. With a card like this, one can get a ton of high-speed (much better than SATA but not as fast as direct NVMe) storage in a regular machine.

bick_nyers 4y ago
Could one hypothetically install an M.2 -> PCIE x16 riser and install a GPU?
- amluto 4y ago
  I don’t see why not. OTOH, if you try this on an NVMe “hardware RAID” slot, you may get hilarious results.
  - bick_nyers 4y ago
    Stripe your GPU in a Raid 0 configuration for maximum performance, if your GPU doesn't have ECC VRAM, consider mirroring them :)

eqvinox 4y ago

> Almost like it's custom designed […] https://www.microchip.com/en-us/product/PM42100 https://www.microchip.com/en-us/product/PM42100 It's a standard COTS part. Coincidentally, the PCIe 5.0 variant is in "Samples available", i.e. not full production yet, which is very likely the reason for this card only being PCIe 4.0. https://www.microchip.com/en-us/product/PM50100 https://www.microchip.com/en-us/product/PM50100

rasz 4y ago
Doesnt help that PM42100 is $7.5K and out of stock.
- eqvinox 4y ago
  That's the price of the development/evaluation kit. Those are produced in small numbers, have provisions for everything and debugging the kitchen sink, and thus always this expensive. With the PM40100 being $800 (single unit, no bulk pricing), the PM41100 / PM42100 are probably < $1500. (They do seem to have more features, not quite clear without proper datasheet sadly.)

deleted 4y ago

[deleted]