6 ms·

I wonder how this might affect GPU access to system RAM. Right now, people wanting to large language models on GPUs are constrained by the amount of VRAM the c

I wonder how this might affect GPU access to system RAM. Right now, people wanting to large language models on GPUs are constrained by the amount of VRAM the cards have. GPU access to system RAM is enough of a bottleneck that you might as well run your application on the regular CPU. But GPU access to system RAM is fast enough, then this opens more possibilities for GPU acceleration for large models.

appplication 3y ago
I know Apple doesn’t really do GPUs the same way other companies do, but isn’t this generally the idea behind their integrated memory SOCs?
alexgartrell 3y ago
Latency and cache coherency are the other things that make this hard. Cache coherency can theoretically be resolved by CXL, so maybe we’ll get there that way.
- Tuna-Fish 3y ago
  AI models do not need coherent memory, the access pattern is regular enough that you can make do with explicit barriers. The bigger problem is that by the time PCIe 7.0 will be actually available, 242GB/s per direction will probably not be sufficient for anything interesting.
  - paulmd 3y ago
    > AI models do not need coherent memory, the access pattern is regular enough that you can make do with explicit barriers. response to both this and the sibling: even for training, I remember some speculation that explicit synchronization might not even be needed, especially in the middle stages of training (past the early part, before fine tuning). it's not "correct" but the gradient descent will eventually fix it anyway, as long as the error signal doesn't exceed the rate of gradient descent. And "error signal" here isn't just the noise itself but the error in gradient descent caused by incorrect sync - if the average delta in the descent is very slow, so is the amount of error it introduces, right? actually iirc there was some thought that the noise might help bounce the model out of local optimums. little bit of a Simulated Annealing idea there. > The bigger problem is that by the time PCIe 7.0 will be actually available, 242GB/s per direction will probably not be sufficient for anything interesting. yea it's this - SemiAnalysis/Dylan Patel actually has been doing some great pieces on this. background: networks really don't scale past about 8-16 nodes. 8 is a hypercube, that's easy. You can do 16 with ringbus or xy-grid arrangements (although I don't think xy-grid has proven satisfactory for anything except systolic arrays imo). But as you increase the number of nodes past 8, link count blows out tremendously, bisection bandwidth stagnates, worst-case distance blows out tremendously, etc. So you want tiers, and you want them to be composed of nodes that are as large as you can make them, because you can't scale the node count infinitely. https://www.cs.cmu.edu/afs/cs/academic/class/15418-s12/www/lectures/18_interconnects.pdf https://www.cs.cmu.edu/afs/cs/academic/class/15418-s12/www/l... Past about 16 nodes you just go to a switched-fabric thing that's a crossbar or a butterfly tree or whatever - and it's still a big fat chip itself, the 2nd gen nvswitch (pascal era) was about as many transistors as a Xeon 1650v1. They are on gen 3 or 4 now, and they have three of these in a giant mixing network (butterfly or fat tree or something) for just titanic amounts of interconnect between 2 racks. I don't even want to know what the switching itself pulls, it's not 300kW but it's definitely not insignificant either. any HPC interconnect really needs to be a meaningful fraction of memory bandwidth speed if you want to treat it like a "single system". Doesn't have to be 100%, but like, needs to be 1/3 or 1/4 of normal memory BW at least. One of the theories around AMD's MCM patents was that the cache port and the interconnect port should be more or less the same thing - because you need to talk to interconnect at pretty much the same rate you talk to cache. So a cache chiplet and an interconnect could be pretty much the same thing in silicon (I guess today we'd say they're both Infinity Link - which is not the same as Infinity Fabric BTW, again the key difference being coherency). But that kinda frames the discussion in terms of the requirements here. https://hexus.net/tech/news/graphics/147643-amd-patent-outlines-gpu-active-bridge-chiplet-design/ https://hexus.net/tech/news/graphics/147643-amd-patent-outli... https://community.amd.com/t5/general-discussions/amd-files-patent-for-its-own-gpu-chiplet-implementation/m-p/436624 https://community.amd.com/t5/general-discussions/amd-files-p... anyway, to your point: pcie/cxl will never be as fast as ethernet because the signal integrity requirements are orders of magnitude tighter, pcie is a very short-range link etc and requires comparatively larger PHY to drive it than ethernet does for the same bandwidth. ethernet serdes apparently have 3x the bandwidth as PCIe (and CXL) serdes per mm of beachfront, and GPU networking highly favors bandwidth above almost any other concern (and they utterly don't care about latency). The denser you make the bandwidth, the more links (or fatter links) you can fit on a given chip. And more links basically translates to larger networks, meaning more total capacity, better TCO, etc. Sort of a gustafson's law thing. (and it goes without saying that regardless, this all burns a tremendous amount of power. data movement is expensive.) https://www.semianalysis.com/p/cxl-is-dead-in-the-ai-era https://www.semianalysis.com/p/cxl-is-dead-in-the-ai-era the shape this is taking is basically computronium. giant chips, massive interconnects between them. It's not that chiplet is a bad idea, but what's better than lots of little chiplets fused into a single processor? lots of big chiplets fused into a single processor. And in fact that pattern gets repeated fractally. MI300X and B200 both take two big dies and fuse them together into what feels like a single GPU. Then you take a bunch of those GPUs and fuse those together into a local node via NVSwitch. https://www.semianalysis.com/p/nvidias-optical-boogeyman-nvl72-infiniband https://www.semianalysis.com/p/nvidias-optical-boogeyman-nvl... Standard HPC stuff... other than the density. They are thinking it might actually scale to at least 300 kW per rack... and efficiency actually improves when you do this (just like packaging!) because data movement is hideously expensive. You absolutely want to keep everything "local" (at every level) and talk over the interconnects as little as possible. https://www.fabricatedknowledge.com/p/the-data-center-is-the-new-compute https://www.fabricatedknowledge.com/p/the-data-center-is-the... MLID interviewed an NVIDIA engineer after RDNA3 came out, iirc they more or less said "they looked at chiplet, they didn't think it was worth it yet, so they didn't do it. and gonna do chiplets in their own way, and aren't going to be constrained or limited to chasing the approaches AMD uses". And my interpretation of that quote has always been - they see what they are doing as building a giant GPU out of tons of "chiplets", where each chiplet is a H100 board or whatever. NVLink is their Infinity Link, Ethernet is their Infinity Fabric/IFOP. The idea of a processor as a bunch of disaggregated tiny chiplets is great for yields, it's terrible for performance and efficiency. "tile" in the sense of having 64 tiles on your server processor is dead dead dead, tiles need to be decent-sized chunks of silicon on their own, because that reduces data movement a ton (and network node count etc). And while of course packaging lets you stack a couple dies... it also blows up power consumption in other areas, because if each chiplet is slower then you are moving more data around. The chip might be more efficient, but the system isn't as efficient as just building computronium out of big chunks. it's been an obvious lesson from the start even with ryzen, and RDNA3 should have really driven it home: data movement (cross-CCX/cross-CCD) is both performance-bottlenecking and power-intensive, so making the chiplets too small is a mistake. Navi 32 is barely a viable product etc, and that's without even hitting the prices that most people want to see from it. Driving 7700XT down to $349 or $329 or whatever is really really tough (it'll get there during clearance but they can't do it during the prime of its life), and idle power/low-load power sucks. You want the chunks to be at least medium sized - and really as big as you can afford to make them. Yields get lower the bigger you get, of course, but does anybody care about yielded price right now? Frankly I am surprised NVIDIA isn't pursing Cerebras-style wafer-scale right now tbh. again, lots of words to say: you want the nodes to be as fat as possible, because you probably only get 8 or 16 nodes per tier anyway. So the smaller you make the nodes, the less performance available at each tier. And that means slower systems with more energy spent moving data. The analyst (not NVIDIA)'s claim is that he thinks water-cooled 300kw would be more efficient than current systems. it's CRAY time! (no nvidia, no!!!) https://en.wikipedia.org/wiki/Cray-2#/media/File:Cray2.jpg https://en.wikipedia.org/wiki/Cray-2#/media/File:Cray2.jpg https://en.wikipedia.org/wiki/File:EPFL_CRAY-II_2.jpg https://en.wikipedia.org/wiki/File:EPFL_CRAY-II_2.jpg https://en.wikipedia.org/wiki/File:Cray-2_module_side_view.jpg https://en.wikipedia.org/wiki/File:Cray-2_module_side_view.j... (e: power consumption was 150-200 kW for Cray-2 so NVIDIA's got a ways to go (currently 100 kW, rumored 200 kW) to even to reach the peak of historical "make it bent so there's less data movement" style hyperdense designs. Tbh that makes me suspect that analyst is probably right, it's both possible and might well improve efficiency, but due to the data movement factors this time, rather than latency. Ironic.)
    - Dylan16807 3y ago
      > networks really don't scale past about 8-16 nodes. 8 is a hypercube, that's easy. You can do 16 with ringbus or xy-grid arrangements (although I don't think xy-grid has proven satisfactory for anything except systolic arrays imo). But as you increase the number of nodes past 8, link count blows out tremendously, bisection bandwidth stagnates, worst-case distance blows out tremendously, etc. I don't think cube/hypercube is optimal, the opposite corner is too many hops away. For 8 nodes, 3 links each, you're better of crossing some of the links to bring the max distance down to 2. Even better is a Petersen graph that can do 10 nodes at max distance 2. If a proper 16-node hypercube is acceptable, 4 links per node and most distances 2-3 hops, then better arrangement lets you fit up to 41 nodes at max distance 3. If you allow for 5 links or max distance 4 you can roughly double that node count, and if you allow both you can have over 200 nodes. https://en.wikipedia.org/wiki/Table_of_the_largest_known_graphs_of_a_given_diameter_and_maximal_degree https://en.wikipedia.org/wiki/Table_of_the_largest_known_gra... If you let the max distance climb, then it will seriously bloat your bandwidth requirements. But you can fit quite a lot of nodes at small max distances. > anyway, to your point: pcie/cxl will never be as fast as ethernet because the signal integrity requirements are orders of magnitude tighter, pcie is a very short-range link etc and requires comparatively larger PHY to drive it than ethernet does for the same bandwidth. Why does the short range link have much tighter requirements, and what would it take to sloppen them up?
- foobiekr 3y ago
  The announced Grace-Hopper chip appears to use a CXL or Arm-specific-CXL-alike
- mlyle 3y ago
  Coherency doesn't matter at all for a lot of cases. E.g. for inference, model weights are the vast majority of the memory use and are unchanging.
- Dylan16807 3y ago
  As long as the GPU-local memory can hold a couple layers at a time, I don't think the latency to the currently-inactive layers matters very much, only the bandwidth.
dist-epoch 3y ago
Currently the memory controller is inside the CPU, so the GPU would have to go through the CPU to get to the memory. You would still be limited by the CPU memory bandwidth, and you will share it with it.
- AnotherGoodName 3y ago
  They just moved that silicon to the CPU side is all. The same issues existed in the northbridge days particularly in multi-CPU setups where you'd have to go over hyper transport if the memory access was on the other controller. Not saying these issues don't exist. It's just that they really haven't changed much here except moved logic from the motherboard to the CPU itself.
- mlyle 3y ago
  > You would still be limited by the CPU memory bandwidth, and you will share it with it. The word I take issue with here is "still", in that today GPUs accessing main memory are not limited by CPU memory bandwidth but instead by the PCIe link bandwidth. I'm assuming that we will, in practice, still be limited by PCIe link bandwidth; but perhaps much less so.
  - numpad0 3y ago
    That problem is solved by CXL that allow PCIe RAM card on the bus, not faster RAM on CPU through bus
    - mlyle 3y ago
      CXL is nice. It doesn't change my belief that accessing RAM over PCIe will not be a win for very long.
      - imtringued 3y ago
        I don't know what you're on about. It is significantly faster to access memory directly via CXL than to use message passing with JSON or gRPC. Also CXL allows seamless pointer sharing between GPU and CPU in a fine grained way. These are obvious wins independent of memory bandwidth.
faeriechangling 3y ago
As PCIe gets faster so does memory so it continues being a bottleneck.
KeplerBoy 3y ago
Regular desktop ddr5 dusk Channel memory bandwidth is only around 60 GB/s. I doubt it will keep up with PCIe speeds.
latchkey 3y ago
AMD MI300x has 192GB HBM3 and 8 in a chassis. 1.5 TB. GigaIO has a PCIe solution that allows you to connect up to 32 of them together to appear as devices on a single machine. 6.1 TB. Eventually this will expand to 64 of them. 12.2 TB. I can't wait for faster PCIe. Later this year, AMD will release MI300x with HBM3e. Disclosure: building a bare metal cloud service provider around this stuff
- throwaway2037 3y ago
  > Disclosure: building a bare metal cloud service provider around this stuff Please share on HN! We would like to read about it and discuss.
  - latchkey 3y ago
    Thanks! It is in my profile. If you have any specific questions, ask here or reach out directly.

appplication 3y ago

I know Apple doesn’t really do GPUs the same way other companies do, but isn’t this generally the idea behind their integrated memory SOCs?

alexgartrell 3y ago

Latency and cache coherency are the other things that make this hard. Cache coherency can theoretically be resolved by CXL, so maybe we’ll get there that way.

Tuna-Fish 3y ago
AI models do not need coherent memory, the access pattern is regular enough that you can make do with explicit barriers. The bigger problem is that by the time PCIe 7.0 will be actually available, 242GB/s per direction will probably not be sufficient for anything interesting.
- paulmd 3y ago
  > AI models do not need coherent memory, the access pattern is regular enough that you can make do with explicit barriers. response to both this and the sibling: even for training, I remember some speculation that explicit synchronization might not even be needed, especially in the middle stages of training (past the early part, before fine tuning). it's not "correct" but the gradient descent will eventually fix it anyway, as long as the error signal doesn't exceed the rate of gradient descent. And "error signal" here isn't just the noise itself but the error in gradient descent caused by incorrect sync - if the average delta in the descent is very slow, so is the amount of error it introduces, right? actually iirc there was some thought that the noise might help bounce the model out of local optimums. little bit of a Simulated Annealing idea there. > The bigger problem is that by the time PCIe 7.0 will be actually available, 242GB/s per direction will probably not be sufficient for anything interesting. yea it's this - SemiAnalysis/Dylan Patel actually has been doing some great pieces on this. background: networks really don't scale past about 8-16 nodes. 8 is a hypercube, that's easy. You can do 16 with ringbus or xy-grid arrangements (although I don't think xy-grid has proven satisfactory for anything except systolic arrays imo). But as you increase the number of nodes past 8, link count blows out tremendously, bisection bandwidth stagnates, worst-case distance blows out tremendously, etc. So you want tiers, and you want them to be composed of nodes that are as large as you can make them, because you can't scale the node count infinitely. https://www.cs.cmu.edu/afs/cs/academic/class/15418-s12/www/lectures/18_interconnects.pdf https://www.cs.cmu.edu/afs/cs/academic/class/15418-s12/www/l... Past about 16 nodes you just go to a switched-fabric thing that's a crossbar or a butterfly tree or whatever - and it's still a big fat chip itself, the 2nd gen nvswitch (pascal era) was about as many transistors as a Xeon 1650v1. They are on gen 3 or 4 now, and they have three of these in a giant mixing network (butterfly or fat tree or something) for just titanic amounts of interconnect between 2 racks. I don't even want to know what the switching itself pulls, it's not 300kW but it's definitely not insignificant either. any HPC interconnect really needs to be a meaningful fraction of memory bandwidth speed if you want to treat it like a "single system". Doesn't have to be 100%, but like, needs to be 1/3 or 1/4 of normal memory BW at least. One of the theories around AMD's MCM patents was that the cache port and the interconnect port should be more or less the same thing - because you need to talk to interconnect at pretty much the same rate you talk to cache. So a cache chiplet and an interconnect could be pretty much the same thing in silicon (I guess today we'd say they're both Infinity Link - which is not the same as Infinity Fabric BTW, again the key difference being coherency). But that kinda frames the discussion in terms of the requirements here. https://hexus.net/tech/news/graphics/147643-amd-patent-outlines-gpu-active-bridge-chiplet-design/ https://hexus.net/tech/news/graphics/147643-amd-patent-outli... https://community.amd.com/t5/general-discussions/amd-files-patent-for-its-own-gpu-chiplet-implementation/m-p/436624 https://community.amd.com/t5/general-discussions/amd-files-p... anyway, to your point: pcie/cxl will never be as fast as ethernet because the signal integrity requirements are orders of magnitude tighter, pcie is a very short-range link etc and requires comparatively larger PHY to drive it than ethernet does for the same bandwidth. ethernet serdes apparently have 3x the bandwidth as PCIe (and CXL) serdes per mm of beachfront, and GPU networking highly favors bandwidth above almost any other concern (and they utterly don't care about latency). The denser you make the bandwidth, the more links (or fatter links) you can fit on a given chip. And more links basically translates to larger networks, meaning more total capacity, better TCO, etc. Sort of a gustafson's law thing. (and it goes without saying that regardless, this all burns a tremendous amount of power. data movement is expensive.) https://www.semianalysis.com/p/cxl-is-dead-in-the-ai-era https://www.semianalysis.com/p/cxl-is-dead-in-the-ai-era the shape this is taking is basically computronium. giant chips, massive interconnects between them. It's not that chiplet is a bad idea, but what's better than lots of little chiplets fused into a single processor? lots of big chiplets fused into a single processor. And in fact that pattern gets repeated fractally. MI300X and B200 both take two big dies and fuse them together into what feels like a single GPU. Then you take a bunch of those GPUs and fuse those together into a local node via NVSwitch. https://www.semianalysis.com/p/nvidias-optical-boogeyman-nvl72-infiniband https://www.semianalysis.com/p/nvidias-optical-boogeyman-nvl... Standard HPC stuff... other than the density. They are thinking it might actually scale to at least 300 kW per rack... and efficiency actually improves when you do this (just like packaging!) because data movement is hideously expensive. You absolutely want to keep everything "local" (at every level) and talk over the interconnects as little as possible. https://www.fabricatedknowledge.com/p/the-data-center-is-the-new-compute https://www.fabricatedknowledge.com/p/the-data-center-is-the... MLID interviewed an NVIDIA engineer after RDNA3 came out, iirc they more or less said "they looked at chiplet, they didn't think it was worth it yet, so they didn't do it. and gonna do chiplets in their own way, and aren't going to be constrained or limited to chasing the approaches AMD uses". And my interpretation of that quote has always been - they see what they are doing as building a giant GPU out of tons of "chiplets", where each chiplet is a H100 board or whatever. NVLink is their Infinity Link, Ethernet is their Infinity Fabric/IFOP. The idea of a processor as a bunch of disaggregated tiny chiplets is great for yields, it's terrible for performance and efficiency. "tile" in the sense of having 64 tiles on your server processor is dead dead dead, tiles need to be decent-sized chunks of silicon on their own, because that reduces data movement a ton (and network node count etc). And while of course packaging lets you stack a couple dies... it also blows up power consumption in other areas, because if each chiplet is slower then you are moving more data around. The chip might be more efficient, but the system isn't as efficient as just building computronium out of big chunks. it's been an obvious lesson from the start even with ryzen, and RDNA3 should have really driven it home: data movement (cross-CCX/cross-CCD) is both performance-bottlenecking and power-intensive, so making the chiplets too small is a mistake. Navi 32 is barely a viable product etc, and that's without even hitting the prices that most people want to see from it. Driving 7700XT down to $349 or $329 or whatever is really really tough (it'll get there during clearance but they can't do it during the prime of its life), and idle power/low-load power sucks. You want the chunks to be at least medium sized - and really as big as you can afford to make them. Yields get lower the bigger you get, of course, but does anybody care about yielded price right now? Frankly I am surprised NVIDIA isn't pursing Cerebras-style wafer-scale right now tbh. again, lots of words to say: you want the nodes to be as fat as possible, because you probably only get 8 or 16 nodes per tier anyway. So the smaller you make the nodes, the less performance available at each tier. And that means slower systems with more energy spent moving data. The analyst (not NVIDIA)'s claim is that he thinks water-cooled 300kw would be more efficient than current systems. it's CRAY time! (no nvidia, no!!!) https://en.wikipedia.org/wiki/Cray-2#/media/File:Cray2.jpg https://en.wikipedia.org/wiki/Cray-2#/media/File:Cray2.jpg https://en.wikipedia.org/wiki/File:EPFL_CRAY-II_2.jpg https://en.wikipedia.org/wiki/File:EPFL_CRAY-II_2.jpg https://en.wikipedia.org/wiki/File:Cray-2_module_side_view.jpg https://en.wikipedia.org/wiki/File:Cray-2_module_side_view.j... (e: power consumption was 150-200 kW for Cray-2 so NVIDIA's got a ways to go (currently 100 kW, rumored 200 kW) to even to reach the peak of historical "make it bent so there's less data movement" style hyperdense designs. Tbh that makes me suspect that analyst is probably right, it's both possible and might well improve efficiency, but due to the data movement factors this time, rather than latency. Ironic.)
  - Dylan16807 3y ago
    > networks really don't scale past about 8-16 nodes. 8 is a hypercube, that's easy. You can do 16 with ringbus or xy-grid arrangements (although I don't think xy-grid has proven satisfactory for anything except systolic arrays imo). But as you increase the number of nodes past 8, link count blows out tremendously, bisection bandwidth stagnates, worst-case distance blows out tremendously, etc. I don't think cube/hypercube is optimal, the opposite corner is too many hops away. For 8 nodes, 3 links each, you're better of crossing some of the links to bring the max distance down to 2. Even better is a Petersen graph that can do 10 nodes at max distance 2. If a proper 16-node hypercube is acceptable, 4 links per node and most distances 2-3 hops, then better arrangement lets you fit up to 41 nodes at max distance 3. If you allow for 5 links or max distance 4 you can roughly double that node count, and if you allow both you can have over 200 nodes. https://en.wikipedia.org/wiki/Table_of_the_largest_known_graphs_of_a_given_diameter_and_maximal_degree https://en.wikipedia.org/wiki/Table_of_the_largest_known_gra... If you let the max distance climb, then it will seriously bloat your bandwidth requirements. But you can fit quite a lot of nodes at small max distances. > anyway, to your point: pcie/cxl will never be as fast as ethernet because the signal integrity requirements are orders of magnitude tighter, pcie is a very short-range link etc and requires comparatively larger PHY to drive it than ethernet does for the same bandwidth. Why does the short range link have much tighter requirements, and what would it take to sloppen them up?
foobiekr 3y ago
The announced Grace-Hopper chip appears to use a CXL or Arm-specific-CXL-alike
mlyle 3y ago
Coherency doesn't matter at all for a lot of cases. E.g. for inference, model weights are the vast majority of the memory use and are unchanging.
Dylan16807 3y ago
As long as the GPU-local memory can hold a couple layers at a time, I don't think the latency to the currently-inactive layers matters very much, only the bandwidth.

dist-epoch 3y ago

Currently the memory controller is inside the CPU, so the GPU would have to go through the CPU to get to the memory. You would still be limited by the CPU memory bandwidth, and you will share it with it.

AnotherGoodName 3y ago
They just moved that silicon to the CPU side is all. The same issues existed in the northbridge days particularly in multi-CPU setups where you'd have to go over hyper transport if the memory access was on the other controller. Not saying these issues don't exist. It's just that they really haven't changed much here except moved logic from the motherboard to the CPU itself.
mlyle 3y ago
> You would still be limited by the CPU memory bandwidth, and you will share it with it. The word I take issue with here is "still", in that today GPUs accessing main memory are not limited by CPU memory bandwidth but instead by the PCIe link bandwidth. I'm assuming that we will, in practice, still be limited by PCIe link bandwidth; but perhaps much less so.
- numpad0 3y ago
  That problem is solved by CXL that allow PCIe RAM card on the bus, not faster RAM on CPU through bus
  - mlyle 3y ago
    CXL is nice. It doesn't change my belief that accessing RAM over PCIe will not be a win for very long.
    - imtringued 3y ago
      I don't know what you're on about. It is significantly faster to access memory directly via CXL than to use message passing with JSON or gRPC. Also CXL allows seamless pointer sharing between GPU and CPU in a fine grained way. These are obvious wins independent of memory bandwidth.

faeriechangling 3y ago

As PCIe gets faster so does memory so it continues being a bottleneck.

KeplerBoy 3y ago

Regular desktop ddr5 dusk Channel memory bandwidth is only around 60 GB/s. I doubt it will keep up with PCIe speeds.

latchkey 3y ago

AMD MI300x has 192GB HBM3 and 8 in a chassis. 1.5 TB. GigaIO has a PCIe solution that allows you to connect up to 32 of them together to appear as devices on a single machine. 6.1 TB. Eventually this will expand to 64 of them. 12.2 TB. I can't wait for faster PCIe. Later this year, AMD will release MI300x with HBM3e. Disclosure: building a bare metal cloud service provider around this stuff

throwaway2037 3y ago
> Disclosure: building a bare metal cloud service provider around this stuff Please share on HN! We would like to read about it and discuss.
- latchkey 3y ago
  Thanks! It is in my profile. If you have any specific questions, ask here or reach out directly.