4 ms·

Bedrock? It's like a vibe-coded "router" app. It really doesn't provide anything that is not provided by countless other companies. AWS is falling behind even

Bedrock? It's like a vibe-coded "router" app. It really doesn't provide anything that is not provided by countless other companies. AWS is falling behind even in their most traditional area: renting compute capacity. For example, I can't easily run models that need GPUs without launching classic EC2 instances. Fargate or Lambda _still_ don't support GPUs. Sagemaker Serverless exists but has some weird limits (like 10GB limit on Docker images).

iLoveOncall 1y ago
Bedrock is not at all a router. They do provide a routing capability now, but at its core it's a wrapper around models so you can interact with any model with the same unique API. > For example, I can't easily run models that need GPUs without launching classic EC2 instances. Yeah okay, but you can run most entreprise-level models via Bedrock.
- Aeolun 1y ago
  Only if you want them to go to random inference regions. God forbid you would want inference in a single region. Then you need to be satisfied with 12 month old models that have been superseded 2 times already.
internetter 1y ago
AWS doesn't need to do anything innovative and the enterprises still come. Every product AWS sells has a similar offering from a competitor. But businesses stick with amazon because its all in one. They get bills from one company, trust their security with one company, ect. The only thing that matters to AWS is its reputation.
- cyberax 1y ago
  This works up to a point. I'm extremely familiar with AWS, but we simply _can't_ use it to train our models because it costs 2-3 times more than their competitors. All while requiring us to basically bring up all the infrastructure around maintaining the training cluster ourselves.
nickysielicki 1y ago
Frankly, this is strictly a positive signal to me. Fargate and lambda are fundamentally very different from EC2/nitro under the hood, with a very different risk profile in terms of security. The reason you can't run GPU workloads on top of fargate and lambda is because exposing physical 3rd-party hardware to untrusted customer code dramatically increases the startup and shutdown costs (ie: validating that the hardware is still functional, healthy, and hasn't been tampered with in any way). That means scrubbing takes a long time and you can't handle capacity surges as easily as you can with paravirtualized traditional compute workloads. There are a lot of business-minded non-technical people running AWS, some of which are sure to be loudly complaining about this horrible loss of revenue... which simply lets you know that when push comes to shove, the right voices are still winning inside AWS (eg: the voices that put security above everything else, where it belongs).
- cyberax 1y ago
  > Frankly, this is strictly a positive signal to me. How? > The reason you can't run GPU workloads on top of fargate and lambda is because exposing physical 3rd-party hardware to untrusted customer code dramatically increases the startup and shutdown costs This is BS. Both NVidia and AMD offer virtualization extensions. And even without that, they can simply power-cycle the GPUs after switching tenants. Moreover, Fargate is used for long-running tasks, and it definitely can run on a regular Nitro stack. They absolutely can provide GPUs for them, but it likely requires a lot of internal work across teams to make it happen. So it doesn't happen. I worked at AWS, in a team responsible for EC2 instance launching. So I know how it all works internally :)
  - nickysielicki 1y ago
    You'd have to build totally separate datacenters with totally different hardware than what they have today. You're not thinking about the complexity introduced by the use of pcie switches. For starters, you don't have enough bandwidth to saturate all gpus concurrently, they're sharing pcie root complex bandwidth, which is a non-starter if you want to define any kind of reasonable SLA. You can't really enforce limits, either. Even if you're able to tolerate that and sell customers on it, the security side is worse. All customer GPU transactions would be traversing a shared switch fabric, which means noisy bursty neighbors, timing side-channels, etc., etc., etc.

iLoveOncall 1y ago

Bedrock is not at all a router. They do provide a routing capability now, but at its core it's a wrapper around models so you can interact with any model with the same unique API. > For example, I can't easily run models that need GPUs without launching classic EC2 instances. Yeah okay, but you can run most entreprise-level models via Bedrock.

Aeolun 1y ago
Only if you want them to go to random inference regions. God forbid you would want inference in a single region. Then you need to be satisfied with 12 month old models that have been superseded 2 times already.

internetter 1y ago

AWS doesn't need to do anything innovative and the enterprises still come. Every product AWS sells has a similar offering from a competitor. But businesses stick with amazon because its all in one. They get bills from one company, trust their security with one company, ect. The only thing that matters to AWS is its reputation.

cyberax 1y ago
This works up to a point. I'm extremely familiar with AWS, but we simply _can't_ use it to train our models because it costs 2-3 times more than their competitors. All while requiring us to basically bring up all the infrastructure around maintaining the training cluster ourselves.

nickysielicki 1y ago

Frankly, this is strictly a positive signal to me. Fargate and lambda are fundamentally very different from EC2/nitro under the hood, with a very different risk profile in terms of security. The reason you can't run GPU workloads on top of fargate and lambda is because exposing physical 3rd-party hardware to untrusted customer code dramatically increases the startup and shutdown costs (ie: validating that the hardware is still functional, healthy, and hasn't been tampered with in any way). That means scrubbing takes a long time and you can't handle capacity surges as easily as you can with paravirtualized traditional compute workloads. There are a lot of business-minded non-technical people running AWS, some of which are sure to be loudly complaining about this horrible loss of revenue... which simply lets you know that when push comes to shove, the right voices are still winning inside AWS (eg: the voices that put security above everything else, where it belongs).

cyberax 1y ago
> Frankly, this is strictly a positive signal to me. How? > The reason you can't run GPU workloads on top of fargate and lambda is because exposing physical 3rd-party hardware to untrusted customer code dramatically increases the startup and shutdown costs This is BS. Both NVidia and AMD offer virtualization extensions. And even without that, they can simply power-cycle the GPUs after switching tenants. Moreover, Fargate is used for long-running tasks, and it definitely can run on a regular Nitro stack. They absolutely can provide GPUs for them, but it likely requires a lot of internal work across teams to make it happen. So it doesn't happen. I worked at AWS, in a team responsible for EC2 instance launching. So I know how it all works internally :)
- nickysielicki 1y ago
  You'd have to build totally separate datacenters with totally different hardware than what they have today. You're not thinking about the complexity introduced by the use of pcie switches. For starters, you don't have enough bandwidth to saturate all gpus concurrently, they're sharing pcie root complex bandwidth, which is a non-starter if you want to define any kind of reasonable SLA. You can't really enforce limits, either. Even if you're able to tolerate that and sell customers on it, the security side is worse. All customer GPU transactions would be traversing a shared switch fabric, which means noisy bursty neighbors, timing side-channels, etc., etc., etc.