7 ms·

Building storage-first serverless apps with HTTP APIs service integrations

41 points by kiyanwang 6y ago

xellisx 6y ago
What I want is a storageless-first serverless connectionless computerless apps.
- wolfgang42 6y ago
  I think you may have just invented the whiteboard.
throwawaysea 6y ago
There is so much dependency on proprietary technologies here that anyone pursue anything serious should just stay away from the suggested architectures. If AWS were to raise prices in the future or to kick you off the platform, it would be that much harder to move elsewhere.
- NomDePlum 6y ago
  Are those scenarios more or less likely than AWS lowering prices and in the future not kicking you off. If it takes less effort and time to build something with proprietary technology surely that is a benefit? You can then assess whether removing that dependency improves your position.
- jedberg 6y ago
  Would you say the same thing to someone building all their software to compile on Intel? What if Intel doubles the prices of their CPUs? Then what? The lock in argument is sort of silly. Everything has some sort of lock in. It's just a question of whether the benefits outweigh the potential costs of the lock in. In the case of AWS, they give you incredible velocity since you don't have to built any of the infrastructure before you get started.
  - ryukafalz 6y ago
    > Would you say the same thing to someone building all their software to compile on Intel? If they’re targeting Intel processors only and not x86 more generally? Yes, I would. Otherwise, there’s at least one credible competitor (AMD) that’ll gladly sell you compatible processors, who I’m sure would be happy to see the increased sales if Intel jacked up their prices.
    - jedberg 6y ago
      AWS has two competitors, so I'd make the same argument back to you.
      - harshitaneja 6y ago
        You can run something compiled for x86 on AMD without modifications. That is not true for custom proprietary tools from cloud providers. All have more or less equivalent offerings but the cost to migrate are not negligible in a large number of cases. So lock in is more of a concern here. And even with processor architectures, I have started considering ARM migration as ARM has started becoming more popular in cloud. This is not to say I don't believe these tools have their benefits, just that the benefits need to be really weighed against the cost on lock-in, which in my opinion are quite understated with those who choose to use it without a second thought.
      - ryukafalz 6y ago
        As a sibling comment pointed out, applications built for x86 PCs don't even need to be recompiled to run on compatible processors by another manufacturer. (Heck, even across architectures you can usually recompile relatively easily provided the software is written in a high-level language with a compiler available there.) That is very much not the case with cloud providers' APIs; they're largely provider-specific. The closest equivalent in the cloud provider ecosystem is probably Kubernetes; there are provider-specific extensions (and I recommend avoiding those where possible) but largely things work the same across providers. I'd say a more representative example of lock-in in the PC ecosystem is that of proprietary OS vendors. If you're targeting specifically iOS/Android (w/ Play Services)/macOS/Windows/etc, then yes, you're beholden to the vendor (to varying degrees in each case). You may accept that risk, but you should always be aware of it.
- social_quotient 6y ago
  Their track record would suggest the opposite, and while I get what you’re saying on the price side I feel AWS is pretty trust worthy here? Open to a counter thought here if I’m missing something. I can’t speak to the getting kicked off side.
- deegles 6y ago
  I know it's not evidence that it won't happen in the future, but how often has AWS raised prices on a service? I would bet it's less than a handful of times, and never on an established product.
abraae 6y ago
These technologies are fascinating and make good sense for some APIs for sure. But if I look at our APIs, and I don't think we're unusual, they characteristically are not "fire and forget". Instead, the API call actually returns some data in the API response! Even for POST /widgets, I want to pass back an identifier for the just created widget, not a 200 code that means "request accepted and queued but no data available right now". The uses for APIs that don't return data seem pretty limited in our realm (SaaS). One example is handling incoming notifications/webhooks from other systems. By their nature these are asynchronous and the caller (the external system) does not usually expect a meaningful response, so for sure we can use SQS to buffer the incoming API calls. But for the APIs that we offer, virtually all of them have an online response. In my experience this is not an unusual pattern (for SaaS APIs anyway). I feel that these technologies are somewhat biased towards super scale systems that are highly decoupled and asynchronous - perhaps the sort of systems that Amazon are used to dealing with in-house - rather than simple old business systems.
- wgjordan 6y ago
  "The advantage of this pattern is increased application resiliency", as spelled out in the first paragraph. So yes, these technologies are 'biased towards' systems that require this degree of resiliency rather than 'simple old business systems'.
  - abraae 6y ago
    Simple old business systems require resilience too.
- sk5t 6y ago
  > not a 200 code that means "request accepted and queued but no data available right now" Properly that would be HTTP 202 Accepted :)
talawahtech 6y ago
This is awesome and long overdue. Providing a simple way to go directly from an HTTP endpoint to SQS, Event Bridge or Kinesis makes it much easier to build more robust webhooks or tracking endpoints. I have tried doing similar things in the past,using API Gateway and SQS without Lambda, and it was a huge pain. I wish they had skipped coining the "storage-first" buzzword though. I mean, I get it, but I feel like it hurts more than it helps. My first assumption was that they were just talking about writing data to S3 and then using Lambda to process after.
deegles 6y ago
How is the the SQS pattern bidirectional?
koluna 6y ago
This is all nice, but here is the important part you gotta remember - the more you take dependencies on AWS and its integrations, the harder it will be to actually have any flexibility and say in your own systems design. AWS is interested in taking your money first and foremost as well as having you depend on it from one side to another of your stack. Instead I’d look at containerizing my workflows and only using cloud tools on the edge (like load balancing).

41 points by kiyanwang 6y ago

xellisx 6y ago

What I want is a storageless-first serverless connectionless computerless apps.

wolfgang42 6y ago
I think you may have just invented the whiteboard.

throwawaysea 6y ago

There is so much dependency on proprietary technologies here that anyone pursue anything serious should just stay away from the suggested architectures. If AWS were to raise prices in the future or to kick you off the platform, it would be that much harder to move elsewhere.

NomDePlum 6y ago
Are those scenarios more or less likely than AWS lowering prices and in the future not kicking you off. If it takes less effort and time to build something with proprietary technology surely that is a benefit? You can then assess whether removing that dependency improves your position.
jedberg 6y ago
Would you say the same thing to someone building all their software to compile on Intel? What if Intel doubles the prices of their CPUs? Then what? The lock in argument is sort of silly. Everything has some sort of lock in. It's just a question of whether the benefits outweigh the potential costs of the lock in. In the case of AWS, they give you incredible velocity since you don't have to built any of the infrastructure before you get started.
- ryukafalz 6y ago
  > Would you say the same thing to someone building all their software to compile on Intel? If they’re targeting Intel processors only and not x86 more generally? Yes, I would. Otherwise, there’s at least one credible competitor (AMD) that’ll gladly sell you compatible processors, who I’m sure would be happy to see the increased sales if Intel jacked up their prices.
  - jedberg 6y ago
    AWS has two competitors, so I'd make the same argument back to you.
    - harshitaneja 6y ago
      You can run something compiled for x86 on AMD without modifications. That is not true for custom proprietary tools from cloud providers. All have more or less equivalent offerings but the cost to migrate are not negligible in a large number of cases. So lock in is more of a concern here. And even with processor architectures, I have started considering ARM migration as ARM has started becoming more popular in cloud. This is not to say I don't believe these tools have their benefits, just that the benefits need to be really weighed against the cost on lock-in, which in my opinion are quite understated with those who choose to use it without a second thought.
    - ryukafalz 6y ago
      As a sibling comment pointed out, applications built for x86 PCs don't even need to be recompiled to run on compatible processors by another manufacturer. (Heck, even across architectures you can usually recompile relatively easily provided the software is written in a high-level language with a compiler available there.) That is very much not the case with cloud providers' APIs; they're largely provider-specific. The closest equivalent in the cloud provider ecosystem is probably Kubernetes; there are provider-specific extensions (and I recommend avoiding those where possible) but largely things work the same across providers. I'd say a more representative example of lock-in in the PC ecosystem is that of proprietary OS vendors. If you're targeting specifically iOS/Android (w/ Play Services)/macOS/Windows/etc, then yes, you're beholden to the vendor (to varying degrees in each case). You may accept that risk, but you should always be aware of it.
social_quotient 6y ago
Their track record would suggest the opposite, and while I get what you’re saying on the price side I feel AWS is pretty trust worthy here? Open to a counter thought here if I’m missing something. I can’t speak to the getting kicked off side.
deegles 6y ago
I know it's not evidence that it won't happen in the future, but how often has AWS raised prices on a service? I would bet it's less than a handful of times, and never on an established product.

abraae 6y ago

These technologies are fascinating and make good sense for some APIs for sure. But if I look at our APIs, and I don't think we're unusual, they characteristically are not "fire and forget". Instead, the API call actually returns some data in the API response! Even for POST /widgets, I want to pass back an identifier for the just created widget, not a 200 code that means "request accepted and queued but no data available right now". The uses for APIs that don't return data seem pretty limited in our realm (SaaS). One example is handling incoming notifications/webhooks from other systems. By their nature these are asynchronous and the caller (the external system) does not usually expect a meaningful response, so for sure we can use SQS to buffer the incoming API calls. But for the APIs that we offer, virtually all of them have an online response. In my experience this is not an unusual pattern (for SaaS APIs anyway). I feel that these technologies are somewhat biased towards super scale systems that are highly decoupled and asynchronous - perhaps the sort of systems that Amazon are used to dealing with in-house - rather than simple old business systems.

wgjordan 6y ago
"The advantage of this pattern is increased application resiliency", as spelled out in the first paragraph. So yes, these technologies are 'biased towards' systems that require this degree of resiliency rather than 'simple old business systems'.
- abraae 6y ago
  Simple old business systems require resilience too.
sk5t 6y ago
> not a 200 code that means "request accepted and queued but no data available right now" Properly that would be HTTP 202 Accepted :)

talawahtech 6y ago

This is awesome and long overdue. Providing a simple way to go directly from an HTTP endpoint to SQS, Event Bridge or Kinesis makes it much easier to build more robust webhooks or tracking endpoints. I have tried doing similar things in the past,using API Gateway and SQS without Lambda, and it was a huge pain. I wish they had skipped coining the "storage-first" buzzword though. I mean, I get it, but I feel like it hurts more than it helps. My first assumption was that they were just talking about writing data to S3 and then using Lambda to process after.

deegles 6y ago

How is the the SQS pattern bidirectional?

koluna 6y ago

This is all nice, but here is the important part you gotta remember - the more you take dependencies on AWS and its integrations, the harder it will be to actually have any flexibility and say in your own systems design. AWS is interested in taking your money first and foremost as well as having you depend on it from one side to another of your stack. Instead I’d look at containerizing my workflows and only using cloud tools on the edge (like load balancing).