4 ms·
We recently used API gateway to publish our first "public" facing API for some of our partners to integrate with. We we're using swagger to describe our interna
by josiahpeters 9y ago
We recently used API gateway to publish our first "public" facing API for some of our partners to integrate with. We we're using swagger to describe our internal APIs. A subset of those APIs needed to be exposed to partners. We also wanted to enforce a different security model and leverage rate limiting for these requests to protect us from unexpected growth.
Integrating API gateway into our deployment took a little bit of work, but we're happy to have it part of our automated deployment process. We push updates of our swagger spec with AWS CLI to a separate API per environment (dev, staging, and prod-west, prod-east). Each of environment has one API gateways with several stages that we use for green/blue deployments. The stages are green, blue, and public. We deploy to the inactive color (green or blue) smoke test and promote that stage to public. This simulates the same process we do in production and the swap to public is akin to our DNS swap from one side to the other. We were very happy that it wasn't a huge time investment to fit this workflow into our deployment process.
Somehow we missed the fact that because API gateway was tied to CloudFront that it did not support multi regional failover. If we have a service outage in our primary region we swap all of our traffic with DNS to our secondary region (from us-west-2 to us-east-1). In order for us to do a regional failover with API gateway we would need to deprovision the endpoint api.domain.com in the failing region and provision api.domain.com in our failover region. I'm expecting if we have to do a regional failover there is a very good chance that the request to deprovision the endpoint will likely fail. See my AWS forum reply and other's asking for the same feature: https://forums.aws.amazon.com/message.jspa?messageID=761925#761925 https://forums.aws.amazon.com/message.jspa?messageID=761925#...
My team is very much looking forward to this functionality. We have duplicated our infrastructure across multiple regions to mitigate a region outage. We were planning on relying on a dns change to route traffic to a new region.
With the limitations that you have described, our failover process is much more complicated and uncertain.
Given the two regions primary (us-west-2) and secondary (us-east-1). We will need to attempt to disassociate our api.domain.com from primary API Gateway and wait for that operation to complete. After that we must associate our secondary API Gateway with api.domain.com. Its very possible and likely issues on primary that caused us to fail over to secondary may prevent the disassociating from the primary. This is an extremely undesirable side effect as it could leave us with unhandled requests or error responses to api.domain.com.
I wanted to post this here for others suggest alternative workarounds or ideas. We will also follow up with a support ticket.
They are aware of the issue and it seems like something on their roadmap, but that could mean 1-2 years out. Thankfully this partner API doesn't have near the same SLA and HA requirements we have for other features of our SaaS product. If our SLA requirements were higher we would need to consider a different solution. We have handwaved some ideas that we could use if there was an extensive outage which likely equates to standing up a reverse proxy that listens on api.domain.com and proxies traffic to the API gateway url for our secondary region. That certainly isn't ideal and doubles the latency that we have with API Gateway already proxying requests to our internal API.