4 ms·

The UDP side of this doesn’t sound very hard. Most challenging part is probably the outbound HTTP requests (if I understood the problem correctly?).

gafferongames 3y ago
You are absolutely correct.
- liendolucas 3y ago
  Not related to the discussion but wanted say that I came across your username and rung a bell on my head. I've read and used the information of one of your posts (https://www.gafferongames.com/post/fix_your_timestep/ https://www.gafferongames.com/post/fix_your_timestep/) and found it very well written and useful!
  - gafferongames 3y ago
    Thank you sir! I've stopped writing on gafferongames.com and will be writing on mas-bandwidth.com from now on, focusing more on scalable backend techniques fused with multiplayer games. I wrote a post about XDP for games last week as well: https://mas-bandwidth.com/xdp-for-game-programmers/ https://mas-bandwidth.com/xdp-for-game-programmers/
mannyv 3y ago
The UDP part is hard because load balancers generally don't deal with UDP. And you can't trigger lambdas via udp (i'd have to check this, because i've never tried). Sending the http request is easy, stuff incoming requests into an sqs queue from UDP side then have a lambda eating queue entries and sending them to the client. Because it's udp you don't need to keep endpoint state. If api gateway doesn't do UDP you could network load balance the udp to a ecs autoscaling group with your udp listener in it. You'd have to figure out what metrics you need to trigger autoscaling. You could also just do route53 dns routing and have some oversized instances as your handler; it depends on your budget + how much time you have.
- mannyv 3y ago
  Well actually NAT on the client side might make this more complicated.
  - gafferongames 3y ago
    Let's just say that there are some challenges on both the UDP side, and on the HTTP forwarding side.
    - mannyv 3y ago
      In the NAT case just keep a list of the active UDP receivers and client IPs/source ports (which you need anyway). Then when the backend processor needs to send a response just sent it to the receiver in question to send back to the client, assuming their NAT works normally. I'd probably stick them in redis, since it's fast and seems to be able to handle some ridiculous number of simultaneous clients. They don't need to be super-persistent since the map can be rebuilt next time a packet comes in. Assuming the UDP packets come in on a timely basis you should be able to connect to the client relatively well. NAT tunnels expire in a minute or two generally, so as long as processing doesn't take more than a few seconds the NAT mapping should be fine. I'd configure the client to send every few seconds, just so if we fail over we can still talk (which latency = those seconds). I'd have to know more about the time envelope that's required tho. Shouldn't take more than 2 weeks to implement/test in node. Learning go might add a week or two.
      - mannyv 3y ago
        NAT makes this a bit more complicated because you can't use a load balancer with NAT for outgoing packets since AFAIK you can't send stuff to the LB and tell it what source port to use. IPv6 makes it easier and harder. I'm not sure what happens if the ipv6 address isn't reachable due to firewall ingress rules, but I'm pretty sure you can't poke a hole in the NAT that doesn't exist. I would like to think that most home users prevent ipv6 ingress, but I have no way to test that. It'll be interesting to look at the solution and see how they handled that.

gafferongames 3y ago

You are absolutely correct.

liendolucas 3y ago
Not related to the discussion but wanted say that I came across your username and rung a bell on my head. I've read and used the information of one of your posts (https://www.gafferongames.com/post/fix_your_timestep/ https://www.gafferongames.com/post/fix_your_timestep/) and found it very well written and useful!
- gafferongames 3y ago
  Thank you sir! I've stopped writing on gafferongames.com and will be writing on mas-bandwidth.com from now on, focusing more on scalable backend techniques fused with multiplayer games. I wrote a post about XDP for games last week as well: https://mas-bandwidth.com/xdp-for-game-programmers/ https://mas-bandwidth.com/xdp-for-game-programmers/

mannyv 3y ago

The UDP part is hard because load balancers generally don't deal with UDP. And you can't trigger lambdas via udp (i'd have to check this, because i've never tried). Sending the http request is easy, stuff incoming requests into an sqs queue from UDP side then have a lambda eating queue entries and sending them to the client. Because it's udp you don't need to keep endpoint state. If api gateway doesn't do UDP you could network load balance the udp to a ecs autoscaling group with your udp listener in it. You'd have to figure out what metrics you need to trigger autoscaling. You could also just do route53 dns routing and have some oversized instances as your handler; it depends on your budget + how much time you have.

mannyv 3y ago
Well actually NAT on the client side might make this more complicated.
- gafferongames 3y ago
  Let's just say that there are some challenges on both the UDP side, and on the HTTP forwarding side.
  - mannyv 3y ago
    In the NAT case just keep a list of the active UDP receivers and client IPs/source ports (which you need anyway). Then when the backend processor needs to send a response just sent it to the receiver in question to send back to the client, assuming their NAT works normally. I'd probably stick them in redis, since it's fast and seems to be able to handle some ridiculous number of simultaneous clients. They don't need to be super-persistent since the map can be rebuilt next time a packet comes in. Assuming the UDP packets come in on a timely basis you should be able to connect to the client relatively well. NAT tunnels expire in a minute or two generally, so as long as processing doesn't take more than a few seconds the NAT mapping should be fine. I'd configure the client to send every few seconds, just so if we fail over we can still talk (which latency = those seconds). I'd have to know more about the time envelope that's required tho. Shouldn't take more than 2 weeks to implement/test in node. Learning go might add a week or two.
    - mannyv 3y ago
      NAT makes this a bit more complicated because you can't use a load balancer with NAT for outgoing packets since AFAIK you can't send stuff to the LB and tell it what source port to use. IPv6 makes it easier and harder. I'm not sure what happens if the ipv6 address isn't reachable due to firewall ingress rules, but I'm pretty sure you can't poke a hole in the NAT that doesn't exist. I would like to think that most home users prevent ipv6 ingress, but I have no way to test that. It'll be interesting to look at the solution and see how they handled that.