12 ms·

Writing scalable backends in UDP

• mannyv 3y ago
  I forgot why we were doing this again hahaha. What's the point of this exercise? I mean it's fun, but I don't need another job right now. Is it a puzzle?
  • gafferongames 3y ago
    It's a puzzle. Getting to the solution requires some pretty creative thinking. Nobody in the comments has posted the solution yet (or come close to it).
    • mannyv 3y ago
      I don't think anyone who can do it has enough free time to do it. It'd be fun, but I'd rather be figuring out how to make box joints on my table saw. I do want to see how the solution code handles IPv6 client ingress issues, though.
      • gafferongames 3y ago
        I agree. I'd rather be gardening, but my version of the solution is almost done :)
• wuiheerfoj 3y ago
  Now that QUIC is widely supported this is probably a lot easier than in the days of yore, though 100m rps is no joke
  • iforgotpassword 3y ago
    How would quic help here? It's relatively much overhead compared to this "one packet in, one packet out" scenario.
• neonsunset 3y ago
  The restriction to Go is downright self-defeating. There are better languages that allow much higher performance ceiling for this kind of task. Also seconding notes regarding encryption and the use of QUIC.
  • gafferongames 3y ago
    I agree, although it's actually quite easy to do it in Golang once you have the right approach. I asked the question in Golang because this was an interview question for Network Next, and the Network Next backend is implemented in Golang.
    • neonsunset 3y ago
      I would not touch Go, thanks. Either Rust or C# would be much easier in terms of vertical scaling strategy to reduce replica count and increase individual pod utilization - both languages offer better tools for scaling on individual many-core instances, reaching much higher throughput. Edit: (though in wider context naturally the switch away from existing stack in a company would never make sense as long as it is a compiled language with sufficiently good average performance)
      • gafferongames 3y ago
        You're more than welcome to demonstrate a solution in your language of choice
        • neonsunset 3y ago
          If that's the case given interview scenario, then I withdraw my complaints :)
  • Sean-Der 3y ago
    Go isn't the problem. The issues I have seen is that you are I/O bound by the kernel. One company did Pion WebRTC + https://www.dpdk.org/ https://www.dpdk.org/ and is now doing 4x throughput. They tried switching to Rust (and still doing userspace networking) and it didn't make a big difference.
• ClumsyPilot 3y ago
  I mean this is cool, but where is the solution/ answer/ approach? Where is info about encryption, for example - very few libraries support DTLS or any UDO compatible encryption at all.
  • gafferongames 3y ago
    A complete worked solution with source code and terraform scripts will be published April 16th (one week from today). This gives everybody on Hacker News time to come up with their own solution, if they want to try. When I asked this as an interview question, I gave candidates permission to implement this without encryption (although asking about encryption is definitely +1). Note that the response size is significantly smaller than the request size (intentional), so you don't have to worry about being used in UDP amplification attacks. (Being concerned and asking about UDP amplification is another +1)
    • gafferongames 3y ago
      The full solution is here: https://github.com/mas-bandwidth/udp/blob/main/001/README.md https://github.com/mas-bandwidth/udp/blob/main/001/README.md
• sheepscreek 3y ago
  Why is this idea so out there? Most streaming used to happen over UDP. It might still. Anytime you have a use-case where latency trumps reliability (and security), you should use UDP. That brings me to another point: HTTPS isn’t preferred because it’s the defacto standard now, it’s the secure bit, the “S” in HTTPS, which makes it a better option than using plain HTTP over TCP. By going completely bespoke means you’ll need to implement your own protocol that implements some level of security and redundancy in UDP communication.
  • deleted 3y ago
    [deleted]
  • hawk_ 3y ago
    > latency trumps reliability Let's say I don't care about the initial handshake latency, have disabled nagle and have no packet loss. Is there still a measurable latency difference between UDP and TCP? It would appear that there is (based on yours and other comments) - but I don't know why.
    • deleted 3y ago
      [deleted]
    • Joel_Mckay 3y ago
      Primarily traffic cost, CDN services were much more important during the early days... and punching through NAT came with its own liabilities.
    • fulafel 3y ago
      No, assumign an already estabilished connection, TCP is just as fast if there's no packet loss, except for the tiny amount of time that the additional TCP stack logic takes on the CPU. See eg here: https://cloud.google.com/blog/products/networking/using-netperf-and-ping-to-measure-network-latency https://cloud.google.com/blog/products/networking/using-netp...
      • vetrom 3y ago
• bjornsing 3y ago
  The UDP side of this doesn’t sound very hard. Most challenging part is probably the outbound HTTP requests (if I understood the problem correctly?).
  • gafferongames 3y ago
    You are absolutely correct.
    • liendolucas 3y ago
      Not related to the discussion but wanted say that I came across your username and rung a bell on my head. I've read and used the information of one of your posts (https://www.gafferongames.com/post/fix_your_timestep/ https://www.gafferongames.com/post/fix_your_timestep/) and found it very well written and useful!
      • gafferongames 3y ago
        Thank you sir! I've stopped writing on gafferongames.com and will be writing on mas-bandwidth.com from now on, focusing more on scalable backend techniques fused with multiplayer games. I wrote a post about XDP for games last week as well: https://mas-bandwidth.com/xdp-for-game-programmers/ https://mas-bandwidth.com/xdp-for-game-programmers/
  • mannyv 3y ago
    The UDP part is hard because load balancers generally don't deal with UDP. And you can't trigger lambdas via udp (i'd have to check this, because i've never tried). Sending the http request is easy, stuff incoming requests into an sqs queue from UDP side then have a lambda eating queue entries and sending them to the client. Because it's udp you don't need to keep endpoint state. If api gateway doesn't do UDP you could network load balance the udp to a ecs autoscaling group with your udp listener in it. You'd have to figure out what metrics you need to trigger autoscaling. You could also just do route53 dns routing and have some oversized instances as your handler; it depends on your budget + how much time you have.
• throwaway63467 3y ago
  We used a similar challenge which tasked a candidate to build a VPN over UDP (without the complex / annoying bits like the TUN interface). If you ignore encryption it’s rather trivial. I guess in this problem the challenge is mostly the backend connection, otherwise you could possibly do this on a single VM though Golang wouldn’t be my first choice for that, the GC becomes quite annoying when processing streams at high frequency.
  • dilyevsky 3y ago
    Would you accept https://github.com/WireGuard/wireguard-go/blob/master/tun/netstack/examples/http_server.go https://github.com/WireGuard/wireguard-go/blob/master/tun/ne... as a solution? =) Edit: also i personally find TUN very convenient.
• jolux 3y ago
  Answer: HTTP/3
  • gafferongames 3y ago
    Cool. Show me
• Joel_Mckay 3y ago
  One essentially pushes the rate limiting token bucket and session management up the ISO/OSI stack. Things like iodine tunnels can trip IDS/smart-routers/firewalls to flag hardware for malformed traffic. Seen people try this before, and it ends rather predictably unreliable/slow.
• dilyevsky 3y ago
  > While I'm confident that an experienced senior engineer could find a solution over a weekend Over weekend? Either I'm misunderstanding it (are you supposed to write your own NAT-pinning solution? TURN?) or this sholudn't take more than couple of hours.
  • deleted 3y ago
    [deleted]
  • gafferongames 3y ago
    Cool. Show me
    • dilyevsky 3y ago
      I actually received something similar for take-home challenge years ago so it may not be as "out there" as you think it is =) (at least if you work in telecom/networking space)
      • gafferongames 3y ago
        The specific challenge here is: "You are tasked with creating a client/server application in Golang that runs in Google Cloud. The client in this application must communicate with the server over UDP. Each client sends 100 requests per-second. Each request is 100 bytes long. The server processes each request it receives and forwards it over HTTP to the backend. The backend processes the request, and returns a response containing the FNV1a 64 bit hash of the request data. The server returns the response it receives from the backend down to the client over UDP. Implement the client, server and backend in Golang and Terraform such that it scales to more than 1M clients. Provide estimates of the cost to run the solution each month at a steady load of 1M clients, as well as some options you would recommend as next steps to reduce the cost. IMPORTANT: When you load test this system, make sure you are communicating between the client, server and backend using internal addresses to avoid egress bandwidth charges."
        • dilyevsky 3y ago
• eikenberry 3y ago
  > While I'm confident that an experienced senior engineer could find a solution over a weekend, What they mean to say is ".. an experienced senior engineer *with my exact skill set* could ..." Senior engineers vary wildly in their skills. Would you expect a senior engineer with experience with (to pick a similar set) Ansible, Rust and AWS to implement this over the weekend using Golang, Terraform and GCP? Obviously not and, just as obviously, you don't need such a wide difference to quickly bleed from "over a weekend" to "over a week". These sorts of bizarrely complex and specific code challenges would at least serve to greatly reduce the number of resumes you have to review. So that is one upside. Unfortunately what that selects for is probably not as obvious as it would seem as many of their candidates would spend that full week on the problem and I wouldn't apply for fear of a workplace culture that values their employees time that little.
  • gafferongames 3y ago
    What I mean to say is that we're hiring Golang engineers. I mean, is that OK? To have a technology stack that you are hiring for (Google Cloud, Golang, Terraform etc), and to look for candidates with experience with that stack, or who would consider learning that stack? ps. This interview question was designed to also be a great question to learn that stack with, but I would not expect a senior programmer would be able to do so in one weekend -- hence I gave programmers as much time as they wanted, in many cases, several months to play around with the stack and see if they enjoyed it in their spare time. Some engineers even asked for some budget to play around with it in Google Cloud and I provided it. Or should we just code each service in Rust or Elixir or Node or F# or whatever language stack a candidate prefers that day?
    • eikenberry 3y ago
      It's not the tech they are using but the size of the project. This project requires simple tasks but spread over multiple technologies such that most developers won't have experiences with all (or even many) of them. The key thing here is the time. Having just done a ton of interviewing I can tell you that almost every code challenge has some form of this issue and so you learn to just multiply the time they say by 3-4 (as a rough average). Thus taking ones that say 'a few hours', which means a day or two at most, are feasible but the ones that say 'a day or two' or 'a weekend' are not. Also remember candidate are usually not only applying to your job. They will easily have 3-4 of these going in parallel and with each taking up that much time is untenable.
• beoberha 3y ago
  I’m more curious about the effectiveness of this question as an interviewer. I’d have to imagine you get a fairly strong signal to noise ratio as whoever takes the time to get a working solution must be pretty qualified for your role. That said, you are probably leaving a ton of talent on the table by asking them to do (relatively) so much work.
  • gafferongames 3y ago
    It's definitely a pretty serious filter, and it's designed to be. The cost of hiring somebody who just cannot be effective with our stack is so high, that it's important that we test for somebody being able to learn our stack, and really wanting to work at Network Next. This question did a very good job of that.
    • gafferongames 3y ago
      I would also mention that there is a type of programmer, that we definitely do want to hire, that finds a challenge like this really easy, might even bang something out in a few hours in a different stack, and show us something amazing. We were open to that as well.
      • gafferongames 3y ago
        As an example for that sort of programmer, consider an engineer at the caliber and experience level of Marek @ Cloudflare, who would absolutely eat a question like this for breakfast... https://blog.cloudflare.com/author/marek-majkowski https://blog.cloudflare.com/author/marek-majkowski This question is designed to help find those people, in addition to people who are really willing to learn the stack with no experience and who are excellent problem solvers.
• Matthias247 3y ago
  I've been working in that domain for a long time - including being one of the main architects for HTTP/3+QUIC in a public CDN offering. And I'll agree with everyone that this is very niche question, and a great answer seems out of reach for most "senior engineers". Translating a UDP packet into a HTTP request and back is reasonably easy. Yes, maybe one can do that in a coding interview with some pseudo code. But scaling it and making it reliable is yet another dimension. Any candidate would need to understand that a single UDP socket itself would probably already a bottleneck for just running this on single machine, and figure out how to increase concurrency there. That's not easy - the amount of engineers knowing how SO_REUSEPORT works and when it doesn't work is low. After that you start to dig into how you can actually spread load between hosts. Would an answer like I "I hope my cloud provider solves this for me" be good enough? Probably not. If it actually is, do candidates have to both understand the cloud providers native APIs and Terraform (mentioned in the blog post). Seems pretty unnecessary, terraform is just one tool out of the myriad of tools which can be used to configure cloud services. Not everyone will have used it before. Or would it even expect candidates to do a long writeup about the pro's and con's of client-side load balancing? Are applicants required to talk about upstream connection pooling? Describe and implement a CDN like multi-tier architecture? Last but far from least is that the requested architecture is very easy to misuse for denial of service and amplification attacks. Just being able to describe how to monitor and mitigate that is already a very very hard task, that very few specialists have worked on so far. It's very fuzzy what would be good enough if this is a "homework task". At least in a synchronous interview the interviewer could give feedback that they are satisfied. So I think in a synchronous interview the question might be ok - but there will probably just be time to either talk about coding or about system architecture.
  • gafferongames 3y ago
    Generally I've provided this as a homework task, with the ability to email me and ask specific questions to help guide the candidate over a period of whatever time the candidate wants. There are definitely degrees of correctness and completeness and depending on the candidate experience and level, certain solutions are acceptable. For example, a totally naive implementation in golang that doesn't quite hit the scalability requirements would be a good conversation starter and would pass a mid level or junior candidate. A senior or above "badass" candidate would be expected to hit the scalability requirements. An incredible candidate would teach us something new about this problem that we don't already know.