8 ms·

Serving 6.8M requests per second at 9 Gbps from a single Azure VM

127 points by profquail 11y ago

ckluis 11y ago
If possible - they should try and fix the C#/windows techempower implementations which are not anywhere near their results.
- benaadams 11y ago
  (Author of article) Working on it, currently the #1 contributor to AspNet Core's new webserver, and it is doing about 2.5M rps on equivalent techempower hardware https://github.com/aspnet/KestrelHttpServer/graphs/contributors https://github.com/aspnet/KestrelHttpServer/graphs/contribut...
  - ckluis 11y ago
    It looks like they are freezing techempower’s next round in a couple weeks. And congrats on the awesome work!
yread 11y ago
The actual code they're using https://github.com/benaadams/benchmarks/blob/managed-rio-experimental/experimental/ManagedRIOHttpServer/RIOServer.cs https://github.com/benaadams/benchmarks/blob/managed-rio-exp...
SEJeff 11y ago
They had me until the moment where I read "While our game servers are built more like HFT Quant trading servers" yet they are in the cloud... They clearly have no idea what they're talking to regarding finance, where latency matters a whole lot more than throughput. "HFT Quant Trading Servers" optimize things down to the micro and even nanosecond level from the hardware to the bios all the way up to the networking interconnects and applications + c libraries. Optimization to this level is simply impossible from the cloud, where even with VT and SRIOV, the performance penalty is simply far too high (and measurably so!). Disclaimer: I'm a technologist who has worked in the "HFT Quant / Electronic Trading" Industry the past 8 or so years. I literally build infrastructure like this for a living.
- happytrails 11y ago
  Someone has to be a buzzkill :(
  - atom_enger 11y ago
    Buzzkill or a realist? Personally I prefer when someone more knowledgeable on the topic corrects me about a statement I've made. If the OP is making analogies that aren't comparable, I'd rather know than not so I can dig in with a healthy bit of skepticism.
    - corysama 11y ago
      SEjeff is seriously over-reacting to reject the entire article on the basis of a bad analogy that isn't even central to the material. If the article was pushing this system for use in HFT, that would be bad. But, this is just someone who doesn't work in the field that SEjeff is very, very deeply involved in mentioning that field offhand, in a clumsy way, once.
      - ceedan 11y ago
        bingo~
- brianwawok 11y ago
  Seconded, where is the Solar Flare card bypassing the OS for network ops? (Does that even work in windows? Doubtful)
  - benaadams 11y ago
    Uses the Registered Input/Output (RIO) API Extensions in Windows for user mode TCP/IP kernel bypass. PacketDirect is the next step for Server 2016 which moves closer to the NIC https://www.youtube.com/watch?v=KaXfDjIhn0U https://www.youtube.com/watch?v=KaXfDjIhn0U
    - 11y ago
vvanders 11y ago
TCP for twitch game? Ugh. Cool that you're getting a ton of throughput but you don't use UDP for throughput, you use it so that the one dropped packet doesn't back up all of your time-sensitive data. You want to drop packets that are out of order since you're using dead-reckoning to keep the game state psuedo-in-sync. Seriously, there's a reason everyone uses UDP(or IPX!) since the days of doom.
- corysama 11y ago
  The whole article is about doing a test to see if they can get away with using TCP instead of UDP. They end up pretty pleased with the results. Though, they don't spend nearly as much time talking about latency as they do about throughput. > For our high throughput needs, prevailing wisdom would suggest you need to write your server in C++, use UDP rather than TCP and run on a very high spec bare metal box running linux. We are already running the client in javascript; so if we ran over TCP (websockets), wrote the server code in managed C#, ran on Windows on a VM in the cloud – is this just madness?
- benaadams 11y ago
  Its a constraint of the browser technology stack; not only do we have to use websockets for real-time (TCP/IP) but we also have to go over SSL/TLS as intermediary hardware/software (routers, proxies etc) still routinely break websocket connections otherwise. (Disclaimer: Author of article)
  - vvanders 11y ago
    I really hope your game designers know how to mask latency. If you're going over TCP/IP then expect to need to resolve a game model that can sustain at least ~750ms of latency. I totally understand the browser constraint part but you're going to hit some serious issues once you start having real-world clients that have >3% packet drop. It gets even worse when you start adding different discrete layers on a single TCP connection(chat, larger game state not related to twitch parts, etc). Each one of those channels has the chance to drop a single packet and then your whole world comes to a stop until you can roundtrip that single packet that's unrelated to the realtime info you care about. FWIW most stacks end up implemented channels over UDP with unreliable+unordered, reliable+unordered and reliable+ordered for the different latency needs of the specific channel. I've done a fair bit of this stuff in a production environment and have to echo my original comment. TCP is great for lockstep but if you're doing dead-reckoning it will fall apart on anything other than a LAN. Edit: I also don't mean this as a dig, just that I've seen many teams go through this and it's usually at a point where it's too late to make any changes.
nacs 11y ago
The article has small bits of useful info but the article and site reeks of submarine (Microsoft Azure) advertising for me.. Their homepage has a testimonial/review-blurb of the game as the very first thing from a Microsoft "White paper". Below that there are 2 videos, one of which is an "Interview Rob Fraser - Microsoft". And of course the article itself focuses on Azure with another 12 mentions of Azure and how it runs Windows, .Net and C# so well (I'd hope so). Also, the 7 mill requests/second aren't real world numbers but just a software benchmark. Oh and the benchmark was run on an instance that has 16 cores, 3TB of SSD storage, and 224 GB of RAM. That's more like a dedicated server than a single cloud server instance (and waay more expensive than dedicated at $3061 per month per instance).
- brazzledazzle 11y ago
  Microsoft may have given them free or heavily discounted access in exchange for being able to use them to advertise. That was apparently the case with TitanFall.
pjmlp 11y ago
Very interesting post. I love that they could have the resources and time to explore using safer languages for their use cases.
jtwebman 11y ago
So wait was this all just serving Hello World?
- profquail 11y ago
  That's basically an echo server, which is a standard networking test. Leaving out the overhead of actually processing the payload means you're able to see how well the networking stack (and the parts of your program which interface with and manage it) can perform. Getting a clean number from a benchmark like this also makes it easier to estimate how much performance overhead is induced by the other parts of your program.
  - jtwebman 11y ago
    I think it is just a fake number. Everything has running code also what if the VM you spin up is in a different part of the data center or next to another heavy network app? To me this seems like a waste of time to look at because depending on the logic it will not matter if the server can do twice as many hello world requests when it has to wait 30ms for business logic anyways. I want to see a real running system and the change out the web server as well as the hosting vm and see who comes out ahead with real systems. This just shows use this web server for making a CDN, nothing more.

127 points by profquail 11y ago

If possible - they should try and fix the C#/windows techempower implementations which are not anywhere near their results.

benaadams 11y ago
(Author of article) Working on it, currently the #1 contributor to AspNet Core's new webserver, and it is doing about 2.5M rps on equivalent techempower hardware https://github.com/aspnet/KestrelHttpServer/graphs/contributors https://github.com/aspnet/KestrelHttpServer/graphs/contribut...
- ckluis 11y ago
  It looks like they are freezing techempower’s next round in a couple weeks. And congrats on the awesome work!

yread 11y ago

The actual code they're using https://github.com/benaadams/benchmarks/blob/managed-rio-experimental/experimental/ManagedRIOHttpServer/RIOServer.cs https://github.com/benaadams/benchmarks/blob/managed-rio-exp...

SEJeff 11y ago

They had me until the moment where I read "While our game servers are built more like HFT Quant trading servers" yet they are in the cloud... They clearly have no idea what they're talking to regarding finance, where latency matters a whole lot more than throughput. "HFT Quant Trading Servers" optimize things down to the micro and even nanosecond level from the hardware to the bios all the way up to the networking interconnects and applications + c libraries. Optimization to this level is simply impossible from the cloud, where even with VT and SRIOV, the performance penalty is simply far too high (and measurably so!). Disclaimer: I'm a technologist who has worked in the "HFT Quant / Electronic Trading" Industry the past 8 or so years. I literally build infrastructure like this for a living.

happytrails 11y ago
Someone has to be a buzzkill :(
- atom_enger 11y ago
  Buzzkill or a realist? Personally I prefer when someone more knowledgeable on the topic corrects me about a statement I've made. If the OP is making analogies that aren't comparable, I'd rather know than not so I can dig in with a healthy bit of skepticism.
  - corysama 11y ago
    SEjeff is seriously over-reacting to reject the entire article on the basis of a bad analogy that isn't even central to the material. If the article was pushing this system for use in HFT, that would be bad. But, this is just someone who doesn't work in the field that SEjeff is very, very deeply involved in mentioning that field offhand, in a clumsy way, once.
    - ceedan 11y ago
      bingo~
brianwawok 11y ago
Seconded, where is the Solar Flare card bypassing the OS for network ops? (Does that even work in windows? Doubtful)
- benaadams 11y ago
  Uses the Registered Input/Output (RIO) API Extensions in Windows for user mode TCP/IP kernel bypass. PacketDirect is the next step for Server 2016 which moves closer to the NIC https://www.youtube.com/watch?v=KaXfDjIhn0U https://www.youtube.com/watch?v=KaXfDjIhn0U
  - 11y ago

vvanders 11y ago

TCP for twitch game? Ugh. Cool that you're getting a ton of throughput but you don't use UDP for throughput, you use it so that the one dropped packet doesn't back up all of your time-sensitive data. You want to drop packets that are out of order since you're using dead-reckoning to keep the game state psuedo-in-sync. Seriously, there's a reason everyone uses UDP(or IPX!) since the days of doom.

corysama 11y ago
The whole article is about doing a test to see if they can get away with using TCP instead of UDP. They end up pretty pleased with the results. Though, they don't spend nearly as much time talking about latency as they do about throughput. > For our high throughput needs, prevailing wisdom would suggest you need to write your server in C++, use UDP rather than TCP and run on a very high spec bare metal box running linux. We are already running the client in javascript; so if we ran over TCP (websockets), wrote the server code in managed C#, ran on Windows on a VM in the cloud – is this just madness?
benaadams 11y ago
Its a constraint of the browser technology stack; not only do we have to use websockets for real-time (TCP/IP) but we also have to go over SSL/TLS as intermediary hardware/software (routers, proxies etc) still routinely break websocket connections otherwise. (Disclaimer: Author of article)
- vvanders 11y ago
  I really hope your game designers know how to mask latency. If you're going over TCP/IP then expect to need to resolve a game model that can sustain at least ~750ms of latency. I totally understand the browser constraint part but you're going to hit some serious issues once you start having real-world clients that have >3% packet drop. It gets even worse when you start adding different discrete layers on a single TCP connection(chat, larger game state not related to twitch parts, etc). Each one of those channels has the chance to drop a single packet and then your whole world comes to a stop until you can roundtrip that single packet that's unrelated to the realtime info you care about. FWIW most stacks end up implemented channels over UDP with unreliable+unordered, reliable+unordered and reliable+ordered for the different latency needs of the specific channel. I've done a fair bit of this stuff in a production environment and have to echo my original comment. TCP is great for lockstep but if you're doing dead-reckoning it will fall apart on anything other than a LAN. Edit: I also don't mean this as a dig, just that I've seen many teams go through this and it's usually at a point where it's too late to make any changes.

nacs 11y ago

The article has small bits of useful info but the article and site reeks of submarine (Microsoft Azure) advertising for me.. Their homepage has a testimonial/review-blurb of the game as the very first thing from a Microsoft "White paper". Below that there are 2 videos, one of which is an "Interview Rob Fraser - Microsoft". And of course the article itself focuses on Azure with another 12 mentions of Azure and how it runs Windows, .Net and C# so well (I'd hope so). Also, the 7 mill requests/second aren't real world numbers but just a software benchmark. Oh and the benchmark was run on an instance that has 16 cores, 3TB of SSD storage, and 224 GB of RAM. That's more like a dedicated server than a single cloud server instance (and waay more expensive than dedicated at $3061 per month per instance).

brazzledazzle 11y ago
Microsoft may have given them free or heavily discounted access in exchange for being able to use them to advertise. That was apparently the case with TitanFall.

pjmlp 11y ago

Very interesting post. I love that they could have the resources and time to explore using safer languages for their use cases.

jtwebman 11y ago

So wait was this all just serving Hello World?

profquail 11y ago
That's basically an echo server, which is a standard networking test. Leaving out the overhead of actually processing the payload means you're able to see how well the networking stack (and the parts of your program which interface with and manage it) can perform. Getting a clean number from a benchmark like this also makes it easier to estimate how much performance overhead is induced by the other parts of your program.
- jtwebman 11y ago
  I think it is just a fake number. Everything has running code also what if the VM you spin up is in a different part of the data center or next to another heavy network app? To me this seems like a waste of time to look at because depending on the logic it will not matter if the server can do twice as many hello world requests when it has to wait 30ms for business logic anyways. I want to see a real running system and the change out the web server as well as the hosting vm and see who comes out ahead with real systems. This just shows use this web server for making a CDN, nothing more.