3 ms·
As someone who has worked on HFT systems for a decade, if there is one piece of advice I can give on the matter, it is this: test with production-like data. Hav
by hliyan 11y ago
As someone who has worked on HFT systems for a decade, if there is one piece of advice I can give on the matter, it is this: test with production-like data. Having processes, 'performance-culture' and unit-wise perf testing is all good and fine, but unless you test your features with production-like loads, you can never be sure.
I've seen features that were regression/unit tested and code-reviewed to death fail in production because all that testing was done at < 10 transactions/sec with a dataset of about 100, when typical production values were in the neighborhood of 1000/tx/sec with a dataset of perhaps 10,000.
At the very beginning of the design process, someone needs to ask, "How frequently will this feature get invoked? What kind of latency can the user tolerate?"
- clevernickname 11y agoVery true, and a huge part of why, say, game engines have a reputation for being efficient, and, say, web software has a reputation for being inefficient. Performance clearly matters (or should matter) to both (dictating what a game can do, or dictating how much it costs to run your web servers and thus remain profitable while sustaining growth), yet it's a lot easier to throw more data into a game until it breaks than it is to simulate a million web sessions.
- amelius 11y agoBut what if testing with production-like data takes 1000x the time of testing with faked-test-data?
- hliyan 11y agoThen one makes the judgement call: cost of testing w/ production-like data > cost of potential production issues and decide whether it's prudent to omit or scale down the tests.
- pjc50 11y agoWell, if it's a real-time system it shouldn't take too long to test with production-like data; and if it does take too long that probably indicates a problem. For an HFT system I'd have the dev version doing overnight testing with the day's data or chunk thereof, in delayed-realtime mode.
- brianwawok 11y agoSpending time on them exchange side, prod like data isn't enough. We had bugs not hit with prod like data hit us in prod. To get the real truth, test in prod. Obviously you need good infrastructure to do so.. But you can't fake a prod exchange. Too much bandwidth and too many users each on their own agenda.