3 ms·
You can try it for yourself! It's my experience that people use the term "fast" and "faster" differently than I do. Go get some data[1] and see how long it ta
by geocar 10y ago
You can try it for yourself!
It's my experience that people use the term "fast" and "faster" differently than I do.
Go get some data[1] and see how long it takes you to process 600 million rows of anything. For example, take the 100 most popular symbols and find their last bid price.
Once it's in memory, top open source SQL engines take 5-10 seconds, but writing it in C you can do it within 20msec.
I'd be really curious to see how fast .net actually is: I'm told Spark took 35 seconds to work that out, and it's referred to as "Lightning-fast".
I don't use magnitudes except when being illustrative because multiplying magnitudes like 10x and 50x become more exaggerated as the data gets bigger. When the cpu stalls, and waits for memory to come in, or the "runtime" takes a diversion to garbage collect some things, we're really talking about doing things that don't need to be doing.
When I say something is "fast", I usually mean it isn't doing anything that doesn't need to be done, or I mean I can justify everything that it is doing that doesn't need to be done (as, for example, waiting for other slower things, or making it easier for me to write).
.net may do fewer things that don't need doing than (say) perl, but programming in C we can actually make "fast" programs; we can simply choose to not do anything that doesn't need doing, and that means .net will never catch up.
[1]: ftp://ftp.nyxdata.com/Historical%20Data%20Samples/Daily%20TAQ/
- cm2187 10y agoIt would be interesting to run the test, but for that it would take someone proficient in both C and .net to run a consistent algorithm on the same machine (I can only do the latter). But to just process raw data, I'd expect .net to be pretty fast. There is a bit of an overhead for the JIT compilation, but at the end you run machine code. You will loose a little bit of performance for things like array boundary checking, but surely it can't be more than a fraction. I would assume differences would appear when you are running a problem that requires a lot of memory allocation / de-allocation, creating and deleting objects.
- pjmlp 10y agoHow many programming hours were spent writing the 20ms C program vs the 5-10s in other programming languages. Do those 5-10s vs 20ms actually matter for the data consumer in the overall use case scenario?
- geocar 10y ago> How many programming hours were spent writing the 20ms C program vs the 5-10s in other programming languages. Sorry for the trick, but it actually wasn't C at all, but a simple, non-optimising interpreted language that beats the pants off all these fancy optimising compilers who can work around shit programmers making shit programs. The <20msec solution written in K is as follows: select last bid by sym from quote where date=d,sym in S I'd love to talk about how to write programs that are fast, but by all means, let's talk about how we don't really need our programs to be fast in the first place.