6 ms·
Since it is a topic I'm interested in I took the time to read all 4 parts, the author manages to summarize it in a paragraph which would have been helpful at th
by ChuckMcM 10y ago
Since it is a topic I'm interested in I took the time to read all 4 parts, the author manages to summarize it in a paragraph which would have been helpful at the beginning:
When somebody says “Iterating through a linked list is a O(N) operation” what they mean to say is “The number of instructions needed to be executed grows linearly with the size of the list.”. That is a correct statement. The argument I’m trying to make is that it would be a mistake to also assume that the amount of time needed would grow linearly with the size of the list as well. This is an important distinction. If you only care about the number of instructions executed that’s fine, you can use Big-O for that! If you care about the time taken, that’s fine too, and you can use Big-O for that too!
Sadly, he doesn't take this knowledge to its conclusion. Let's introduce the notation Oi() for the Big-O notation in instructions, and Ot() for the Big-O notation for time.
Lemma: For all f(N), if Oi(f(N)) > Oi(g(N)), Ot(f(N) will be > Ot(g(N)).
Or put another way, it's important not to confuse complexity scaling with time scaling, but the more complex the computation, the longer it will take.
- gnud 10y agoDon't think your conclusion holds - and that's actually something the author brings up. It might be quicker in time (not in the amount of instructions executed) to calculate something multiple times, or to do a more complex calculation or what have you, in order to touch less memory.
- DanWaterworth 10y ago> if Oi(f(N)) > Oi(g(N)), Ot(f(N) will be > Ot(g(N)). That doesn't hold. Some instructions take longer than others.
- jacobparker 10y agoIf all instructions complete within some constant deadline independent of N/the input then it does hold even if they take different amounts of time.
- DanWaterworth 10y agoIt doesn't hold for binary tree lookups vs B-tree lookups.
- malisper 10y agoThe hypothesis of the statement doesn't hold. O(n log n) isn't greater than O(n log n).
- DanWaterworth 10y agoYou have to evaluate it in a model with an explicit cache width size, then O(log_b(n)) < O(log(n))
- GauntletWizard 10y agolog_b(n) * log(b) = log(n). log(b) is a constant in our algorithmic analysis, and can therefore be removed. There will never be a log_b(x^2) that grows at slower than log(x), no matter how big b is.
- DanWaterworth 10y agoWhich is why you have to evaluate in a model with an explicit cache width. In such a model, b is not a constant. The cache-oblivious model [1] is a fairly well-known example, but that won't work in this case, since we need to know b to set the tree width. Any of the other external memory models will do. [1] https://en.wikipedia.org/wiki/Cache-oblivious_algorithm https://en.wikipedia.org/wiki/Cache-oblivious_algorithm
- ChuckMcM 10y agoI think the notation is imprecise. The quantity the author describes is commonly called the working set. His explicit thesis is that the time to process the working set is related to the size of the working set regardless of computational complexity. He spends the entire first part of his discussion belaboring this point with a linked list example. He goes on to do a systems analysis of the results and concludes that time of execution is system dependent not complexity dependent. Again using the same algorithm engaging with a larger and larger memory system. Generally engineers would experience this important principal in a more binary way, once their program started swapping to disk, its performance would fall through the floor. If you hold the size of the datum constant (and that is important), then processing fewer of them will take less time than processing more of them. The argument you make, as I understand it is as follows; Consider two algorithms that process N and P datums respectively to achieve the same computational result, and where N > P. There exists an algorithm such that P datums takes less per unit time to process than N. Resulting in a violation of my Lemma because Oi(P) > Oi(N) but Ot(P) < Ot(N). And you cite cache obliviousness as the principle that enables that violation. Would you agree that I have accurately summed up your argument?
- ChuckMcM 10y agoThis is how I would go about stating the proof. 1) The article shows that the time to process N datums of data is related to the size of the total data processed. And that as the size of the data set grows, the time to process it grows with the root of N. 2) Computational complexity's order of magnitude is relative number of datums need to be processed in order to complete the computation. 3) Less complex computations process fewer datums, fewer datums mean a smaller total memory footprint. Therefore, processing fewer datums will lower your total memory processing cost which will lower your total time. Q.E.D.
- SamReidHughes 10y agoQ.E.Nope. Memory can get reused.
- pedrocr 10y agoDoes that really follow? The sqrt(N) factor is the overhead of memory access only, so if you're comparing an algorithm where every operation is a memory access to another where that is not the case it would seem you could have the inverse relation in Oi and Ot. Edit: A counterexample then: Oi(f(N)) = N^(2/3) (memory accessing operations) Oi(g(N)) = N (non memory accessing operations) Ot(f(N)) = N^(7/6) Ot(g(N)) = N So Oi(f(N)) < Oi(g(N)) but Ot(f(N)) > Ot(g(N))
- Kenji 10y agoYour Lemma doesn't hold if the time to execute one instruction is dependent on N - which is pretty much the entire point the blog post makes.
- aschampion 10y agoA complexity measure is a measure of complexity over a model. Mostly we talk about input size models, but we can (and do) talk about other models. It isn't a question of complexity scaling vs. time scaling, but of complexity scaling over an input size model vs. a machine architecture model. There are tons of well-researched machine architectural models for complexity analysis. In the case of the article, any one of the many external memory (EM) or hierarchical memory models would be appropriate. These can capture latencies and bandwidth of hierarchical memories and storage, and in some cases even eviction, contention, partition, etc. This isn't some esoteric point. In high performance computing, distributed algorithms, or even systems & architecture it's pretty much expected that if you talk about complexity, you should do so under some sort of EM model.
- ChuckMcM 10y agoWell stated. That's a much better way to say it.
- Terr_ 10y agoI just assume that every big-O equation has some sort of units (possibly implicit) and that neither the inputs nor the results are dimensionless quantities. We just tend to assume that "N" is clear from context and that the result is some number of similarly-loose "instructions".