3 ms·

The author gets the definition of big-O completely backwards. Every single time the author says "Big-O" he means "little-o". Here are the definitions: https:

by zeroer 10y ago

The author gets the definition of big-O completely backwards. Every single time the author says "Big-O" he means "little-o". Here are the definitions: https://cathyatseneca.gitbooks.io/data-structures-and-algorithms/content/analysis/notations.html https://cathyatseneca.gitbooks.io/data-structures-and-algori... Basically, big-O is a claim that your process is at least -this- fast. Small-o is a claim that your process can't be faster than -this-. He means the latter. > Still not convinced? Think I'm misunderstanding Big-O? Yea, I do.

jo909 10y ago
Please see https://en.wikipedia.org/wiki/Best,_worst_and_average_case https://en.wikipedia.org/wiki/Best,_worst_and_average_case Big-O is valid notation for _many aspects_ of an algorithm. The author applied it to a very practical one and explained what exactly he describes very well.
- reikonomusha 10y ago
  I think it's a massive and needless abuse of standard computer science terminology to re-define it as something completely different.
  - MrManatee 10y ago
    I'm not entirely sure what you are referring to. You might be referring to the fact that the author's definition of big-O doesn't say anything about constant factors or asymptotics. This makes the definition incorrect, or at least sloppy. But judging by usage, it seems that he actually knows and is using the standard definition. The error is just in that one sentence, and it doesn't affect the rest of the argument. You might also be objecting to the fact that he makes a distinction between time and instruction count, and is using big-O notation for both. I don't think there's anything nonstandard about making this distinction when it needs to be made. Take, for example, Karmarkar's algorithm: https://en.wikipedia.org/wiki/Karmarkar%27s_algorithm https://en.wikipedia.org/wiki/Karmarkar%27s_algorithm
- DanWaterworth 10y ago
  Re-read the article, bearing in mind that O(sqrt n) is a set that contains O(1).
  - angry_octet 10y ago
    Yes, but it isn't very helpful to think of it that way, we're not competing to find the weakest constraint.
- zeroer 10y ago
  The best/worst/average issue is orthogonal to the definition of what big-O means. The time it take an algorithm has best/worst/average cases, and each of these is a function and thus is a member of big-O classes. The definition of big-O has a completely standardized and unambiguous definition, and the author of the article somehow has a four-part blog post on big-O without knowing it.
smallnamespace 10y ago
You're right, but almost all colloquial usage of Big-O seems to be close to theta-O because people pick the tightest bound they can. That's why everyone writes that comparison sorts are O(n lg N), even though it's also technically O(exp N) as well.
- zeroer 10y ago
  Yea, I thought about that, but he doesn't mean theta, either. What he's giving is a physical impossibility proof -- you can't do memory access better than sqrt(N). When he gives a physical example of a machine of arbitrary size that accesses memory in sqrt(N) time, then we can talk about theta.
  - Animats 10y ago
    Well, cube root of N; you can stack memory. The speed of light limitation on distance to DRAM memory is real, but so far seems to affect mostly supercomputers. Are there any large server boards where speed of light lag is the limit on memory capacity?
    - zeroer 10y ago
      Read part two of the article. You can't stack memory indefinitely -- you'll get a black hole.
- perfectfire 10y ago
  > it's also technically O(exp N) as well. I wonder if this is what one interviewer was trying to get at when he asked me over and over again if a binary search was worst case O(log N). I just kept saying that if you took more than log N steps to perform a binary search then you by definition did not perform a binary search. He was off his rocker so I kind of doubt he was looking for the difference between little-o, theta, and big-O. It's good stuff to remember and might impress a more sane interviewer some day though.
rntz 10y ago
Your claims about what little-o and big-O mean don't match the explanation at https://cathyatseneca.gitbooks.io/data-structures-and-algorithms/content/analysis/notations.html https://cathyatseneca.gitbooks.io/data-structures-and-algori... That page says o(f) == O(f) and not Theta(f). When you say "small-o is a claim that your process can't be faster than -this-", that seems to match the definition of Omega(f). Did you mean that the author should have used big-Omega? As a side-note, I've always thought that {big-O,Theta,Omega,small-o} notation was very confusing. We should just have a notation for "the asymptotic equivalence class of a function", let's say A(f). Then to say f \in O(g), we say A(f) <= A(g). To say f \in Theta(g), we say A(f) = A(g). f \in o(g) becomes A(f) < A(g). And so forth. Instead of lots of new notation, we add a single function A and re-use notation for orderings which everyone already knows. (Of course, I'm omitting the definition of the ordering on these equivalence classes, so maybe there is some difficulty in defining that...)
- contravariant 10y ago
  >As a side-note, I've always thought that {big-O,Theta,Omega,small-o} notation was very confusing. We should just have a notation for "the asymptotic equivalence class of a function", let's say A(f). Then to say f \in O(g), we say A(f) <= A(g). To say f \in Theta(g), we say A(f) = A(g). f \in o(g) becomes A(f) < A(g). And so forth. Instead of lots of new notation, we add a single function A and re-use notation for orderings which everyone already knows. (Of course, I'm omitting the definition of the ordering on these equivalence classes, so maybe there is some difficulty in defining that...) Your A is pretty much equivalent to \Theta. Saying f \in \Theta(g) is equivalent to saying \Theta(f) = \Theta(g). Edit: as an aside O(f) = O(g) is also equivalent to f \in \Theta(g), and O(f) \subset O(g) is equivalent to f \in O(g).
  - rntz 10y ago
    For equality, yes, Theta works! But I care not just about equality but about comparison. In particular, it would be nice if the "natural" ordering on A(f) had the property that A(f) <= A(g) iff f is O(g) Theta(g) is a set, and the most natural ordering on sets is subsetting/inclusion. However, Theta(x) is not a subset of Theta(x^2), even though x is O(x^2)! So the natural ordering on sets doesn't do what I want. It might be possible to define an ordering on Theta-classes that does what I want, though. Edit: I just saw your comment about O. I think O might just work! So instead of remembering big-O, Theta, and Omega, I can just use O(f) <= O(g) for f is O(g) O(f) = O(g) for f is Theta(g) O(f) >= O(g) for f is Omega(g) Is that true? (I think small-o might be slightly more complicated, unfortunately. It has a more unusual definition: https://en.wikipedia.org/wiki/Big_O_notation#Little-o_notation https://en.wikipedia.org/wiki/Big_O_notation#Little-o_notati...)

by zeroer 10y ago

jo909 10y ago

Please see https://en.wikipedia.org/wiki/Best,_worst_and_average_case https://en.wikipedia.org/wiki/Best,_worst_and_average_case Big-O is valid notation for _many aspects_ of an algorithm. The author applied it to a very practical one and explained what exactly he describes very well.

reikonomusha 10y ago
I think it's a massive and needless abuse of standard computer science terminology to re-define it as something completely different.
- MrManatee 10y ago
  I'm not entirely sure what you are referring to. You might be referring to the fact that the author's definition of big-O doesn't say anything about constant factors or asymptotics. This makes the definition incorrect, or at least sloppy. But judging by usage, it seems that he actually knows and is using the standard definition. The error is just in that one sentence, and it doesn't affect the rest of the argument. You might also be objecting to the fact that he makes a distinction between time and instruction count, and is using big-O notation for both. I don't think there's anything nonstandard about making this distinction when it needs to be made. Take, for example, Karmarkar's algorithm: https://en.wikipedia.org/wiki/Karmarkar%27s_algorithm https://en.wikipedia.org/wiki/Karmarkar%27s_algorithm
DanWaterworth 10y ago
Re-read the article, bearing in mind that O(sqrt n) is a set that contains O(1).
- angry_octet 10y ago
  Yes, but it isn't very helpful to think of it that way, we're not competing to find the weakest constraint.
zeroer 10y ago
The best/worst/average issue is orthogonal to the definition of what big-O means. The time it take an algorithm has best/worst/average cases, and each of these is a function and thus is a member of big-O classes. The definition of big-O has a completely standardized and unambiguous definition, and the author of the article somehow has a four-part blog post on big-O without knowing it.

smallnamespace 10y ago

You're right, but almost all colloquial usage of Big-O seems to be close to theta-O because people pick the tightest bound they can. That's why everyone writes that comparison sorts are O(n lg N), even though it's also technically O(exp N) as well.

zeroer 10y ago
Yea, I thought about that, but he doesn't mean theta, either. What he's giving is a physical impossibility proof -- you can't do memory access better than sqrt(N). When he gives a physical example of a machine of arbitrary size that accesses memory in sqrt(N) time, then we can talk about theta.
- Animats 10y ago
  Well, cube root of N; you can stack memory. The speed of light limitation on distance to DRAM memory is real, but so far seems to affect mostly supercomputers. Are there any large server boards where speed of light lag is the limit on memory capacity?
  - zeroer 10y ago
    Read part two of the article. You can't stack memory indefinitely -- you'll get a black hole.
perfectfire 10y ago
> it's also technically O(exp N) as well. I wonder if this is what one interviewer was trying to get at when he asked me over and over again if a binary search was worst case O(log N). I just kept saying that if you took more than log N steps to perform a binary search then you by definition did not perform a binary search. He was off his rocker so I kind of doubt he was looking for the difference between little-o, theta, and big-O. It's good stuff to remember and might impress a more sane interviewer some day though.

rntz 10y ago

Your claims about what little-o and big-O mean don't match the explanation at https://cathyatseneca.gitbooks.io/data-structures-and-algorithms/content/analysis/notations.html https://cathyatseneca.gitbooks.io/data-structures-and-algori... That page says o(f) == O(f) and not Theta(f). When you say "small-o is a claim that your process can't be faster than -this-", that seems to match the definition of Omega(f). Did you mean that the author should have used big-Omega? As a side-note, I've always thought that {big-O,Theta,Omega,small-o} notation was very confusing. We should just have a notation for "the asymptotic equivalence class of a function", let's say A(f). Then to say f \in O(g), we say A(f) <= A(g). To say f \in Theta(g), we say A(f) = A(g). f \in o(g) becomes A(f) < A(g). And so forth. Instead of lots of new notation, we add a single function A and re-use notation for orderings which everyone already knows. (Of course, I'm omitting the definition of the ordering on these equivalence classes, so maybe there is some difficulty in defining that...)

contravariant 10y ago
>As a side-note, I've always thought that {big-O,Theta,Omega,small-o} notation was very confusing. We should just have a notation for "the asymptotic equivalence class of a function", let's say A(f). Then to say f \in O(g), we say A(f) <= A(g). To say f \in Theta(g), we say A(f) = A(g). f \in o(g) becomes A(f) < A(g). And so forth. Instead of lots of new notation, we add a single function A and re-use notation for orderings which everyone already knows. (Of course, I'm omitting the definition of the ordering on these equivalence classes, so maybe there is some difficulty in defining that...) Your A is pretty much equivalent to \Theta. Saying f \in \Theta(g) is equivalent to saying \Theta(f) = \Theta(g). Edit: as an aside O(f) = O(g) is also equivalent to f \in \Theta(g), and O(f) \subset O(g) is equivalent to f \in O(g).
- rntz 10y ago
  For equality, yes, Theta works! But I care not just about equality but about comparison. In particular, it would be nice if the "natural" ordering on A(f) had the property that A(f) <= A(g) iff f is O(g) Theta(g) is a set, and the most natural ordering on sets is subsetting/inclusion. However, Theta(x) is not a subset of Theta(x^2), even though x is O(x^2)! So the natural ordering on sets doesn't do what I want. It might be possible to define an ordering on Theta-classes that does what I want, though. Edit: I just saw your comment about O. I think O might just work! So instead of remembering big-O, Theta, and Omega, I can just use O(f) <= O(g) for f is O(g) O(f) = O(g) for f is Theta(g) O(f) >= O(g) for f is Omega(g) Is that true? (I think small-o might be slightly more complicated, unfortunately. It has a more unusual definition: https://en.wikipedia.org/wiki/Big_O_notation#Little-o_notation https://en.wikipedia.org/wiki/Big_O_notation#Little-o_notati...)