10 ms·
How likely do you run into array bigger than 2gb?
by flamedoge 10y ago
How likely do you run into array bigger than 2gb?
- minipci1321 10y agoProbably not very likely, but keep in mind that this method could also be used without actually allocating the array -- akin to the 'offsetof()' macro. (Which is undefined behavior.)
- dragandj 10y agoToday, with ML, big data and similar applications, that might be often.
- quotemstr 10y agoThe "how likely is it, really?" response to questions of technical correctness has always bothered me. It takes a mindset completely alien to mine to say "Here's a race condition. Sure, it's undefined behavior, but the race is narrow, so it's rare" or to say "Sure, memory allocation can theoretically fail, but in practice almost never does" or to say "fsync is too slow and most computers have batteries these days". Software is unreliable enough as it is due to problems beneath our notice. It seems reckless to avoid fixing problems that we do notice. Sure, you could argue that rare problems are rare and that users probably won't notice them --- this attitude is penny-wise and pound-foolish, because you can't meaningfully reason about a system that's only probably correct.
- mjburgess 10y agoThe problem you're latching on to I think is how the context for caculating a probability can vary. If it were really as likely as, say, the sun exploding that X happened then it would be of no use to expend time on X. BUT very often people speaking about the probability of events given suspicious constraints. While a memory allocation might not fail in most situations it will fail often in some situations. And a one-in-a-million chance is almost guaranteed when there are millions of uses.
- spdionis 10y agoThat's why it's called one in a million...
- tholmes 10y agoOne in a million happens quite often if you're processing something like ~100k requests a second.
- jschwartzi 10y agoBut it's extremely unlikely when you're processing 10 requests a week, such as might be the case for the web server in a consumer-grade router.
- paulddraper 10y ago...by hundreds of thousands of customers.
- quotemstr 10y agoIt's amazing how skilled blackhats are at converting "rare bug that doesn't affect the UI" into "massive DDoS cannon".
- jschwartzi 10y agoAnd you can see how risk analyses by senior engineers with tons of embedded experience who are used to working with systems that are not networked leads to problems when their systems are later networked.
- spdionis 10y agoIn fact I agreed with the parent and just posted a tongue in cheek remark. One in a million literally means that at ~100k requests a second it will happen once every 10 seconds.
- maxlybbert 10y ago
- Swizec 10y agoEngineering is about tradeoffs. How many once-in-a-thousand bugs do you fix before you tackle the one-in-a-million? Or one in a billion? What about if it takes $10/bug to fix every 1:1000 bug and $100,000 to fix one 1:1000000 bug? Correctness is great in theory, but in practice it's a matter of what's important.
- mannykannot 10y agoIf you are only looking at probability and cost-to-fix you are overlooking something important - the cost if/when it happens.
- verdax_1 10y agoThis is really emphasized in things like dmfea and other failure mode analysis documents or regulated industry. They want you to document the likelihood, your ability to recover from the failure, as well as the cost o the failure. You can say that you didn't want to pay for someone fixing some unlikely fail mode but that's small consultation to the people whose lives your product is ruining.
- Dylan16807 10y ago>he "how likely is it, really?" response to questions of technical correctness has always bothered me. But the question is important in another context: language design. Why is this undefined behavior something that exists in the first place? Objects larger than PTRDIFF_MAX could just not be allowed! This avoids the problem and makes code easier to reason about, with pretty much no downside.
- verdax_1 10y agoI like the way you're thinking, but that sort of thing probably doesn't get past a committee. "Hey we might not be able to think of an application but that doesn't mean our users won't have a legitimate reason for doing it ... Motion passed."
- vram22 10y agoNot likely, but possible. This reminds me of the bug that was found in the binary search algorithm a few years ago, IIRC, in Java. The interesting thing is that binary search is probably one of the earliest-invented algorithms. Yet, in the book Writing Efficient Programs by Jon Bentley (which I mentioned in a recent HN comment), he says that in a class he taught to several industrial programmers with many years of experience, some had bugs in their implementations of binary search that he set them as an exercise. Not sure but I think I remember reading in the article about the Java binary search issue, that even his algorithm had the bug that was found in the Java version. Why it was not found earlier is (maybe) because it only occurred with an extremely large array, IIRC. Don't have a link right now, but it can probably be found by searching for the right phrase.
- vram22 10y agoJust did a google search, and it even partially auto-completed this search for me: bug in java binary search and showed a related search in the drop-down, 'programming pearls ...', a book by Jon Bentley, which seems to confirm what I said above (though I saw it in his other book, "Efficient Programs", IIRC - he might have mentioned the same issue in the Programming Pearls book too). Edit: and the Wikipedia article confirms it too: https://en.wikipedia.org/wiki/Binary_search_algorithm#Implementation_issues https://en.wikipedia.org/wiki/Binary_search_algorithm#Implem...
- pigz 10y agohttps://research.googleblog.com/2006/06/extra-extra-read-all-about-it-nearly.html https://research.googleblog.com/2006/06/extra-extra-read-all...
- vram22 10y agoYes, thanks, that closely describes the assignment I read that he gave.
- jbn 10y agoI think this is it: http://bugs.java.com/bugdatabase/view_bug.do?bug_id=6412541 http://bugs.java.com/bugdatabase/view_bug.do?bug_id=6412541
- cperciva 10y agoA few months ago I was doing FFTs on arrays larger than 4GB. Amusingly, this uncovered a bug in the LLVM optimizer: It was looking at stride lengths to figure out if accesses were independent, and truncated a 4GB stride down to 0.
- ksml 10y agoI would be extremely interested to hear how you found this bug. Sounds like a difficult bug to track down, and I always learn from good debugging stories.
- cperciva 10y agoIt was pretty easy to track down: clang38 was exiting with Assertion failed: (Distance > 0 && "The distance must be non-zero"), function areStridedAccessesIndependent, file /wrkdirs/usr/ports/devel/ llvm38/work/llvm-3.8.0.src/lib/Analysis/LoopAccessAnalysis.cpp, line 1004. Looking at the file it was easy to see what was being asserted, and to see that the type was a 32-bit integer; since I knew I was dealing with huge FFTs, the problem was obvious. Let this be a lesson: Asserting that impossible things don't happen makes debugging much easier when they do happen!
- flamedoge 10y agoWell good thing it was ReleaseWithAsserts build!
- icedchai 10y agoOn a 64-bit platform (anything modern), ptrdiff_t is going to be 64-bit so this will not be an issue (ok, 63-bit... but you get my point.)
- mannykannot 10y agoIn this context, the difference between 63 bits and 64 bits is not trivial.
- yoklov 10y agoBoth are bigger than the x64 address space...
- kabdib 10y agoOften enough; "pack" files in video games are often many GB. Memory-map one of those and there you are . . .
- bonzini 10y agoThe parent is taking about >2gb on a 32-bit machine.
- icebraining 10y agoSo what? You can have up to 64GB of RAM on a 32-bit machine: https://en.wikipedia.org/wiki/Physical_Address_Extension https://en.wikipedia.org/wiki/Physical_Address_Extension
- bonzini 10y agoBut those have to be mapped a slice at a time into virtual addresses.
- loeg 10y agoIt's basically bogus to have a single object bigger or equal to half of address space (represented by size_t) in C. 32-bit platforms should detect and abort in such conditions (compiler/linker for static objects, malloc() implementation for dynamic allocations).
- icebraining 10y agoWhy? If you're running a system with PAE, half of a 32-bit address space is a small fraction of the whole addressable memory.
- loeg 10y agoIt's either addressable or it isn't. My understanding of typical PAE systems is that userspace is still limited to 32 bits of address space per process. Any system where userspace is not limited to 32 bits should have a larger than 32-bit size_t. (PAE systems are not true 32-bit platforms.)
- sfifs 10y agoOh surprisingly easily. Say you're handling a few billion cookies in RAM or manipulating DNA data.
- hsivonen 10y agoIt's quite easy to serve over 2 GB of spaces over the network. (gzip, brotli)