5 ms·
Gcc compile error with 2GB of data
- __rkaup__ 15y agoIs including all that data in the object code really necessary?
- kodablah 15y agoAbsolutely not. This appears to be generated code from a data set to avoid what most developers would do at runtime.
- kelnos 15y agoFrom some of the comments, it's suggested that the problem is too much code, and that the data (of which there is a lot, granted) isn't contributing to the problem. I'm not sure if that's the case though. If the data is part of the problem, I wonder if he can write a new linker script to rearrange the sections in the file so all code is below the signed 32-bit (~2GB) boundary. Though that raises the question... will it be able to address the data? Does initialized data access use 32- or 64-bit offsets in the small/medium models? At any rate, it seems gcc 4.6 supports the x86_64 "large model", which should solve the problem without code/data changes.
- dekayed 15y agoGenerally not. My last project involved building a framework in which scientists could run calculations for certain types of risk. A prototype built by some of said scientists involved code like seen in the SO question. After looking at the types of calculations being done, we figured out that most of the equations that were used were similar and could be generalized. We also moved all the coefficients and parameters for all the equations into configuration files. I suspect that a thorough evaluation of this code would reveal something similar as in my case.
- losethos 15y agoI mostly use MAlloc() in LoseThos. By default, LoseThos puts global variables on the code heap. There's a flag you set to do globals off the data heap. OptOn(OPTf_HEAP_GLBLS); or something like that. Normally, you JIT compile everything. If you insist on doing it old school, the binary is flat and has gaps for glbl variables. If you're really old school and store files in names without Z, the binary will be uncompressed and those globals will be huge. Use JIT.
- bsiemon 15y agoThis sorta reminds me of the silly things people do with JavaScript.
- nddrylliog 15y agoSuch as... an MP3 decoder? https://github.com/nddrylliog/jsmad https://github.com/nddrylliog/jsmad
- dexen 15y agotl;dr: the error is returned by the linker rather than compiler itself. It is not a bug, just size limitation of the default memory model. Linux x86_64 provides `large model' -- as pointed out by VJo, but it is not supported by GCC before 4.6, http://stackoverflow.com/questions/6296837/gcc-compile-error-with-2gb-of-data/6297704#6297704 http://stackoverflow.com/questions/6296837/gcc-compile-error...
- joeyh 15y agoWhat a great technical problem, except for this bit: > Btw, I didn't try to hide behind "this is scientific computing -- no way to optimize". It's just that the basis for this code is something that comes out of a "black box" where I have no real access to Black box != science.
- eru 15y agoI wish that was true.
- esrauch 15y agoA research assistantship I held was based on classified data; all of the published work had to be approved by the DoD and the actual data we used wasn't allowed to be published which made our results entirely unreproducable.
- deleted 15y ago[deleted]
- sevenproxies 15y agoScience is a blackbox. We make theories about the world we live in based upon the results of experiments. Richard Feynman commented method of thinking about science on one of his videos. http://www.youtube.com/watch?v=o1dgrvlWML4 http://www.youtube.com/watch?v=o1dgrvlWML4
- malkia 15y agoI suggested to try and use LuaJIT. It has very good double precision floating point, and it looks like SSE/SIMD were not used directly from the code. LuaJIT should garbage collect unused code, if not it always regenerates it. It should be able to compile and execute tons of code as he has (and it looks like all his stuff is generated, but for C++).
- premchai21 15y agoLast I checked, the LuaJIT allocator on AMD64 platforms uses only a small part of the address space for the Lua heap, partly so that more efficient type-punned representations can be used internally. I don't remember what the limit is exactly, but it's only a few GB (and beyond that the GC starts having trouble anyway). I don't know whether this applies to the machine-code JIT output, or to external cdata arrays, but it's something to watch out for here.
- juiceandjuice 15y agoThose expressions look a lot like an alternating series to me. You should be able to generate an expression to produce them fairly easily.