11 ms·
Does my data fit in RAM?
- ailef 4y agoBasically every fits in RAM up to 24TB.
- donkarma 4y ago64TB because of the mainframe
- jhbadger 4y agoI was disappointed that the page didn't start offering vintage computers for very small datasets given that it has bytes and kilobytes as options ("your data is too large for a VIC-20, but a Commodore 64 should handle it")
- louwrentius 4y agoThat is actually a funny idea, I didn't think about that. I only revived and refreshed what somebody else came up with and made before me.
- rob_c 4y agocute, but really, it's not data if it's not measured in PB it's training data or a derrived dataset.
- rob_c 4y agohilarious to get voted down for this and the equivalent comment is still, please stop wasting storage on data that is never going to be needed. If the LHC can find the Higgs Boson before hitting the exa-scale then it's not needed to identify "not a hotdog".
- baisq 4y agoWhy is table.html loaded as an external resource instead of being in index.html proper?
- deleted 4y ago[deleted]
- louwrentius 4y agoI can't remember why I did that, probably to keep the data separate from the rest of the code.
- chrismorgan 4y agoThe technique used means that it always takes a while to load it, as it’s reloading it after every change, which is generally taking around two seconds for me. You make it still worse by using 'change' and 'keyup' events, so any key press (e.g. left/right arrow) triggers it. It should use the 'input' event. Even if it were done as a separate page, it should still be shown or hidden via the CSS display property in order to avoid this reloading.
- louwrentius 4y agoThanks for the tip!
- game-of-throws 4y agoI just confirmed that 640 KB fits in RAM. That's enough for me.
- mech422 4y agoThanks Bill!
- tester756 4y ago>Gates himself has strenuously denied making the comment. In a newspaper column that he wrote in the mid-1990s, Gates responded to a student's question about the quote: "I've said some stupid things and some wrong things, but not that. No one involved in computers would ever say that a certain amount of memory is enough for all time." Later in the column, he added, "I keep bumping into that silly quotation attributed to me that says 640K of memory is enough. There's never a citation; the quotation just floats like a rumor, repeated again and again."
- rmetzler 4y ago640K ought to be enough for anybody.
- antisthenes 4y agoIf you're wondering, the cutoff is 64 TiB. That's the amount of RAM on an IBM Power E980 System.
- baisq 4y agoHow much does that cost?
- edmundsauto 4y ago$8.5 million according to a sibling comment.
- civilized 4y agoHas anyone tried firing up Pandas or something to load a multi-TB table? Would be interested to see if you run into some hidden snags.
- jdeaton 4y agoI've done this though the data in the table was split across DataFrames in many concurrent processes. https://stackoverflow.com/questions/49438954/python-shared-memory-dictionary-for-mapping-big-data https://stackoverflow.com/questions/49438954/python-shared-m...
- itamarst 4y agoThere's just a huge amount of waste in many cases which is very easy to fix. For example, if we have a list of fractions (0.0-1.0): * Python list of N Python floats: 32×N bytes (approximate, the Python float is 24 bytes + 8-byte pointer for each item in the list) * NumPy array of N double floats: 8×N bytes * Hey, we don't need that much precision, let's use 32-bit floats in NumPy: 4×N * Actually, values of 0-100 are good enough, let's just use uint8 in NumPy and divide by 100 if necessary to get the fraction: N bytes And now we're down to 3% of original memory usage, and quite possibly with no meaningful impact on the application. (See e.g. https://pythonspeed.com/articles/python-integers-memory/ https://pythonspeed.com/articles/python-integers-memory/ and https://pythonspeed.com/articles/pandas-reduce-memory-lossy/ https://pythonspeed.com/articles/pandas-reduce-memory-lossy/ for longer prose versions that approximate the above.)
- adamsmith143 4y agoOk now I have 100s of columns. I should do this for every single one in every single dataset I have?
- staticassertion 4y agoYes?
- itamarst 4y agoIt takes like 5 minutes, and once you are in the habit it's something you do automatically as you write the code and so it doesn't actually cost you extra time. Efficient representation should be something you build into your data model, it will save you time in the long run. (Also if you have 100s of columns you're hopefully already benefiting from something like NumPy or Arrow or whatever, so you're already doing better than you could be... )
- maerF0x0 4y ago> It takes like 5 minutes, and once you are in the habit it's something you do automatically as you write the code and so it doesn't actually cost you extra time. This is the argument I've been having my whole career with people who claim the better way is "too hard and too slow" . I'm like "gee, funny how the thing you do the most often you're fastest at... could it be that you'd be just as fast at a better thing if you did it more than never?" .
- AdamJacobMuller 4y agohttps://www.redbooks.ibm.com/redpapers/pdfs/redp5510.pdf https://www.redbooks.ibm.com/redpapers/pdfs/redp5510.pdf I want one of these. a system with 1TB of ram is 133k, 8.5mil for a system with 64TB of ram?
- didgetmaster 4y agoI still remember the first advertisement I saw for 1TB of disk space. I think it was about 1997 and about the biggest individual drive you could buy was 2GB. The system was the size of a couple of server racks and they put 500 of those disks in it. It cost over $1M for the whole system.
- nimish 4y agoThat's insanely overpriced. A 128gb lrdimm is $1000. So a tb on a commodity 8 mem slot board would be 8k plus a few thousand for the cpu and chassis.
- adgjlsfhk1 4y agoNote that when you start talking about multiple 10s of TiB of ram, you start having to buy super high density dims which are very expensive (because not many of them get made and anyone who needs one has lots of money)
- nimish 4y agoYes, true. But at that point you have to run on exotic 4p/8p systems since you start running out of memory channels. 256GB DIMMs go for ~3k, so 133k for just 1TB is still a bit much.
- chaxor 4y agoAbsolutely not. You can purchase a system with 1TB of ram, and some decent CPUs etc for ~25k. My lab just did this. That's far overpriced. 133k is closer to what you would spend if you used the machine with 1tb "in the cloud".
- staticassertion 4y agomine does not
- louwrentius 4y agoThe original site made by lukegb inspired me because of the down-to-earth simplicity. Scaling vertically is often so much easier and better in so many dimensions than creating a complex distributed computing setup. This is why I recreated the site when it went down quite a while ago. The recent article "Use One Big Server"[0] inspired me to (re)submit this website to HN because it addresses the same topic. I like this new article so much because in this day and age of the cloud, people tend to forget how insanely fast and powerful modern servers have become. And if you don't have budget for new equiment, the second-hand stuff from a few years back is stil beyond amazing and the prices are very reasonable compared to cloud cost. Sure, running bare metal co-located somewhere has it's own cost, but it's not that of a big deal and many issues can be dealt with using 'remote hands' services. To be fair, the article admits that in the end it's really about your organisation's specific circumstances and thus your requirements. Physical servers and/or vertical scaling may not (always) be the right answer. That said, do yourself a favour, and do take this option seriously and at least consider it. You can even do an experiment: buy some second-hand gear just to gain some experience with hardware if you don't have it already and do a trial in a co-location. Now that we are talking, yourdatafitsinram.net runs on a Raspberry Pi 4 which in turn is running on solar power.[1] (The blog and this site are both running on the same host) [0]: https://news.ycombinator.com/item?id=32319147 https://news.ycombinator.com/item?id=32319147 [1]: https://louwrentius.com/this-blog-is-now-running-on-solar-power.html https://louwrentius.com/this-blog-is-now-running-on-solar-po...
- karamanolev 4y ago> many issues can be dealt with using 'remote hands' services. I have a few second-hand HP/Dell/Supermicro systems running colocated. I find that for all software issues, remote management / IPMI / KVM over IP is perfectly sufficient. Remote hands are needed only for actual hardware issues, most of which is "replace this component with an identical one". Usually HDD, if you're running those. Overall, I'm quite happy with the setup and it's very high on the value/$ spectrum.
- louwrentius 4y agoYes, I bet a lot of people aren't even aware of IPMI/KVM over IP capabilities that servers have for decades, which makes hardware management (manual or automated!) much easier. Remote hands is for the inevitable hardware failure (Disk, PSU, Fan) or human error (you locked yourself out somehow remotely from IPMI). P.S. I have a HP Proliant DL380 G8 with 128 GB of memory and 20 physical cores as a lab system for playing with many virtual machines. I turn it on and off on demand using IPMI.
- vlunkr 4y agoAmazing. This has been the solution to postgres issues for me. Just add enough memory that everything, or at least everything that is accessed frequently can fit in RAM. Suddenly everything is cached and fast.
- somekyle 4y agoIs the point of this that you can do large-scale data processing without the overhead of distribution if you're willing to pay for the kind of hardware that can give you fast random access to all of it?
- nattaylor 4y agoYes, take a look at the "inspired by" tweet [0] [0] https://twitter.com/garybernhardt/status/600783770925420546 https://twitter.com/garybernhardt/status/600783770925420546
- z3t4 4y agoYour data might even fit in the CPU L3 cache... But most likely you want your data to be persistent. But how often do you actually "pull the plug" on your servers!? And what happens when SSD's are fast enough ? Will we see a whole new architecture where the working memory is integrated into the CPU and the main memory is persistent ?
- mnd999 4y agoThat was the promise of optane. Unfortunately nobody bought it.
- tester756 4y agoWhat do you mean by "nobody"? Significant % of top 500 fortune used it
- mnd999 4y agoNot enough to stop Intel from killing it. Had they been using it seriously at volume this would not have happened.
- altmind 4y agoa lot of people were scratching their heads on how to put optane to use. a fast ssd? or a slow DRAM? with a propietary api? and requiring support from the hardware platform? the whole product line was inconsistent as they use the same name for both storage devices and NVDIMM.
- bee_rider 4y agoOn one hand it would be cool to have some persistence in the CPU. On the other -- imagine if rebooting a computer didn't make the problems all go away. What a nightmare.
- marcinzm 4y agoWe went with this approach. Pandas hit GIL limits which made it too slow. Then we moved to Dask and hit GIL limits on the scheduler process. Then we moved to Spark and hit JVM GC slowdowns on the amount of allocated memory. Then we burned it all down and became hermits in the woods.
- deleted 4y ago[deleted]
- mumblemumble 4y agoI have decided that all solutions to questions of scale fall into one of two general categories. Either you can spend all your money on computers, or you can spend all your money on C/C++/Rust/Cython/Fortran/whatever developers. There's one severely under-appreciated factor that favors the first option: computers are commodities that can be acquired very quickly. Almost instantaneously if you're in the cloud. Skilled lower-level programmers are very definitely not commodities, and growing your pool of them can easily take months or years.
- jbverschoor 4y agoBuying hardware won't give you the same performance benefits as a better implementation/architecture. And if the problem is big enough, buying hardware will cause operational problems, so you'll need more people. And most likely you're not gonna wanna spend on people, so you get a bunch of people who won't fix the problem, but buy more hardware.
- marcinzm 4y ago>And if the problem is big enough, buying hardware will cause operational problems, so you'll need more people. And most likely you're not gonna wanna spend on people, so you get a bunch of people who won't fix the problem, but buy more hardware. That's why people love the cloud.
- jbverschoor 4y ago
- hyperman1 4y agoFunny, it lets you click to negative amounts of RAM. My -1 PiB fits in RAM, so having it as a unit is not useless. (It also accepts fractions but not octal)
- chrismorgan 4y agoIt’s using an HTML <input type=number> element, which requires that the value be a valid floating-point number <https://html.spec.whatwg.org/multipage/common-microsyntaxes.html#valid-floating-point-number https://html.spec.whatwg.org/multipage/common-microsyntaxes....>. You’ll note from this that exponent notation is allowed, something people are unlikely to expect. For restricting negative numbers, it should probably had the attribute min=0; then negative zero would be the lowest you could go.
- dang 4y agoRelated: Does my data fit in RAM? - https://news.ycombinator.com/item?id=22309883 https://news.ycombinator.com/item?id=22309883 - Feb 2020 (162 comments)
- Cwizard 4y agoAnyone have any recommendations for a SQL engine that works on in-memory data and has a simple/monolithic architecture? Our data is about 50-100gb (uncompressed) and thus easily fits into memory. I am sure we could do our processing using something like polars or pandas in memory quite quickly but we prefer a SQL interface. Using postgres is still quite slow even when it has more than enough memory available compared to something like duckdb. Duckdb has other limitations however. I’ve been eying MemSQL but that also seems to be targeted more towards multi machine deployments.
- mritchie712 4y agowhat limit are you hitting with duckdb?
- giraffe_lady 4y agosqlite?
- chaxor 4y agoSQLite is almost always the answer
- specialist 4y agoH2 is trivial to embed. Written in Java. It's one of my favorite things. http://www.h2database.com/html/features.html#in_memory_databases http://www.h2database.com/html/features.html#in_memory_datab...
- whichdan 4y agoMySQL has a MEMORY[0] storage engine which might be of interest? [0] https://dev.mysql.com/doc/refman/8.0/en/memory-storage-engine.html#memory-storage-engine-performance-characteristics https://dev.mysql.com/doc/refman/8.0/en/memory-storage-engin...
- boredumb 4y agowould be neat if I could do say, 6gb, and see the machines that are closest in size instead of only the upper limit
- none_to_remain 4y agoSeveral years ago my job then got a dev and prod server with a terabyte of RAM. I liked the dev server because a few times I found myself thinking "this would be easy to debug if I had an insane amount of RAM" and then I would remember I did.
- MathYouF 4y agoWhat kind of things are easier to debug with lots of RAM and how would you do it?
- none_to_remain 4y agoBasically working with code manipulating a couple dozen GB of data and then keeping a couple dozen copies of that to examine it after various stages of manipulation.
- Goz3rr 4y agoAm I the only one here using Chrome or is everyone else just ignoring the table being broken? The author used an <object> tag which just results in Chrome displaying "This plugin is not supported". I'm unsure why they didn't just use an iframe instead.
- louwrentius 4y agoI can only state for myself that on my Mac running chrome, the site works OK. I don't get any plugin messages.
- louwrentius 4y agoExtra anecdote: Around 2000, a guy told me he was asked to support very significant performance issues with a server running a critical application. He quickly figured out that the server ran out of memory. Option 1 was to rewrite the application to use less memory. He chose option two: increase the server memory, going from 64 MB to 128 MB (Yes MB). At that time, 128 MB was an ungodly amount of memory and memory was very expensive. But it was still cheaper to just throw RAM at the problem than to spend many hours rewriting the application.
- navjack27 4y agoIn 2000? 128 ungodly? What!
- nomel 4y agoThe price curve was in sharp decline around that time, easily making "around 2000" +/- 5 years, possible: https://jcmit.net/memoryprice.htm https://jcmit.net/memoryprice.htm For example, four years earlier, it would be >$3k. A year before that, >$6k. This was an amazing, and terrible, time to be into computers.
- nsxwolf 4y ago64 TiB fits in a Dell Poweredge R840 with 6TB max RAM... how exactly?
- louwrentius 4y agoPlease take a look further down the list. For your use case, a Power System E980 may just be enough or too small. :-)
- rbanffy 4y agoDammit! The biggest z16 can only address 40TB!
- mciancia 4y agoMaybe someone mixed up regular dram and optane dimms?
- sophacles 4y agoI find it mind boggling that one can purchase a server with more RAM than the sum of all working storage media in my house.
- bob1029 4y agoThis kind of realization that "yes, it probably will" has recently inspired me to hand-build various database engines wherein the entire working set lives in memory. I do realize others have worked on this idea too, but I always wanted to play with it myself. My most recent prototypes use a hybrid mechanism that dramatically increases the supported working set size. Any property larger than a specific cutoff would be a separate read operation to the durable log. For these properties, only the log's 64-bit offset is stored in memory. There is an alternative heuristic that allows for the developer to add attributes which signify if properties are to be maintained in-memory or permitted to be secondary lookups. As a consequence, that 2TB worth of ram can properly track hundreds or even thousands of TB worth of effective data. If you are using modern NVMe storage, those reads to disk are stupid-fast in the worst case. There's still a really good chance you will get a hit in the IO cache if you application isn't ridiculous and has some predictable access patterns.
- saltcured 4y agoI don't mean to discourage personal exploration in any way, but when doing this sort of thing it can also be illuminating to consider the null hypothesis... what happens if you let the conventional software use a similarly enlarged RAM budget or fast storage? SQLite or PostgreSQL can be given some configuration/hints to be more aggressive about using RAM while still having their built-in capability to spill to storage rather than hit a hard limit. Or on Linux (at least), just allowing the OS page cache to sprawl over a large RAM system may make the IO so fast that the database doesn't need to worry about special RAM usage. For PostgreSQL, this can just be hints to the optimizer to adjust the cost model and consider random access to be cheaper when comparing possible query plans. Once you do some sanity check benchmarks of different systems like that, you might find different bottlenecks than expected, and this might highlight new performance optimization quests you hadn't even considered before. :-)
- bob1029 4y ago> what happens if you let the conventional software use a similarly enlarged RAM budget or fast storage? Oh I absolutely have gone down this road as well. The biggest thing for me is taking advantage of the other benefits you can get with in-memory working sets, such as arbitrary pointer machine representations. When working with a traditional SQL engine (even one tuned for memory-only operation), there are many rules you have to play by or things will not go well.
- elktea 4y agowillyourbudgetletoyufityourdatainram.net
- glonq 4y agoDAE remember "how many tapes do I need to hold this much data"?
- rbanffy 4y agoHow many years it'll take to back everything up?
- rbanffy 4y agoDon't forget memory bandwidth. You may be able to shoehorn your data in a single machine's memory, but it may take a very long time to process all of it. Sometimes sharding will yield better results. It will still take a long time to read through 64 TB of memory, even if you are just NOPing on it.
- vkoskiv 4y agoThe word ‘data’ is plural, so it might be more correct to say “Do my data fit in RAM?”
- bdowling 4y agoThe only people who treat data as plural are academics, usually social scientists, and overeducated journalists. Sentences like, "What do these data tell us?" are awkward for everyone else. Like "information", "data" is correctly used as a singular mass noun.
- dkbrk 4y agoDepends on context. If, for your purposes it would be reasonable to talk about a single datum, then sure, it's plural. But in common use it's a singular mass noun.
- deathanatos 4y agoI actually worked on a project that did this. We used a database called "Tile38" [1] which used an R-Tree to make geospatial queries speedy. It was pretty good. Our dataset was ~150 GiB, I think? All in RAM. Took a while to start the server, as it all came off disk. Could have been faster. (It borrowed Redis's query language, and its storage was just "store the commands the recreate the DB, literally", IIRC. Dead simple, but a lot of slack/wasted space there.) Overall not a bad database. Latency serving out of RAM was, as one should/would expect, very speedy! [1]: https://tile38.com/ https://tile38.com/
- langsoul-com 4y agoWhat's the differences in cost between server with mainly hdd VS ram storage? Ie with vultur, hertzer etc I though the issue with ram was the much higher costs. Also, could you put all of postgres dB in ram, like with redis? Thus no need for separate redis server since your entire dB is fast in ram.
- KronisLV 4y agoHonestly when all that you can afford are VPSes with 4 GB of RAM, reading this feels a bit silly and mean. Though at scale, sure, one could even run a RAM disk for speeding up lots of traditional software, with some caveats.