15 ms·
Does my data fit in RAM?
- 1MachineElf 7y agoI've no idea if the intended audience here would ever run their workloads on Solaris, but like the IBM POWER systems, Oracle & Fujitsu SPARC servers also max out at 64TB of RAM. I didn't see those included here.
- ramraj07 7y agoLegit question: I have a dataset that's a terabyte in size spread over multiple tables, but my queries often involve complex self joins and filters; for various reasons, I'd prefer to be able to write my queries in SQL (or spark code) because it's the most expressive system I've seen. What tool should I use it to load this dataset on RAM and run these queries?
- chmod775 7y agoMost DB engines will use what RAM is available and even if they don't, your OS's page cache will make sure stuff is fast anyways. > What tool should I use it to load this dataset on RAM and run these queries? The question should really be: What tool should I use to make this fast? Postgres can be pretty fast when used correctly and you can make your data fit.
- falcolas 7y agoExactly this. DBs are really good at utilizing all the memory you give them. The query planners might give you some fits when they try and use disk tables for complicated joins, but you can work around them.
- efficax 7y agoBoth mysql and pgsql bypass the page cache if they can and maintain their own page caches. You have to do this, otherwise you’re double caching! That is, you’d have your own page cache, which you need to manage calls to read() and to know when to flush pages, while the OS would also have the same pages in its own cache. (mongodb I believe uses direct mmap access instead of a pagecache, and lmdb does this as well)
- saber6 7y agoI am not a big-data guy but wouldn't it be along the lines of A) get a big honking server B) fire up "X" SQL server C) Allocate 95-98% of the RAM to DB cache?
- chmod775 7y agoA single terabyte is a few magnitudes from what you need big-data-anything for. You could probably work with that just fine on your average 64GB ram desktop with an SSD.
- thedance 7y agoA small computer with 1 SSD will take at least 10-20 minutes to make a pass over 1TB of data, if everything is perfectly pipelined.
- deleted 7y ago[deleted]
- keanzu 7y agoSamsung claims their 970 Pro NVMe can read 3.5GB/s sequentially. That's about 300 seconds or 5 minutes per TB.
- thedance 7y agoIt can't though.
- wtallis 7y agoIt can, and their fastest enterprise SSD can write at that speed too, or do sequential reads at 7-8GB/s, or random reads at over 4 GB/s. I just ran `time cp /dev/nvme0n1 /dev/null` on the 1TB 970 Pro. The result: real 4m50.724s user 0m2.001s sys 3m10.282s So with literally zero optimization effort, we've hit the spec (and saturated a PCIe 3.0 x4 link).
- nikhizzle 7y agoYou should use the cache or persist call on the spark data frame/dataset. Persist gives you more control.
- ramraj07 7y agoThis is what most people in my org do, this is orders of magnitude slower than running queries on the same dataset on redshift with optimized presorting and distribution. Redshift doesn't scale for tens or hundreds of parallel users though, so looking for options
- zten 7y agoI'm about 75% joking: restore a new cluster from a snapshot for horizontal scaling. They did launch a feature along those lines, because my joke suggestion probably doesn't scale organizationally: take a look at Concurrency Scaling. This is also a fundamental feature of Snowflake with the separation of compute and storage.
- ramraj07 7y agoExactly the same game plan for us now, evaluating snowflake, but wanted to check if there's a fundamentally different paradigm that can be much more faster and scalablr
- RookyNumbas 7y agoThere are a few steps to consider before you are loading data into ram. Can you partition the data in any useful way? For example if queries use separate ranges of dates, then you can partition data so that queries only need to touch the relevant date range. Can you pre-process any computations? Sometimes tricky things done within the context of multiple joins can be done once and written to a table for later use. Can you materialize any views? Do you have the proper indexes set up for your joins and filters? Are you looking at execution plans for your queries? Sometimes small changes can speed up queries by many orders of magnitude. Smart queries + properly structured data + a well tuned postgres DB is an incredibly powerful tool.
- amznthrowaway5 7y agoCan I set up efficient indexes on parquet data to use with Spark, or is it necessary to use a DB?
- lsb 7y agoYou can get a box with 4TB of ram on EC2 for $4/hr spot, so copy your data into /dev/shm and go hog wild. For lots of databases, most of their time is spent locking and copying data around, so depending on your workload you might getsignificant speedups in Pandas/Numpy if it's just you doing manipulations, and there are multicore just-in-time compilers for lots of Pandas/Numpy operations (like Numba/Dask/etc). If you have lots of weird merging criteria and want the flexibility of SQL I'd say use a modern Postgresql with multicore selects on that 4TB box.
- dijit 7y agoHow long does it take to copy your data in? And what’s the bandwidth cost involved? People like to talk about the elasticity or compute, but startup is not free (or even cheap in most cases).
- cldellow 7y agoIf your data is in S3, my experience is that you can push ~20-40MB/core/sec on most instances. OP is probably talking about an x1e.32xlarge. According to Daniel Vassalo's S3 benchmark [1], it can do about 2.7GB/sec. So your 4TB DB might take ~30min to fetch. Bandwidth is free, you'd pay $2 for the 30 min of compute, and some fractions of pennies for the few hundred S3 requests. [1]: https://github.com/dvassallo/s3-benchmark https://github.com/dvassallo/s3-benchmark
- satanspastaroll 7y agoIt's to note that any data exported out of AWS will be billed at $0.09/GB, or $90/TB
- ramraj07 7y agoCurrently I have it loaded on redshift with as much optimization as possible, and the queries are far more analytical than end-user like (often having to self join on the same dataset). This works okay, but doesn't scale with more than a handful users at a time. I'll probably run some tests with the postgres suggestion but curious if this is still a better alternative or not
- typon 7y agoin memory sqlite is pretty fast
- ken 7y agoYes, but SQLite is probably the least expressive SQL dialect there is. If you're choosing SQL because of its expressiveness, you probably aren't thinking of a dialect with only 5 types (including NULL).
- ramraj07 7y agoI tried this once, creating an index on a 20 billion row table isn't fast :/
- bkanber 7y agoI haven't tried this and don't know if it would work -- but depending on the shape of your data and queries, you might not need certain indices. That is, for some workloads (especially if you're thinking of spot instances), it might be overall faster to skip the indexing and allow the query to do a full table scan. It sounds like maybe you never tried the query without the index, so I'm curious to know if there's any weight behind this theory.
- endorphone 7y agoEvery major database will load the hottest data into RAM, where the scope of "hottest" broadens to whatever amount will fit in RAM. A small percentage of them require you to confirm how much RAM it can use for this cache. Putting the data on a ramdisk just becomes entirely redundant because it's still going to create a second memory cache that it uses. Many operations do a local cache warming by running the common queries over the database before they it is brought online for processing. As a secondary note, people often under-estimate the size of their data because they don't account for all of the keys, indexes and relationships that also would be memory cached in an ideal situation.
- threeseed 7y agoYou can just write Spark SQL, set the executor memory to whatever the machine is and not worry about whether it's in RAM or not. Spark will naturally use RAM first and then disk as needed.
- juped 7y agoPostgres
- quersive 7y ago[Disclaimer: I worked on BigQuery a couple lives ago] I'd give Google BigQuery a shot. Should work fast [seconds] and scale seamlessly to [much] larger datasets and [many] more users. For a 1 TB dataset, I have a hard time imagining crafting a slow query. Maybe something outlandish like 1000[00?] joins. They also have an in-memory "BI Engine" offering, alas limited to 50GB max. On premise, there is Tableau Data Engine. I don't think they offer a SQL interface, you have to buy into the entire ecosystem. Long shot: I've been working on "most expressive query system over multiple tables" as an offshoot of some recent NLP work. Your use case piqued my interest. I'd love to help / understand it better. My contact is in my profile.
- FpUser 7y ago"My contact is in my profile" This makes for a good laugh: Queries for the masses. Contact: f'info@${user}.com'
- dana321 7y agoIf you run on MySQL/InnoDB you can set innodb_buffer_pool_size=1000Gb and it should cache your data after the first query.
- dredmorbius 7y agoI'll be That Guy and ask: what are you doing with the data, and can you change your processing or analysis to reduce the amount of data you need to touch? In my experience, it's nearly always the case that pulling in all data is not necessary, and that thinking through your goals, data, and processing can often reduce both the amount of data touched and the processing run on it massively. Look up the article on why GNU grep is so fast for a bunch of general tricks that can be employed, many of which may apply to data processing generally. Otherwise: 1. Random sampling. The Law of Large Numbers applies and affords copious advantages. There are few problems a sample of 100 - 1,000 cannot offer immense insights on, and even if you need to rely on larger samples for more detailed results, these can guide further analysis at greatly reduced computational cost. 2. Stratified sampling. When you need to include exemplars of various groups, some not highly prevalent within the data. 3. Subset your data. Divide by regions, groups, accounts, corporate divisions, demographic classifications, time blocks (day, week, month, quarter, year, ...), etc. Process chunks at a time. 4. Precompute summary / period data. Computing max, min, mean, standard deviation, and a set of percentiles for data attributes (individuals, groups, age quintiles or deciles, geocoded regions, time series), and then operating on the summarised data, can be tremendously useful. Consider data as an RRD rather than a comprehensive set (may apply to time series or other entities). Creating a set of temporary or analytic datasets / tables can be tremendously useful. As much fun as it is to write a single soup-to-nuts SQL query. 5. Linear scans typically beat random scans. If you can seek sequentially through data rather than mix-and-match, so much the better. With SSD this advantage falls markedly, but isn't completely erased. For fusion type drives (hybrid SSD/HDD) there can still be marked advantages. 6. Indexes and sorts. The rule of thumb I'd grown up with in OLAP was that indexes work when you're accessing up to 10% of a dataset, otherwise a sort might be preferred. Remember that sorts are exceedingly expensive. If at all possible, subset or narrow (see below) data BEFORE sorting. 6. Hash lookups. If one table fits into RAM, then construct a hash table using that (all the better if your tools support this natively -- hand-rolling hashing algorithms is possible, but tedious), and use that to process larger table(s). 7. "Narrow" the data. Select only the fields you need. Most especially, write only the fields you need. In SQL this is as simple as a "SELECT <fieldlist> FROM <table>" rather than "SELECT * FROM <table>". There are times you can also reduce total data throughput by recoding long records (say, geocoded names, there are a few thousands of place names in the US, using Census TIGER data, vs. placenames which may run to 22 characters ("Truth or Consequences", in NM), or even longer for international placenames. You'll need a tool to remap those later. For statistical analysis, converting to analysis variables may be necessary regardless. The number of times I've seen people dragging all fields through extensive data is ... many. Some of this can be performed in SQL, some wants a more data-related language (SAS DATA Step and awk are both largely equivalent here). Otherwise: understanding your platforms storage, memory, and virtual memory subsystems can be useful. Even as simple a practice as running "cat mydatafile > /dev/null" can often speed up subsequent processing.
- manigandham 7y agoYou don't need to store everything into RAM to get fast results. Data warehouse relational databases are designed exactly for this kind of fast SQL analysis over extremely large datasets. They use a variety of techniques like vectorized processing on compressed columnar storage to get you quick results. Google's BigQuery, AWS Redshift, Snowflake (are all hosted), or MemSQL, Clickhouse (to run yourself). Other options include Greenplum, Vertica, Actian, YellowBrick, or even GPU-powered systems like MapD, Kinetica, and Sqream. I recommend BigQuery for no-ops hosted version or MemSQL if you want a local install.
- arnon 7y agoSome of those aren't designed for data that doesn't fit in RAM. They even have it in their name (MemSQL for example).
- manigandham 7y agoAll of those support datasets that don't fit in RAM, or else they would be useless at data warehousing. MemSQL uses rowstores in memory combined with columnstores on disk. Both can be joined together seamlessly and the latest release will automatically choose and transition the table type for you as data size and access patterns change.
- bordercases 7y agoShort answer: It fits in RAM if it's <= 12288 GB
- slumdev 7y agoThe original Twitter thread is funny. This site doesn't add anything. It looks like SEO more than anything, and now that HN has linked to it, it's been successful.
- samspenc 7y agoCould you share the link to the original Twitter thread? I guess the link was changed on HN before some of us got here.
- louwrentius 7y agoThe original site yourdatafitsinram.com is now a get rich quick scheme site. Also, the site that remained wasn't ever updated. So I took some time and updated some links. I liked updating/creating this because I love the simplicity of the concept / thought behind it.
- excalibur 7y ago> Short answer: It fits in RAM if it's <= 12288 GB For a few given system configurations, and a given definition of "fits".
- siffland 7y agoFunny story, about 6 years ago we got a HP DL980 server with 1TB of memory to move from an Itanium HP-UX server. The test database was Oracle and about 600GB in size. We loaded the data and they had some query test they would run and the first time took about 45 minutes (which was several hours faster than the HPUX), They made changes and all the rest of the runs took about 5 minutes for their test to complete. Finally someone asked me and i manually dropped the buffers and cache and back to about 45 minutes. Their changes did not do anything, everything was getting cached. It was cool, but one needs to know what is happening with their data. I am just glad they asked before going to management saying their tests only took 5 minutes.
- Dylan16807 7y agoWhat an unfortunate cache-filling algorithm, though. With eight drive slots, 40 minutes of IO is about 31 megabytes per disk per second.
- rconti 7y agoThose were some poorly built systems. I worked on probably 10 of them and they not-infrequently had .. major issues. HP's support model was to send a tech out with 2 sticks of RAM, and try them in different places to try to trace memory failures... across 4 (or 8?) cassettes, and 64 sticks of ram, and 20+ minute POST times. We eventually had one server entirely replaced at HP's cost after yelling at them long enough, and that one never worked well enough to ever use in production, either. I'd say we had maybe a 70-80% success rate with those servers. They were beasts, though, with 4TB of RAM as I recall, and 288 cores.
- panzagl 7y agoI wonder how Abe Vigoda is doing...
- ghc 7y agoFine print: But it might cost you $300K CapEx or $800K/yr OpEx. Hope you have a budget!
- lmilcin 7y agoYou know what else costs? Humongous amount of servers to run silly stuff to orchestrate other silly stuff to autoscale yet else silly stuff to do stuff on your stuff that could fit into memory and be processed on a single server (+ backup, of course). Add to that small army of people, because, you know, you need specialists of variety of professions just to debug all integration issues between all those components that WOULD NOT BE NEEDED if you just decided to put your stuff in memory. Frankly, the proportion of projects that really need to work on data that could not fit in memory of a single machine is very low. I work for one of the largest banks in the world processing most of its trades from all over the world and guess what, all of it fits in RAM.
- threeseed 7y agoI really don't understand comments like this. Yes your company's data may fit in RAM. But does every intermediate data set also fit in RAM ? Because I've also worked at a bank and we had thousands of complex ETLs often needing tens to hundreds of intermediate sets along the way. There is no AWS server that can keep all of that inflight at one time. And what about your Data Analysts/Scientists. Can all of their random data sets reside in RAM on the same server too ?
- lmilcin 7y agoNever said EVERY data set fits in RAM, but that doesn't mean MOST of them don't. There is a trend, when the application is inefficient, to spend huge amount of resources on scaling it instead of making the application more efficient.
- andyjpb 7y agoBuy them a machine each. $100K has always been "cheap" for a "business computer" and today you can get more computer for that money than ever. $100K of hardware (per year or so) is small-fry compared to almost every other R&D industry out there. Just compare with the cost of debuggers, oscilloscopes and EMC labs for electronic engineers.
- bluedino 7y ago$2,000 each for 128GB LRDIMMs, 48 of those will be $100,000 and then you'll need another $20,000 to buy the rest of the server it goes in.
- belltaco 7y agoFinally a machine that can run Chrome.
- dijit 7y agoCan still “only” have 6000 tabs. (Someone did it recently with 1.5TiB of ram on a Mac Pro. Then Linus from Linus tech tips did it on Windows with 2TiB) https://youtu.be/ldDTgp5irKA https://youtu.be/ldDTgp5irKA
- deleted 7y ago[deleted]
- wmf 7y agoIf the result is faster than a $250K Hadoop cluster then you're still ahead.
- trhway 7y agothe way things today, the Hadoop cluster will be just a bit faster on that thing. 25 years ago i was suggesting clients to upgrade from 4MB to 6-8MB as it was improving their experience with our business software, these days i've already suggested a couple of customers to upgrade from 6TB and 8TB respectively ... as it would improve their experience with our business software. What's funny is that customer experience with business software back then was better than today.
- Aperocky 7y agoEverything is way too much bloat now
- malux85 7y ago
- justizin 7y agoah and it is so fucking lovely to be hired by a company which has been running everything on a single machine for 2 years.
- lunixbochs 7y agoLooks like I'm all set with my -12PB dataset fitting in ram.
- jagged-chisel 7y agoIs that “negative twelve” or did you mean ~?
- jkaptur 7y ago12TiB, not PiB! Hope you read this before you spent $50,000,000 dollars/month (although you might be able to negotiate a discount).
- randyzwitch 7y agoWhile pithy, the implication that you are going to process 12 TB of data in RAM using mostly single-threaded tools doesn't reflect reality.
- lotyrin 7y agoWhere exactly does that implication come from? Are you from a world where you need a map reduce framework + cluster to have parallelism of any kind?
- randyzwitch 7y agoNo, but I frequently see people implying that you can do your data science in Python and R as long as you can fit the data in RAM. As you mention, it's not RAM that's the limiting factor for larger data volumes, it's finding tools that exploit parallelism.
- lotyrin 7y agoI have done plenty of parallel work in python and R, so I'm still not sure what you mean.
- randyzwitch 7y agoYes, that's my point. It's too simplistic to say "well, the data fits in RAM", you have to add parallelism to make the workload tolerable. In the past, some people have done that using MapReduce or Spark, GNU parallel or just writing parallel code in their favorite language. But RAM by itself isn't the only limiting factor to whether a problem is solvable in a reasonable amount of time.
- Aeolun 7y agoIf you are using python and R, can’t you make your own parallelism?
- ksec 7y agoThe problem is DRAM price hasn't drop one bit. The lowest price floor per GB has been similar for the past decade. Roughly at $2.8/GB in 2012, 2016, and 2019. And all DRAM manufacturers has been enjoying a very profitable period. And yet our Data size continue to grow. We can fit more Data inside memory not because DRAM capacity has increase, but we are simply increasing memory channels.
- gameswithgo 7y agobuying the ram probably cheaper than buying database liceneses and/or the machines to run them and/or the time you spend making things fast enough when using them.
- markus_zhang 7y agojust wait for China to enter the mkt in thia decade...
- yorwba 7y agoYou mean 2 years ago? https://www.anandtech.com/show/12455/chinese-xian-uniic-semiconductors-starts-to-sell-ddr4-memory-chips-modules https://www.anandtech.com/show/12455/chinese-xian-uniic-semi... It doesn't seem to have brought prices down much.
- keanzu 7y agoEveryone knows that DRAM prices have been in a collapse since early this year, but last week DRAM prices hit a historic low point on the spot market. Based on data the Memory Guy collected from spot-price source InSpectrum, the lowest spot price per gigabyte for branded DRAM reached $2.59 last week. https://thememoryguy.com/dram-prices-hit-historic-low/ https://thememoryguy.com/dram-prices-hit-historic-low/ You've selected out the low points on the graph: 2012, 2016, and 2019, most of the time DRAM has not been available at these prices. Now is definitely the time to load up on RAM.
- jimnotgym 7y ago> Everyone knows that DRAM prices have been in a collapse I'm afraid I had no idea that had happened at all. > since early this year, Since..when?..now? Feb 12th is pretty early in the year AFAIAC
- iaabtpbtpnn 7y agoSure, my data fits in RAM if you take one of those example machines, which we have, and which are currently hosting dozens of production VMs, and instead dedicate the whole thing to my database. I'd love that, but it's never going to happen.
- zaltekk 7y agoThe Amazon listing has an Azure instance type and links to Azure docs. A quick search shows that you can get at least 24TiB from AWS: https://aws.amazon.com/ec2/instance-types/high-memory https://aws.amazon.com/ec2/instance-types/high-memory
- louwrentius 7y agoThanks for pointing out my sloppy mistake. I have chosen for the cloud options to only select virtual instances that can be spun up on demand. The high-memory instances you link to are purpose-build. On the other hand, it is true, they do exist.
- jszymborski 7y ago"Can I afford to fit my data in RAM?" is a whole other site, I presume...
- vb6sp6 7y ago"Can you afford to not put your data in RAM" is another one.
- birdyrooster 7y ago"Did you even need all of this data?" is the one after that one
- dredmorbius 7y agoI'd argue it's the first.
- takeda 7y agoThe point is that the alternative is far more expensive (taking from the tweet linked on the page that inspired it)
- skwb 7y agoI remember back in the early 2010's that a large selling point of SAS (besides the ridiculous point that R/python were freeware and therefore cannot be trusted on important projects ) was that it can chew through large data sets that perhaps couldn't be moved into RAM (but maybe it takes a week or whatever....). This was a fairly salient point, and remember circa 2012/2013 struggling to fit large bioinfomatics data into an older iMac with base R.
- vasili111 7y agoI have not tested it myself but now there is a disk.frame ( https://github.com/xiaodaigh/disk.frame https://github.com/xiaodaigh/disk.frame ). As far as I know, other options exist too.
- dredmorbius 7y agoSAS Institute have long claimed this. It's been provably bullshit for decades. In practice, an awk script frequently ran circles around processing. On a direct basis, awk corresponds quite closely to the SAS DATA Step (and was intended to be paired with tools such as S, the precursor to R, for similar types of processing). The fact that awk had associative arrays (which SAS long lacked, it's since ... come up with something along those lines) and could perform extremely rapid sort-merge or pattern matches (equivalent to SAS data formats, which internally utilise a b-tree structure) helped. With awk, sort, unique, and a few hand-rolled statistics awk libraries / scripts, you can replace much of the functionality of SAS. And that's without even touching R or gnuplot, each of which offer further vast capabilities. And at an aggreable annual license fee.
- staticassertion 7y agoPretty cool that single boxes have > 10 terabytes of RAM. I would definitely imagine that most workloads rarely need more than a few to a few hundred TB in memory, since you may have petabytes of data but you probably touch very little of it.
- H8crilA 7y agoOne of the best things you can often do for latency/throughput improvement is to just mlock() the data. Yes, it fits. Get a more expensive machine and 5x the throughput in a single day with a config change.
- dmos62 7y agoI just upgraded my laptop to 3 GB. Feeling a bit behind on the times.
- scarejunba 7y agoYou can fit 48 TB on a HPE MC990 X though I'm pretty sure that's got one of those NUMA architectures that SGI had with the UV 3000 or whatever. I remember jokingly telling my team to spend the millions of dollars we did expanding our clusters with one of these and just processing in RAM. I honestly don't think I did the analysis to make sure it would be actually better. It was 'jokingly' because we couldn't afford three of these machines anyway. The clusters had the property that we could lose some large fraction of nodes and still operate, we could expand slightly sub-linearly, etc. which are all lovely properties. It would have been neat, though. Ahhhh imagine the luxury of just loading things into memory and crunching the whole thing in minutes instead of hours. Gives me shivers.
- andyjpb 7y agoThis is why IBM have historically done so well. You didn't have to have all the capital. They'd rent you the machine on a long lease and you could say you'd bought a million dollar computer. Sun tried to get into similar markets but it was always a tougher deal on minis and micros.
- rbanffy 7y agoIIRC, the largest IBM z15 can have up to 40 TiB of RAM. The website needs updating.
- einpoklum 7y ago> Does my data fit in RAM? 1. If you have to ask, then either it doesn't now, or it doesn't sometimes. So assume it doesn't. 2. If you can use a cluster, then maybe. 3. In some senses, it doesn't matter. How so? Reading and writing from RAM is very slow, latency-wise, for today's processors. If I can bend the truth a little, it's a bit like a fast SSD. So, if you can up the bandwidth to disk enough, it becomes kind of comparable. Well, if you can use 16 PCIe 4.0 lanes, it's roughly 24 GB/sec effective bandwidth, which is roughly half of your memory bandwidth. Now it's true that in real-life systems it's usually just 4 lanes, but it's very doable to change that with a nice card. 4. DIMM-form-factor non-volatile memory may increase memory sizes much more.
- yongjik 7y agoEven more importantly, does your data have to fit in RAM? There are tons of problems that need to process large data, but touch each item just once (or a few times). You can go a really long way by storing them in disk (or some cloud storage like S3) and writing a script to scan through them. I know, pretty obvious, but somehow escapes many devs.
- sevensor 7y agoThere's also the "not all memory is RAM" trick: plan ahead with enough swap to fit all the data you intend to process, and just pretend that you have enough RAM. Let the virtual memory subsystem worry about whether or not it fits in RAM. Whether this works well or horribly depends on your data layout and access patterns.
- mrmrcoleman 7y agoInteresting. Can you provide some examples of where this is the correct approach?
- tmountain 7y agoThis is how mongodb originally managed all its data. It used memory mapped files to store the data and let the underlying OS memory management facilities do what they were designed to do. This saved the mongodb devs a ton of complexity in building their own custom cache and let them get to market much faster. The downside is that since virtual memory is shared between processes, other competing processes could potentially mess with your working set (pushing warm data out, etc). The other downside is that since your turning over the management of that “memory” to the OS, you lose fine grained control that can be used to optimize for your specific use case.
- xchaotic 7y agoExcept nowadays with Docker / Kubr you can safely assume the db engine will be the only tenant of a given vm /pod whatever so I think it’s better to let OS do memory management than fight it
- Jupe 7y agoSome pedantry... Raw RAM space is not the issue... it's indexing and structure of the data that makes it process-able. If you just need to spin through the data once, there's no need to even put all of it in RAM - just stream it off disk and process it sequentially. If you need to join the data, filter, index, query it, you'll need a lot more RAM than your actual data. Database engines have their own overhead (system tables, query processors, query parameter caches, etc.) And, this all assumes read-only. If you want to update that data, you'll need even more for extents, temp tables, index update operations, etc.
- LeoPanthera 7y agoI often use RAM disks for production services. I'm sure that I shouldn't. It feels very lazy, and I know that I'm probably missing out on the various "right" ways to do things. But it works so well and it's so easy. It's a really difficult habit to kick. Obligatory joke: "Oh boy, virtual memory! Now I can have a really big RAM disk!"
- deepsun 7y agoThere is AWS and Azure, but no GCP. Why?
- EdwardDiego 7y agoTelling me that my (example) 24TB data set fits into RAM, because it fits into 24TB on an AWS instance designed for SAP that's so expensive that it's price on inquiry, isn't overly helpful. May I suggest that a) they consider applications need RAM too and b) that if the price is POI, it's probably better to mention "But you know, 188 m5.8xlarges might be cheaper"
- sriku 7y agoThe question of whether my storage mechanism fits budget is more relevant, no? It costs less than 4cents per hour to store 1TB on S3 but an X1 with 1TB of RAM costs $13 or so per hour on Aws. So the issue is whether what you pay for is worth the result you're computing.
- known 7y agoReminds me of https://en.wikipedia.org/wiki/List_of_Ultrabook_models https://en.wikipedia.org/wiki/List_of_Ultrabook_models
- hkt 7y agoHonestly thought this would be a page that says "yes"