5 ms·
> When your data doesn’t fit in memory: the basic techniques Uhm... Isn't it called a database? Jokes aside, one of the core use cases of things like database
by gwbas1c 7y ago
> When your data doesn’t fit in memory: the basic techniques
Uhm... Isn't it called a database?
Jokes aside, one of the core use cases of things like databases is random access to a part of data that's too large to fit in RAM.
- dev_dull 7y agoMy assumption was that the system had finite ram available, whether that's serialized in a database or read directly into the program.
- gwbas1c 7y agoSo? Typically "database" implies a product like SQLite or Postgres. These handle many multiples of RAM size. In-memory databases are a thing, but that's a very specialized use case.
- dfox 7y agoSomewhat surprising fact about most PostgreSQL deployments is that most of the frequently accessed tuples will fit into RAM and it handles this usecase really well (to the extent that people who don’t know any better do not see any issue with LIKE ‘%whatewer%’ over 100M tuple table, it just takes 1.5s, so what...)
- PhearTheCeal 7y agoPretty much all SQL DBs support pagination, and many libraries that abstract the DB away hide this as an iterator.
- VikingCoder 7y agoHi, once upon a time, I worked with medical image data. I literally had a DBA suggest storing the volumetric data in a database. I worked to diligently to explain to him why he was incorrect. When he resisted my explanations and tried to push forward with his plan, I worked diligently to get him fired. I was successful.
- Twisol 7y agoFor those of us who don’t work with medical image data, can you share why he was incorrect? I realize that relational databases are not the right box to fit certain kinds of data into, but you have to put your data somewhere that allows it to be efficiently manipulated. What is that if not a “data base”?
- pbourke 7y agoStuff like this, back in the day: a RAIDed filesystem. Nowadays: a blob object store (like S3 or Azure Blob Store) or distributed FS such as HDFS
- aspaceman 7y agoMedical imaging data is typically dense 3D grids of density samples. A relational database could hold a pointer to the file, but not the 5GB-500GB dataset files. One dataset is typically a folder of various files, metadata, and other information representing an entire filesystem. Back in the day, the datasets would be RAID across several machines. Now they're virtual filesystems. The database could manage this info, but not hold the actual dataset itself. I imagine this is what the commentter means. The DBA probably said "it all needs to be in the database", and that just doesn't make sense.
- xapata 7y agoHolds true for lots of data varieties; the database maintains links to compressed archives.
- user5994461 7y agoShort version is that a medical image, like a typical MRI, is a 3D picture at least 1024 x 1024 x 1024 pixels, meaning each image is multiple gigabytes. SQL databases don't do well at storing huge binary blobs like that.
- Twisol 7y agoRight, but SQL databases aren't the only kind of database. I gather that the guy in the original anecdote was referring to a relational database specifically, but not all databases are relational. The filesystem is a perfectly good hierarchical document-based database, and it's worth thinking about it that way in both theory and in practice.