3 ms·

Metadata, not data, is what drags your database down

BeefWellington 5y ago
This entire post is essentially an ad for a product they've developed and should come wrapped in a giant [citation needed] because they don't really source any of their claims. Ten years ago, the typical ratio between data and metadata was 1,000:1. This means that a data unit (file, block, or object) that is 32k in size would have metadata of around 32 bytes. Existing data engines were able to handle these amounts of data quite effectively. Since then, however, the ratio has shifted significantly towards metadata. The ratio can now vary from 1000:1 when the object size is large and 1:10 when the object is really small. The explosion of metadata has a direct and immediate impact on our data infrastructures. I'd love to see this breakdown and for what specific use cases. The entire post goes on at length about how your "database engine overheating is dangerous" because the metadata could inflate well past the number of records you keep but it doesn't really prove out that assertion or provide use-case examples. It's little better than advertising for a given service. Another example: In addition to the continuous increase in the volume of unstructured data—such as documents and audio/video files—the rapid propagation of connected devices and IoT sensors creates a metadata sprawl that is expected to accelerate going forward. The data itself is typically very small (for example, an alphanumeric read of a sensor), but it is accompanied by large chunks of metadata (location, timestamp, description) that might be even larger than the data itself. These are additional pieces of data and may be as or more important than the sensor read itself. This is not what database metadata is, even according to the author's own statement of what metadata is: So what’s causing our data engines to heat up? The main reason is the overwhelming pace of data growth, especially in metadata, which is the silent data engine killer. Metadata refers to any piece of information about the data—such as indexes, for example—that makes it easier to find and work with data. This means that metadata doesn’t have a pre-fixed schema to fit a database (which is usually in a key-value format); rather, it’s a general description of the data that is created by various systems and devices. In the example given, the pieces of information being stored do in fact have a pre-fixed schema. So do indexes. This is a very poorly constructed argument for creating a new product. Either Metadata is "extra stuff you need to store unnecessarily" or it's specifically metadata used in RDBMS parlance, which would be things like indexes, table schema, etc. The author appears to be using inflation of storage numbers because of the former to convince you it will affect the latter. I'm open to there being concrete examples of what the author asserts but I'd actually like to see them. Most of this article is just vague statements and don't even fit with one another. Database sizing is not new; It's been a thing for decades, one might argue nearly two centuries or more if we're considering data systems that aren't purely electronic -- I would suggest that the Dewey Decimal system is essentially the underpinning inspiration for relational database indexes and I'm sure there's examples out there that predate it.

This entire post is essentially an ad for a product they've developed and should come wrapped in a giant [citation needed] because they don't really source any of their claims. Ten years ago, the typical ratio between data and metadata was 1,000:1. This means that a data unit (file, block, or object) that is 32k in size would have metadata of around 32 bytes. Existing data engines were able to handle these amounts of data quite effectively. Since then, however, the ratio has shifted significantly towards metadata. The ratio can now vary from 1000:1 when the object size is large and 1:10 when the object is really small. The explosion of metadata has a direct and immediate impact on our data infrastructures. I'd love to see this breakdown and for what specific use cases. The entire post goes on at length about how your "database engine overheating is dangerous" because the metadata could inflate well past the number of records you keep but it doesn't really prove out that assertion or provide use-case examples. It's little better than advertising for a given service. Another example: In addition to the continuous increase in the volume of unstructured data—such as documents and audio/video files—the rapid propagation of connected devices and IoT sensors creates a metadata sprawl that is expected to accelerate going forward. The data itself is typically very small (for example, an alphanumeric read of a sensor), but it is accompanied by large chunks of metadata (location, timestamp, description) that might be even larger than the data itself. These are additional pieces of data and may be as or more important than the sensor read itself. This is not what database metadata is, even according to the author's own statement of what metadata is: So what’s causing our data engines to heat up? The main reason is the overwhelming pace of data growth, especially in metadata, which is the silent data engine killer. Metadata refers to any piece of information about the data—such as indexes, for example—that makes it easier to find and work with data. This means that metadata doesn’t have a pre-fixed schema to fit a database (which is usually in a key-value format); rather, it’s a general description of the data that is created by various systems and devices. In the example given, the pieces of information being stored do in fact have a pre-fixed schema. So do indexes. This is a very poorly constructed argument for creating a new product. Either Metadata is "extra stuff you need to store unnecessarily" or it's specifically metadata used in RDBMS parlance, which would be things like indexes, table schema, etc. The author appears to be using inflation of storage numbers because of the former to convince you it will affect the latter. I'm open to there being concrete examples of what the author asserts but I'd actually like to see them. Most of this article is just vague statements and don't even fit with one another. Database sizing is not new; It's been a thing for decades, one might argue nearly two centuries or more if we're considering data systems that aren't purely electronic -- I would suggest that the Dewey Decimal system is essentially the underpinning inspiration for relational database indexes and I'm sure there's examples out there that predate it.