5 ms·

The problem is named named values often come with significant performance penalties in terms of storage. With data points in the billions it becomes significant

by cerved 5y ago

Someone 5y ago
I think that’s less of a problem than often claimed. For formats such as csv, parquet or a SQL database that store the names of fields only once, overhead is constant, regardless of number of items. So, it only can be a problem for formats such as xml or json that repeat the names of fields in every record. Those happen to be formats with variable record length, so you can’t index into such files; the only way to process them is in their entirety. If so, and storage size is a problem, you can compress the files. Your typical LZW variant will, if the files are large enough, eventually encode both the “(lat=“ and “,lon=“ parts to single codes of (typically) 12 bits each. That will happen after the fifth occurrence of such strings, so fairly soon). That’s 24 bits of overhead per item. Significant, but if you use xml or json, chances are you’re already giving up way more by storing floating point values as text strings. So, that leaves json or xml files that each store only a few items per file. With a typical file system block size of 8 kB, those already give up 4 kB on average per file.
- cerved 5y ago
  > If so, and storage size is a problem, you can compress the files I'm literally in the process of processing 100gb compressed spatial json and it's not enjoyable
eyelidlessness 5y ago
That’s a fair point. In my opinion, more languages should support something like TypeScript’s labeled tuple elements[1] for use cases like that. 1: I wasn’t able to find this in the main docs for some reason? Only the release notes. https://www.typescriptlang.org/docs/handbook/release-notes/typescript-4-0.html#labeled-tuple-elements https://www.typescriptlang.org/docs/handbook/release-notes/t...
- WorldMaker 5y ago
  Yeah, it's a API communication issue that tools like Typescript are designed to solve with "type information". Typescript's labeled tuples are a relatively recent addition, but an extremely welcome one especially here in this specific [lat, lon] versus [lon, lat] communication problem.
  - eyelidlessness 5y ago
    It’s also super handy for preserving parameter names when dealing with rest/spread in both runtime and static type positions, and even for naming new ones in static types for composition.

The problem is named named values often come with significant performance penalties in terms of storage. With data points in the billions it becomes significant

by cerved 5y ago

The problem is named named values often come with significant performance penalties in terms of storage. With data points in the billions it becomes significant

Someone 5y ago

I think that’s less of a problem than often claimed. For formats such as csv, parquet or a SQL database that store the names of fields only once, overhead is constant, regardless of number of items. So, it only can be a problem for formats such as xml or json that repeat the names of fields in every record. Those happen to be formats with variable record length, so you can’t index into such files; the only way to process them is in their entirety. If so, and storage size is a problem, you can compress the files. Your typical LZW variant will, if the files are large enough, eventually encode both the “(lat=“ and “,lon=“ parts to single codes of (typically) 12 bits each. That will happen after the fifth occurrence of such strings, so fairly soon). That’s 24 bits of overhead per item. Significant, but if you use xml or json, chances are you’re already giving up way more by storing floating point values as text strings. So, that leaves json or xml files that each store only a few items per file. With a typical file system block size of 8 kB, those already give up 4 kB on average per file.

cerved 5y ago
> If so, and storage size is a problem, you can compress the files I'm literally in the process of processing 100gb compressed spatial json and it's not enjoyable

eyelidlessness 5y ago

That’s a fair point. In my opinion, more languages should support something like TypeScript’s labeled tuple elements[1] for use cases like that. 1: I wasn’t able to find this in the main docs for some reason? Only the release notes. https://www.typescriptlang.org/docs/handbook/release-notes/typescript-4-0.html#labeled-tuple-elements https://www.typescriptlang.org/docs/handbook/release-notes/t...

WorldMaker 5y ago
Yeah, it's a API communication issue that tools like Typescript are designed to solve with "type information". Typescript's labeled tuples are a relatively recent addition, but an extremely welcome one especially here in this specific [lat, lon] versus [lon, lat] communication problem.
- eyelidlessness 5y ago
  It’s also super handy for preserving parameter names when dealing with rest/spread in both runtime and static type positions, and even for naming new ones in static types for composition.