4 ms·

A big part of my job is to ingest data from hundreds of providers to process. A provider that will push data to my S3 bucket, as parquet files, is the best I c

A big part of my job is to ingest data from hundreds of providers to process. A provider that will push data to my S3 bucket, as parquet files, is the best I can think of. I don't want no provider managed database to host, stupidly huge zipped CSV dumps, or whatever weird technology the provider can think of. Seriously reading a file from S3 and stream processing the parquet is not the peak of engineering. What is there to complain about... Some of the insanity I had to deal with: - 350TB of zipped CSVs which columns and delimiter vary across time, shipped on HDD by mail - Self hosting a MSSQL server for the provider to update - Good old FTP polling at more or less predictable times - Fancy you-name-it-shiny-new-tech remote database accounts - The crappy client http website with custom SQL inspired query language to download zipped extracts - The web 2.0 rest API, billed per request - Snowflake data warehouse data lake big data Please, don't be that guy, just push parquet flat files in my bucket.

racional 2y ago
Unfortunately if "that guy" is your customer then your real job is to make them happy. Which means, by definition, putting up with their crazy-ass formats, services, and data stores. Or they're some company / agency providing the data for free essentially, but whom you in any case have precisely zero hope of bending them toward your preferences on this matter on any foreseeable timeline. Or perchance they're internal - you're stuck with lobbying and politics (again, to get them to bend your way). Meanwhile you've got stuff to build and such. Given the associated costs and benefits; and the fact that it's just a day job, after all -- stupidly huge CSV files may not be so bad.
- Galanwe 2y ago
  > Unfortunately if "that guy" is your customer then your real job is to make them happy. Usually the power relationship is not layout like that. Data providers rarely compete on technology, because their differentiating factor is the uniqueness of the data itself. The clients will put up with whatever insanity the provider will deliver the data with anyway. The history repeats itself: The older a provider, the more involved and complex its data offering. Once they have dozens of datasets, packages, clustering, a fleet of sales in the wild negociating with clients and a whole billing system on top of that, they just won't change it. That's how you end up with these insanities.

racional 2y ago

Unfortunately if "that guy" is your customer then your real job is to make them happy. Which means, by definition, putting up with their crazy-ass formats, services, and data stores. Or they're some company / agency providing the data for free essentially, but whom you in any case have precisely zero hope of bending them toward your preferences on this matter on any foreseeable timeline. Or perchance they're internal - you're stuck with lobbying and politics (again, to get them to bend your way). Meanwhile you've got stuff to build and such. Given the associated costs and benefits; and the fact that it's just a day job, after all -- stupidly huge CSV files may not be so bad.

Galanwe 2y ago
> Unfortunately if "that guy" is your customer then your real job is to make them happy. Usually the power relationship is not layout like that. Data providers rarely compete on technology, because their differentiating factor is the uniqueness of the data itself. The clients will put up with whatever insanity the provider will deliver the data with anyway. The history repeats itself: The older a provider, the more involved and complex its data offering. Once they have dozens of datasets, packages, clustering, a fleet of sales in the wild negociating with clients and a whole billing system on top of that, they just won't change it. That's how you end up with these insanities.