7 ms·

Ah, I remember this specific file from my time at the ECB 15-ish years ago. IIRC it was by far the most downloaded file on the ECB website. Tons of people, inc

Ah, I remember this specific file from my time at the ECB 15-ish years ago. IIRC it was by far the most downloaded file on the ECB website. Tons of people, including many financial institutions, downloaded it daily, and used it to update their own systems. IIRC #2 in the minutes immediately after the daily scheduled time for publishing this file, there was a massive traffic spike. It was a conscious decision to make it a simple CSV file (once unzipped): it made it possible to serve the file reliably, fast, and with little resources needed. The small team responsible at the time for the ECB’S public website was inordinately proud of the technical decisions made to serve this data in a single static file. And rightly so.

andylynch 3y ago
I know this file too, I was one of them. Of all the data sources we used, it was the best to work with.
nlehuen 3y ago
It's a bit scary to imagine the consequences if this file was somehow corrupted (wrong column headers for instance)!
- thghtihadanacct 3y ago
  Just wait until you inherit a service that, sans documentation, pulls in a web resource file that suddenly is no longer available :(
  - dizhn 3y ago
    scale8 recently archived their GitHub repo without explanation and right after that the url for a GDPR opt-in list they were using changed. The product is now dead because the code doesn't handle the situation where that file might not be available. (Scale8 is a web analytics and tag management software like google analytics)
    - dizhn 3y ago
      I forgot to mention this error check even breaks the admin dashboard meaning you can't even see historical data anymore. Just in case anybody else is in the same boat you can change the url in the code to the new one yourself.
mstade 3y ago
Do you know why they decided to host a zip file instead of just hosting the CSV and relying on HTTP compression?
- theamk 3y ago
  HTTP compression is optional, so they either have to compress on the fly (wasting cpu) or provide multiple versions (complicating setup and deployment) or make some HTTP clients not work. simgle zip file is really the easiest solution for cases when the file must absolutely be compressed
  - sethhochberg 3y ago
    Many webservers allow you to serve a compressed file (stored on disk) and _decompress_ when a client specifically can't support the compressed encoding. Since most clients should support compression, this means you only use the CPU for the rarer case where the uncompressed data is required. Eg, http://nginx.org/en/docs/http/ngx_http_gunzip_module.html http://nginx.org/en/docs/http/ngx_http_gunzip_module.html
    - nikau 3y ago
      Or you can just zip it explicitly and remove complexity and future issues it they have to move to another web server platform.
      - MengerSponge 3y ago
        But what if an analyst needs to access this data and run their regressions on a potato? Surely that use case is worth adding a few libraries to handle.
        CJefferson 3y ago
        What machine could anyone be running an analysis on where unzip is a limiting factor?
tharkun__ 3y ago
This is something that powers a lot of data interchange but lots of people don't realize because they never work with older systems like that. It's not shiny. No frameworks. I used to work for a large, old company whose products you have all bought and buy (and which shall remain nameless) about 15 years ago. I worked on data interchange between the systems used for record keeping of said products and various other downstream or parallel systems (same purpose, separate system because left over from merger/acquisition). So basically bulk data import and export. The product was 15 years old at that point. Data interchange was via various fixed width or delimited (CSV but the C might be some other character or character sequence) files transferred to and from various SFTP servers. It's been a while but there were probably like 20 or 30 different such data sources or exports going in and out. Worked like a charm. I bet they're still used without much change. The frontend was being rewritten at the time (old one was in Smalltalk).
- keepamovin 3y ago
  I'm using this type of simple approach to build a SaaS right now. We need to spin up many VPS and provision them, and the fastest way to do that is with rsync and ssh. But we didn't stop there: this SaaS for our open source browser product is entirely built like this: behind the scenes it's a collection of bash scripts that implement and execute the business operation of the SaaS. So basically, it's a command-line interface to the SaaS. Think of it this way, say I didn't have a website, with login, and "click a button to open a browser", but instead people would write me letters, send me cheques, or call me on the phone. Then I can serve their requests manually, at the command line. The reason I made it like this was: - clear separation between thin web front-end and actual business logic - nice command-line interface (options, usage, help, clear error messages) to business logic for maintenance and support to jump on and fix things - inheritance of operating system permissions and user process isolation - highly testable implementation Maybe this is dumb, but I really like it. To me it's an architecture and approach that makes sense. I'm sure this is not new, and I think a lot of good quality operations must be built via this way. I highly align with the author's stance of the composition of a few simple command line tools to get the job done. Perhaps we can call this "unix driven development", or "unix-philosophy backend engineering" browser product: https://github.com/dosyago/BrowserBoxPro https://github.com/dosyago/BrowserBoxPro (saas coming soonish)
leokennis 3y ago
Wow. I’m just thinking of the hoops I’d have to go through to host changing data as a static CSV/ZIP at my employer. Architects would complain ZIP isn’t a format conform to their specs for this purpose. Compliance would complain there needs to be a check that no private informatie leaked. Risk that I should prevent bad actors from downloading the file. Web people that I need an approved change to add stuff to the site.
- deleted 3y ago
  [deleted]
- TeMPOraL 3y ago
  Yeah, but modern, approved practices don't really address any of that - they just hide it under a pile of extra complexity that's too big for most devs and managers' attention span, effectively disappearing it.
scotty79 3y ago
> IIRC #2 in the minutes immediately after the daily scheduled time for publishing this file, there was a massive traffic spike. Perfect usecase for bittorrent protocol imho. Funny thing is you can serve files with bittorrent protocol from Amazon S3 without any additional complexity or charges I think.
geysersam 3y ago
Do you remember why they decided to zip it instead of just relying on the compression in http?
- ric2b 3y ago
  Does HTTP compression perform well if you're streaming a file? Wouldn't it need to have access to the entire file to have the same result as zipping it?
  - geysersam 3y ago
    Yes it performs well. You don't need access to the entire file.

andylynch 3y ago

I know this file too, I was one of them. Of all the data sources we used, it was the best to work with.

nlehuen 3y ago

It's a bit scary to imagine the consequences if this file was somehow corrupted (wrong column headers for instance)!

thghtihadanacct 3y ago
Just wait until you inherit a service that, sans documentation, pulls in a web resource file that suddenly is no longer available :(
- dizhn 3y ago
  scale8 recently archived their GitHub repo without explanation and right after that the url for a GDPR opt-in list they were using changed. The product is now dead because the code doesn't handle the situation where that file might not be available. (Scale8 is a web analytics and tag management software like google analytics)
  - dizhn 3y ago
    I forgot to mention this error check even breaks the admin dashboard meaning you can't even see historical data anymore. Just in case anybody else is in the same boat you can change the url in the code to the new one yourself.

mstade 3y ago

Do you know why they decided to host a zip file instead of just hosting the CSV and relying on HTTP compression?

theamk 3y ago
HTTP compression is optional, so they either have to compress on the fly (wasting cpu) or provide multiple versions (complicating setup and deployment) or make some HTTP clients not work. simgle zip file is really the easiest solution for cases when the file must absolutely be compressed
- sethhochberg 3y ago
  Many webservers allow you to serve a compressed file (stored on disk) and _decompress_ when a client specifically can't support the compressed encoding. Since most clients should support compression, this means you only use the CPU for the rarer case where the uncompressed data is required. Eg, http://nginx.org/en/docs/http/ngx_http_gunzip_module.html http://nginx.org/en/docs/http/ngx_http_gunzip_module.html
  - nikau 3y ago
    Or you can just zip it explicitly and remove complexity and future issues it they have to move to another web server platform.
    - MengerSponge 3y ago
      But what if an analyst needs to access this data and run their regressions on a potato? Surely that use case is worth adding a few libraries to handle.
      - CJefferson 3y ago
        What machine could anyone be running an analysis on where unzip is a limiting factor?

tharkun__ 3y ago

This is something that powers a lot of data interchange but lots of people don't realize because they never work with older systems like that. It's not shiny. No frameworks. I used to work for a large, old company whose products you have all bought and buy (and which shall remain nameless) about 15 years ago. I worked on data interchange between the systems used for record keeping of said products and various other downstream or parallel systems (same purpose, separate system because left over from merger/acquisition). So basically bulk data import and export. The product was 15 years old at that point. Data interchange was via various fixed width or delimited (CSV but the C might be some other character or character sequence) files transferred to and from various SFTP servers. It's been a while but there were probably like 20 or 30 different such data sources or exports going in and out. Worked like a charm. I bet they're still used without much change. The frontend was being rewritten at the time (old one was in Smalltalk).

keepamovin 3y ago
I'm using this type of simple approach to build a SaaS right now. We need to spin up many VPS and provision them, and the fastest way to do that is with rsync and ssh. But we didn't stop there: this SaaS for our open source browser product is entirely built like this: behind the scenes it's a collection of bash scripts that implement and execute the business operation of the SaaS. So basically, it's a command-line interface to the SaaS. Think of it this way, say I didn't have a website, with login, and "click a button to open a browser", but instead people would write me letters, send me cheques, or call me on the phone. Then I can serve their requests manually, at the command line. The reason I made it like this was: - clear separation between thin web front-end and actual business logic - nice command-line interface (options, usage, help, clear error messages) to business logic for maintenance and support to jump on and fix things - inheritance of operating system permissions and user process isolation - highly testable implementation Maybe this is dumb, but I really like it. To me it's an architecture and approach that makes sense. I'm sure this is not new, and I think a lot of good quality operations must be built via this way. I highly align with the author's stance of the composition of a few simple command line tools to get the job done. Perhaps we can call this "unix driven development", or "unix-philosophy backend engineering" browser product: https://github.com/dosyago/BrowserBoxPro https://github.com/dosyago/BrowserBoxPro (saas coming soonish)

leokennis 3y ago

Wow. I’m just thinking of the hoops I’d have to go through to host changing data as a static CSV/ZIP at my employer. Architects would complain ZIP isn’t a format conform to their specs for this purpose. Compliance would complain there needs to be a check that no private informatie leaked. Risk that I should prevent bad actors from downloading the file. Web people that I need an approved change to add stuff to the site.

deleted 3y ago
[deleted]
TeMPOraL 3y ago
Yeah, but modern, approved practices don't really address any of that - they just hide it under a pile of extra complexity that's too big for most devs and managers' attention span, effectively disappearing it.

scotty79 3y ago

> IIRC #2 in the minutes immediately after the daily scheduled time for publishing this file, there was a massive traffic spike. Perfect usecase for bittorrent protocol imho. Funny thing is you can serve files with bittorrent protocol from Amazon S3 without any additional complexity or charges I think.

geysersam 3y ago

Do you remember why they decided to zip it instead of just relying on the compression in http?

ric2b 3y ago
Does HTTP compression perform well if you're streaming a file? Wouldn't it need to have access to the entire file to have the same result as zipping it?
- geysersam 3y ago
  Yes it performs well. You don't need access to the entire file.