3 ms·
When my gf and I met she had recently started her Phd in environmental sustainability science and was specifically looking into the fishing quota system in Icel
by MrDresden 6y ago
When my gf and I met she had recently started her Phd in environmental sustainability science and was specifically looking into the fishing quota system in Iceland with mathematical modeling in mind.
The catch, haul and other related data is available on the relevant government agency's webpage down to ship granularity level through a rather nice web interface. It however contains vast amounts of data, and there is no way to get the data in bulk form (even asking the agency resulted in a 'No this is sensitive data that we can't share in bulk form' answer).
So I was rather shocked when I heard her say that she had started copying the data by hand. Knowing that it would take her ages, and would probably never be possible for her to do, I wrote a small python scraper & data transforming tool and set about scraping couple of GiBs of data into a normalized sqlite database, which I then gave her to use with her R code.
We have been together ever since (4+ years).