8 ms·

A lot of comments in here are poking fun at how little data it is relative to a commercial data mining operation. The data they process and what they do with it

A lot of comments in here are poking fun at how little data it is relative to a commercial data mining operation. The data they process and what they do with it is worth more to society than any number of petabytes crunched to target ads. Processing petty petabytes is not praiseworthy. If you focus on the headline, you'll miss the point. The point is they used open source technology to process public data for reporting once the government stopped updating its own tools.

VectorLock 8y ago
Pettybytes. I like that.
- deleted 8y ago
  [deleted]
dang 8y ago
Maybe we'll just take the gigabytes per day bit out of the title so it doesn't trigger people.
- deleted 8y ago
  [deleted]
- virtuabhi 8y ago
  Would it be possible to put the gigabytes back in the title? In general, ETL of gigabytes of data can involve complicated operations, e.g., use of statistical models. And, the utility of data is not determined by the size of data. One has to be a pretty petty person to make fun of this article for data sizes.
  - rainbowmverse 8y ago
    That was almost every comment before I posted. All of them seem to be grayed out now.
  - dang 8y ago
    You're right on the facts, but that's not the way a forum like this works. Like it or not (and probably no one likes it), the way it works is that a minor irrelevant provocation in the title completely determines the discussion. The solution is to take out the minor provocation, even if in principle we shouldn't need to.
danso 8y ago
As someone who has done a lot of data processing in journalism, I've found the engineering issues aren't usually about scale, but involve the harder problems around data cleaning/wrangling/updating. Particularly interoperability with opaque government systems, and transformation/delivery to a variety of users, including ones with a high variation in technical skill (i.e. journalists), and an extremely picky tolerance for public-facing errors in the finished product. I started ProPublica's Dollars for Docs [0], and the initial project involved <1M rows. Gathering the data involved writing scrapers for more than a dozen independent sources (i.e. drug companies) using whatever format they felt like publishing (HTML, PDF, Flash applet). This data had to be extracted and collated without error for the public-facing website, as mistakenly listing a doctor had a very high chance of legal action. Hardest part by far was distributing the data and coordinating the research among reporters and interns internally, and also with about 10 different newsrooms. I had to work with outside investigative journalists who, when emailed a CSV text file, thought I had attached a broken Excel file. Today, the D4D has millions of records, and the government now its own website [1] for the official dissemination of the standardized data. I have a few shell scripts that can download the official raw data -- about ~30GB of text when unzipped -- and import it into a SQLite DB in about 20 minutes. The data for the first D4D investigation probably could've fit in a single Google Sheet, but it still took months to properly wrangle. But the computational bottleneck wasn't the size of data. One of other tricky issues is that data management isn't easy in a newsroom. Devops is not only not a traditional priority, but anyone working with data has to do it fairly fast, and they have to move on almost immediately to another project/domain when done. There's not a lot of incentive or resources to get past a collection of hacky scripts, so it's really cool (to several-years-ago me) to see a guide about how to get things started in a more proper, maintainable way. [0] https://projects.propublica.org/docdollars/ https://projects.propublica.org/docdollars/ [1] https://www.cms.gov/openpayments/ https://www.cms.gov/openpayments/ edit: for a more technical detailed example of newsroom data issues, check out Adrian Holovaty's (creator of Django) 3-part essay, "Sane Data Updates Are Harder than You Think", which details the ETL process for Chicago crime data: https://source.opennews.org/articles/sane-data-updates-are-harder-you-think/ https://source.opennews.org/articles/sane-data-updates-are-h... Here's a great write-up by Jeremy Merrill, who helped overhaul the D4D project after I left. Unlike me, Jeremy was a proper engineer: https://www.propublica.org/nerds/heart-of-nerd-darkness-why-dollars-for-docs-was-so-difficult https://www.propublica.org/nerds/heart-of-nerd-darkness-why-...
TAForObvReasons 8y ago
Not just "open source technology", battle-tested tried-and-true technologies. Articles like these should remind us that we don't need to keep rebuilding tools to solve the same set of problems -- sometimes, all it takes is some familiarity with what exists
- rainbowmverse 8y ago
  I honestly didn't know Make did all that. I thought it was just a build script thing for complex software. This was enlightening.
WillEngler 8y ago
Hear hear! It's a good solution for this domain using reliable, free tools. And other folks without a lot of compute resources or experience setting up Spark clusters or whatever can easily adapt their approach. Hats off.
qubax 8y ago
> The data they process and what they do with it is worth more to society than any number of petabytes crunched to target ads. No it isn't. You are getting defensive for no reason. If propublica ceases to exist or never existed, it wouldn't matter a single bit to the world. You could even argue the world would be better off. > Processing petty petabytes is not praiseworthy. From a technical point and many other ways, it is. I don't get why you are getting offended by people making a jab at the scant amount of data. Last I checked, hacker news is a technology oriented site. And from a technology point of view, what pro publica is doing is a joke. It's a toy amount of data. Why not just say pro publica is not a technology company and hence people shouldn't expect technological feats of wonder? > The point is they used open source technology to process public data for reporting once the government stopped updating its own tools. Which is something I could have done on a lazy afternoon all by myself. It isn't anything to be impressed about. But good for them anyways.
- LMYahooTFY 8y ago
  So people can't make a point about any aspects of accomplishment that are not engineering feats? Whether or not Propublica has produced something of value for society seems, at an absolute minimum, highly debatable. Your comment is the one that seems defensive....
- mc92nxk 8y ago
  The way qubax sees the world is as important as qubax makes propublica’s view out to be You’re being too literal. Yes if nothing exists it doesn’t matter Look in the mirror and realize you’re just one of thousands that could do this in an afternoon Given the “big picture” context, your personal computer skills aren’t much to brag about either. Literally good with computers. Get in line. Were they being defensive? Or offering a context to consider the value from?
- danso 8y ago
  ProPublica is not a technology company, it's a non-profit investigative journalism outlet. They and countless other journalism/civic orgs would likely be happy for you to show them up by whipping up usable ETL scripts relevant in their respective domains. Since it all involves public open data you don't have to wait for anyone's permission.
nineteen999 8y ago
The production databases on the project I build and support around are around 100GB in size - tiny. But if they didn't work correctly - ambulances wouldn't arrive at the correct location on time, nor fire engines nor emergency services workers. Covering an area of 1.25 million square kilometres, supporting 40,000 first responders helping to protect 8 million people. Of course the databases are not the only important part of an emergency services network such as ours, but they are a critical component. I would rather work on a project like this any day than working to prop up some faceless advertising/data collection behemoth such as Facebook or Google.

VectorLock 8y ago

Pettybytes. I like that.

deleted 8y ago
[deleted]

dang 8y ago

Maybe we'll just take the gigabytes per day bit out of the title so it doesn't trigger people.

deleted 8y ago
[deleted]
virtuabhi 8y ago
Would it be possible to put the gigabytes back in the title? In general, ETL of gigabytes of data can involve complicated operations, e.g., use of statistical models. And, the utility of data is not determined by the size of data. One has to be a pretty petty person to make fun of this article for data sizes.
- rainbowmverse 8y ago
  That was almost every comment before I posted. All of them seem to be grayed out now.
- dang 8y ago
  You're right on the facts, but that's not the way a forum like this works. Like it or not (and probably no one likes it), the way it works is that a minor irrelevant provocation in the title completely determines the discussion. The solution is to take out the minor provocation, even if in principle we shouldn't need to.

danso 8y ago

As someone who has done a lot of data processing in journalism, I've found the engineering issues aren't usually about scale, but involve the harder problems around data cleaning/wrangling/updating. Particularly interoperability with opaque government systems, and transformation/delivery to a variety of users, including ones with a high variation in technical skill (i.e. journalists), and an extremely picky tolerance for public-facing errors in the finished product. I started ProPublica's Dollars for Docs [0], and the initial project involved <1M rows. Gathering the data involved writing scrapers for more than a dozen independent sources (i.e. drug companies) using whatever format they felt like publishing (HTML, PDF, Flash applet). This data had to be extracted and collated without error for the public-facing website, as mistakenly listing a doctor had a very high chance of legal action. Hardest part by far was distributing the data and coordinating the research among reporters and interns internally, and also with about 10 different newsrooms. I had to work with outside investigative journalists who, when emailed a CSV text file, thought I had attached a broken Excel file. Today, the D4D has millions of records, and the government now its own website [1] for the official dissemination of the standardized data. I have a few shell scripts that can download the official raw data -- about ~30GB of text when unzipped -- and import it into a SQLite DB in about 20 minutes. The data for the first D4D investigation probably could've fit in a single Google Sheet, but it still took months to properly wrangle. But the computational bottleneck wasn't the size of data. One of other tricky issues is that data management isn't easy in a newsroom. Devops is not only not a traditional priority, but anyone working with data has to do it fairly fast, and they have to move on almost immediately to another project/domain when done. There's not a lot of incentive or resources to get past a collection of hacky scripts, so it's really cool (to several-years-ago me) to see a guide about how to get things started in a more proper, maintainable way. [0] https://projects.propublica.org/docdollars/ https://projects.propublica.org/docdollars/ [1] https://www.cms.gov/openpayments/ https://www.cms.gov/openpayments/ edit: for a more technical detailed example of newsroom data issues, check out Adrian Holovaty's (creator of Django) 3-part essay, "Sane Data Updates Are Harder than You Think", which details the ETL process for Chicago crime data: https://source.opennews.org/articles/sane-data-updates-are-harder-you-think/ https://source.opennews.org/articles/sane-data-updates-are-h... Here's a great write-up by Jeremy Merrill, who helped overhaul the D4D project after I left. Unlike me, Jeremy was a proper engineer: https://www.propublica.org/nerds/heart-of-nerd-darkness-why-dollars-for-docs-was-so-difficult https://www.propublica.org/nerds/heart-of-nerd-darkness-why-...

TAForObvReasons 8y ago

Not just "open source technology", battle-tested tried-and-true technologies. Articles like these should remind us that we don't need to keep rebuilding tools to solve the same set of problems -- sometimes, all it takes is some familiarity with what exists

rainbowmverse 8y ago
I honestly didn't know Make did all that. I thought it was just a build script thing for complex software. This was enlightening.

WillEngler 8y ago

Hear hear! It's a good solution for this domain using reliable, free tools. And other folks without a lot of compute resources or experience setting up Spark clusters or whatever can easily adapt their approach. Hats off.

qubax 8y ago

> The data they process and what they do with it is worth more to society than any number of petabytes crunched to target ads. No it isn't. You are getting defensive for no reason. If propublica ceases to exist or never existed, it wouldn't matter a single bit to the world. You could even argue the world would be better off. > Processing petty petabytes is not praiseworthy. From a technical point and many other ways, it is. I don't get why you are getting offended by people making a jab at the scant amount of data. Last I checked, hacker news is a technology oriented site. And from a technology point of view, what pro publica is doing is a joke. It's a toy amount of data. Why not just say pro publica is not a technology company and hence people shouldn't expect technological feats of wonder? > The point is they used open source technology to process public data for reporting once the government stopped updating its own tools. Which is something I could have done on a lazy afternoon all by myself. It isn't anything to be impressed about. But good for them anyways.

LMYahooTFY 8y ago
So people can't make a point about any aspects of accomplishment that are not engineering feats? Whether or not Propublica has produced something of value for society seems, at an absolute minimum, highly debatable. Your comment is the one that seems defensive....
mc92nxk 8y ago
The way qubax sees the world is as important as qubax makes propublica’s view out to be You’re being too literal. Yes if nothing exists it doesn’t matter Look in the mirror and realize you’re just one of thousands that could do this in an afternoon Given the “big picture” context, your personal computer skills aren’t much to brag about either. Literally good with computers. Get in line. Were they being defensive? Or offering a context to consider the value from?
danso 8y ago
ProPublica is not a technology company, it's a non-profit investigative journalism outlet. They and countless other journalism/civic orgs would likely be happy for you to show them up by whipping up usable ETL scripts relevant in their respective domains. Since it all involves public open data you don't have to wait for anyone's permission.

nineteen999 8y ago

The production databases on the project I build and support around are around 100GB in size - tiny. But if they didn't work correctly - ambulances wouldn't arrive at the correct location on time, nor fire engines nor emergency services workers. Covering an area of 1.25 million square kilometres, supporting 40,000 first responders helping to protect 8 million people. Of course the databases are not the only important part of an emergency services network such as ours, but they are a critical component. I would rather work on a project like this any day than working to prop up some faceless advertising/data collection behemoth such as Facebook or Google.