4 ms·

I used to work as a web editor at a daily newspaper. One of my daily responsibilities was to write and post short blurbs about lottery results and beach surf c

by jawns 13y ago

I used to work as a web editor at a daily newspaper. One of my daily responsibilities was to write and post short blurbs about lottery results and beach surf conditions. They both were extremely formulaic, so I wrote push-button scripts that would fetch the data, parse it, and generate stories. Saved me time, which I used to do other, more important things. Some people have said, "Why not just display the raw data?" Well, to save you the trouble of having to analyze the data. For instance, suppose I've written a Powerball results script that lists the winning numbers, along with which states had winners. If I'm just spitting out the raw data, then people might miss the fact that one of the winners was from our state -- whereas if I'm generating a story, I'm going to make that the lede. Similarly, the magic of a company like Narrative Science and its Quill service is not in taking a ton of data and filling in a bunch of blanks with the values, Mad Libs style. It's in analyzing the data and figuring out what the most important parts are, and constructing a story around those findings. In other words, it's not hard for a bot to write, "The Tigers played the Wolves yesterday. The Tigers won 1-0. The Tigers' John Johnson scored a home run." It's more difficult to write, "The Tigers' Tom Thompson pitched the first no-hitter of his career yesterday in a 1-0 game against the Wolves. Remarkably, the Wolves' pitcher, Dobbie Dobson, was moments away from forcing the game into extra innings with his own no-hitter, when, in the ninth inning with two outs and two strikes on the board, the Tigers' John Johnson hit a home run." (I'm not a baseball writer, so I'm probably bungling it, but you get the point.)

exelius 13y ago
News in this sense is just catching up to the financial world. Big finance has had bots that parse news feeds and raw data for YEARS as part of their high frequency trading (HFT) platforms as a form of arbitrage. So let's say a major earthquake happens in San Francisco and the Google HQ falls into a bottomless pit. The trading algorithms will know about the earthquake within seconds, and they will also know about likely know about Google's obliteration from speech-to-text of police scanners. The millisecond these reports hit the wire, the HFT algorithms would likely short the hell out of Google's stock immediately while also placing positive bets on Google's competitors as well as construction companies and building suppliers. And this was going on 10 years ago; they're probably light years ahead of this by now. It's also not very hard to write the second piece given the right machine learning technology. Feed a decade's worth of human-generated articles into an algorithm (which could also be paired with marketing metrics like page views, avg time spent on page, etc.) and the algorithm picks out the types of statistics it should look up from the databases and how those statistics should be incorporated into the story. Heck, if you wanted to get really fancy, you could try to pick the major themes of the game out of a transcript of the play-by-play or specific emotional moments of the game by measuring the level of crowd noise or twitter traffic. A computer will have better information recall, better research skills and better perspective to write stories that are more appealing to people. Heck, they could even write a story on the fly designed to appeal specifically to the individual's preferences. I understand that there's a place for reporters, but I don't think it's going to be written event-based reporting for much longer.
kafkaesque 13y ago
I'm new to programming and web development, but just out of curiosity, if I wanted to write a script that fetches data and writes/generates product descriptions, is there an online resource that explains how to do this in general terms? Also, would writing it be something a beginner/novice like myself could do or does it require pretty advanced skills? Thanks for any info you can provide.

by jawns 13y ago

exelius 13y ago

News in this sense is just catching up to the financial world. Big finance has had bots that parse news feeds and raw data for YEARS as part of their high frequency trading (HFT) platforms as a form of arbitrage. So let's say a major earthquake happens in San Francisco and the Google HQ falls into a bottomless pit. The trading algorithms will know about the earthquake within seconds, and they will also know about likely know about Google's obliteration from speech-to-text of police scanners. The millisecond these reports hit the wire, the HFT algorithms would likely short the hell out of Google's stock immediately while also placing positive bets on Google's competitors as well as construction companies and building suppliers. And this was going on 10 years ago; they're probably light years ahead of this by now. It's also not very hard to write the second piece given the right machine learning technology. Feed a decade's worth of human-generated articles into an algorithm (which could also be paired with marketing metrics like page views, avg time spent on page, etc.) and the algorithm picks out the types of statistics it should look up from the databases and how those statistics should be incorporated into the story. Heck, if you wanted to get really fancy, you could try to pick the major themes of the game out of a transcript of the play-by-play or specific emotional moments of the game by measuring the level of crowd noise or twitter traffic. A computer will have better information recall, better research skills and better perspective to write stories that are more appealing to people. Heck, they could even write a story on the fly designed to appeal specifically to the individual's preferences. I understand that there's a place for reporters, but I don't think it's going to be written event-based reporting for much longer.

kafkaesque 13y ago

I'm new to programming and web development, but just out of curiosity, if I wanted to write a script that fetches data and writes/generates product descriptions, is there an online resource that explains how to do this in general terms? Also, would writing it be something a beginner/novice like myself could do or does it require pretty advanced skills? Thanks for any info you can provide.