4 ms·

Nice review of what sounds like an interesting book. One small objection: "Cleaning data is the fundamental task that defines data science as a role: statistic

Nice review of what sounds like an interesting book. One small objection: "Cleaning data is the fundamental task that defines data science as a role: statisticians and economists buy clean data from somebody." Man I wish. I spent more than a year of my PhD (cumulatively and not all at once) cleaning various kinds of data. I joked w/ an adviser once that I thought I had signed up for an economics PhD but it turned out I was getting a PhD in data cleaning. Finding pre-cleaned, nice data for a project is like finding a $100 bill on the street. Doesn't happen.

netizen-936824 5y ago
I started doing some work with ML models and immediately realised that around 80-90% of my code was dedicated entirely to data cleaning and preparation. Glad I got some practice before diving into a PhD honetly
Mc91 5y ago
I met a Renaissance guy at a party once. His description of data cleaning was "it's like you have a pile of gold bars buried beneath a couple of tons of shit".
bko 5y ago
I really liked the book. I wrote my own review trying to dissect their secret sauce and I believe it to be their data pipeline. The focused on it from the very beginning and wasn't sexy at all. From the book: > Piecing together a custom-built database, Straus purchased historic commodity-price data on magnetic tape from an Indiana-based firm called Dunn & Hargitt, then merged it with the historic information others in the firm already had amassed. ... Using an Apple II computer, Straus and others wrote a program to collect and store their growing data trove. > Over time, Straus and his colleagues created and discovered additional historical pricing data, helping Ax develop new predictive models relying on Carmona’s suggestions. Some of the weekly stocktrading data they’d later find went back as far as the 1800s, reliable information almost no one else had access to. At the time, the team couldn’t do much with the data, but the ability to search history to see how markets reacted to unusual events would later help Simons’s team build models to profit from market collapses and other unexpected events, helping the firm trounce markets during those periods. They also took market prices as nearly devoid of real life meaning. I was originally very skeptical of this approach as it requires immense discipline and rigor to avoid data mining. You see economists do that where they create a signal for something like the 23 month lag of GDP. Why 23 month lag? Because 1-22 were insignificant. But they seem to get away with it due to their strong academic background. > Here’s what was really unique: The paper didn’t try to identify or predict these states using economic theory or other conventional methods, nor did the researchers seek to address why the market entered certain states. Simons and his colleagues used mathematics to determine the set of states best fitting the observed pricing data; their model then made its bets accordingly. The whys didn’t matter, Simons and his colleagues seemed to suggest, just the strategies to take advantage of the inferred states. https://mleverything.substack.com/p/how-renaissance-technologies-solved-the-market-part-1-2814eb271dc3 https://mleverything.substack.com/p/how-renaissance-technolo...

netizen-936824 5y ago

I started doing some work with ML models and immediately realised that around 80-90% of my code was dedicated entirely to data cleaning and preparation. Glad I got some practice before diving into a PhD honetly

Mc91 5y ago

I met a Renaissance guy at a party once. His description of data cleaning was "it's like you have a pile of gold bars buried beneath a couple of tons of shit".

bko 5y ago

I really liked the book. I wrote my own review trying to dissect their secret sauce and I believe it to be their data pipeline. The focused on it from the very beginning and wasn't sexy at all. From the book: > Piecing together a custom-built database, Straus purchased historic commodity-price data on magnetic tape from an Indiana-based firm called Dunn & Hargitt, then merged it with the historic information others in the firm already had amassed. ... Using an Apple II computer, Straus and others wrote a program to collect and store their growing data trove. > Over time, Straus and his colleagues created and discovered additional historical pricing data, helping Ax develop new predictive models relying on Carmona’s suggestions. Some of the weekly stocktrading data they’d later find went back as far as the 1800s, reliable information almost no one else had access to. At the time, the team couldn’t do much with the data, but the ability to search history to see how markets reacted to unusual events would later help Simons’s team build models to profit from market collapses and other unexpected events, helping the firm trounce markets during those periods. They also took market prices as nearly devoid of real life meaning. I was originally very skeptical of this approach as it requires immense discipline and rigor to avoid data mining. You see economists do that where they create a signal for something like the 23 month lag of GDP. Why 23 month lag? Because 1-22 were insignificant. But they seem to get away with it due to their strong academic background. > Here’s what was really unique: The paper didn’t try to identify or predict these states using economic theory or other conventional methods, nor did the researchers seek to address why the market entered certain states. Simons and his colleagues used mathematics to determine the set of states best fitting the observed pricing data; their model then made its bets accordingly. The whys didn’t matter, Simons and his colleagues seemed to suggest, just the strategies to take advantage of the inferred states. https://mleverything.substack.com/p/how-renaissance-technologies-solved-the-market-part-1-2814eb271dc3 https://mleverything.substack.com/p/how-renaissance-technolo...