4 ms·

The further I get into my career as a data scientist (formerly a software engineer) the more I think I see what Kay was getting at in this thread. I spend a hu

The further I get into my career as a data scientist (formerly a software engineer) the more I think I see what Kay was getting at in this thread. I spend a huge chunk of my time swimming in a sea of data that people have carelessly amassed on the assumption that data is inherently valuable. And frequently I come to the conclusion that this data has negative value. The people who collected it failed to record enough information about the data's provenance. So I don't know what kind of processes produced it, how and why it was collected, any transformations that might have happened on the way to storing it, etc. Without that information, I simply cannot know for sure what any of it really means with enough precision to be able to draw valid conclusions from it. The best I can do is advise caution, say that this data can help us form hypotheses whose only true use is helping us form a plan to collect new data that will help us answer the question we have. What I'm typically asked to do instead is make a best guess and call it good. The former option is just the scientific method. The latter option is the very essence of pseudoscience. That second word in my job title gives me some anxiety. I'm keenly aware that most people are more fond of the word "science" than they are of the intellectual practice we call science. The people who like to amass data are no exception.

throwanem 2y ago
This makes sense to me. Hickey seems to have taken his cite of "data"'s definition as "a thing given" as axiomatic, requiring no further thought on the implicit following questions like "given by whom?" and "by what means?", and this severely limits the scope of his analysis versus Kay's, this I think being what had them talking past one another. In industry, the incentives seem very rarely to line up such that questions like those are welcome.
- KingMob 2y ago
  Yeah. As a general rule of thumb, dictionaries are simplistic, and extremely lagging, signals about everything a word can be about. No offense to dictionaries, since their goal is to be a succinct, useful, and universal summary of words, but it's usually a mistake to trot them out in an argument. Would you take a dictionary's definition as the final matter on a complex philosophical topic, like epistemology? Or its starting point? It gets even worse in the realm of something like politics, where different groups have contended over, and actively fought to redefine, the meanings of words over time.
nonrandomstring 2y ago
> negative value Yes. In cybersecurity we already say data is a toxic asset. It can be 'wrong' or cause harm in so many more ways than a narrow band of intended good. This thread touches a concurrent topic from less-wrong about pianos and quality. Reality is infinitely nuanced, and the finer detail the more important to the person who "cares" (Pirsig said quality and care were flip sides of the same thing and Quine had a similar thought about how all data has meaning in line-spectrum of context.) "Data" today is collected without care, for it's use, quality or effects. The horror is that we are training machines on that very low quality data and expecting high quality results.
fellowniusmonk 2y ago
I was the data and analytics part of a global team at HomeAway as we were struggling to finally release a "free listing / pay per booking" model to catch up with airbnb, I wired up tracking and a whole bunch of stuff for behavioral analysis including our GA implementation at the time. Before launch we kept seeing a step in the on-boarding flow where we saw massive drop-off and I kept redflagging it, eventually the product engineering team responsible for that step came back with a bunch of splunk logs saying they couldn't see the drop off and our analytics must be wrong "because it's js", which was just an objectively weird take. For "silo" reasons this splunk logging was used by product and no one else trusted it or did anything actionable with it as far as I could tell other than internally measuring app response times. I would not unflag the step and one PM in particular started getting very upset about this and saying our implementation was wrong and he roped a couple senior engineers in for support. I personally started regression testing that page based on our data and almost immediately caught that any image upload over ~1mb was not working and neither was mobile safari, turned out they had left their mvp code in place and it used flash or something stupid so it would break on image size and some browsers just wouldn't work at all. It was updated a couple weeks before launch and the go live was as good as could be expected. To this day I have no clue how this particular team had so misconfigured their server side logging that it HID the problem, but you see it all the time, if you don't know what you're doing and don't know how to validate things your data will actually sabotage you.
- QuercusMax 2y ago
  Did they just assume "no errors on the server side, so the problem can't exist"? That's bizarre.
  - fellowniusmonk 2y ago
    It was odder than no errors, they weren't seeing any funnel drop off at all. It wasn't worth investigating and fixing for them, at the time I figured they were excluding traffic incorrectly or didn't know how to properly query for "session" data... could have been any number of things though.
albert_e 2y ago
> I spend a huge chunk of my time swimming in a sea of data ... > The people who collected it failed to record enough information about the data's provenance. >> failed to record enough information >>> enough information careful what you wish for :)
yifanl 2y ago
This reminded me of the classic blogpost: https://ansuz.sooke.bc.ca/entry/23 https://ansuz.sooke.bc.ca/entry/23 There's intent that can't be stored: I can write down the word potato on a shopping list, and again on a recipe, and they represent totally different ideas with exactly the same characters. I interpret the first by going to the market, the other in the kitchen. I'm sure there are many people eagerly trying to solve this problem by just storing more metadata so we can interpret which interpreter we want, but now we need increasingly more layers of interpreters, and now you're asking for a machine that simulates the universe. I find myself aligning that just processing "data" forever has limits, and our continued refusal to recognize that is going to be very costly.
- andrelaszlo 2y ago
  > I can write down the word potato on a shopping list, and again on a recipe, and they represent totally different ideas with exactly the same characters. I think uncle Ludwig would agree. https://en.m.wikipedia.org/wiki/Philosophical_Investigations#Meaning_as_use https://en.m.wikipedia.org/wiki/Philosophical_Investigations...
teqsun 2y ago
May I ask why you moved from software engineering to data science?
- indoordin0saur 2y ago
  Not the OP but as a data engineer I can tell you that a data scientist with strong SWE credentials is an asset on any DS team.
- morkalork 2y ago
  Personally it was because I was staring down a career of making endless CRUD apps and became disillusioned. I read all those cool data science articles in the 2010s and thought it was way more interesting. Joke's on me though, now I'm still disillusioned and a data scientist.
  - antimemetics 2y ago
    Don’t worry you didn’t make a bad choice, it’s disillusionment all the way down in this industry
- bunderbunder 2y ago
  I like statistics.
Terr_ 2y ago
> The people who collected it failed to record enough information about the data's provenance. This feels a bit a debate that I (general SDE) keep having with Product folks who propose some sort of magic system that collates and displays "Applicants" across a bunch of unauthenticated form-submits and third-party customer databases, "because we already have the data". Yeah, but most if it is fundamentally untrustworthy and/or must be aggressively siloed to prevent cross-customer data poisoning. We could try to an in-house identity-graph system, but we'd at least need to record something about levels of confidence for different steps or relationships or assumptions. For example, it would be very bad for privacy if a visitor could put my public e-mail address into a form/wizard, and then the next step "helpfully" autofills or asks to confirm data like the associated (real) name or (real) home address. Alternately, someone could submit data with a correct phone number or e-mail address, but named "Turdy McPooperson" at "123 Ignore This Application Drive." Now the real user comes by, gets pissed when the system "greets" them with an insult, and anything they do get thrown in the trash by users who see it displayed under a combined profile named Turdy McPooperson.
layer8 2y ago
This seems just like the realization that data and information are not the same thing. Maybe we need information scientists? ;)
- antimemetics 2y ago
  Dada scientist
morkalork 2y ago
I wasn't there for the beginning but I got dropped into a corp that had amassed a "data lake" with 20k tables of almost worthless data. One senior data scientist lost their pet project to what turned out to be contaminated data that leaked outcomes into his model features. Basically checked out mentally and eventually quit after. It was a hopeless environment, engeers in one country were building products and completely silo'd away from the people who were supposed to use their data.
aaron695 2y ago
[dead]

throwanem 2y ago

This makes sense to me. Hickey seems to have taken his cite of "data"'s definition as "a thing given" as axiomatic, requiring no further thought on the implicit following questions like "given by whom?" and "by what means?", and this severely limits the scope of his analysis versus Kay's, this I think being what had them talking past one another. In industry, the incentives seem very rarely to line up such that questions like those are welcome.

KingMob 2y ago
Yeah. As a general rule of thumb, dictionaries are simplistic, and extremely lagging, signals about everything a word can be about. No offense to dictionaries, since their goal is to be a succinct, useful, and universal summary of words, but it's usually a mistake to trot them out in an argument. Would you take a dictionary's definition as the final matter on a complex philosophical topic, like epistemology? Or its starting point? It gets even worse in the realm of something like politics, where different groups have contended over, and actively fought to redefine, the meanings of words over time.

nonrandomstring 2y ago

> negative value Yes. In cybersecurity we already say data is a toxic asset. It can be 'wrong' or cause harm in so many more ways than a narrow band of intended good. This thread touches a concurrent topic from less-wrong about pianos and quality. Reality is infinitely nuanced, and the finer detail the more important to the person who "cares" (Pirsig said quality and care were flip sides of the same thing and Quine had a similar thought about how all data has meaning in line-spectrum of context.) "Data" today is collected without care, for it's use, quality or effects. The horror is that we are training machines on that very low quality data and expecting high quality results.

fellowniusmonk 2y ago

I was the data and analytics part of a global team at HomeAway as we were struggling to finally release a "free listing / pay per booking" model to catch up with airbnb, I wired up tracking and a whole bunch of stuff for behavioral analysis including our GA implementation at the time. Before launch we kept seeing a step in the on-boarding flow where we saw massive drop-off and I kept redflagging it, eventually the product engineering team responsible for that step came back with a bunch of splunk logs saying they couldn't see the drop off and our analytics must be wrong "because it's js", which was just an objectively weird take. For "silo" reasons this splunk logging was used by product and no one else trusted it or did anything actionable with it as far as I could tell other than internally measuring app response times. I would not unflag the step and one PM in particular started getting very upset about this and saying our implementation was wrong and he roped a couple senior engineers in for support. I personally started regression testing that page based on our data and almost immediately caught that any image upload over ~1mb was not working and neither was mobile safari, turned out they had left their mvp code in place and it used flash or something stupid so it would break on image size and some browsers just wouldn't work at all. It was updated a couple weeks before launch and the go live was as good as could be expected. To this day I have no clue how this particular team had so misconfigured their server side logging that it HID the problem, but you see it all the time, if you don't know what you're doing and don't know how to validate things your data will actually sabotage you.

QuercusMax 2y ago
Did they just assume "no errors on the server side, so the problem can't exist"? That's bizarre.
- fellowniusmonk 2y ago
  It was odder than no errors, they weren't seeing any funnel drop off at all. It wasn't worth investigating and fixing for them, at the time I figured they were excluding traffic incorrectly or didn't know how to properly query for "session" data... could have been any number of things though.

albert_e 2y ago

> I spend a huge chunk of my time swimming in a sea of data ... > The people who collected it failed to record enough information about the data's provenance. >> failed to record enough information >>> enough information careful what you wish for :)

yifanl 2y ago

This reminded me of the classic blogpost: https://ansuz.sooke.bc.ca/entry/23 https://ansuz.sooke.bc.ca/entry/23 There's intent that can't be stored: I can write down the word potato on a shopping list, and again on a recipe, and they represent totally different ideas with exactly the same characters. I interpret the first by going to the market, the other in the kitchen. I'm sure there are many people eagerly trying to solve this problem by just storing more metadata so we can interpret which interpreter we want, but now we need increasingly more layers of interpreters, and now you're asking for a machine that simulates the universe. I find myself aligning that just processing "data" forever has limits, and our continued refusal to recognize that is going to be very costly.

andrelaszlo 2y ago
> I can write down the word potato on a shopping list, and again on a recipe, and they represent totally different ideas with exactly the same characters. I think uncle Ludwig would agree. https://en.m.wikipedia.org/wiki/Philosophical_Investigations#Meaning_as_use https://en.m.wikipedia.org/wiki/Philosophical_Investigations...

teqsun 2y ago

May I ask why you moved from software engineering to data science?

indoordin0saur 2y ago
Not the OP but as a data engineer I can tell you that a data scientist with strong SWE credentials is an asset on any DS team.
morkalork 2y ago
Personally it was because I was staring down a career of making endless CRUD apps and became disillusioned. I read all those cool data science articles in the 2010s and thought it was way more interesting. Joke's on me though, now I'm still disillusioned and a data scientist.
- antimemetics 2y ago
  Don’t worry you didn’t make a bad choice, it’s disillusionment all the way down in this industry
bunderbunder 2y ago
I like statistics.

Terr_ 2y ago

> The people who collected it failed to record enough information about the data's provenance. This feels a bit a debate that I (general SDE) keep having with Product folks who propose some sort of magic system that collates and displays "Applicants" across a bunch of unauthenticated form-submits and third-party customer databases, "because we already have the data". Yeah, but most if it is fundamentally untrustworthy and/or must be aggressively siloed to prevent cross-customer data poisoning. We could try to an in-house identity-graph system, but we'd at least need to record something about levels of confidence for different steps or relationships or assumptions. For example, it would be very bad for privacy if a visitor could put my public e-mail address into a form/wizard, and then the next step "helpfully" autofills or asks to confirm data like the associated (real) name or (real) home address. Alternately, someone could submit data with a correct phone number or e-mail address, but named "Turdy McPooperson" at "123 Ignore This Application Drive." Now the real user comes by, gets pissed when the system "greets" them with an insult, and anything they do get thrown in the trash by users who see it displayed under a combined profile named Turdy McPooperson.

layer8 2y ago

This seems just like the realization that data and information are not the same thing. Maybe we need information scientists? ;)

antimemetics 2y ago
Dada scientist

morkalork 2y ago

I wasn't there for the beginning but I got dropped into a corp that had amassed a "data lake" with 20k tables of almost worthless data. One senior data scientist lost their pet project to what turned out to be contaminated data that leaked outcomes into his model features. Basically checked out mentally and eventually quit after. It was a hopeless environment, engeers in one country were building products and completely silo'd away from the people who were supposed to use their data.

aaron695 2y ago

[dead]