14 ms·
The related problem that I see actually more often is the "you don't have big data" problem. You know, in data science, you see people spending hours writing p
by shubb 4y ago
The related problem that I see actually more often is the "you don't have big data" problem.
You know, in data science, you see people spending hours writing pandas scripts that replicate a few clicks in excel for a one of analysis. You see datasets of a few gigabytes being processed with spark when SQL would be fine. You see ML techniques being thrown at questions that could be answered simply and reliably with basic statistical tests.
Especially in the B2C space a lot of companies, departments, products don't actually have a lot of customers and certainly not many decision makers. The N number is always going to be low. You can just talk to people. Let's say you are doing pretty well and running a SaS with 1000 corporate customers paying a million each - that's a billion dollar revenue - you can just talk to them. Certainly you can just talk to every single person who signs the cheque and those are the only people that matter.
And which is easier - putting together a thorough suite of A/B tests or getting some real customers to use your app on video and talking to them about what they are finding annoying, useful, missing? I see less people do that than you'd think.
- bell-cot 4y agoUnfortunately, your reality-driven approach has ~zero emotional appeal for most managers, exec's, and alpha-data-scientist wanna-be's.
- germinalphrase 4y agoWhy? Inadequately “technical”?
- jkingsbery 4y agoI think there are lots of reasons why. One possible reason: no one whose job it is to write Python scripts was ever promoted for making an Excel spreadsheet when that is the simpler and more practical approach. And no manager of people who write Python scripts is going to be able to use that Excel spreadsheet to sell "I need more responsibility and head count." People tend to follow incentives, rather than focusing on making wise decisions.
- butUhmErm 4y agoExcel has a history of forced format updates, breaking incompatibility. I know people who banned it because they got tired of marching to MSs upgrade beat. Python 2 to 3 upgrade aside, can’t really say the same about the language. There are a number of good arguments out there that might violate an engineers perception, which one might call a cognitive data model built through training and experience. There is no theory that makes any given engineering path “wiser” than others. Just engineers chasing incentives to be engineers.
- TimTheTinker 4y ago> People tend to follow incentives, rather than focusing on making wise decisions. This is the key issue. Solving it isn't easy -- it requires people who are wise, and wisdom is a scarce commodity.
- wtetzner 4y agoEven wise people likely follow the incentives. What is wise about doing something that your employer doesn’t reward in exchange for doing something that they will reward?
- TimTheTinker 4y agoIt's wise to do what's morally right, regardless of the consequences.
- marcosdumay 4y ago- "You just talked to them and concluded this? What certainty you can have on this conclusion, and how can we trust you just didn't want it to be true from the start?" A few slides showing the data, a boring 10 minutes about methodology, and finally the conclusion brings an air of reliability that you can't replicate for knowledge instead of data.
- strikelaserclaw 4y agoOur field is filled with people who want to use the most technical approach possible to solve a non issue, their paychecks probably depend on it.
- robertlagrant 4y agoData has CYA appeal.
- bell-cot 4y agoNeeding to CYA also has pretty low emotional appeal for managers, exec's, and alpha-data-scientist wanna-be's. (Until it's just about too late, obviously.) And recall Mark Twain's old quip about lies & statistics. The more & bigger data that the folks who control the data & analysis have, the easier it is to make sure that those meet their own emotional & political needs.
- ekianjo 4y agoTalking to people is not going to help you either. You end up getting a lot of noise and making sense of what you hear is difficult. When you keep probing you will get to hear stuff thats not really critical and just often made up because you ask too many questions. Classical trap of market research.
- peteradio 4y agoYou both make good arguments, there must be a middle here. I doubt you can uncover what your customer wants very well without just talking to them, but maybe they wind up misleading you sometimes. A/B testing to discover a customer wants a whole different paradigm isn't possible.
- Cthulhu_ 4y agoThis is true; what customers SAY they want doesn't necessarily corellate with what they will actually use or pay for. I mean I worked on an app where in one part, the end user could upload CSV files to be used. What they SAID they wanted was basically a full data management system and RESTful API to enforce constraints, data validation, record retrieval and updating, etc. What they probably wanted was an excel sheet. I dislike how my employer was like "yeah sure if you pay for it" to them.
- dspillett 4y ago> what customers SAY they want doesn't necessarily corellate with what they will actually use A key cause of this in many cases is that the stake-holders you talk to do not work closely with the end users of the system. Talking to the right people can help a lot, though unfortunately as a 3rd party this is not usually anywhere near your realm of control. The other issue is them knowing what they have and wish to store, but not knowing what outputs are going to be needed down the line. That is harder to fix, but having some good industry knowledge within your company can be a great help on such matters – you can then sometimes preempt client needs if the people holding that knowledge are keeping an active eye on changes (for instance new/planned regulations that might be coming into force in X weeks/months/years).
- 4y ago
- stevofolife 4y agoWhy not both…
- shubb 4y agoMaybe what I wrote comes off a bit one sided - I'm really urging people to do what actually makes sense in their specific context - which can be both!
- packetlost 4y ago> You know, in data science, you see people spending hours writing pandas scripts that replicate a few clicks in excel for a one of analysis I mean, having an Excel doc at all usually implies hour(s) of work formatting the data in structured manner. Sometimes collective decades of work depending on how much heavy lifting your 15GB .xlsx is doing.
- selykg 4y agoThis is why I've adopted R and Python for the data work I do. I have a bunch of exported data (CSV files) that I use. Manipulating the structure and format is 90% of the work. I wrote the scripts once, now I can reuse that for everything instead of playing games getting those CSV files (dates in particular) to play nicely. Even a one off analysis is actually FASTER in Pandas because I've done the work of farting around with the formatting. Now I can just write the necessary analysis code, rather than deal with the formatting. That said, my data analytics work is seriously small potatoes compared to many. But I can write a quick pivot table using Dplyr faster than I can do it in Excel.
- majormajor 4y agoOften that work exists regardless of if a table of processed data that engineering formatted and schema-fied is dumped out to Excel or queried over SQL into Pandas... I've seen this myself: the person who "naively" downloads that table and plays around in excel finds interesting things that the person who was using Pandas hadn't, because the code to manipulate columns and do certain types of calcs is actually more time consuming to write and modify than making a bunch of new columns in Excel with a bunch of formulas! A good data scientist will have a more rigorous approach to their notebooks and practice reuse and so on... but that's not necesssarily easy.
- didgetmaster 4y ago> the person who "naively" downloads that table and plays around in excel finds interesting things that the person who was using Pandas hadn't,... I think they call that serendipity. Never underestimate its power. https://didgets.substack.com/p/data-science-and-serendipity https://didgets.substack.com/p/data-science-and-serendipity
- k__ 4y agoWould you say the big data threshold moves every year? That would explain why people think a <1TB is big data.
- ramesh31 4y ago>Would you say the big data threshold moves every year? It moves with Moore's law. Big data is anything that cannot reasonably fit into memory for a single server, so yes that number is well over 1TB now.
- orangepurple 4y agoI know this isn't the correct definition but I think of "big data" as the set of data which takes me more than 15 minutes to query on average with a moderately complex Postgres SQL join on well indexed information. I use JSONB in Postgres regularly and have indices on that too. So far I have gotten really far with increasing Postgres work_mem to a gig or more, a fast SSD, and strategically placed materialized views. These kinds of operations in Pandas make my computer billow smoke by comparison.
- didgetmaster 4y agoWhat used to be 'big data' is now just 'normal data'. https://didgets.substack.com/p/big-data https://didgets.substack.com/p/big-data
- Too 4y agoI don’t think many give much thought to what it really means. They just use the term because it sounds cool, either to themselves or to their superiors. Same as with Machine learning.
- gampleman 4y agoI say this about every other day at work (we even have only internal users so it's part of their job to talk to us). So far impact: zero....
- dxbydt 4y agofrankly, there’s only a tiny handful of these mythical saas “1000 users each paying 1 mullion dollars” companies. the vast, vast majority of saas startups are serving millions of “users” - i put that in quotes because these aren’t real users or customers. they are real people checking out your product - but they aren’t users or customers. if you set up a gas station near the off ramp of some major interstate, say I-65 North, you will see cars pulling in to fill up on gas. maybe buying a coffee. now, these aren’t your customers in the traditional sense of a Target or Walmart customer. Because you will never see them again. They were driving from town A to town B via the interstate- they started running out of gas and needed to refuel, so they are in your gas station now. Once they gas up, off they go. They aren’t going to come back to you and establish a customer relationship or something. We’ve all been to tons of gas stations on the interstate and we’ll probably never go back to the same one twice - unless we are plying the same route everyday like a truck driver. So the task is to find and convert these truck drivers, who are the true repeat customers. I was working on an android app which had like millions of unique cookies. When they hired me they said we have million of users. No you don’t. If you put out an android app in some popular domain, say news, entertainment, tax accounting etc- people will download and “use” your app. they are checking it out. they aren’t users, in the sense they aren’t using it everyday or want to have a relationship with you, pay subscription etc. conversion stats are minuscule, like 0.01%. So maybe 1 out of 10000 users is the truck driver. The vast majority will never ever use your app again. To do data science with these millions of rows of user interactions and find some nuggets just because you know your way around pandas or sklearn is a fool’s pursuit. To ask foolish questions of your data, like why are all these people churning, is silly - they aren’t your users, they haven’t converted, they are just checking it out. In that sense, its a waste of time and resources to do so much data crunching. Look at actual conversions, which are probably a few thousand people, not millions. Reach out to those thousands and maybe a few tens will give feedback and then continue to iterate on the product based on that.
- thiele 4y agoThis is a really good analogy (gas station customers) that I haven’t heard before. I’ve often tried to describe this ‘low intent’ group but never had a good way to make it relatable.