4 ms·

I agree with all what you said, except the part about 10 records vs 10 000, that I think need better clarification. depending on the amount of computation need

by gryn 5y ago

I agree with all what you said, except the part about 10 records vs 10 000, that I think need better clarification. depending on the amount of computation needed/task executed the number of record to update can make things very different. updating 1 million or 10 000 records that need very heavy computation and communication to other systems under some duration or latency constraints is very different from updating 10 of them. you may need to optimize and/or rewrite your whole systems to handle such things.

orev 5y ago
I was hesitant to include that example because I was concerned it would be nitpicked in this way. The vast majority of business records are not being sent through machine learning algorithms or being used to factor prime numbers, they’re simply lists of basic things like names, addresses, phone numbers, etc. In all of those cases, the difference between 10s or 1000s is trivial. These are the types of things people use Excel for daily. In cases where the data processing is actually significant, then sure, bring in your data scientist.
- gryn 5y ago
  yes I'm aware it's nitpicking, which is why I said I agree and that it just need more clarification. but you don't need machine learning to hit these cases just a mismatch between what want to do and how/where it's stored. examples: - a noSQL db that cant directly do what you want and now you need to download lots of document to string them together before updating that field. - a SQL data schema not matching your tasks where need lots and lots of joins and aggregations. - data scattered across multiple web APIs that may have rate limits, now you may have to use a batch API if it exists and find a new route. - data scraped from multiple places, going from 10 to 10 000s now will get you banned from most website, now you need to rewrite your code to circumvent that. none of these examples involve data science, heavy processing algorithms, just moving data and selecting from it. and they seems like the kind of things business/people with no coding experience might want to do. (I want to keep track of prices from competitor, department X,Y, and Z have these info in their databases and gave us read-only access since they don't have time to do things for us, etc ...)

by gryn 5y ago

orev 5y ago

I was hesitant to include that example because I was concerned it would be nitpicked in this way. The vast majority of business records are not being sent through machine learning algorithms or being used to factor prime numbers, they’re simply lists of basic things like names, addresses, phone numbers, etc. In all of those cases, the difference between 10s or 1000s is trivial. These are the types of things people use Excel for daily. In cases where the data processing is actually significant, then sure, bring in your data scientist.

gryn 5y ago
yes I'm aware it's nitpicking, which is why I said I agree and that it just need more clarification. but you don't need machine learning to hit these cases just a mismatch between what want to do and how/where it's stored. examples: - a noSQL db that cant directly do what you want and now you need to download lots of document to string them together before updating that field. - a SQL data schema not matching your tasks where need lots and lots of joins and aggregations. - data scattered across multiple web APIs that may have rate limits, now you may have to use a batch API if it exists and find a new route. - data scraped from multiple places, going from 10 to 10 000s now will get you banned from most website, now you need to rewrite your code to circumvent that. none of these examples involve data science, heavy processing algorithms, just moving data and selecting from it. and they seems like the kind of things business/people with no coding experience might want to do. (I want to keep track of prices from competitor, department X,Y, and Z have these info in their databases and gave us read-only access since they don't have time to do things for us, etc ...)