6 ms·
Machine learning, concluded: Did no-code tools beat manual analysis?
- monkeydust 4y ago"The compute time I spent for the whole project came to about $1,000" Any comparative costs not using aws? Seems expensive for what he did.
- levesque 4y agoFrom what I could see, the dataset has 300 data points (and less than 100 attributes/features), so it should take less than a minute to train a state of the art model on a normal computer. Cost: 0$
- monkeydust 4y agoRight, it's a small dataset no GPU, guess your paying for aws software but I think there are plenty of open source equivalents that could match/beat these results with similar level of effort.
- petra 4y agoAny affordable(for a small business) no-code/low-code ML for small business?
- blackbear_ 4y agoThis one perhaps [1]? I only tried it for ten minutes but it seemed nice. [1] https://github.com/biolab/orange3 https://github.com/biolab/orange3
- moffkalast 4y agoAh Orange, that takes me back to university days. What a pain that thing was to work with.
- levesque 4y agoNo joke, it was the clunkiest library ever. I wonder if they ended up improving it.
- spywaregorilla 4y agoTraining an ML model is very easy. You can learn to pass things through an sklearn decision tree model over a couple hours. Interpreting it is a more battle hardened task. Don't blindly trust such tools.
- jhfdbkofdchk 4y agoKNIME is a free, fairly well used tool in some sectors, for doing low and no-code analysis and machine learning models. https://en.wikipedia.org/wiki/KNIME https://en.wikipedia.org/wiki/KNIME
- CarbonCycles 4y agoI think the only reason why he used Sagemaker is that it provided him an easy out of the box set of tools that required minimum-to-no setup. If the author took the time, he could have stitched together a completely zero-cost alternative that could have run locally on his box, but that would have gone against the argument that this is so easy that "even a monkey can do this". Don't get me wrong...AWS has some very nice libraries, especially their DL and encoding/decoding options. I just think it was a bit heavy using Sagemaker especially for such a trivial use-case...maybe even the free Kaggle notebooks would have been a better alternative?
- weego 4y agoClearly, there's a lot of magic going on behind the curtain with Canvas that is not being exposed. The data handling is all automated, so it's hard to tell what was done to improve its use of the data set with missing data and to produce its output. So as a tool for business analysts, it's great So we're going to be swapping business decisions being made via excel bugs to business decisions being made via black box ML systems.
- dougmwne 4y agoDare I say that might be an improvement? I have observed many businesses being run from broken reporting or from someone’s hunch they invented reporting to support that amounts to random decision making. Even a weak correlation from a black box could be better.
- kjkjadksj 4y agoNo chance any of these models are sufficiently powered to even test the questions the operators are interested in asking
- eyeball 4y agoHow much does this cost on aws? Recent threads on datarobot make me wonder how viable their product is when stuff like this is available from the major cloud providers.
- whoevercares 4y agoI know Alteryx is quite expensive, over time probably nobody can beat AWS on pricing
- matmatmatmat 4y agoI look forward to seeing what will be able to beat AWS in the long run. I don't know what it could be. I guess, maybe, if they stop innovating in the space others will catch up and then price will become the differentiator? Idk, right now AWS is a money printing machine.
- spywaregorilla 4y ago> Speaking to Ars, Amazon Web Services Senior Product Manager Danny Smith explained a non-obvious conundrum: Waiting for a data scientist to analyze one's data "is like waiting for an IT project," he said, noting that biz analyst customers have told him things like, "'We have to wait 18 months to just get to the front of the queue.' And then the data scientists, who are all scarce resources, they're like, 'Well, OK, what's the project's ROI?'" > Smith continued: "And the business analyst is like, 'If I knew that, I wouldn't need your help!... I've got this data, I've got a business problem. I've got questions for this data. I don't know what it's worth until we actually do the analysis.' So the business analyst is thinking about just accelerating to the point where they can actually argue for or justify their use of a scarce data scientist." Well as a data scientist I can tell you the ROI of this project is 0 because its a worthless model. Or really negative because it seems like it'd be pretty dangerous to use in practice. The critical error here is having the business analyst scope this out at all They don't know what they're doing. They don't know how to evaluate the results or predict the likely outcomes.
- nojito 4y ago>They don't know how to evaluate the results or predict the likely outcomes. Hard disagree. Interpreting baseline models is very simple these days. Granted our models are based on tabular data so YMMV.
- spywaregorilla 4y agoOh yeah? So what are you going to do with this heart attack model? Because I sure hope the answer isn't "model said heart attack so the answer is heart attack"
- nojito 4y agoWe don’t for clinical outcomes. There are way more interesting things to model in healthcare than clinical outcomes.
- woeirua 4y agoThis is the major problem with the way data science is managed in most organizations. You typically have business analysts or other management types queuing up work that they _think_ could go somewhere, but they have typically not done the necessary legwork to figure out if a market actually exists for the solution that you're going to build. If a business analyst can't tell you: 1 - What problem you're trying to solve. 2 - What a valuable solution to that problem requires (in gory detail), including model metrics. 3 - What a customer is willing to pay for that solution. 4 - What the timeframe looks like for a valuable solution. 5 - How many resources (people, money) the company is willing to devote to this project. Then the ROI on that project is zero, because there is a very high likelihood that any solution you build will not satisfy the demands of the market.
- CarbonCycles 4y agoThis is a very dangerous article as it tries to make an argument that a person with minimum to no training can start to run these black box systems. One of the biggest fallacies that I have run into within this space is the failure to understand the problem statement. For example, this article does not mention or address models performances on very unbalanced datasets. Most classic M/S/D-L models will perform mediocrely if care is not given to understanding the statistical distributions underlying the data (or even understand the dynamics of the system such as seasonality). In addition, the author does not address how biases are introduced by using AWS' (or insert Google, Azure, etc) algorithms...not all algorithms are codified equally. Finally, this article demonstrates how ppl are trying to trivialize the complexity of very complex algorithms, statistics, etc with a plug-n-play...guess it doesn't help that many companies (I'm looking at you Meta) treat their DS as high caliber business analysts/intelligence units. Okay, time to step away from the keyboard...
- whoevercares 4y agoIn the article, the report from Data Wrangler actually tells about class imbalance and report a whole other bunch of ML specific issues like target leakage. There’s a blog post on it (granted there are other pre canned tool you could use to do this) https://aws.amazon.com/blogs/machine-learning/accelerate-data-preparation-with-data-quality-and-insights-in-amazon-sagemaker-data-wrangler/ https://aws.amazon.com/blogs/machine-learning/accelerate-dat.... Canvas also seem to have nice histograms in their UI
- CarbonCycles 4y agoThere is a difference between reporting a stat and understanding the stat. Another thing that many ppl (including ppl who work in this field) fail to understand is the theoretical (or statistical) underpinnings of the algorithms they are trying to deploy. Many assume IID or NID to make the problem tractable, but that's not how the world typically works even on the very very large data scale. More things on why many data science teams/groups/fail because too many ppl treat it as a BI/BA organization...I really should step away from the keyboard now. LOL.
- 4y ago
- EGreg 4y agoThe comments here all sound like "this thing can never replace me". But if it's an inscrutable black box that performs better on the bottom line metrics (which is something AI can be trained for) then how is it different than all the other AIs that play chess better etc. but can't explain how they do it, because it's a strategy encoded in a million-dimensional vector space?
- CarbonCycles 4y agoNo, I think the arguments you are reading is that this article was poorly written (granted its Ars) and a more appropriate article would have considered broader perspectives. This feels like something an undergrad would write for their business management class...btw, your chess analogy is flawed in that there are millions of patterns that a computer can simulate for next play. I wouldn't really call it AI per se.
- notahacker 4y agoMost real world business problems don't involve board positions and rules and winning moves as simple as chess or Go, and do involve a lot of people that need to be persuaded. So the ability to answer "why's that the winning move?" or "what would change that conclusion?" or "how confident are you are data is reliable enough?" is a lot more useful than a slightly better [over]fit. Even with this example where it's analysing probabilities of heart attacks which can be [retrospectively] validated with precision, I absolutely do want the doctor who says "you're high risk, these are the main reasons why and how you mitigate them" than just a model with a lower false negative rate that can't tell me why.
- spywaregorilla 4y agoThe problem is that the real job that the data scientist does is what the article suggests the business analyst is doing. The business analyst doesn't have a clue what they're doing, and training the ML model is not the difficult part.
- matmatmatmat 4y agoIn line with the other comments, let me give you a recent example from my work. We had a dataset with just about 30 records and about twice as many features, about 1:3 class balance between positive and negative. Not an impossible situation at all, but there was a big premium on performance, so we threw the book at it. Lo and behold, SVM with some particular parameters comes out on top. Ship it, right? Not so fast, what decision did the model learn? Turns out, the model was learning decision boundaries that did not make sense. It was overfitting because the model did not know about what kinds of decision boundaries were allowable. In the end, someone looked at a 2D scatter plot, drew a line and called it a day. It took a fraction of the effort and time of running through a box of models. It'll save the company $MM and it's trivially explainable to engineers and execs. AutoML with blind application would've completely botched this.
- woeirua 4y agoThis article just reinforces that management thinks that data scientists spend most of their time building models. In reality, data scientists spend the majority of their time munging and wrangling disparate data types (that are typically a total mess), understanding the data well enough to fix problems, translating business requirements to code, and converting model outputs to human understandable presentations. Building models is a trivial amount of the effort for most projects, typically 10% or less of the time. Most organizations will eventually move over to AutoML type solutions, but then will be shocked when they fail to achieve any significant gains in productivity because they're optimizing one of the shortest steps in the whole process.
- CarbonCycles 4y agoLOL...typically i would agree but i find DL requires a lot more intense engagement (for now) followed by lots of idle time reading hacker news as the models complete training. DS/ML/BI/WTF EVER has all become overly incestious filled with buzzwords generated by hiring managers/executives who really have no idea what they're talking about when they write the job descriptions (or they're being seriously cheap by blending job functions together such as Head of Data and Analytics...wtf is all that about. From a business function sure sounds great, technically it's a recipe to set ppl up for failure).
- AtNightWeCode 4y agoI think that this Q explains why Gartner and other garbage analytic companies are so full bs. Hold them accountable to what they said 5 years ago! Also laughable about the 90% threshold that seems to STILL be the magic number for success within ML. Try applying that in your private space.