21 ms·

Machine Learning: The Great Stagnation

484 points by puttycat 6y ago

marvel_boy 6y ago
Winter is coming, again.
- jmortin 6y ago
  Yeah, if I got a penny everytime I heard that one...
  - pjmlp 6y ago
    Python is the new Lisp.
  - Rochus 6y ago
    Promises made by AI caused more spent pennies during the last 60 years. And people continue to be inspired by such promises.
- WanderPanda 6y ago
  Just a quick belief of myself (might change tomorrow): ML winter will not come until we hit the flattening of the specialized hardware s-curve. I know people believe that ML can still scale in funding (for even bigger models than GPT-3 with current hardware) but that shouldn’t be our hope. I also can not imagine that the exponential efficiency increase in architectures with optimized structure can continue for a couple more years (happy to be proven wrong here). Also the scaling in dataset sizes will ultimately halt (or might be already) for supervised learning which might give a bump for reinforcement learning where there is usually unlimited data (due to simulation)
  - mlthoughts2018 6y ago
    Most commercial use of ML, including neural nets, is small data application-specific business logic. Think phishing / spam / fraud detection, anomaly detection, semantic search, image search, keyword labeling, classifying or segmenting customer data. These applications have well-understood business value propositions. Much, much less often the ML application focused on truly large data. I think we might see a winter in super big applications like self-driving cars or voice assistants, but ML in general is just a boring, non-controversial business tool with hundreds of valuable applications. You’ll still need statistical specialists to train and operate models and ensure systems avoid pitfalls like overfitting, poor convergence, multicollinearity, confounders, etc. So I doubt this will have much impact on ML job market. Companies that invest in ML will continue to run circles around companies that don’t. You’ll just see the unjustified over-focus on SOTA neural networks die down and become just another boring tool in the toolbox like everything else in ML.
    - WanderPanda 6y ago
      I agree, or at least I hope you are right. I have some doubts about investment into ML without the hype (at least for legacy big corps) but probably their hype-driven ML efforts are/were misguided by poor incentives. But I agree again that companies that invest in ML from a grassroots/first principles way should run circles around the legacy ones. I think the „boring“ part you mention might be what I refer to as ML/AI winter. But then again do you think there is much space left in the ML toolbox apart from neural networks? I think at least for e.g. computer vision we can agree that neural nets ate all the cake, right?
- superbcarrot 6y ago
  That's unlikely. There are many machine learning applications that have been shown to be good enough for commercial use and they aren't going anywhere. The worst case for the field is that progress slows down, people realise that their expectations were unrealistic and the hype inevitably dies down. Which has to happen eventually. So even if ML isn't the hottest thing or a massively growing field, it will still be used for a long time.
  - otabdeveloper4 6y ago
    > There are many machine learning applications that have been shown to be good enough for commercial use and they aren't going anywhere. If you could name three of them I'd be really grateful. Serious question; everything surrounding ML seems to be only good for (non-monetizable) art projects. As art it is amazing, not going to lie, but "commercial use" seems like a huge stretch.
    - Tenoke 6y ago
      Here's 3 off the top of my head, but there's more especially when you get into less flashy territory. Translation (Google translate, DeepL) Automatically generated product descriptions, sometimes also edited by humans (Alibaba) Image Tagging (Facebook photos)
      - otabdeveloper4 6y ago
        Okay, I should have worded my comment more carefully. These applications seem to firmly fall into the "I'm willing to compromise on quality if I don't have to pay a living person a wage" niche, so they're value-destroying, not value-creating. Are there examples of value-creating applications for ML? (From a business point of view; obviously the "shitty translations but at no cost" proposition creates value for the average Internet user.)
        Tenoke 6y ago
        I don't understand this. Do you think that e.g. the average engineer is value destroying because if the business hires a more expensive and experienced one they will do a better job? In either case it is only 'value-destroying' if the business has unlimited resources.
- 314 6y ago
  Winter is always coming, the important question is when it will arrive. Current ML research is still destroying new problems with ease so the current velocity is high. It will slow down first, before people start to question why the new crop of problems are too hard, and then the cycle will start again.
  - ausbah 6y ago
    I think there's already a good amount of discussion around where current ML methods. stuff like lack of sample efficiency, adjusting for distribution shift, etc.
- indymike 6y ago
  Not sure that it's as much stagnation as commoditization.
laichzeit0 6y ago
> It’s important to avoid becoming Gary Marcus and criticize existing technique that work without proposing something else that works even better. Hilarious. Has Gary Marcus actually done anything, in practical terms, like actual code or something, that outperforms the DL approaches he attacks so viciously?
- smeeth 6y ago
  I keep coming across versions of this comment. Why does it matter? It’s not like the validity or invalidity of Gary’s criticisms would be different if he had done other totally separate ML research.
- Rochus 6y ago
  Marcus argues convincingly and appears less biased than others who benefit from the current high level of investment in DNN. It seems to be true that no better proposals for solutions have come from his side so far. But I think his criticism per se is valuable, especially his reminder that one cannot simply ignore sixty years of research. EDIT: and of course he is not the only renowned scientist who calls for reflection; here is a quote from an interview with Judea Pearl: "AI is currently split. First, there are those who are intoxicated by the success of machine learning and deep learning and neural nets. They don’t understand what I’m talking about. They want to continue to fit curves. But when you talk to people who have done any work in AI outside statistical learning, they get it immediately. I have read several papers written in the past two months about the limitations of machine learning." (https://www.theatlantic.com/technology/archive/2018/05/machine-learning-is-stuck-on-asking-why/560675/ https://www.theatlantic.com/technology/archive/2018/05/machi...)
  - Tenoke 6y ago
    Gary Marcus has been proven wrong on claims he's made and is mostly just nay-saying with adhoc reasoning made up to support it. >appears less biased than others who benefit from the current high level of investment in DNN. Yes, instead he blatantly tries to benefit from the counter-investment in AI skepticism.
jarym 6y ago
> Academics sacrifice material opportunity costs in exchange for intellectual freedom. Most academics I’ve come across only think they’re doing this. My perception is they are too insecure about their self-worth to pursue material opportunities. I admit, the number of academic types I know is not vast so maybe it’s too small a subset to make any judgments
- rowanG077 6y ago
  The core problem is that you don't have intellectual freedom. You won't get funded if you are not researching the hot new thing.
  - dimitrios1 6y ago
    Exactly, and that's assuming you are one of the very select few that gets to even select and lead the research areas. Most of the opportunities are working along side professors and department heads that already have an idea of what you should be working on. If you are at one of the elite universities with billion dollar endowments that don't have a lot of these funding problems, then you are working on what the corporate donors suggest the University work on. The university will sell it as "staying relevant for the future job market"
    - chriskanan 6y ago
      If one has a supportive advisor and adequate non-grant support (e.g., a fellowship), it is possible to work on self-drive ambitious projects. However, typically an advisor only wants to invest significant time if the work is interesting to them. Also, often I've been approached by students who want to do their own thing in terms of high risk work, but they don't have adequate skills or enough existing work to graduate with. Mt goal is to ensure students advance knowledge, graduate, and get the kind of job they want. Ambitious projects that fail will significantly impair positive outcomes. The billion dollar endowments don't usually go toward supporting research directly.
  - deleted 6y ago
    [deleted]
hehehaha 6y ago
I see a lot of parallels between ML/DL and cryptocurrency research.
- komuher 6y ago
  Can u elaborate on that?
  - gbolcer 6y ago
    I can't speak for them, but I took it this way: there are certain industries that are simply incentivized to produce more of their own industry. The criticisms of the financial industry where they are not really creating anything of value other than optimizing more value out of existing money. Adtech industry where a whole generation of technical researchers spent time figuring out how to optimize clicks. Cryptoeconomy and blockchain where large amounts of money are created out of perceived value and gargantuan efforts are made to build, not solutions to real-world problems (yes I know there are some), but ways to increase the shared, total value of the technology or individual cryptocurrency.
- londons_explore 6y ago
  > cryptocurrency research We're thinking very hard about if the price of our deep-lolcoin is going to go up enough to buy a new a vacation house....
- superbcarrot 6y ago
  At least with machine learning there is some actual substance underneath the hype.
knuthsat 6y ago
There's a nice talk by Yann LeCun where he goes on to explain really well why deep learning has such fast progress. [0] He goes on to explain how theory always comes later. I thought that information theory came before practice but turns out it also came after. (there was a bunch of heuristics for sending messages with teletypes) A nice example is Roman technological advances in architecture and materials that predate the use of geometry or mathematics completely. All advancements a result of tinkering and heuristics. [0]: https://www.youtube.com/watch?v=gG5NCkMerHU https://www.youtube.com/watch?v=gG5NCkMerHU
- mettamage 6y ago
  So in that sense forming a theory about something is integrating all the practical stuff you've seen before. To me that make sense, IMO that "practical stuff" are what experiments are, aren't they? Probably the experiments are far from perfect, if remotely good even, but that's how empiricism should work? I think? (hmm... not entirely sure)
  - curiousllama 6y ago
    It can go both ways. E.g. often, experimental physicists will try to prove/disprove a priori predictions from theoretical physicists. Theory is just an explanation of hw the world works: you can come up with that explanation as a reason for observations, or as a logical consequence of other theories (which is then verified by observation).
- _0ffh 6y ago
  I've once read a paper in the late 90s or so which openly said ML research was partly "experimental maths".
- gnramires 6y ago
  It's not so simple as "what comes first?" (theory or experiment). What generally happens is that we stumble upon something, build a rudimentary but useful or interesting piece, then try to understand it, generalize it, and expand it (and usually succeed, with science). Both happen iteratively. Information theory came after the telegraph and early communication systems. However, we could not have built modern communication devices without Information theory (the insights and design principles). We build, then we theorize, then we build better, etc. it's not a simple procedure. Computers were developed similarly: there were all sorts of ad-hoc logical apparatus, we built boolean theory to explain it, and then we did all sorts of experiments trying to build computers. Their architects were largely mathematicians with very good ideas of mathematical design principles (and creative new mathematical ideas), not a group stringing together electrical elements and seeing what happens. The same goes for the development of ML/Deep learning, and many other technological marvels. Although "accidental" discoveries do happen, they happen from a methodical set, with knowledge, intuition, and good priors.
osipov 6y ago
> If an idea is guaranteed to work then it moves from the realm of research to engineering. The idea of the Dyson Sphere is guaranteed to work, so I guess the author is under the impression that there will not be research involved.
underdeserver 6y ago
Missing (2020)?
- Rochus 6y ago
  Two month ago?
Ericson2314 6y ago
I still feel like much of AI is a plot to dumb down the modern economy. We want our business people to be just as effective as our quants; we want nothing to require real intellectual labor. The idea that you traditionally have these programmers who spout mumbo-jumo all day, cost a lot of money, and seem to always be planning stuff behind your back is threatening, and all the more so because you are utterly dependent on them. ML breaks their control over the means of production. Now, that's not to say I am against labor saving devices. I most certainly am for them, but an an economy in which everyone is in a deep learning arms race is an irrational shit show that could only result in less productivity. (It's possible a single central planner AI could do better, because at least the training data would be "real world" and not output of other deep learning black box actors. But of course single-planner economies have a huge amount of other downsides.)
- mlthoughts2018 6y ago
  > “ The idea that you have these programmers who spout mumbo-jumo all day, cost a lot of money, and seem to always be planning stuff behind your back is threatening, and this breaks their control over the means of production.” That is a very bizarre description of ML engineers. In every company I’ve worked at, ML is a team or teams that partners with product managers and other engineering teams to learn about problems they need solved. It’s very systematic, boring, and tied heavily to those other teams as the leaders and decision makers. First you look for high level value propositions, like automating a decision process, removing a customer friction point, creating key metrics where simple metrics are intractable, or various multi-modal information retrieval goals. You identify opportunities in these kinds of high level areas in lock-step with product managers and other engineering leaders. Then you move on to identify sources of data that can be leveraged, and eventually (much later) you get to the smaller set of work training a model, validating with acceptance tests and hardening the implementation for safe production deployment. If people are spouting “mumbo jumbo” and making big ML model decisions without lock-step synchronization with other stakeholders, that sounds like organizational dysfunction, not any type of issue with ML. I also find it odd that you bring up “cost[ing] a lot of money” .. that’s very out of place among everything else mentioned. That seems more like insecurity or jealousy over the market demand for ML talent, and wanting to cut other people down rather than work with them or acknowledge the level of effort it required to get that level of expertise in ML.
screye 6y ago
There is about to be a 'great pivot' in ML. There has been a rabid frenzy of throwing money at anything that has ML in it. Soon investors and CEOs will realize that ML is effective in narrow ways and that not everything needs ML. They will also realize that 1 ML team + ML as a service (Azure ML, Sagemaker, Google AI platform) is cheaper and works more reliably. The services will keep improving and an underpaid mediocre ML-Engineering team can work to keep the production system up and running. Basically, ML teams might lose jobs just as DB/Cluster admins did with the advent of serverless compute as service. I expect it to (already happening) create a day trading company like hierarchy. The Fair/Brain/OpenAIs will pay 7 figures to the top grads to be first to market. BigN ML product teams will expand and stay as well paid as they are. Then there will be a huge drop as we move to offshore ML product teams that are viewed as cost centers by the remainder of 99% companies. These will be most of the jobs available. In such a system, a pure ML scientist (usually a PhD) will only exist at the top companies. So if you are not in the top 1-3% percentile, you will not have a pure ML job. However, there will still be hybrid DS-SDE jobs (ML engineering, ML product maintenance, ML-as-a-service user) or hybrid DS-PM jobs (Analysts, Consultants, data driven business decision makers). So, anyone who is not in that top 1-3% will have to pivot to one of these 3 roles. ______________ I won't call this an AI winter. But, it will definitely become boring majority of those employed in the field..
- ramoz 6y ago
  Mostly disagree. > CEOs will realize that ML is effective in narrow ways and that not everything needs ML. Any stable business isn't unjustifiably syncing costs here. I project FY21 rise an AI-funded efforts in large businesses. > They will also realize that 1 ML team + ML as a service Yes/no. This is has more platform implications vs actual ML. > ML teams might lose jobs Assumes ML Jobs only do some form of R&D. Data is a utility, and advanced analytics is valuable. Stable ML Jobs dont just work on deep learning. > I expect it to (already happening)... partially agree. already happening. But cost centers are only taking on what was standardized yesterday. Tomorrow still requires advanced analytics capabilities. > In such a system moot point. This is the system. I see a pivot in focused efforts. More optimism in an early AI commodity vs stagnation. We're moving from research to integration. Further areas to improve AI in applications (with continuous feedback training) and many domains of advanced analytics.
magicsmoke 6y ago
I dropped out of a PhD in RFIC/MMIC design. Gradient student descent and lack of first principles reasoning hits too close to home. Maybe it's just the nature of highly empirical engineering disciples to just throw many hours of grad student and simulation time blindly at problems in hopes that one lucky fella finds a new optimization.
mbeex 6y ago
"Neural Network weights are learnt via Gradient Descent and Neural Network architectures are learnt via Graduate Student Descent." Like it!
b3kart 6y ago
Some good points in the article, but I disagree with the tone and the conclusion. > we’ve rewarded and lauded incremental researchers as innovators, increased their budgets so they can do even more incremental research There isn't a scientific field where every single paper is groundbreaking. It's a Brownian motion of small incremental innovations, until eventually we stumble upon something big (like deep learning). In no way is machine learning unique in this. Sounds like the author is simply disappointed that, like in any other profession, day-to-day of a researcher is a slog and not a perennial intellectual festival. We've been in an exciting deep learning craze for a while, but it's silly to expect it to last forever. Back to the grind now. > Machine Learning Researchers can now engage in risk-free, high-income, high-prestige work Not sure what author means by "risk-free". Yes, if you're not publishing enough you're most likely not going to starve. Is that a bad thing? Is the survival instinct the only good motivator for doing good research? I would argue that there's plenty of risk, in that people who don't publish good research don't get very far in their academic careers, which in my view is good enough motivation. "They must do good research or starve" is a rather cynical take, especially from someone who seems to not be doing too badly for themselves. I'd rather more fields provided similar benefits. Maybe then going into science wouldn't be associated with so much sacrifice, so more smart people would choose science over investment banking or such, and we'd make more scientific progress faster. > CNNs use convolutions which are a generalization of matrix multiplication. A nitpick: CNNs are most definitely not a generalization of matrix multiplication. In fact, the opposite: you can view CNNs as a matrix multiplication with a particular matrix structure.
- AndrewKemendo 6y ago
  >We've been in an exciting deep learning craze for a while, but it's silly to expect it to last forever. Back to the grind now. This was effectively my response to hardmaru when this topic came up on reddit [1] Basically 2010-2018 was an open field for ML/DL research with old(ish) methods being rapidly applied to low hanging fruit and large datasets with newly cheap compute. Deepmind and others are actually making new methods but by and large are remixes of those same old approaches. The majority of different research out there trying other approaches (Numenta, OpenCog, Causal Calculus, anything Schmidhuber etc...) don't really get any love because it doesn't fit within the mass tensorflow/torch framework. [1]https://twitter.com/AndrewKemendo/status/1349387455552745473 https://twitter.com/AndrewKemendo/status/1349387455552745473
curiousllama 6y ago
> We’ve gamified and standardized the process so much that it’s starting to resemble case studies at consulting interviews. This is precisely true, as someone who has passed both screens for competitive jobs. Cracking the coding interview <> Case in Point Live coding <> Do 3-digit multiplication in your head (eg 347 * 469) Sorting algorithms <> M&A Evaluation Frameworks I could go on... You just memorize a bunch of crap that's vaguely (but not really) applicable but is super random, and then you just keep asking "do you want me to keep going" in various tones until they tell you to stop.
- jbn 6y ago
  >> We’ve gamified and standardized the process so much that it’s starting to resemble case studies at consulting interviews. Love the way this is put.. it is so true :(
tannhaeuser 6y ago
Here's something to put ML and related tech to work: to quantify the influence of shilling, astroturfing, brigading, gatekeeping, and all kinds of political campaigning on the 'net, the devastating consequences of which are only now seen by the general public. Though I have to say that I've personally seen ML invading conferences not related to ML per se, with over half of submissions employing ML techniques to random problems in the primary field which however were in itself clearly of no interest to the presenter and only a vehicle for their graduation even more than usual. So I'm admitting to see MLer as mostly in it for advancing their careers; glad to be convinced otherwise.
DrNuke 6y ago
A number of industrial domains have largely benefitted from this tool and outsiders can even set up their own virtual r&d lab at almost no upfront investment these days. As usual, a tool is (still...) only a tool, and it (still, just...) needs domain knowledge to work in full and produce noticeable ROI?
RGamma 6y ago
Sounds like the field has accelerated itself into an intellectual dead end. Not enough to know how to walk . You need to know where you want to go, and figure out the existence of any path in the first place or you might end up squaring the circle.
gbolcer 6y ago
Graduate Student Descent -- there's also the other saying: Every moment you spend writing, publishing, or working on your dissertation is a moment that you are getting behind on doing unique and cutting edge research.
- lalo2302 6y ago
  The only difference between doing science and fooling around is when you write it down
- dr_hooo 6y ago
  But also "The difference between screwing around and science is writing it down."
  - gbolcer 6y ago
    I like yours better.
FartyMcFarter 6y ago
This article makes a lot of points, but nothing that implies actual stagnation? I wish the writing was more concise and to the point.
PIKAL 6y ago
What a nothing burger.
CyberRabbi 6y ago
Interesting article but the part where he says “all” you need to learn are matrix multiplications seems vacuously true to the point of meaninglessness. All numerical methods across every field essentially boil down to matrix multiplications. Your math ability needs to be such that you can transform a problem into a form that can be computed numerically using matrix multiplications, which requires more skill (sometimes significantly more) than simply knowing how matrix multiplications work. Sometimes this ability to reframe complex problems numerically using matrix multiplications is the lions share of the research!
- b3kart 6y ago
  Good point. Almost as useful as saying "all you need to learn is Boolean algebra" when it comes to computing.
- z77dj3kl 6y ago
  All you need is ZFC...
  - tartaglia 6y ago
    All you need is ZF...
adamc 6y ago
So, this is the claim I thought was interesting: "Every paper is SOTA, has strong theoretical guarantees, an intuitive explanation, is interpretable and fair but almost none are mutually consistent with each other" How can that be true unless the "theory" itself isn't really worked out?
kfk 6y ago
I fear ML and AI are taking the lion share of analytics resources in most big companies and that's just not right. We still have basic problems with data integrity and blending. Getting a real end to end view of any business process is still a herculean task. Try matching something like invoices with marketing leads, good luck...
- curiousllama 6y ago
  If it makes you feel any better, in practice, most ML projects fail and just turn into (super valuable) reporting like you describe. If you want a reporting solution, buy AI.
tempodox 6y ago
BERT engineer? That acronym is essentially ungooglable, since you just find every person ever named Bert. This is what I finally found: BERT, which stands for Bidirectional Encoder Representations from Transformers. https://arxiv.org/abs/1810.04805 https://arxiv.org/abs/1810.04805
- lordgrenville 6y ago
  Searching for "BERT machine learning" would have worked.
JHonaker 6y ago
I kind of agree with the author's major sentiment: that ML research is stuck in a rut with incremental improvement. However, the longer the article goes on, the less and less I agree with any of their statements. They start of criticizing the incremental improvers. They advocate later that if "stack more layers" beats a method, the method isn't good while completely ignoring anything other than the standard SOTA metrics like data efficiency, computational efficiency, model representation efficiency, etc. And, while I agree there is a "fake rigor" problem in ML research, the particular examples that they bring up aren't extremely good exemplars in my opinion. Instead, they seem to have a problem with the standard operating procedure of mathematics while missing the point that it is what got the "stack more layers" school here in the first place. Simplifying a problem so you can understand it, and then relaxing the assumptions and seeing if you can figure out what implications that has is how advances are made. Plus, they have some hot takes and statements that are just plain wrong. > With Automatic Differentiation, the backward pass is essentially free and is as engaging to compute as 50 digit number long division. Deriving long complicated gradients is fake rigor that was useful before we had computers. What? First of all, AD is not a solved problem and using it is not "essentially free." There's a huge performance overhead when adding AD to a system. Try using a second order method with AD. I hope your Hessian actually finished computing. > Julia on the other hand is a language made for scientific computing where everything is automatically differentiable by default. This is just plain false. I'm a huge proponent of Julia, and there are some great AD packages, but in no way is everything automatically differentiable (even with the nice packages), nor is that a design goal. The work on Flux.jl (a package) is extremely impressive though, and there are particular features of Julia that allow some awesome package interoperability (e.g. the fact that some ODE solvers can be differentiated through with Flux).
- ChrisRackauckas 6y ago
  >some ODE solvers Some? Which ones aren't? Could you open an issue?
  - JHonaker 6y ago
    I was unsure if every ODE solver was able to be differentiated through. The "some" was me trying to avoid making a blanket statement about all, when I wasn't sure. Sorry for the confusion.
DaedPsyker 6y ago
I've talked to colleagues about this before but I find it's mostly a cyclical nature fuelled by tunnel vision. Every good couple of years researchers come up with a good advancement or fresh concept that reignites the community. However all that happens till the next breakthrough is basic tweaks. The amount of junk papers that slightly adjust the method that gets a few decimal place improvement then call it a snazzy name would fill a mountain. Drawing blood from a stone. People get so obsessed with specific methods thinking its the new great thing they don't stop and think it's probably not the only way. Marketing and media is the worst though for general public perception. The amount of times they would warp ML into a magical pangea. "ML will make you skinny!" I mean it's just mathematical approximations, been around a while. Don't get me wrong, I like this field. I'm fortunate that I get to apply it to a problem that helps people but at times I just want to shout from the rooftops that it's not a God it wont make all your dreams come true. Then they get annoyed it doesn't and we a bunch of stagnation articles like this.
zetazzed 6y ago
Astronomy has entered a period of great stagnation. More and more grad students are investing huge amount of time and building larger and larger radio telescopes just to learn more about black hole formation and the properties of pulsars. Little consideration is given to how to more efficiently use inexpensive consumer telescopes purchased at Walmart! Where are the big discoveries? New planets in our solar system? A new structure for the Milky Way? I sincerely hope that the Julia language will allow us to discover ET radio signals, because we all know that Python lacks this capability.
- formalsystem 6y ago
  This made me lol - We should write a second post together
- deleted 6y ago
  [deleted]
SoSoRoCoCo 6y ago
This article is dead-on, but I think it is missing a fairly large segment of where ML is actually working well: anomaly detection and industrial defect detection. While I agree that everyone was shocked, myself included, when we saw how well SSD and YOLO worked, the last mile problem is stagnating. What I mean is: 7 years ago I wrote an image pipeline for a company using traditional AI methods. It was extremely challenging. When we saw SSDMobileNet do the same job 10x faster with a fraction of the code, our jaws dropped. Which is why the dev ship turned on a dime: there's something big in there. The industry is stagnated for exactly the reasons brought up: we don't know how to squeeze out the last mile problem because NNs are EFFING HARD and research is very math heavy: e.g., it cannot be hacked by a Zuck-type into a half-assed product overnight, it needs to be carefully researched for years. This makes programmers sad, because by nature we love to brute force trial-and error our code, and homey don't play that game with machine learning. However, places where it isn't stagnating are things like vibration and anomaly detection. This is a case where https://github.com/YumaKoizumi/ToyADMOS-dataset https://github.com/YumaKoizumi/ToyADMOS-dataset really shines because it adds something that didn't exist before, and it doesn't have to be 100% perfect: anything is better than nothing. At Embedded World last year I saw tons of FPGA solutions for rejecting parts on assembly lines. Since every object appears nearly in canonical form (good lighting, centered, homogeneous presentation), NN's are kicking ass bigtime in that space. It is important to remember Self-Driving Car Magic is just the consumer-facing hype machine. ML/NNs are working spectacularly well in some domains.
- gok 6y ago
  > This makes programmers sad, because by nature we love to brute force trial-and error our code, and homey don't play that game with machine learning. Huh? If anything I would say ML is way more trial-and-error focused than imperative programming.
  - SoSoRoCoCo 6y ago
    Well, if your only experience is reading python opencv stack overflow posts, then of course...
gbolcer 6y ago
Technology Review summary of AI and ML adoption into business. It indicates, while you leading thinkers have figured it out, business is a long ways from being anywhere near peak ML. https://mittrinsights.s3.amazonaws.com/AIagenda2020/GlobalAIagenda.pdf https://mittrinsights.s3.amazonaws.com/AIagenda2020/GlobalAI...
antonzabirko 6y ago
Hasn't this been the state ef academia for a long time now? At least for ML.
formalsystem 6y ago
Hey everyone this is OP, was a really nice surprise seeing my article generating so much discussion. I unfortunately don't think I'll have time to answer everyone but feel free to reply to this comment if you'd like to ask me anything. It seems like the article was a bit polarizing, some of the comments made me realize I made a few imprecise statements and I'll fix those. Other comments didn't approve of my tone and that's a bit harder to fix since I tried writing this article in the same way I usually speak. I've written a lot of more dry content for my robotics ebook and I was purposeful in trying something different for this article. Some of the comments actually generalized my observations to management and software more generally and it's always nice to see people taking my ideas further than I thought they could go. One major thing I'd like to point out is I don't think "ML is dumb" is the right conclusion, there are lots of incentives making it so more of ML is stagnating but this is certainly not true for the field at large. Interesting ideas need to involve a certain amount of risk. The latter half of my article showcases a few ML adjacent projects which I think are absolutely fascinating. And if you're interested in reading more stuff by me my robotics and machine learning ebook is very easy to read http://robotoverlordmanual.com/ http://robotoverlordmanual.com/ and will teach you all you need to start building robots at home
- mloncode 6y ago
  Great job writing this article.
- arolihas 6y ago
  Where did you find the memes?
  - formalsystem 6y ago
    I made most of the edgy ones and had shared them throughout 2020 on my Twitter - only Stack more layers and Squirrel are not original content and ofc the really polished ones like how many angels dance on the head of a pin and the daft punk reference The writing process for this article was meme first, content second
karmasimida 6y ago
Matrix multiplication is nothing to be laughed at. It is how anyone performs linear projection anyway. Fair enough to say, currently ML research is fixated on exhausting combinations of blocks to squeeze a marginal improve on a few public benchmarks. But it is like saying software has been stagnated since we are not getting new ISA created. Building ML application remains as challenging as ever, it is just modeling itself has become streamlined before everything else.
klik99 6y ago
It's about time someone formalized Graduate Student Descent, please someone write a paper diving into all it's properties! "Graduate Student Descent is one of the most reliable ways of getting state of the art performance in Machine Learning today and it’s also ... fully parallelizable"
whoevercares 6y ago
Are you sure that overall deep learning customers are spending more than traditional ML customers? My guess: it’s not even close - financial sector is still order of magnitude larger as paying customer than those hot CV/NLP companies.
deeeeplearning 6y ago
Lots of salty kids in here trying to justify spending 6 years as an anonymous hyperparameter tuning drone in the Stanford AI Lab.
pitspotter 6y ago
>The construction of Academia was predicated on providing a downside hedge or safety net for researchers. Where they can pursue ambitious ideas where the likelihood of success if secondary to the boldness of the vision Is there a citation for this? Much I would like it to be true, from the little I've read it seems that plenty of universities/colleges were originally religious in nature and were intended to defend orthodoxy, even to combat specific heresies. Definitely not to take intellectual risks.
totorovirus 6y ago
lol I am a so called BERT engineer and I can't agree more to this wow
academia_hack 6y ago
There's so much stuff about the world that we don't understand and want to and that even trivial machine learning techniques can help us figure out. Yet, at least in my field (computer security), it has become incredibly difficult to publish research that applies machine learning without having some needlessly obtuse methodological twist to make it seem more complicated. Even low hanging fruit needs picking and I'm worried discoveries with intrinsic value are being left to rot in the name of feeling "scientific."
master_yoda_1 6y ago
I like till it mention keras, fast.ai. No offense to keras, fast.ai they are very good for beginners in ML. But for serious work you need more (try to write autoregressive generative model etc). Here again bullshit article trending on HN front page, shame
BlueTemplar 6y ago
Richard Feynman — 'Philosophy of science is about as useful to scientists as ornithology is to birds.' Well, this blogpost shows where this analogy breaks down : birds (AFAWK) don't try to discuss ornithology. "bureaucrats running the asylum" is normal science ! Thomas Kuhn has shown how it works in The Structure of Scientific Revolutions : https://samzdat.com/2018/05/19/science-under-high-modernism/ https://samzdat.com/2018/05/19/science-under-high-modernism/ The problem is that without Kuhn our expectation are set by pop history of science, which only remembers 'anormal', extraordinary science : the paradigm changes. (Otherwise, this is a great blogpost.)
fuy 6y ago
fast.ai link actually points to http://book.realworldhaskell.org/ http://book.realworldhaskell.org/ What gives? Edit: having read to the end, there's a mention of Haskell book, so just an editing error then.

484 points by puttycat 6y ago

marvel_boy 6y ago

Winter is coming, again.

jmortin 6y ago
Yeah, if I got a penny everytime I heard that one...
- pjmlp 6y ago
  Python is the new Lisp.
- Rochus 6y ago
  Promises made by AI caused more spent pennies during the last 60 years. And people continue to be inspired by such promises.
WanderPanda 6y ago
Just a quick belief of myself (might change tomorrow): ML winter will not come until we hit the flattening of the specialized hardware s-curve. I know people believe that ML can still scale in funding (for even bigger models than GPT-3 with current hardware) but that shouldn’t be our hope. I also can not imagine that the exponential efficiency increase in architectures with optimized structure can continue for a couple more years (happy to be proven wrong here). Also the scaling in dataset sizes will ultimately halt (or might be already) for supervised learning which might give a bump for reinforcement learning where there is usually unlimited data (due to simulation)
- mlthoughts2018 6y ago
  Most commercial use of ML, including neural nets, is small data application-specific business logic. Think phishing / spam / fraud detection, anomaly detection, semantic search, image search, keyword labeling, classifying or segmenting customer data. These applications have well-understood business value propositions. Much, much less often the ML application focused on truly large data. I think we might see a winter in super big applications like self-driving cars or voice assistants, but ML in general is just a boring, non-controversial business tool with hundreds of valuable applications. You’ll still need statistical specialists to train and operate models and ensure systems avoid pitfalls like overfitting, poor convergence, multicollinearity, confounders, etc. So I doubt this will have much impact on ML job market. Companies that invest in ML will continue to run circles around companies that don’t. You’ll just see the unjustified over-focus on SOTA neural networks die down and become just another boring tool in the toolbox like everything else in ML.
  - WanderPanda 6y ago
    I agree, or at least I hope you are right. I have some doubts about investment into ML without the hype (at least for legacy big corps) but probably their hype-driven ML efforts are/were misguided by poor incentives. But I agree again that companies that invest in ML from a grassroots/first principles way should run circles around the legacy ones. I think the „boring“ part you mention might be what I refer to as ML/AI winter. But then again do you think there is much space left in the ML toolbox apart from neural networks? I think at least for e.g. computer vision we can agree that neural nets ate all the cake, right?
superbcarrot 6y ago
That's unlikely. There are many machine learning applications that have been shown to be good enough for commercial use and they aren't going anywhere. The worst case for the field is that progress slows down, people realise that their expectations were unrealistic and the hype inevitably dies down. Which has to happen eventually. So even if ML isn't the hottest thing or a massively growing field, it will still be used for a long time.
- otabdeveloper4 6y ago
  > There are many machine learning applications that have been shown to be good enough for commercial use and they aren't going anywhere. If you could name three of them I'd be really grateful. Serious question; everything surrounding ML seems to be only good for (non-monetizable) art projects. As art it is amazing, not going to lie, but "commercial use" seems like a huge stretch.
  - Tenoke 6y ago
    Here's 3 off the top of my head, but there's more especially when you get into less flashy territory. Translation (Google translate, DeepL) Automatically generated product descriptions, sometimes also edited by humans (Alibaba) Image Tagging (Facebook photos)
    - otabdeveloper4 6y ago
      Okay, I should have worded my comment more carefully. These applications seem to firmly fall into the "I'm willing to compromise on quality if I don't have to pay a living person a wage" niche, so they're value-destroying, not value-creating. Are there examples of value-creating applications for ML? (From a business point of view; obviously the "shitty translations but at no cost" proposition creates value for the average Internet user.)
      - Tenoke 6y ago
        I don't understand this. Do you think that e.g. the average engineer is value destroying because if the business hires a more expensive and experienced one they will do a better job? In either case it is only 'value-destroying' if the business has unlimited resources.
314 6y ago
Winter is always coming, the important question is when it will arrive. Current ML research is still destroying new problems with ease so the current velocity is high. It will slow down first, before people start to question why the new crop of problems are too hard, and then the cycle will start again.
- ausbah 6y ago
  I think there's already a good amount of discussion around where current ML methods. stuff like lack of sample efficiency, adjusting for distribution shift, etc.
indymike 6y ago
Not sure that it's as much stagnation as commoditization.

laichzeit0 6y ago

> It’s important to avoid becoming Gary Marcus and criticize existing technique that work without proposing something else that works even better. Hilarious. Has Gary Marcus actually done anything, in practical terms, like actual code or something, that outperforms the DL approaches he attacks so viciously?

smeeth 6y ago
I keep coming across versions of this comment. Why does it matter? It’s not like the validity or invalidity of Gary’s criticisms would be different if he had done other totally separate ML research.
Rochus 6y ago
Marcus argues convincingly and appears less biased than others who benefit from the current high level of investment in DNN. It seems to be true that no better proposals for solutions have come from his side so far. But I think his criticism per se is valuable, especially his reminder that one cannot simply ignore sixty years of research. EDIT: and of course he is not the only renowned scientist who calls for reflection; here is a quote from an interview with Judea Pearl: "AI is currently split. First, there are those who are intoxicated by the success of machine learning and deep learning and neural nets. They don’t understand what I’m talking about. They want to continue to fit curves. But when you talk to people who have done any work in AI outside statistical learning, they get it immediately. I have read several papers written in the past two months about the limitations of machine learning." (https://www.theatlantic.com/technology/archive/2018/05/machine-learning-is-stuck-on-asking-why/560675/ https://www.theatlantic.com/technology/archive/2018/05/machi...)
- Tenoke 6y ago
  Gary Marcus has been proven wrong on claims he's made and is mostly just nay-saying with adhoc reasoning made up to support it. >appears less biased than others who benefit from the current high level of investment in DNN. Yes, instead he blatantly tries to benefit from the counter-investment in AI skepticism.

jarym 6y ago

> Academics sacrifice material opportunity costs in exchange for intellectual freedom. Most academics I’ve come across only think they’re doing this. My perception is they are too insecure about their self-worth to pursue material opportunities. I admit, the number of academic types I know is not vast so maybe it’s too small a subset to make any judgments

rowanG077 6y ago
The core problem is that you don't have intellectual freedom. You won't get funded if you are not researching the hot new thing.
- dimitrios1 6y ago
  Exactly, and that's assuming you are one of the very select few that gets to even select and lead the research areas. Most of the opportunities are working along side professors and department heads that already have an idea of what you should be working on. If you are at one of the elite universities with billion dollar endowments that don't have a lot of these funding problems, then you are working on what the corporate donors suggest the University work on. The university will sell it as "staying relevant for the future job market"
  - chriskanan 6y ago
    If one has a supportive advisor and adequate non-grant support (e.g., a fellowship), it is possible to work on self-drive ambitious projects. However, typically an advisor only wants to invest significant time if the work is interesting to them. Also, often I've been approached by students who want to do their own thing in terms of high risk work, but they don't have adequate skills or enough existing work to graduate with. Mt goal is to ensure students advance knowledge, graduate, and get the kind of job they want. Ambitious projects that fail will significantly impair positive outcomes. The billion dollar endowments don't usually go toward supporting research directly.
- deleted 6y ago
  [deleted]

hehehaha 6y ago

I see a lot of parallels between ML/DL and cryptocurrency research.

komuher 6y ago
Can u elaborate on that?
- gbolcer 6y ago
  I can't speak for them, but I took it this way: there are certain industries that are simply incentivized to produce more of their own industry. The criticisms of the financial industry where they are not really creating anything of value other than optimizing more value out of existing money. Adtech industry where a whole generation of technical researchers spent time figuring out how to optimize clicks. Cryptoeconomy and blockchain where large amounts of money are created out of perceived value and gargantuan efforts are made to build, not solutions to real-world problems (yes I know there are some), but ways to increase the shared, total value of the technology or individual cryptocurrency.
londons_explore 6y ago
> cryptocurrency research We're thinking very hard about if the price of our deep-lolcoin is going to go up enough to buy a new a vacation house....
superbcarrot 6y ago
At least with machine learning there is some actual substance underneath the hype.

knuthsat 6y ago

There's a nice talk by Yann LeCun where he goes on to explain really well why deep learning has such fast progress. [0] He goes on to explain how theory always comes later. I thought that information theory came before practice but turns out it also came after. (there was a bunch of heuristics for sending messages with teletypes) A nice example is Roman technological advances in architecture and materials that predate the use of geometry or mathematics completely. All advancements a result of tinkering and heuristics. [0]: https://www.youtube.com/watch?v=gG5NCkMerHU https://www.youtube.com/watch?v=gG5NCkMerHU

mettamage 6y ago
So in that sense forming a theory about something is integrating all the practical stuff you've seen before. To me that make sense, IMO that "practical stuff" are what experiments are, aren't they? Probably the experiments are far from perfect, if remotely good even, but that's how empiricism should work? I think? (hmm... not entirely sure)
- curiousllama 6y ago
  It can go both ways. E.g. often, experimental physicists will try to prove/disprove a priori predictions from theoretical physicists. Theory is just an explanation of hw the world works: you can come up with that explanation as a reason for observations, or as a logical consequence of other theories (which is then verified by observation).
_0ffh 6y ago
I've once read a paper in the late 90s or so which openly said ML research was partly "experimental maths".
gnramires 6y ago
It's not so simple as "what comes first?" (theory or experiment). What generally happens is that we stumble upon something, build a rudimentary but useful or interesting piece, then try to understand it, generalize it, and expand it (and usually succeed, with science). Both happen iteratively. Information theory came after the telegraph and early communication systems. However, we could not have built modern communication devices without Information theory (the insights and design principles). We build, then we theorize, then we build better, etc. it's not a simple procedure. Computers were developed similarly: there were all sorts of ad-hoc logical apparatus, we built boolean theory to explain it, and then we did all sorts of experiments trying to build computers. Their architects were largely mathematicians with very good ideas of mathematical design principles (and creative new mathematical ideas), not a group stringing together electrical elements and seeing what happens. The same goes for the development of ML/Deep learning, and many other technological marvels. Although "accidental" discoveries do happen, they happen from a methodical set, with knowledge, intuition, and good priors.

osipov 6y ago

> If an idea is guaranteed to work then it moves from the realm of research to engineering. The idea of the Dyson Sphere is guaranteed to work, so I guess the author is under the impression that there will not be research involved.

underdeserver 6y ago

Missing (2020)?

Rochus 6y ago
Two month ago?

Ericson2314 6y ago

I still feel like much of AI is a plot to dumb down the modern economy. We want our business people to be just as effective as our quants; we want nothing to require real intellectual labor. The idea that you traditionally have these programmers who spout mumbo-jumo all day, cost a lot of money, and seem to always be planning stuff behind your back is threatening, and all the more so because you are utterly dependent on them. ML breaks their control over the means of production. Now, that's not to say I am against labor saving devices. I most certainly am for them, but an an economy in which everyone is in a deep learning arms race is an irrational shit show that could only result in less productivity. (It's possible a single central planner AI could do better, because at least the training data would be "real world" and not output of other deep learning black box actors. But of course single-planner economies have a huge amount of other downsides.)

mlthoughts2018 6y ago
> “ The idea that you have these programmers who spout mumbo-jumo all day, cost a lot of money, and seem to always be planning stuff behind your back is threatening, and this breaks their control over the means of production.” That is a very bizarre description of ML engineers. In every company I’ve worked at, ML is a team or teams that partners with product managers and other engineering teams to learn about problems they need solved. It’s very systematic, boring, and tied heavily to those other teams as the leaders and decision makers. First you look for high level value propositions, like automating a decision process, removing a customer friction point, creating key metrics where simple metrics are intractable, or various multi-modal information retrieval goals. You identify opportunities in these kinds of high level areas in lock-step with product managers and other engineering leaders. Then you move on to identify sources of data that can be leveraged, and eventually (much later) you get to the smaller set of work training a model, validating with acceptance tests and hardening the implementation for safe production deployment. If people are spouting “mumbo jumbo” and making big ML model decisions without lock-step synchronization with other stakeholders, that sounds like organizational dysfunction, not any type of issue with ML. I also find it odd that you bring up “cost[ing] a lot of money” .. that’s very out of place among everything else mentioned. That seems more like insecurity or jealousy over the market demand for ML talent, and wanting to cut other people down rather than work with them or acknowledge the level of effort it required to get that level of expertise in ML.

screye 6y ago

There is about to be a 'great pivot' in ML. There has been a rabid frenzy of throwing money at anything that has ML in it. Soon investors and CEOs will realize that ML is effective in narrow ways and that not everything needs ML. They will also realize that 1 ML team + ML as a service (Azure ML, Sagemaker, Google AI platform) is cheaper and works more reliably. The services will keep improving and an underpaid mediocre ML-Engineering team can work to keep the production system up and running. Basically, ML teams might lose jobs just as DB/Cluster admins did with the advent of serverless compute as service. I expect it to (already happening) create a day trading company like hierarchy. The Fair/Brain/OpenAIs will pay 7 figures to the top grads to be first to market. BigN ML product teams will expand and stay as well paid as they are. Then there will be a huge drop as we move to offshore ML product teams that are viewed as cost centers by the remainder of 99% companies. These will be most of the jobs available. In such a system, a pure ML scientist (usually a PhD) will only exist at the top companies. So if you are not in the top 1-3% percentile, you will not have a pure ML job. However, there will still be hybrid DS-SDE jobs (ML engineering, ML product maintenance, ML-as-a-service user) or hybrid DS-PM jobs (Analysts, Consultants, data driven business decision makers). So, anyone who is not in that top 1-3% will have to pivot to one of these 3 roles. ______________ I won't call this an AI winter. But, it will definitely become boring majority of those employed in the field..

ramoz 6y ago
Mostly disagree. > CEOs will realize that ML is effective in narrow ways and that not everything needs ML. Any stable business isn't unjustifiably syncing costs here. I project FY21 rise an AI-funded efforts in large businesses. > They will also realize that 1 ML team + ML as a service Yes/no. This is has more platform implications vs actual ML. > ML teams might lose jobs Assumes ML Jobs only do some form of R&D. Data is a utility, and advanced analytics is valuable. Stable ML Jobs dont just work on deep learning. > I expect it to (already happening)... partially agree. already happening. But cost centers are only taking on what was standardized yesterday. Tomorrow still requires advanced analytics capabilities. > In such a system moot point. This is the system. I see a pivot in focused efforts. More optimism in an early AI commodity vs stagnation. We're moving from research to integration. Further areas to improve AI in applications (with continuous feedback training) and many domains of advanced analytics.

magicsmoke 6y ago

I dropped out of a PhD in RFIC/MMIC design. Gradient student descent and lack of first principles reasoning hits too close to home. Maybe it's just the nature of highly empirical engineering disciples to just throw many hours of grad student and simulation time blindly at problems in hopes that one lucky fella finds a new optimization.

mbeex 6y ago

"Neural Network weights are learnt via Gradient Descent and Neural Network architectures are learnt via Graduate Student Descent." Like it!

b3kart 6y ago

Some good points in the article, but I disagree with the tone and the conclusion. > we’ve rewarded and lauded incremental researchers as innovators, increased their budgets so they can do even more incremental research There isn't a scientific field where every single paper is groundbreaking. It's a Brownian motion of small incremental innovations, until eventually we stumble upon something big (like deep learning). In no way is machine learning unique in this. Sounds like the author is simply disappointed that, like in any other profession, day-to-day of a researcher is a slog and not a perennial intellectual festival. We've been in an exciting deep learning craze for a while, but it's silly to expect it to last forever. Back to the grind now. > Machine Learning Researchers can now engage in risk-free, high-income, high-prestige work Not sure what author means by "risk-free". Yes, if you're not publishing enough you're most likely not going to starve. Is that a bad thing? Is the survival instinct the only good motivator for doing good research? I would argue that there's plenty of risk, in that people who don't publish good research don't get very far in their academic careers, which in my view is good enough motivation. "They must do good research or starve" is a rather cynical take, especially from someone who seems to not be doing too badly for themselves. I'd rather more fields provided similar benefits. Maybe then going into science wouldn't be associated with so much sacrifice, so more smart people would choose science over investment banking or such, and we'd make more scientific progress faster. > CNNs use convolutions which are a generalization of matrix multiplication. A nitpick: CNNs are most definitely not a generalization of matrix multiplication. In fact, the opposite: you can view CNNs as a matrix multiplication with a particular matrix structure.

AndrewKemendo 6y ago
>We've been in an exciting deep learning craze for a while, but it's silly to expect it to last forever. Back to the grind now. This was effectively my response to hardmaru when this topic came up on reddit [1] Basically 2010-2018 was an open field for ML/DL research with old(ish) methods being rapidly applied to low hanging fruit and large datasets with newly cheap compute. Deepmind and others are actually making new methods but by and large are remixes of those same old approaches. The majority of different research out there trying other approaches (Numenta, OpenCog, Causal Calculus, anything Schmidhuber etc...) don't really get any love because it doesn't fit within the mass tensorflow/torch framework. [1]https://twitter.com/AndrewKemendo/status/1349387455552745473 https://twitter.com/AndrewKemendo/status/1349387455552745473

curiousllama 6y ago

> We’ve gamified and standardized the process so much that it’s starting to resemble case studies at consulting interviews. This is precisely true, as someone who has passed both screens for competitive jobs. Cracking the coding interview <> Case in Point Live coding <> Do 3-digit multiplication in your head (eg 347 * 469) Sorting algorithms <> M&A Evaluation Frameworks I could go on... You just memorize a bunch of crap that's vaguely (but not really) applicable but is super random, and then you just keep asking "do you want me to keep going" in various tones until they tell you to stop.

jbn 6y ago
>> We’ve gamified and standardized the process so much that it’s starting to resemble case studies at consulting interviews. Love the way this is put.. it is so true :(

tannhaeuser 6y ago

Here's something to put ML and related tech to work: to quantify the influence of shilling, astroturfing, brigading, gatekeeping, and all kinds of political campaigning on the 'net, the devastating consequences of which are only now seen by the general public. Though I have to say that I've personally seen ML invading conferences not related to ML per se, with over half of submissions employing ML techniques to random problems in the primary field which however were in itself clearly of no interest to the presenter and only a vehicle for their graduation even more than usual. So I'm admitting to see MLer as mostly in it for advancing their careers; glad to be convinced otherwise.

DrNuke 6y ago

A number of industrial domains have largely benefitted from this tool and outsiders can even set up their own virtual r&d lab at almost no upfront investment these days. As usual, a tool is (still...) only a tool, and it (still, just...) needs domain knowledge to work in full and produce noticeable ROI?

RGamma 6y ago

Sounds like the field has accelerated itself into an intellectual dead end. Not enough to know how to walk . You need to know where you want to go, and figure out the existence of any path in the first place or you might end up squaring the circle.

gbolcer 6y ago

Graduate Student Descent -- there's also the other saying: Every moment you spend writing, publishing, or working on your dissertation is a moment that you are getting behind on doing unique and cutting edge research.

lalo2302 6y ago
The only difference between doing science and fooling around is when you write it down
dr_hooo 6y ago
But also "The difference between screwing around and science is writing it down."
- gbolcer 6y ago
  I like yours better.

FartyMcFarter 6y ago

This article makes a lot of points, but nothing that implies actual stagnation? I wish the writing was more concise and to the point.

PIKAL 6y ago

What a nothing burger.

CyberRabbi 6y ago

Interesting article but the part where he says “all” you need to learn are matrix multiplications seems vacuously true to the point of meaninglessness. All numerical methods across every field essentially boil down to matrix multiplications. Your math ability needs to be such that you can transform a problem into a form that can be computed numerically using matrix multiplications, which requires more skill (sometimes significantly more) than simply knowing how matrix multiplications work. Sometimes this ability to reframe complex problems numerically using matrix multiplications is the lions share of the research!

b3kart 6y ago
Good point. Almost as useful as saying "all you need to learn is Boolean algebra" when it comes to computing.
z77dj3kl 6y ago
All you need is ZFC...
- tartaglia 6y ago
  All you need is ZF...

adamc 6y ago

So, this is the claim I thought was interesting: "Every paper is SOTA, has strong theoretical guarantees, an intuitive explanation, is interpretable and fair but almost none are mutually consistent with each other" How can that be true unless the "theory" itself isn't really worked out?

kfk 6y ago

I fear ML and AI are taking the lion share of analytics resources in most big companies and that's just not right. We still have basic problems with data integrity and blending. Getting a real end to end view of any business process is still a herculean task. Try matching something like invoices with marketing leads, good luck...

curiousllama 6y ago
If it makes you feel any better, in practice, most ML projects fail and just turn into (super valuable) reporting like you describe. If you want a reporting solution, buy AI.

tempodox 6y ago

BERT engineer? That acronym is essentially ungooglable, since you just find every person ever named Bert. This is what I finally found: BERT, which stands for Bidirectional Encoder Representations from Transformers. https://arxiv.org/abs/1810.04805 https://arxiv.org/abs/1810.04805

lordgrenville 6y ago
Searching for "BERT machine learning" would have worked.

JHonaker 6y ago

I kind of agree with the author's major sentiment: that ML research is stuck in a rut with incremental improvement. However, the longer the article goes on, the less and less I agree with any of their statements. They start of criticizing the incremental improvers. They advocate later that if "stack more layers" beats a method, the method isn't good while completely ignoring anything other than the standard SOTA metrics like data efficiency, computational efficiency, model representation efficiency, etc. And, while I agree there is a "fake rigor" problem in ML research, the particular examples that they bring up aren't extremely good exemplars in my opinion. Instead, they seem to have a problem with the standard operating procedure of mathematics while missing the point that it is what got the "stack more layers" school here in the first place. Simplifying a problem so you can understand it, and then relaxing the assumptions and seeing if you can figure out what implications that has is how advances are made. Plus, they have some hot takes and statements that are just plain wrong. > With Automatic Differentiation, the backward pass is essentially free and is as engaging to compute as 50 digit number long division. Deriving long complicated gradients is fake rigor that was useful before we had computers. What? First of all, AD is not a solved problem and using it is not "essentially free." There's a huge performance overhead when adding AD to a system. Try using a second order method with AD. I hope your Hessian actually finished computing. > Julia on the other hand is a language made for scientific computing where everything is automatically differentiable by default. This is just plain false. I'm a huge proponent of Julia, and there are some great AD packages, but in no way is everything automatically differentiable (even with the nice packages), nor is that a design goal. The work on Flux.jl (a package) is extremely impressive though, and there are particular features of Julia that allow some awesome package interoperability (e.g. the fact that some ODE solvers can be differentiated through with Flux).

ChrisRackauckas 6y ago
>some ODE solvers Some? Which ones aren't? Could you open an issue?
- JHonaker 6y ago
  I was unsure if every ODE solver was able to be differentiated through. The "some" was me trying to avoid making a blanket statement about all, when I wasn't sure. Sorry for the confusion.

DaedPsyker 6y ago

I've talked to colleagues about this before but I find it's mostly a cyclical nature fuelled by tunnel vision. Every good couple of years researchers come up with a good advancement or fresh concept that reignites the community. However all that happens till the next breakthrough is basic tweaks. The amount of junk papers that slightly adjust the method that gets a few decimal place improvement then call it a snazzy name would fill a mountain. Drawing blood from a stone. People get so obsessed with specific methods thinking its the new great thing they don't stop and think it's probably not the only way. Marketing and media is the worst though for general public perception. The amount of times they would warp ML into a magical pangea. "ML will make you skinny!" I mean it's just mathematical approximations, been around a while. Don't get me wrong, I like this field. I'm fortunate that I get to apply it to a problem that helps people but at times I just want to shout from the rooftops that it's not a God it wont make all your dreams come true. Then they get annoyed it doesn't and we a bunch of stagnation articles like this.

zetazzed 6y ago

Astronomy has entered a period of great stagnation. More and more grad students are investing huge amount of time and building larger and larger radio telescopes just to learn more about black hole formation and the properties of pulsars. Little consideration is given to how to more efficiently use inexpensive consumer telescopes purchased at Walmart! Where are the big discoveries? New planets in our solar system? A new structure for the Milky Way? I sincerely hope that the Julia language will allow us to discover ET radio signals, because we all know that Python lacks this capability.

formalsystem 6y ago
This made me lol - We should write a second post together
deleted 6y ago
[deleted]

SoSoRoCoCo 6y ago

This article is dead-on, but I think it is missing a fairly large segment of where ML is actually working well: anomaly detection and industrial defect detection. While I agree that everyone was shocked, myself included, when we saw how well SSD and YOLO worked, the last mile problem is stagnating. What I mean is: 7 years ago I wrote an image pipeline for a company using traditional AI methods. It was extremely challenging. When we saw SSDMobileNet do the same job 10x faster with a fraction of the code, our jaws dropped. Which is why the dev ship turned on a dime: there's something big in there. The industry is stagnated for exactly the reasons brought up: we don't know how to squeeze out the last mile problem because NNs are EFFING HARD and research is very math heavy: e.g., it cannot be hacked by a Zuck-type into a half-assed product overnight, it needs to be carefully researched for years. This makes programmers sad, because by nature we love to brute force trial-and error our code, and homey don't play that game with machine learning. However, places where it isn't stagnating are things like vibration and anomaly detection. This is a case where https://github.com/YumaKoizumi/ToyADMOS-dataset https://github.com/YumaKoizumi/ToyADMOS-dataset really shines because it adds something that didn't exist before, and it doesn't have to be 100% perfect: anything is better than nothing. At Embedded World last year I saw tons of FPGA solutions for rejecting parts on assembly lines. Since every object appears nearly in canonical form (good lighting, centered, homogeneous presentation), NN's are kicking ass bigtime in that space. It is important to remember Self-Driving Car Magic is just the consumer-facing hype machine. ML/NNs are working spectacularly well in some domains.

gok 6y ago
> This makes programmers sad, because by nature we love to brute force trial-and error our code, and homey don't play that game with machine learning. Huh? If anything I would say ML is way more trial-and-error focused than imperative programming.
- SoSoRoCoCo 6y ago
  Well, if your only experience is reading python opencv stack overflow posts, then of course...

gbolcer 6y ago

Technology Review summary of AI and ML adoption into business. It indicates, while you leading thinkers have figured it out, business is a long ways from being anywhere near peak ML. https://mittrinsights.s3.amazonaws.com/AIagenda2020/GlobalAIagenda.pdf https://mittrinsights.s3.amazonaws.com/AIagenda2020/GlobalAI...

antonzabirko 6y ago

Hasn't this been the state ef academia for a long time now? At least for ML.

formalsystem 6y ago

Hey everyone this is OP, was a really nice surprise seeing my article generating so much discussion. I unfortunately don't think I'll have time to answer everyone but feel free to reply to this comment if you'd like to ask me anything. It seems like the article was a bit polarizing, some of the comments made me realize I made a few imprecise statements and I'll fix those. Other comments didn't approve of my tone and that's a bit harder to fix since I tried writing this article in the same way I usually speak. I've written a lot of more dry content for my robotics ebook and I was purposeful in trying something different for this article. Some of the comments actually generalized my observations to management and software more generally and it's always nice to see people taking my ideas further than I thought they could go. One major thing I'd like to point out is I don't think "ML is dumb" is the right conclusion, there are lots of incentives making it so more of ML is stagnating but this is certainly not true for the field at large. Interesting ideas need to involve a certain amount of risk. The latter half of my article showcases a few ML adjacent projects which I think are absolutely fascinating. And if you're interested in reading more stuff by me my robotics and machine learning ebook is very easy to read http://robotoverlordmanual.com/ http://robotoverlordmanual.com/ and will teach you all you need to start building robots at home

mloncode 6y ago
Great job writing this article.
arolihas 6y ago
Where did you find the memes?
- formalsystem 6y ago
  I made most of the edgy ones and had shared them throughout 2020 on my Twitter - only Stack more layers and Squirrel are not original content and ofc the really polished ones like how many angels dance on the head of a pin and the daft punk reference The writing process for this article was meme first, content second

karmasimida 6y ago

Matrix multiplication is nothing to be laughed at. It is how anyone performs linear projection anyway. Fair enough to say, currently ML research is fixated on exhausting combinations of blocks to squeeze a marginal improve on a few public benchmarks. But it is like saying software has been stagnated since we are not getting new ISA created. Building ML application remains as challenging as ever, it is just modeling itself has become streamlined before everything else.

klik99 6y ago

It's about time someone formalized Graduate Student Descent, please someone write a paper diving into all it's properties! "Graduate Student Descent is one of the most reliable ways of getting state of the art performance in Machine Learning today and it’s also ... fully parallelizable"

whoevercares 6y ago

Are you sure that overall deep learning customers are spending more than traditional ML customers? My guess: it’s not even close - financial sector is still order of magnitude larger as paying customer than those hot CV/NLP companies.

deeeeplearning 6y ago

Lots of salty kids in here trying to justify spending 6 years as an anonymous hyperparameter tuning drone in the Stanford AI Lab.

pitspotter 6y ago

>The construction of Academia was predicated on providing a downside hedge or safety net for researchers. Where they can pursue ambitious ideas where the likelihood of success if secondary to the boldness of the vision Is there a citation for this? Much I would like it to be true, from the little I've read it seems that plenty of universities/colleges were originally religious in nature and were intended to defend orthodoxy, even to combat specific heresies. Definitely not to take intellectual risks.

totorovirus 6y ago

lol I am a so called BERT engineer and I can't agree more to this wow

academia_hack 6y ago

There's so much stuff about the world that we don't understand and want to and that even trivial machine learning techniques can help us figure out. Yet, at least in my field (computer security), it has become incredibly difficult to publish research that applies machine learning without having some needlessly obtuse methodological twist to make it seem more complicated. Even low hanging fruit needs picking and I'm worried discoveries with intrinsic value are being left to rot in the name of feeling "scientific."

master_yoda_1 6y ago

I like till it mention keras, fast.ai. No offense to keras, fast.ai they are very good for beginners in ML. But for serious work you need more (try to write autoregressive generative model etc). Here again bullshit article trending on HN front page, shame

BlueTemplar 6y ago

Richard Feynman — 'Philosophy of science is about as useful to scientists as ornithology is to birds.' Well, this blogpost shows where this analogy breaks down : birds (AFAWK) don't try to discuss ornithology. "bureaucrats running the asylum" is normal science ! Thomas Kuhn has shown how it works in The Structure of Scientific Revolutions : https://samzdat.com/2018/05/19/science-under-high-modernism/ https://samzdat.com/2018/05/19/science-under-high-modernism/ The problem is that without Kuhn our expectation are set by pop history of science, which only remembers 'anormal', extraordinary science : the paradigm changes. (Otherwise, this is a great blogpost.)

fuy 6y ago

fast.ai link actually points to http://book.realworldhaskell.org/ http://book.realworldhaskell.org/ What gives? Edit: having read to the end, there's a mention of Haskell book, so just an editing error then.