6 ms·

Deep reinforcement learning will transform manufacturing as we know it

87 points by apsec112 5y ago

jpfr 5y ago
No it won’t transform manufacturing. Deep RL in physical systems is a pipe dream. Deep RL is one of the most data hungry methods. And in physical systems you don’t get enough samples. There are loads of data. But no manager of a shopfloor lets you produce scrap just to potentially learn. There is an opportunity to combine ML with classical engineering models for manufacturing. Think differential equations for chemical processes. Then you end up with something like ML augmented control theory. That does work.
- thoughtstheseus 5y ago
  That’s why you simulate the process.
  - Frost1x 5y ago
    Your simulated environment needs to accurately represent everything you're simulating for RL to be useful, otherwise, it will simply find artifacts of flaws of the sim itself or may fail to discover n-th order effects since they won't ever occur. This is exactly where DNNs have excelled and is most desired to be used: in all the places we don't know the foundational theory to. That may be why they're so successful compared to our traditional reductionist approaches: they might catch all the non-reduced aspects we never add into sims as we know them. We don't need DNNs doing QED or Newtonian mechanics (in general), we have nice solutions to these that in many (not all) cases are pretty darn efficient. In some cases, you could be computationally bound and never achieve a sim representing your desired environment. In many, you lack theory to correctly build a sim. You need RL building the sims that you plan to use RL to explore optimal processes you seek within. We have a few disciplines that are well formulated enough where sims can be useful in but the vast majority are simply going to give you garbage or are just computationally bound. At best, many sims provide guidance that experts need to interpret.
- jgalt212 5y ago
  > Deep RL is one of the most data hungry methods. What's good for AWS is good for America.
- dimatura 5y ago
  It's not easy (or it would have been done already!) but I don't think it's a pipe dream. A lot of the computer vision capabilities that are taken for granted today would've been pipe dreams a few years ago. It's true that the process is data hungry. But this isn't such a big problem in some cases. Having a few dozens or more of robot arms just figuring out stuff by trial and error isn't such a big deal (you don't have to do this at the customer's shop floor...). For self driving cars and drones this isn't such a great idea (though it has been tried to some extent). That's where simulation and more clever algorithms come in. Including, of course some prior knowledge, like you are implying. I don't think anybody expects to "transform manufacturing" by just throwing REINFORCE into a random KUKA arm.
- LudwigNagasena 5y ago
  Manufacturers already install all sorts of sensors to control the production process. Of course if you produce high end luxury cars there is probably not enough data for a useful model, but if you operate e.g. a hot rolling mill it may be useful.
  - jpfr 5y ago
    Hot rolling is a good example. For the last 30 years neural networks have been used to control the process. There’s a nice NeurIPS paper from 1991 [1]. And even with the non-deep networks from back in the day, people felt compelled to include prior knowledge and even used Bayesian priors to deal with a lack of data. [1] http://papers.neurips.cc/paper/447-neural-control-for-rolling-mills-incorporating-domain-theories-to-overcome-data-deficiency.pdf http://papers.neurips.cc/paper/447-neural-control-for-rollin...
behnamoh 5y ago
How about Inverse Reinforcement Learning? MaxEnt and Deep MaxEnt seem to estimate the reward functions pretty well. But I've only seen limited applications of IRL. Researchers mostly focus on gridworld examples or robot arms learning how to do certain tasks from (human) experts. But how can IRL have more influential role in the society?
- deleted 5y ago
  [deleted]
hannibalhorn 5y ago
So after reading the article I started reading up on deep RL, and came across what is essentially a blog post that shows you how to train a model to play Super Mario Bros [1]. I have of course done all the mNIST handwritten number recognition stuff, and have done quite a bit of real world analysis with scikit-learn, but this is... pretty cool! 1. https://pytorch.org/tutorials/intermediate/mario_rl_tutorial.html https://pytorch.org/tutorials/intermediate/mario_rl_tutorial...
- Matumio 5y ago
  RL theory is cool and all, but a large part of the trick seems to be the neural network architecture itself (that is, having a good parametrized model for the policy) and millions of evaluations. For some Atari games it turned out to be almost as sample-efficient to try random weights instead of RL. Not for everything, and e.g. policy gradient is certainly worth learning about. But see for example https://openai.com/blog/evolution-strategies/ https://openai.com/blog/evolution-strategies/ Personally I expect all real-world applications of RL to be trained in simulation, with tricks to make sure they also learn to adapt to reality at startup (meta-learning). For example by simulating each episode with parameters that are slightly off.
robbmorganf 5y ago
Other comments point out valid technical concerns, but even if those were solved, there's still not a clear application in manufacturing. If it has already been automated (e.g. semiconductors, welding car frames) then there's not a lot to gain from Deep RL. On the other hand, if it hasn't been automated yet, who is going to invest hundreds of millions into a Deep RL system (never mind an army of robots) when they can just pay a slightly higher variable fee for humans. For example, sure a robot could theoretically sew jeans for $0.20 vs a human at $15.00 (living wage in US), but does any one firm have the scale to justify $500M to train Deep RL?
- throwaway9980 5y ago
  Nike has a market capitalization of over $200B. I’d guess if they could do away with human labor for $500M they would just do it. Cut labor costs and eliminate their sweatshop labor practice PR problems in one fell swoosh.
  - colinmhayes 5y ago
    Nike does their manufacturing in sweatshops. Probably already cheaper than robots and technicians would cost.
    - Matrixik 5y ago
      Sewing is hard problem for robots but they are already here and promise prices similar to low wage countries: https://softwearautomation.com/ https://softwearautomation.com/
  - patcon 5y ago
    > I’d guess if they could do away with human labor for $500M they would just do it. I see what you did there :)
  - robbmorganf 5y ago
    Remember that Nike only manufactures a small amount itself. Most is contracted out to factories overseas with different incentives.
loehnsberg 5y ago
I have tested numerous RL algorithms on simulated production systems for my PhD thesis. It was a complete failure. Most production planning problems are inherently high-dimensional and come with complex constraints. When you add noise to the problem, like random demand, all generic RL algorithms ran into trouble once you added complexity (like producing multiple products on the same line). So if you’re looking for good examples to break RL codes, throw them a nice stochastic economic lot sizing problem with sequence-dependent setup times. By constrast what works well is policy search over parametric policies that consider domain knowledge. Sounds boring, but is quite hard to beat.
- vonnik 5y ago
  TechCrunch article author here. Our team at Pathmind has applied deep RL to multiple use cases in industrial control and supply chain management, notably various forms of scheduling, and MEIO, respectively. You can see some demo simulations here: https://pathmind.com/examples/ https://pathmind.com/examples/ https://cloud.anylogic.com/model/15ee2fe9-c45c-425d-8b98-9202b3d0a4f9?mode=SETTINGS https://cloud.anylogic.com/model/15ee2fe9-c45c-425d-8b98-920... https://cloud.anylogic.com/model/9c0725cc-7eb5-44ed-a2de-ff81690f2075?mode=SETTINGS https://cloud.anylogic.com/model/9c0725cc-7eb5-44ed-a2de-ff8... https://cloud.anylogic.com/model/8769d942-43c4-4530-bc68-243e19f4dd4d?mode=SETTINGS https://cloud.anylogic.com/model/8769d942-43c4-4530-bc68-243... https://cloud.anylogic.com/model/b7da42a0-734c-461f-9f68-38254285178c?mode=SETTINGS https://cloud.anylogic.com/model/b7da42a0-734c-461f-9f68-382... Yes, these show RL applied to simulations, and yes, real data and real physical plants are more complex than that. But those physical systems are already controlled by optimizers (usually mathematical solvers like IBM Cplex or Gurobi), and deep RL happens to be an optimizer that can handle a few things better: * data variability * multiple objectives in complex scenarios * multiple agents making simultaneous decisions in coordination Solving the third problem gives rise to very interesting emergent behavior among teams of machines, which learn to behave in ways that are almost impossible for a hard-coded set of rules to specify. This is the source of many of the gains that RL produces. I see a lot of preconceptions about RL in this thread that are partially true, but also solvable. Yes, RL is data hungry. I acknowledged that in the TC piece. The problems with data are pretty much the same problems you get with ML in general. It's messy, partial, and usually not gathered with ML in mind. RL is no more impossible than other ML approaches, based on that argument. A valid simulation of a physical system is a source of synthetic data that helps solve that problem. Deep RL is already controlling machines on factory floors, and it will slowly help optimize larger and larger systems, from groups of assembly lines, to networks of plants. In some cases it will serve as a "decisioning layer" (not my word) with HITL, and in others, it will drive more autonomous systems with low latency requirements. Believe me or not, it's happening. Without getting into too many details, the underlying algorithms are a flavor of policy gradient implemented with the open source projects Ray/RLlib (supported by AnyScale) and Gym (from OpenAI). They're great tools and I highly recommend them to anyone who wants to explore deep RL. Fwiw, DeepMind says RL may be the path to AGI, and smarter people than me have placed their bets on that team, which has consistently surpassed expectations about what RL can do. https://www.sciencedirect.com/science/article/pii/S0004370221000862 https://www.sciencedirect.com/science/article/pii/S000437022... Microsoft is making a major push into this area using their Bonsai acquisition to power autonomous systems. https://www.microsoft.com/en-us/ai/autonomous-systems https://www.microsoft.com/en-us/ai/autonomous-systems AWS, while not particularly focused on RL, is making a big push into operational technology (OT) as well. The incentives driving their cloud business will spur them to transform the software that controls the factory floor, or at least prompt existing OT software makers like Siemens to move faster. If you have any questions about how deep RL is being applied, my email is in my profile. Pls hit me up!
maxbaines 5y ago
Pretty much no mention of any manufacturing machinery or processes, beyond a robot arm.
doubtfuluser 5y ago
Many comments here actually state that RL is really data hungry. That is true, and indeed one of the difficulties to overcome. However there is a lot of work in the field with making use of transfer learning / meta learning to tackle this and I have seen successful implementations in the manufacturing industry. The bigger blocker right now is the simulation part itself (and there is also a lot work on that!) Training a DRL for real world physical processes ob real world processes has the Problem, that a failure is extremely expensive. You don’t want to see a manufacturing process moving too fast (because it’s still learning / optimizing) and then breaking a 500k router spindle. Simulation is certainly one of the trajectories to deal with this though. The third big block is then really the cost: for many manufacturing processes there are pretty good parameters found over multiple years. The trade off of the costs of a new DRL system - training it, potential failures, deploying it and training the users - to the gains need to be big enough to justify the use of DRL financially. Again standardization will help with this, but it requires significant R&D costs upfront only few are willing to pay. In the end, in my opinion we will see the application more often, but it takes time and effort to improve and make it cost effective.
deleted 5y ago
[deleted]
phantasilide 5y ago
I’ve worked at a startup providing a similar solution. Even for an ad piece, the article is wildly underestimating the difficulty in getting this to work and the problem is not lack of state of the art methods. Most customers don’t have the type of data to control these systems, even if they think they do. While industry 4.0 has taught companies to measure everything, there is rarely any notion of data integrity or quality in place, and even your smart new ML technique will not be able to control system with very low correlation between observations and controls. The reality for companies providing these ML solutions is that they end up spending all of their resources on fixing each clients’ data problems rendering their smart solution irrelevant.
- vonnik 5y ago
  It's true that a lot of companies are at different stages of what we would call "the journey of digital transformation." Some of them do not know what's happening inside their plants. The problem before the problem -- the upstream chokepoint of ML -- is gathering, collecting and cleaning the right data. That's true across the industry and it's the main reason why ML is not moving faster, and why we don't see more huge ML-based startups. Scale, one of the unicorns, is chiefly devoted to data annotation... But there are a lot of smaller, well defined problems with machine scheduling and/or inventory management where they have exactly the data they need, and they are already feeding it to an optimizer. Deep RL can often outperform against those existing technologies.
thom 5y ago
I think there are lower hanging fruit in manufacturing than this, although someone more familiar with the industry can tell me how realistic this is. I was speaking to my father-in-law recently, who has helped design production lines his entire life. One interesting thing he said was that companies often have huge investments in machines that are effectively no longer supported by manufacturers. Many of these machines are based on simple circuits or microcontrollers, and their behaviour is more or less fixed at this point. But in many cases the performance of the machine is not optimal, with wasteful pauses and inefficient transitions between parts of a job. It occurred to me that there’d be a reasonably useful business in learning the behaviour of the controllers in these machines, and in some cases it must be possible to just rip out chips and replace them with tweaked versions. I don’t believe the IO can be very complex in this generation of machines (we’re talking various cutting machines etc) but you’d be offering savings of millions if you could build a scalable business tweaking them. One approach to that would be treating microcontrollers as black boxes and just using deep learning to pick up the mappings between inputs and outputs. Anyway, completely idle speculation on my part but free business idea for someone if they thought it was in any way possible, and entirely possible this is big business already.
- caseymarquis 5y ago
  This is called 'retrofitting', and it commonly costs $100k to $300k per machine. It's a niche business as people smart enough to pull it off are fairly rare.
caseymarquis 5y ago
If you want to revolutionize manufacturing, you need to open a large number of small job shops each specializing in a different kind of work, then do applied research and engineering in this real world environment. Step one is making the shops successful, step two is throwing a bunch of geniuses into the room who can help the people who understand what's needed build interesting new technologies. Large existing manufacturing firms with the kind of money to do this can't get out of their own way, and will never be a source of innovation. Small companies don't have the funds to throw at maintaining a staff of genius engineers and researchers. Most outsiders with access to capital looking at innovating in the space probably don't even appreciate the difference between a job shop and an OEM, and are likely to lose lots of cash on projects headed by former OEM enterprise middle managers who exclusively have experience getting in the way of production and wasting money. I'd love to be part of a highly funded team of smart people whose job was to reinvent manufacturing in a practical environment.

87 points by apsec112 5y ago

jpfr 5y ago

No it won’t transform manufacturing. Deep RL in physical systems is a pipe dream. Deep RL is one of the most data hungry methods. And in physical systems you don’t get enough samples. There are loads of data. But no manager of a shopfloor lets you produce scrap just to potentially learn. There is an opportunity to combine ML with classical engineering models for manufacturing. Think differential equations for chemical processes. Then you end up with something like ML augmented control theory. That does work.

thoughtstheseus 5y ago
That’s why you simulate the process.
- Frost1x 5y ago
  Your simulated environment needs to accurately represent everything you're simulating for RL to be useful, otherwise, it will simply find artifacts of flaws of the sim itself or may fail to discover n-th order effects since they won't ever occur. This is exactly where DNNs have excelled and is most desired to be used: in all the places we don't know the foundational theory to. That may be why they're so successful compared to our traditional reductionist approaches: they might catch all the non-reduced aspects we never add into sims as we know them. We don't need DNNs doing QED or Newtonian mechanics (in general), we have nice solutions to these that in many (not all) cases are pretty darn efficient. In some cases, you could be computationally bound and never achieve a sim representing your desired environment. In many, you lack theory to correctly build a sim. You need RL building the sims that you plan to use RL to explore optimal processes you seek within. We have a few disciplines that are well formulated enough where sims can be useful in but the vast majority are simply going to give you garbage or are just computationally bound. At best, many sims provide guidance that experts need to interpret.
jgalt212 5y ago
> Deep RL is one of the most data hungry methods. What's good for AWS is good for America.
dimatura 5y ago
It's not easy (or it would have been done already!) but I don't think it's a pipe dream. A lot of the computer vision capabilities that are taken for granted today would've been pipe dreams a few years ago. It's true that the process is data hungry. But this isn't such a big problem in some cases. Having a few dozens or more of robot arms just figuring out stuff by trial and error isn't such a big deal (you don't have to do this at the customer's shop floor...). For self driving cars and drones this isn't such a great idea (though it has been tried to some extent). That's where simulation and more clever algorithms come in. Including, of course some prior knowledge, like you are implying. I don't think anybody expects to "transform manufacturing" by just throwing REINFORCE into a random KUKA arm.
LudwigNagasena 5y ago
Manufacturers already install all sorts of sensors to control the production process. Of course if you produce high end luxury cars there is probably not enough data for a useful model, but if you operate e.g. a hot rolling mill it may be useful.
- jpfr 5y ago
  Hot rolling is a good example. For the last 30 years neural networks have been used to control the process. There’s a nice NeurIPS paper from 1991 [1]. And even with the non-deep networks from back in the day, people felt compelled to include prior knowledge and even used Bayesian priors to deal with a lack of data. [1] http://papers.neurips.cc/paper/447-neural-control-for-rolling-mills-incorporating-domain-theories-to-overcome-data-deficiency.pdf http://papers.neurips.cc/paper/447-neural-control-for-rollin...

behnamoh 5y ago

How about Inverse Reinforcement Learning? MaxEnt and Deep MaxEnt seem to estimate the reward functions pretty well. But I've only seen limited applications of IRL. Researchers mostly focus on gridworld examples or robot arms learning how to do certain tasks from (human) experts. But how can IRL have more influential role in the society?

deleted 5y ago
[deleted]

hannibalhorn 5y ago

So after reading the article I started reading up on deep RL, and came across what is essentially a blog post that shows you how to train a model to play Super Mario Bros [1]. I have of course done all the mNIST handwritten number recognition stuff, and have done quite a bit of real world analysis with scikit-learn, but this is... pretty cool! 1. https://pytorch.org/tutorials/intermediate/mario_rl_tutorial.html https://pytorch.org/tutorials/intermediate/mario_rl_tutorial...

Matumio 5y ago
RL theory is cool and all, but a large part of the trick seems to be the neural network architecture itself (that is, having a good parametrized model for the policy) and millions of evaluations. For some Atari games it turned out to be almost as sample-efficient to try random weights instead of RL. Not for everything, and e.g. policy gradient is certainly worth learning about. But see for example https://openai.com/blog/evolution-strategies/ https://openai.com/blog/evolution-strategies/ Personally I expect all real-world applications of RL to be trained in simulation, with tricks to make sure they also learn to adapt to reality at startup (meta-learning). For example by simulating each episode with parameters that are slightly off.

robbmorganf 5y ago

Other comments point out valid technical concerns, but even if those were solved, there's still not a clear application in manufacturing. If it has already been automated (e.g. semiconductors, welding car frames) then there's not a lot to gain from Deep RL. On the other hand, if it hasn't been automated yet, who is going to invest hundreds of millions into a Deep RL system (never mind an army of robots) when they can just pay a slightly higher variable fee for humans. For example, sure a robot could theoretically sew jeans for $0.20 vs a human at $15.00 (living wage in US), but does any one firm have the scale to justify $500M to train Deep RL?

throwaway9980 5y ago
Nike has a market capitalization of over $200B. I’d guess if they could do away with human labor for $500M they would just do it. Cut labor costs and eliminate their sweatshop labor practice PR problems in one fell swoosh.
- colinmhayes 5y ago
  Nike does their manufacturing in sweatshops. Probably already cheaper than robots and technicians would cost.
  - Matrixik 5y ago
    Sewing is hard problem for robots but they are already here and promise prices similar to low wage countries: https://softwearautomation.com/ https://softwearautomation.com/
- patcon 5y ago
  > I’d guess if they could do away with human labor for $500M they would just do it. I see what you did there :)
- robbmorganf 5y ago
  Remember that Nike only manufactures a small amount itself. Most is contracted out to factories overseas with different incentives.

loehnsberg 5y ago

I have tested numerous RL algorithms on simulated production systems for my PhD thesis. It was a complete failure. Most production planning problems are inherently high-dimensional and come with complex constraints. When you add noise to the problem, like random demand, all generic RL algorithms ran into trouble once you added complexity (like producing multiple products on the same line). So if you’re looking for good examples to break RL codes, throw them a nice stochastic economic lot sizing problem with sequence-dependent setup times. By constrast what works well is policy search over parametric policies that consider domain knowledge. Sounds boring, but is quite hard to beat.

vonnik 5y ago
TechCrunch article author here. Our team at Pathmind has applied deep RL to multiple use cases in industrial control and supply chain management, notably various forms of scheduling, and MEIO, respectively. You can see some demo simulations here: https://pathmind.com/examples/ https://pathmind.com/examples/ https://cloud.anylogic.com/model/15ee2fe9-c45c-425d-8b98-9202b3d0a4f9?mode=SETTINGS https://cloud.anylogic.com/model/15ee2fe9-c45c-425d-8b98-920... https://cloud.anylogic.com/model/9c0725cc-7eb5-44ed-a2de-ff81690f2075?mode=SETTINGS https://cloud.anylogic.com/model/9c0725cc-7eb5-44ed-a2de-ff8... https://cloud.anylogic.com/model/8769d942-43c4-4530-bc68-243e19f4dd4d?mode=SETTINGS https://cloud.anylogic.com/model/8769d942-43c4-4530-bc68-243... https://cloud.anylogic.com/model/b7da42a0-734c-461f-9f68-38254285178c?mode=SETTINGS https://cloud.anylogic.com/model/b7da42a0-734c-461f-9f68-382... Yes, these show RL applied to simulations, and yes, real data and real physical plants are more complex than that. But those physical systems are already controlled by optimizers (usually mathematical solvers like IBM Cplex or Gurobi), and deep RL happens to be an optimizer that can handle a few things better: * data variability * multiple objectives in complex scenarios * multiple agents making simultaneous decisions in coordination Solving the third problem gives rise to very interesting emergent behavior among teams of machines, which learn to behave in ways that are almost impossible for a hard-coded set of rules to specify. This is the source of many of the gains that RL produces. I see a lot of preconceptions about RL in this thread that are partially true, but also solvable. Yes, RL is data hungry. I acknowledged that in the TC piece. The problems with data are pretty much the same problems you get with ML in general. It's messy, partial, and usually not gathered with ML in mind. RL is no more impossible than other ML approaches, based on that argument. A valid simulation of a physical system is a source of synthetic data that helps solve that problem. Deep RL is already controlling machines on factory floors, and it will slowly help optimize larger and larger systems, from groups of assembly lines, to networks of plants. In some cases it will serve as a "decisioning layer" (not my word) with HITL, and in others, it will drive more autonomous systems with low latency requirements. Believe me or not, it's happening. Without getting into too many details, the underlying algorithms are a flavor of policy gradient implemented with the open source projects Ray/RLlib (supported by AnyScale) and Gym (from OpenAI). They're great tools and I highly recommend them to anyone who wants to explore deep RL. Fwiw, DeepMind says RL may be the path to AGI, and smarter people than me have placed their bets on that team, which has consistently surpassed expectations about what RL can do. https://www.sciencedirect.com/science/article/pii/S0004370221000862 https://www.sciencedirect.com/science/article/pii/S000437022... Microsoft is making a major push into this area using their Bonsai acquisition to power autonomous systems. https://www.microsoft.com/en-us/ai/autonomous-systems https://www.microsoft.com/en-us/ai/autonomous-systems AWS, while not particularly focused on RL, is making a big push into operational technology (OT) as well. The incentives driving their cloud business will spur them to transform the software that controls the factory floor, or at least prompt existing OT software makers like Siemens to move faster. If you have any questions about how deep RL is being applied, my email is in my profile. Pls hit me up!

maxbaines 5y ago

Pretty much no mention of any manufacturing machinery or processes, beyond a robot arm.

doubtfuluser 5y ago

Many comments here actually state that RL is really data hungry. That is true, and indeed one of the difficulties to overcome. However there is a lot of work in the field with making use of transfer learning / meta learning to tackle this and I have seen successful implementations in the manufacturing industry. The bigger blocker right now is the simulation part itself (and there is also a lot work on that!) Training a DRL for real world physical processes ob real world processes has the Problem, that a failure is extremely expensive. You don’t want to see a manufacturing process moving too fast (because it’s still learning / optimizing) and then breaking a 500k router spindle. Simulation is certainly one of the trajectories to deal with this though. The third big block is then really the cost: for many manufacturing processes there are pretty good parameters found over multiple years. The trade off of the costs of a new DRL system - training it, potential failures, deploying it and training the users - to the gains need to be big enough to justify the use of DRL financially. Again standardization will help with this, but it requires significant R&D costs upfront only few are willing to pay. In the end, in my opinion we will see the application more often, but it takes time and effort to improve and make it cost effective.

deleted 5y ago

[deleted]

phantasilide 5y ago

I’ve worked at a startup providing a similar solution. Even for an ad piece, the article is wildly underestimating the difficulty in getting this to work and the problem is not lack of state of the art methods. Most customers don’t have the type of data to control these systems, even if they think they do. While industry 4.0 has taught companies to measure everything, there is rarely any notion of data integrity or quality in place, and even your smart new ML technique will not be able to control system with very low correlation between observations and controls. The reality for companies providing these ML solutions is that they end up spending all of their resources on fixing each clients’ data problems rendering their smart solution irrelevant.

vonnik 5y ago
It's true that a lot of companies are at different stages of what we would call "the journey of digital transformation." Some of them do not know what's happening inside their plants. The problem before the problem -- the upstream chokepoint of ML -- is gathering, collecting and cleaning the right data. That's true across the industry and it's the main reason why ML is not moving faster, and why we don't see more huge ML-based startups. Scale, one of the unicorns, is chiefly devoted to data annotation... But there are a lot of smaller, well defined problems with machine scheduling and/or inventory management where they have exactly the data they need, and they are already feeding it to an optimizer. Deep RL can often outperform against those existing technologies.

thom 5y ago

I think there are lower hanging fruit in manufacturing than this, although someone more familiar with the industry can tell me how realistic this is. I was speaking to my father-in-law recently, who has helped design production lines his entire life. One interesting thing he said was that companies often have huge investments in machines that are effectively no longer supported by manufacturers. Many of these machines are based on simple circuits or microcontrollers, and their behaviour is more or less fixed at this point. But in many cases the performance of the machine is not optimal, with wasteful pauses and inefficient transitions between parts of a job. It occurred to me that there’d be a reasonably useful business in learning the behaviour of the controllers in these machines, and in some cases it must be possible to just rip out chips and replace them with tweaked versions. I don’t believe the IO can be very complex in this generation of machines (we’re talking various cutting machines etc) but you’d be offering savings of millions if you could build a scalable business tweaking them. One approach to that would be treating microcontrollers as black boxes and just using deep learning to pick up the mappings between inputs and outputs. Anyway, completely idle speculation on my part but free business idea for someone if they thought it was in any way possible, and entirely possible this is big business already.

caseymarquis 5y ago
This is called 'retrofitting', and it commonly costs $100k to $300k per machine. It's a niche business as people smart enough to pull it off are fairly rare.

caseymarquis 5y ago

If you want to revolutionize manufacturing, you need to open a large number of small job shops each specializing in a different kind of work, then do applied research and engineering in this real world environment. Step one is making the shops successful, step two is throwing a bunch of geniuses into the room who can help the people who understand what's needed build interesting new technologies. Large existing manufacturing firms with the kind of money to do this can't get out of their own way, and will never be a source of innovation. Small companies don't have the funds to throw at maintaining a staff of genius engineers and researchers. Most outsiders with access to capital looking at innovating in the space probably don't even appreciate the difference between a job shop and an OEM, and are likely to lose lots of cash on projects headed by former OEM enterprise middle managers who exclusively have experience getting in the way of production and wasting money. I'd love to be part of a highly funded team of smart people whose job was to reinvent manufacturing in a practical environment.