7 ms·

Using AI and AR to improve ArcellorMittal's production process at a Hackathon

72 points by netb 9y ago

ambicapter 9y ago
Damn, how many years of experience do you need to be able to come up with workable solutions as fast as this?
- MariuszGalus 9y ago
  6 months, here's why: They random forested height data, everyone random forests these days. It's quick and dirty and yields good accuracy. Not only that, everyone seems to support a built in Random Forest training model, programs from pandas/numpy/scikit to R has it built in. Height data seems the easiest to go by, to short and it has a kink that may cause cracking in the future as the metal oxidizes. What were their 3 models? Who knows? But their idea of combining 3 models, that 3 different teams have made is standard in all DataScience competitions. This is called ensembling. ArToolkit for Unreal Engine, someone else made that and they probably just connected a bunch of blocks together with Unreal's script engine. Honestly, 6 months max and you can do this in a day yourself.
  - Drdrdrq 9y ago
    Multiple years, here's why: you need to decide what will work and what will not, as you don't have time to go the wrong route. And that takes experience which it would be difficult to gain in 6 months.
  - altstar 9y ago
    Yeah right: We used two convolutional neural networks, one for each side of the weld. Features extracted by these nets are then concatenated, as well as the meta data. Two fully connected layers are then applied on this vector, giving the final prediction. Everything was trained on the raw images resized to 20x150 pixels.
- onebonsai 9y ago
  I am no expert into AI. The AR part we learned because we are busy in VR. It is a natural evolution in that case, as we can use the workflow we are used to in VR. The AI part however is more complex, as it requires understanding of the concepts and methodologies behind it. We used Deep Learning methodologies for this project. This is an upcoming field in AI, which evolves very quickly. If you are interested in it, it's a great time to start learning the basics of it!
tshanmu 9y ago
extremely inspiring!!
option_greek 9y ago
I didn't understand where the AR was overlaid. Was it on the glasses on which the operator was wearing or on a LED monitor ?
- otoburb 9y ago
  FTA: "Furthermore, we decided that we would require a setup that actually worked. Given that we did not have any AR hardware available (eg AR glasses such as a Microsoft Hololens), we quickly built one ourselves using a production helmet and a webcam." Medium images sometimes take a while to load, but if you wait you can see their "webcam on a hard-hat helmet" setup for the demonstration to the judges. The overlaid AR + video feed then shows up on a monitor.
- mring33621 9y ago
  They did not have VR glasses, so they simulated them using a helmet-mounted webcam and a monitor. The AR weld risk profile (my words) is overlaid on the captured video images. See the article's video with caption 'Screen-capture of the AR experience...'
- AndrewKemendo 9y ago
  Nowhere, it wasn't actually AR. It was a video feed that had the 3D model overlayed on the desktop/laptop the video was feeding from. This isn't just semantics - moving from their setup to an actual AR HMD - even if it was just the current state of the art (Hololens, ODG, Epson etc...) wouldn't be trivial and would likely be a different workflow.
webbrahmin 9y ago
Can you please elaborate on the deep learning part. What kind of network dis you use?
- Simoncarbo 9y ago
  We used two convolutional neural networks, one for each side of the weld. Features extracted by these nets are then concatenated, as well as the meta data. Two fully connected layers are then applied on this vector, giving the final prediction. Everything was trained on the raw images resized to 20x150 pixels.
peterbonney 9y ago
This is a really cool piece of engineering, but I'm bothered by the idea that the going rate for a huge multinational corporation to get a prototype of a potentially extremely valuable technology is less than 10k of prize money [1], plus a "mini-drone race" and "snacks and food". Maybe it's just me, but I find this type of corporate-sponsored hackathon exploitative and distastefull. We're not talking about an event that is producing a social good in an area that is otherwise underinvested. This is purely profit-motivated, and ArcelorMittal can afford to pay a lot more for a prototype like this. 1. https://www.eventbrite.nl/e/the-arcelormittal-challenge-hack4steel-tickets-34228761146 https://www.eventbrite.nl/e/the-arcelormittal-challenge-hack...
- ak39 9y ago
  I'm with you on this. I can't imagine any other complex discipline where experts offer their time & creativity to the lowest bidder so quickly. Programming seems uniquely generous in this regard. Or stupid.
- onebonsai 9y ago
  Thanks for your view. However, it is important to see this event also for what it is. It is a way for Arcelor to think out-of-the-box, and get input from people that are not directly linked to their business. Secondly, it is a way for students to get a glimps of what the corporate projects and challenges are. Additionally, for us, as professionals, it is a way to show, although in a very rough fashion, what is possible, and where we can help. We saw this event more as a team building than work. The result we got at the hackathon is a very rough prototype, not a finished product. In no way, this can directly be implemented on the floor. As such, input from experts is still required, either us, or other experts in the field. Therefore, these hackathons are useful, in my opinion, not only to the company, but also the participants :)
  - csmattryder 9y ago
    > It is a way for Arcelor to think out-of-the-box, and get input from people that are not directly linked to their business. And there's nothing wrong with that, so much so, that we created a title for these people: Consultant. They're highly paid, and available to bring years of domain experience to a business. How much value did ArcellorMittal get in exchange for 8.5k and some pizza? How much would <leading AR firm> charge, in comparison? (Man, programmers desperately need someone to tell them that no, not everyone just "gets" programming. You have a skill, charge for it!)
onebonsai 9y ago
Hey! Thanks guys for your comments! I'm the writer of the article (Evarest). Just a bit of an update in the mean while :) Our AI team is now busy writing a bit more detail on the actual AI part of the project. The project used: Random Forested Height Data Deep Learning Models We trained on the entire dataset of 60.000+ elements, but did pre-processing on the data. I am not placed to answer detailed questions on the AI part, but we will update the article soon with more details. Note that this is not a perfect result, nor that we claim to be all-knowing or have all the answers. This is a result from a hackathon, which involved working in suboptimal conditions under heavy time constraints. As for the AR part, we did not have access to AR helmets. We are aware that not all the data that we showed would work on an AR helmet (currently). However, this tech is totally feasible in the short to medium term. Also, it is completely feasible on tablet or similar. The concept of the hackathon was to show the potential of these technologies, and to integrate the AR and AI, which I find we succeeded in.
- corford 9y ago
  Great hack. Really enjoyed reading this! Noob question: the weld as shown in the AR video looks really detailed (fissures, deformations etc). Was that rendered by you using the heightmap data alone or did you have access to top and bottom images of the welds as part of the training data?
  - loa_in_ 9y ago
    I think it was data from the laser scanner as it was monochromatic (w/ shading)
    - corford 9y ago
      Thanks!
FLUX-YOU 9y ago
I just got into Unreal Engine with the VR template project, which includes grabbing/teleportation functionality by default. It's been really easy to get into and if you have a headset, I'd really recommend trying it out if you've had a VR development itch.
- onebonsai 9y ago
  Indeed, thanks for the comment! Unreal Engine is a great engine to quickly build prototypes in. We are used to work with the included VR Template of Unreal, and it most of the times works great. Also for more serious projects, Unreal did not let us down.
onebonsai 9y ago
UPDATE: Thank you for your comments and remarks ! Based on the inputs of the community, we rewrote the AI part of the article (see section 'Building the AI Assistant'). More details are available now. --> https://lab.onebonsai.com/arcelormittals-hack4steel-hackaton-43d9a0338a4 https://lab.onebonsai.com/arcelormittals-hack4steel-hackaton...
- specialist 9y ago
  I suggest changing "conjest" to "condense".
kerberos84 9y ago
How much money will Arcelor earn from this piece of Software? You just implemented it for free!
- rmorm 9y ago
  Hey ! I am one of the AI guy of the team. I agree that if the solution gets implemented on the lines, they will probably make more net money out of it than what was spent for the challenge (which is more than 8.5k€ and some pizzas as said in other comments BTW, the challenge was probably more expensive to organize). However, let's be realistic, they will have to put a lot more money into this before they get a viable product to insert in their production lines. The solution we have built is way too simple regarding actual operational constraints (e.g.: operator safety).
  - wenc 9y ago
    This. A lot of the other commenters have no idea what it takes to get software productionized in an industrial company. This is more of a proof-of-concept (and a very impressive one, congratulations), but it will take much more development and resources to get it working on the production line.
    - kerberos84 9y ago
      I think nobody here thinks that this piece of code from weekend will be used on Monday on the band. The hardest part is coming up with a solution and that company got it almost for free. I am pretty sure that this company will even aplly for a patent for this solution! The team should at least get a remuneration if this "proof-of-concept" implemented and will actually be used.
deleted 9y ago
[deleted]
evarest 9y ago
Thanks all for your great comments! For some reason, I cannot answer all your questions as the system blocks me from posting, probably because I did not post the original link here... Therefore, I am going to answer as much as possible in one post :) I totally understand the feelings of some about "corporate hackathons". While it is valid that companies get probably more from these hackathons than us, I want to stress the following: - We had the opportunity to work on a challenge with a huge amount of data that I would not otherwise be able to get - We were able to work with great guys I would not have met otherwise - We stretched ourselves to see if we could combine AI and AR, which would probably not have happened otherwise - The implementation of anything built during the weekend would takes months for anyone, even with the code we produced - This is a learning experience for all parties The data we used was based on input from a laser scanner. The scanner provides a height map, which would be shown as a colored PNG. We also had point clouds, but only for a few months, so not much use in AI... We used the PNG to generate a bump and normal map to show in AR. Thanks again and hope that you had fun reading it :)
nartam11 9y ago
Awesome work! I understand its a proof of concept for a hackathon but what's the value of AR here? If AI can indicate a fault with a high success rate how does AR help this situation?
- evarest 9y ago
  Thanks! Good question, the AI part is clear, it will indicate if there is a big chance of break. The AR will help the operator decide without going to the actual break whether the AI is correct. The AI only suggests, does not decide on its own. As such, an operator will be able to see the break in full detail (the top and bottom and up close) to better be able to decide whether to continue production, or do a reweld. Also, sometimes the AI will not be able to provide a good input, as new materials might be welded together and no data is available for them. In that case, again it's up to the operator to actually give the go or no-go. The AI will inform the operator after about 2 seconds, the operator then still has about 8 seconds to finalize his decision. The current prototype is not really usable, as it has some big usability issues. We might work further on improving this prototype with the organizer, depending on their interest.
Odjansse 9y ago
Can you elaborate on: "We received feedback from the AI subteam, and incorporated methods to show parts of the weld that were prone to breaking"? What techniques did they use to extract this information from the network?
- Simoncarbo 9y ago
  Hi! I'm from the AI subteam. We actually didn't have time to implement this part of the solution. Our idea was to use this library: https://github.com/raghakot/keras-vis https://github.com/raghakot/keras-vis

72 points by netb 9y ago

ambicapter 9y ago

Damn, how many years of experience do you need to be able to come up with workable solutions as fast as this?

MariuszGalus 9y ago
6 months, here's why: They random forested height data, everyone random forests these days. It's quick and dirty and yields good accuracy. Not only that, everyone seems to support a built in Random Forest training model, programs from pandas/numpy/scikit to R has it built in. Height data seems the easiest to go by, to short and it has a kink that may cause cracking in the future as the metal oxidizes. What were their 3 models? Who knows? But their idea of combining 3 models, that 3 different teams have made is standard in all DataScience competitions. This is called ensembling. ArToolkit for Unreal Engine, someone else made that and they probably just connected a bunch of blocks together with Unreal's script engine. Honestly, 6 months max and you can do this in a day yourself.
- Drdrdrq 9y ago
  Multiple years, here's why: you need to decide what will work and what will not, as you don't have time to go the wrong route. And that takes experience which it would be difficult to gain in 6 months.
- altstar 9y ago
  Yeah right: We used two convolutional neural networks, one for each side of the weld. Features extracted by these nets are then concatenated, as well as the meta data. Two fully connected layers are then applied on this vector, giving the final prediction. Everything was trained on the raw images resized to 20x150 pixels.
onebonsai 9y ago
I am no expert into AI. The AR part we learned because we are busy in VR. It is a natural evolution in that case, as we can use the workflow we are used to in VR. The AI part however is more complex, as it requires understanding of the concepts and methodologies behind it. We used Deep Learning methodologies for this project. This is an upcoming field in AI, which evolves very quickly. If you are interested in it, it's a great time to start learning the basics of it!

tshanmu 9y ago

extremely inspiring!!

option_greek 9y ago

I didn't understand where the AR was overlaid. Was it on the glasses on which the operator was wearing or on a LED monitor ?

otoburb 9y ago
FTA: "Furthermore, we decided that we would require a setup that actually worked. Given that we did not have any AR hardware available (eg AR glasses such as a Microsoft Hololens), we quickly built one ourselves using a production helmet and a webcam." Medium images sometimes take a while to load, but if you wait you can see their "webcam on a hard-hat helmet" setup for the demonstration to the judges. The overlaid AR + video feed then shows up on a monitor.
mring33621 9y ago
They did not have VR glasses, so they simulated them using a helmet-mounted webcam and a monitor. The AR weld risk profile (my words) is overlaid on the captured video images. See the article's video with caption 'Screen-capture of the AR experience...'
AndrewKemendo 9y ago
Nowhere, it wasn't actually AR. It was a video feed that had the 3D model overlayed on the desktop/laptop the video was feeding from. This isn't just semantics - moving from their setup to an actual AR HMD - even if it was just the current state of the art (Hololens, ODG, Epson etc...) wouldn't be trivial and would likely be a different workflow.

webbrahmin 9y ago

Can you please elaborate on the deep learning part. What kind of network dis you use?

Simoncarbo 9y ago
We used two convolutional neural networks, one for each side of the weld. Features extracted by these nets are then concatenated, as well as the meta data. Two fully connected layers are then applied on this vector, giving the final prediction. Everything was trained on the raw images resized to 20x150 pixels.

peterbonney 9y ago

This is a really cool piece of engineering, but I'm bothered by the idea that the going rate for a huge multinational corporation to get a prototype of a potentially extremely valuable technology is less than 10k of prize money [1], plus a "mini-drone race" and "snacks and food". Maybe it's just me, but I find this type of corporate-sponsored hackathon exploitative and distastefull. We're not talking about an event that is producing a social good in an area that is otherwise underinvested. This is purely profit-motivated, and ArcelorMittal can afford to pay a lot more for a prototype like this. 1. https://www.eventbrite.nl/e/the-arcelormittal-challenge-hack4steel-tickets-34228761146 https://www.eventbrite.nl/e/the-arcelormittal-challenge-hack...

ak39 9y ago
I'm with you on this. I can't imagine any other complex discipline where experts offer their time & creativity to the lowest bidder so quickly. Programming seems uniquely generous in this regard. Or stupid.
onebonsai 9y ago
Thanks for your view. However, it is important to see this event also for what it is. It is a way for Arcelor to think out-of-the-box, and get input from people that are not directly linked to their business. Secondly, it is a way for students to get a glimps of what the corporate projects and challenges are. Additionally, for us, as professionals, it is a way to show, although in a very rough fashion, what is possible, and where we can help. We saw this event more as a team building than work. The result we got at the hackathon is a very rough prototype, not a finished product. In no way, this can directly be implemented on the floor. As such, input from experts is still required, either us, or other experts in the field. Therefore, these hackathons are useful, in my opinion, not only to the company, but also the participants :)
- csmattryder 9y ago
  > It is a way for Arcelor to think out-of-the-box, and get input from people that are not directly linked to their business. And there's nothing wrong with that, so much so, that we created a title for these people: Consultant. They're highly paid, and available to bring years of domain experience to a business. How much value did ArcellorMittal get in exchange for 8.5k and some pizza? How much would <leading AR firm> charge, in comparison? (Man, programmers desperately need someone to tell them that no, not everyone just "gets" programming. You have a skill, charge for it!)

onebonsai 9y ago

Hey! Thanks guys for your comments! I'm the writer of the article (Evarest). Just a bit of an update in the mean while :) Our AI team is now busy writing a bit more detail on the actual AI part of the project. The project used: Random Forested Height Data Deep Learning Models We trained on the entire dataset of 60.000+ elements, but did pre-processing on the data. I am not placed to answer detailed questions on the AI part, but we will update the article soon with more details. Note that this is not a perfect result, nor that we claim to be all-knowing or have all the answers. This is a result from a hackathon, which involved working in suboptimal conditions under heavy time constraints. As for the AR part, we did not have access to AR helmets. We are aware that not all the data that we showed would work on an AR helmet (currently). However, this tech is totally feasible in the short to medium term. Also, it is completely feasible on tablet or similar. The concept of the hackathon was to show the potential of these technologies, and to integrate the AR and AI, which I find we succeeded in.

corford 9y ago
Great hack. Really enjoyed reading this! Noob question: the weld as shown in the AR video looks really detailed (fissures, deformations etc). Was that rendered by you using the heightmap data alone or did you have access to top and bottom images of the welds as part of the training data?
- loa_in_ 9y ago
  I think it was data from the laser scanner as it was monochromatic (w/ shading)
  - corford 9y ago
    Thanks!

FLUX-YOU 9y ago

I just got into Unreal Engine with the VR template project, which includes grabbing/teleportation functionality by default. It's been really easy to get into and if you have a headset, I'd really recommend trying it out if you've had a VR development itch.

onebonsai 9y ago
Indeed, thanks for the comment! Unreal Engine is a great engine to quickly build prototypes in. We are used to work with the included VR Template of Unreal, and it most of the times works great. Also for more serious projects, Unreal did not let us down.

onebonsai 9y ago

UPDATE: Thank you for your comments and remarks ! Based on the inputs of the community, we rewrote the AI part of the article (see section 'Building the AI Assistant'). More details are available now. --> https://lab.onebonsai.com/arcelormittals-hack4steel-hackaton-43d9a0338a4 https://lab.onebonsai.com/arcelormittals-hack4steel-hackaton...

specialist 9y ago
I suggest changing "conjest" to "condense".

kerberos84 9y ago

How much money will Arcelor earn from this piece of Software? You just implemented it for free!

rmorm 9y ago
Hey ! I am one of the AI guy of the team. I agree that if the solution gets implemented on the lines, they will probably make more net money out of it than what was spent for the challenge (which is more than 8.5k€ and some pizzas as said in other comments BTW, the challenge was probably more expensive to organize). However, let's be realistic, they will have to put a lot more money into this before they get a viable product to insert in their production lines. The solution we have built is way too simple regarding actual operational constraints (e.g.: operator safety).
- wenc 9y ago
  This. A lot of the other commenters have no idea what it takes to get software productionized in an industrial company. This is more of a proof-of-concept (and a very impressive one, congratulations), but it will take much more development and resources to get it working on the production line.
  - kerberos84 9y ago
    I think nobody here thinks that this piece of code from weekend will be used on Monday on the band. The hardest part is coming up with a solution and that company got it almost for free. I am pretty sure that this company will even aplly for a patent for this solution! The team should at least get a remuneration if this "proof-of-concept" implemented and will actually be used.

deleted 9y ago

[deleted]

evarest 9y ago

Thanks all for your great comments! For some reason, I cannot answer all your questions as the system blocks me from posting, probably because I did not post the original link here... Therefore, I am going to answer as much as possible in one post :) I totally understand the feelings of some about "corporate hackathons". While it is valid that companies get probably more from these hackathons than us, I want to stress the following: - We had the opportunity to work on a challenge with a huge amount of data that I would not otherwise be able to get - We were able to work with great guys I would not have met otherwise - We stretched ourselves to see if we could combine AI and AR, which would probably not have happened otherwise - The implementation of anything built during the weekend would takes months for anyone, even with the code we produced - This is a learning experience for all parties The data we used was based on input from a laser scanner. The scanner provides a height map, which would be shown as a colored PNG. We also had point clouds, but only for a few months, so not much use in AI... We used the PNG to generate a bump and normal map to show in AR. Thanks again and hope that you had fun reading it :)

nartam11 9y ago

Awesome work! I understand its a proof of concept for a hackathon but what's the value of AR here? If AI can indicate a fault with a high success rate how does AR help this situation?

evarest 9y ago
Thanks! Good question, the AI part is clear, it will indicate if there is a big chance of break. The AR will help the operator decide without going to the actual break whether the AI is correct. The AI only suggests, does not decide on its own. As such, an operator will be able to see the break in full detail (the top and bottom and up close) to better be able to decide whether to continue production, or do a reweld. Also, sometimes the AI will not be able to provide a good input, as new materials might be welded together and no data is available for them. In that case, again it's up to the operator to actually give the go or no-go. The AI will inform the operator after about 2 seconds, the operator then still has about 8 seconds to finalize his decision. The current prototype is not really usable, as it has some big usability issues. We might work further on improving this prototype with the organizer, depending on their interest.

Odjansse 9y ago

Can you elaborate on: "We received feedback from the AI subteam, and incorporated methods to show parts of the weld that were prone to breaking"? What techniques did they use to extract this information from the network?

Simoncarbo 9y ago
Hi! I'm from the AI subteam. We actually didn't have time to implement this part of the solution. Our idea was to use this library: https://github.com/raghakot/keras-vis https://github.com/raghakot/keras-vis