21 ms·

We are starting to operate our CNC machines remotely

127 points by Sharapolas 6y ago

sneak 6y ago
> First, was the network setup. We needed to merge multi-site networks into a mesh topology network where all the nodes could be interconnected which was solved using a hub and spoke topology VPN. Although, it does have its own drawbacks it had a good cost-benefit ratio when it comes to setup and deployment and is working for us for now. How is "mesh" solved with "hub and spoke"? If your hub goes offline, everything stops; the whole point of a mesh is to avoid the SPOF. > Next, was building the dashboard which not only required the making of the UI but required figuring out how we’re going to expose and manage resources in our shop network. We decided to build the backend in the spirit of Kubernetes, so it is easy to manage at scale with the addition of new sensors, machines and full-blown sites themselves. What does that mean? Docker containers?
- d00bianista 6y ago
  IPIP/IPsec and OSPF would do nicely for efficient site-to-site meshing. But, adding docker to the equation makes it much more complicated. I'd prefer just operating the machines over IP and get rid of all remote desktop related latency all together.
- Sharapolas 6y ago
  Indeed, as you noticed the networking setup is primitive, but for the time being it works for us. Currently, the hub is chosen to be as reliable as possible and nodes at the end of the spokes are prepared for hub down (ie. they stop safely ). Kubernetes is more than just Docker containers. Having managed small computing clusters for a decade using custom scripts I find that the value of K8s is how easy it is to manage highly complex applications over an array of loosely connected computing nodes. So in my view k8s includes networking, orchestration and principles of how everything should be done and managed to achieve redundancy, performance and stability. Containers are just a convenience for simple deployment on asymmetric nodes and is only part of K8s.
  - sneak 6y ago
    If you're actually running kubernetes and you say you're running stuff "in the spirit of kubernetes" it gives the strong impression you are not actually running kubernetes. I also didn't say your networking was primitive, just that "hub and spoke" and "mesh" are two entirely different things that you seemed to have conflated entirely.
    - Sharapolas 6y ago
      And that is correct. We are indeed not running kubernetes because it is not supported on our hardware and some assumptions were limiting. So we have a new system, but having kubernetes as our ideal. Hub and spoke is the underlying architecture of the net, but we are achieving mesh like functionality having hub as our central router. The wording might be a bit confusing.
megraf 6y ago
I moonlight at a machine shop. I'm not a machinist, and I'm hardly a machine operator. Machine shops need bodies to do a few things (some of which OP mentioned) loading materials, removing materials, changing broken cutting bits, cleaning the machines (especially if you're cutting something like magnesium). Machine shops also need bodies to build jigs to hold pieces to the machine table while the machines remove material (think 5 axis pieces, especially). Despite all of this – there's an amazing amount of automation at play. Tool changes, coolant, temperature monitoring, the list goes on. I'm currently building some automation hardware and software so we can start to fill the gap of 'dead' time where machinists are pressing buttons, loading in blanks, and are bored out of their minds. It's such an interesting space.
- Quequau 6y ago
  Moonlighting at a machine shop is something that I am very interested in doing. How did you wind up with such a position?
  - paddy_m 6y ago
    I wrote an email to friends years ago about a side job I got at a machine shop while working in tech, it was quite well received. email in my profile, contact me and I'll send it to you.
  - Ccecil 6y ago
    Depends on where you live. I recommend just going around and meeting machinists. Look into local meetings...Society of Manufacturing and such. Typically there are things like shop tours put on monthly (non-pandemic times). Talk to local machine shop instructors and/or metal suppliers and see who might be interested in hiring...they usually know everyone. My area has many very, very large machine shops but it has a larger number of small "mom and pop" shops which just run a few machines. I have found that the people who run either type are equally as approachable...it is hit and miss if they are willing to even let you in the shop at all without NDAs and such...then there are others who have no issues with it. Most of the really difficult to learn stuff in the field is fixturing and CAM setup. Once that is layed out it is mostly just changing parts and watching the machine run...but there is a lot to be learned while that is going on.
soheil 6y ago
Pretty epic. But doesn't it need constant lubrication for the head not to get too warm?
- a2tech 6y ago
  Typically only when cutting metal. And harder metal especially. Wood and the like will often have an spray of air directed at the cutting bit but thats simply to help clear away chips and clean the cutting surface. Helps keep a nice finish and good visibility.
  - megraf 6y ago
    Actually, it's a little different. Using coolant is the norm even when doing mild steels. You can remove material quicker, your part remains cooler, and your tools remain cleaner. There's even cuttings bits that have integral cooling (holes in the cutting surface).
    - a2tech 6y ago
      Right--I support a fair number of machine shops and they're all moving to CNC (although they're using a lot manual machines still). 5 axis is nice and there's a surprising amount of manual tooling changes. We had a problem where one of the integral cooling loops had a tube come off and it sprayed the better part of a 50 gallon drum of coolant all over the shop before someone got over and hit the e-stop.
      - duckfang 6y ago
        That sounds correct. Unless you have a sensor, there's no way to make a meaningful detection and decision to stop something. And sensors that are reliable and durable cost $$$$.
    - a1369209993 6y ago
      Pretty sure even the mildest steel is still a hard metal. Soft metal would be something like copper or aluminum IIRC. (Or lead or gold, moreso, but you probably wouldn't want to be machining those for obvious reasons.)
joshdick 6y ago
Now that's it possible for the machines to be operated remotely, how long till those operator jobs are offshored?
- megraf 6y ago
  you should read the article my friend :)
- Sharapolas 6y ago
  I wouldn't jump to that conclusion so fast. Having hired subcontractors from offshore I can tell that there is always a background difference that decrease the value. Covid has shown that a lot of office work can be done remotely, but that does not mean that all the businesses suddenly will go away offshore. It just becomes a perk of the job. And, thus, we're close to having machine operators working from their beds :)
samdung 6y ago
Hand me the wrench ... oh wait.
spyder 6y ago
It's called lights-out manufacturing: "Factories that employ "lights-out manufacturing" are fully automated and require no human presence on-site." https://en.wikipedia.org/wiki/Lights_out_(manufacturing) https://en.wikipedia.org/wiki/Lights_out_(manufacturing)
- u678u 6y ago
  For some reason this scares me at a core level more than anything else. Seems inevitable that Skynet will control the factory sooner or later.
  - jtsiskin 6y ago
    Imagine a paper clip factory, for example: https://en.wikipedia.org/wiki/Instrumental_convergence#Paperclip_maximizer https://en.wikipedia.org/wiki/Instrumental_convergence#Paper...
    - bsder 6y ago
      Wrong link: https://www.decisionproblem.com/paperclips/ https://www.decisionproblem.com/paperclips/
  - Sharapolas 6y ago
    With IIOT (industry internet of things) that definitely seems like a possible reality. However, as long as Skynet is controlled by ex-factory workers it will be a promotion for everyone.
  - spyder 6y ago
    Yep, one of the example in the Wiki is robots building robots since 2001: "FANUC, a Japanese robotics company, has been operating as a lights-out factory since 2001.[6] Robots are building other robots at a rate of about 50 per 24-hour shift and can run unsupervised for as long as 30 days at a time."
- riskneutral 6y ago
  If I'm not mistaken, Apple's machine shops are almost fully automated. They bought an enormous number of CNC machines to make MacBooks, etc.
yrgulation 6y ago
My master’s degree project was cncs designed to operate like servers in a service oriented architecture. Each machine exposed an api (robot arm for feeding, cnc for machining), jobs were routed using a queue server like rabbit mq (when a machine becomes available it just consumes events). I built the cncs myself using raspberry pis. Basically, i built a blue print of a fully automated factory.
OldHand2018 6y ago
> Those skills need to either be brought in or be taught and as soon as a person has those skills we are back at having a skilled person doing a large amount of unskilled work. I’m completely in awe of this attitude. Are you located in the USA? I know a bunch of people in small part manufacturing (mostly metal and plastic) that are turning away work on a daily basis. There is so much demand in the USA right now that any worker with the skill to increase production rates is like gold.
- musingsole 6y ago
  I think you misunderstand the authors point. There's unskilled aspects of a job that seem like an allocation waste to have skilled labor handle. But to properly account for some costly errors and such, an unskilled worker has to be trained...making them a skilled worker that should be able to better allocate their time elsewhere than the unskilled tasks.
  - OldHand2018 6y ago
    Based on the people I know, paying someone skilled wages to do an unskilled job is more than worth it right now. One of the shops I know of is making small little pieces of aluminum in the central US and shipping them to Foxconn in China for assembly into computers. As fast as they can make them, which is constrained by the number of people they can get on the floor. 2020 was their best year ever, by a wide margin. Things are seriously crazy right now. If you’re wondering why, the answer is most likely tariffs. The more American value in the product, the lower the tariff once you import the computer. Nobody knows if Biden will continue this.
    - Sharapolas 6y ago
      From what you've said semi-automation is even more crucial as the more checks and automations you have the lower the bar of entry for the people on the shop. What a company pays is up to them, but having more options is always for the better.
paddy_m 6y ago
If you're interested in machining, check out the Business of Machining podcast. https://businessofmachining.libsyn.com/ https://businessofmachining.libsyn.com/ Every week John Saunders of NYCCNC talks about his business with John Grimsmo of Grimsmo knives (they make $1000 pocket knives on cnc machines). It is an excellent podcast about machining and business.
- iak8god 6y ago
  Would you have a favorite episode or two to recommend? When someone recommends a series that sounds interesting and I see there are 200+ episodes, it's hard to know where to start!
mchusma 6y ago
This is cool stuff. I don't know the state of the art here, but it seems like making progress on remote factory operations would dramatically speed up our ability to industrialize space.
xupybd 6y ago
I work with large scale flat pack furniture manufacturing. Unattended machining is possible and not uncommon. You have a large tool changer on the CNC, an auto loader to get material on the CNC, and then a stock warehousing robot to fetch the material. You just talk to one of the large equipment manufacturers and they'll sell you this sort of setup. This seems like a strange problem to try and solve. As a former software developer I see problems every day that AI or advanced expert systems could solve I'd love to chat with these guys. If they solved some of the problems with QC or Packing the industry would be clawing at their doors to buy their products. Not to mention the outdated state of most CAM / nesting packages.
- davidtsong 6y ago
  Hey! Would love to chat more about this if you have time
- Sharapolas 6y ago
  Being software developers by nature and having got ourselves into CNC we feel exactly the same. It is crazy to us how easy it is to make a dumb and costly mistake. So we are tackling it one little thing at a time to make operators life slightly less stressful. We're always open for a discussion so feel free to reach out ( the contacts are on the site of the blog )
fudged71 6y ago
This is terrifying and impressive :) We built a similar system for 3D printers. There's nothing quite like intentionally giving access to someone on the other side of the globe and having them make something for you in front of your hands and eyes. It's not technically teleporting, but close enough to have that same magical feeling.
- Sharapolas 6y ago
  Very well said. It is also one of those win-win situations where the business can be more efficient and people have wider opportunities.
hahahahe 6y ago
I am convinced being a skilled machinist with proper engineering background will be one of the hottest career tracks in the next couple of decades. Seems demand for small batch production is only going to increase.

127 points by Sharapolas 6y ago

> First, was the network setup. We needed to merge multi-site networks into a mesh topology network where all the nodes could be interconnected which was solved using a hub and spoke topology VPN. Although, it does have its own drawbacks it had a good cost-benefit ratio when it comes to setup and deployment and is working for us for now. How is "mesh" solved with "hub and spoke"? If your hub goes offline, everything stops; the whole point of a mesh is to avoid the SPOF. > Next, was building the dashboard which not only required the making of the UI but required figuring out how we’re going to expose and manage resources in our shop network. We decided to build the backend in the spirit of Kubernetes, so it is easy to manage at scale with the addition of new sensors, machines and full-blown sites themselves. What does that mean? Docker containers?

d00bianista 6y ago
IPIP/IPsec and OSPF would do nicely for efficient site-to-site meshing. But, adding docker to the equation makes it much more complicated. I'd prefer just operating the machines over IP and get rid of all remote desktop related latency all together.
Sharapolas 6y ago
Indeed, as you noticed the networking setup is primitive, but for the time being it works for us. Currently, the hub is chosen to be as reliable as possible and nodes at the end of the spokes are prepared for hub down (ie. they stop safely ). Kubernetes is more than just Docker containers. Having managed small computing clusters for a decade using custom scripts I find that the value of K8s is how easy it is to manage highly complex applications over an array of loosely connected computing nodes. So in my view k8s includes networking, orchestration and principles of how everything should be done and managed to achieve redundancy, performance and stability. Containers are just a convenience for simple deployment on asymmetric nodes and is only part of K8s.
- sneak 6y ago
  If you're actually running kubernetes and you say you're running stuff "in the spirit of kubernetes" it gives the strong impression you are not actually running kubernetes. I also didn't say your networking was primitive, just that "hub and spoke" and "mesh" are two entirely different things that you seemed to have conflated entirely.
  - Sharapolas 6y ago
    And that is correct. We are indeed not running kubernetes because it is not supported on our hardware and some assumptions were limiting. So we have a new system, but having kubernetes as our ideal. Hub and spoke is the underlying architecture of the net, but we are achieving mesh like functionality having hub as our central router. The wording might be a bit confusing.

megraf 6y ago

I moonlight at a machine shop. I'm not a machinist, and I'm hardly a machine operator. Machine shops need bodies to do a few things (some of which OP mentioned) loading materials, removing materials, changing broken cutting bits, cleaning the machines (especially if you're cutting something like magnesium). Machine shops also need bodies to build jigs to hold pieces to the machine table while the machines remove material (think 5 axis pieces, especially). Despite all of this – there's an amazing amount of automation at play. Tool changes, coolant, temperature monitoring, the list goes on. I'm currently building some automation hardware and software so we can start to fill the gap of 'dead' time where machinists are pressing buttons, loading in blanks, and are bored out of their minds. It's such an interesting space.

Quequau 6y ago
Moonlighting at a machine shop is something that I am very interested in doing. How did you wind up with such a position?
- paddy_m 6y ago
  I wrote an email to friends years ago about a side job I got at a machine shop while working in tech, it was quite well received. email in my profile, contact me and I'll send it to you.
- Ccecil 6y ago
  Depends on where you live. I recommend just going around and meeting machinists. Look into local meetings...Society of Manufacturing and such. Typically there are things like shop tours put on monthly (non-pandemic times). Talk to local machine shop instructors and/or metal suppliers and see who might be interested in hiring...they usually know everyone. My area has many very, very large machine shops but it has a larger number of small "mom and pop" shops which just run a few machines. I have found that the people who run either type are equally as approachable...it is hit and miss if they are willing to even let you in the shop at all without NDAs and such...then there are others who have no issues with it. Most of the really difficult to learn stuff in the field is fixturing and CAM setup. Once that is layed out it is mostly just changing parts and watching the machine run...but there is a lot to be learned while that is going on.

soheil 6y ago

Pretty epic. But doesn't it need constant lubrication for the head not to get too warm?

a2tech 6y ago
Typically only when cutting metal. And harder metal especially. Wood and the like will often have an spray of air directed at the cutting bit but thats simply to help clear away chips and clean the cutting surface. Helps keep a nice finish and good visibility.
- megraf 6y ago
  Actually, it's a little different. Using coolant is the norm even when doing mild steels. You can remove material quicker, your part remains cooler, and your tools remain cleaner. There's even cuttings bits that have integral cooling (holes in the cutting surface).
  - a2tech 6y ago
    Right--I support a fair number of machine shops and they're all moving to CNC (although they're using a lot manual machines still). 5 axis is nice and there's a surprising amount of manual tooling changes. We had a problem where one of the integral cooling loops had a tube come off and it sprayed the better part of a 50 gallon drum of coolant all over the shop before someone got over and hit the e-stop.
    - duckfang 6y ago
      That sounds correct. Unless you have a sensor, there's no way to make a meaningful detection and decision to stop something. And sensors that are reliable and durable cost $$$$.
  - a1369209993 6y ago
    Pretty sure even the mildest steel is still a hard metal. Soft metal would be something like copper or aluminum IIRC. (Or lead or gold, moreso, but you probably wouldn't want to be machining those for obvious reasons.)

joshdick 6y ago

Now that's it possible for the machines to be operated remotely, how long till those operator jobs are offshored?

megraf 6y ago
you should read the article my friend :)
Sharapolas 6y ago
I wouldn't jump to that conclusion so fast. Having hired subcontractors from offshore I can tell that there is always a background difference that decrease the value. Covid has shown that a lot of office work can be done remotely, but that does not mean that all the businesses suddenly will go away offshore. It just becomes a perk of the job. And, thus, we're close to having machine operators working from their beds :)

samdung 6y ago

Hand me the wrench ... oh wait.

spyder 6y ago

It's called lights-out manufacturing: "Factories that employ "lights-out manufacturing" are fully automated and require no human presence on-site." https://en.wikipedia.org/wiki/Lights_out_(manufacturing) https://en.wikipedia.org/wiki/Lights_out_(manufacturing)

u678u 6y ago
For some reason this scares me at a core level more than anything else. Seems inevitable that Skynet will control the factory sooner or later.
- jtsiskin 6y ago
  Imagine a paper clip factory, for example: https://en.wikipedia.org/wiki/Instrumental_convergence#Paperclip_maximizer https://en.wikipedia.org/wiki/Instrumental_convergence#Paper...
  - bsder 6y ago
    Wrong link: https://www.decisionproblem.com/paperclips/ https://www.decisionproblem.com/paperclips/
- Sharapolas 6y ago
  With IIOT (industry internet of things) that definitely seems like a possible reality. However, as long as Skynet is controlled by ex-factory workers it will be a promotion for everyone.
- spyder 6y ago
  Yep, one of the example in the Wiki is robots building robots since 2001: "FANUC, a Japanese robotics company, has been operating as a lights-out factory since 2001.[6] Robots are building other robots at a rate of about 50 per 24-hour shift and can run unsupervised for as long as 30 days at a time."
riskneutral 6y ago
If I'm not mistaken, Apple's machine shops are almost fully automated. They bought an enormous number of CNC machines to make MacBooks, etc.

yrgulation 6y ago

My master’s degree project was cncs designed to operate like servers in a service oriented architecture. Each machine exposed an api (robot arm for feeding, cnc for machining), jobs were routed using a queue server like rabbit mq (when a machine becomes available it just consumes events). I built the cncs myself using raspberry pis. Basically, i built a blue print of a fully automated factory.

OldHand2018 6y ago

> Those skills need to either be brought in or be taught and as soon as a person has those skills we are back at having a skilled person doing a large amount of unskilled work. I’m completely in awe of this attitude. Are you located in the USA? I know a bunch of people in small part manufacturing (mostly metal and plastic) that are turning away work on a daily basis. There is so much demand in the USA right now that any worker with the skill to increase production rates is like gold.

musingsole 6y ago
I think you misunderstand the authors point. There's unskilled aspects of a job that seem like an allocation waste to have skilled labor handle. But to properly account for some costly errors and such, an unskilled worker has to be trained...making them a skilled worker that should be able to better allocate their time elsewhere than the unskilled tasks.
- OldHand2018 6y ago
  Based on the people I know, paying someone skilled wages to do an unskilled job is more than worth it right now. One of the shops I know of is making small little pieces of aluminum in the central US and shipping them to Foxconn in China for assembly into computers. As fast as they can make them, which is constrained by the number of people they can get on the floor. 2020 was their best year ever, by a wide margin. Things are seriously crazy right now. If you’re wondering why, the answer is most likely tariffs. The more American value in the product, the lower the tariff once you import the computer. Nobody knows if Biden will continue this.
  - Sharapolas 6y ago
    From what you've said semi-automation is even more crucial as the more checks and automations you have the lower the bar of entry for the people on the shop. What a company pays is up to them, but having more options is always for the better.

paddy_m 6y ago

If you're interested in machining, check out the Business of Machining podcast. https://businessofmachining.libsyn.com/ https://businessofmachining.libsyn.com/ Every week John Saunders of NYCCNC talks about his business with John Grimsmo of Grimsmo knives (they make $1000 pocket knives on cnc machines). It is an excellent podcast about machining and business.

iak8god 6y ago
Would you have a favorite episode or two to recommend? When someone recommends a series that sounds interesting and I see there are 200+ episodes, it's hard to know where to start!

mchusma 6y ago

This is cool stuff. I don't know the state of the art here, but it seems like making progress on remote factory operations would dramatically speed up our ability to industrialize space.

xupybd 6y ago

I work with large scale flat pack furniture manufacturing. Unattended machining is possible and not uncommon. You have a large tool changer on the CNC, an auto loader to get material on the CNC, and then a stock warehousing robot to fetch the material. You just talk to one of the large equipment manufacturers and they'll sell you this sort of setup. This seems like a strange problem to try and solve. As a former software developer I see problems every day that AI or advanced expert systems could solve I'd love to chat with these guys. If they solved some of the problems with QC or Packing the industry would be clawing at their doors to buy their products. Not to mention the outdated state of most CAM / nesting packages.

davidtsong 6y ago
Hey! Would love to chat more about this if you have time
Sharapolas 6y ago
Being software developers by nature and having got ourselves into CNC we feel exactly the same. It is crazy to us how easy it is to make a dumb and costly mistake. So we are tackling it one little thing at a time to make operators life slightly less stressful. We're always open for a discussion so feel free to reach out ( the contacts are on the site of the blog )

fudged71 6y ago

This is terrifying and impressive :) We built a similar system for 3D printers. There's nothing quite like intentionally giving access to someone on the other side of the globe and having them make something for you in front of your hands and eyes. It's not technically teleporting, but close enough to have that same magical feeling.

Sharapolas 6y ago
Very well said. It is also one of those win-win situations where the business can be more efficient and people have wider opportunities.

hahahahe 6y ago

I am convinced being a skilled machinist with proper engineering background will be one of the hottest career tracks in the next couple of decades. Seems demand for small batch production is only going to increase.