4 ms·

Ungooglable problems still exist. Yesterday, for example, 30 Fanuc i industrial controllers started sending back RST tcp packets and refused to initiate new tcp

Ungooglable problems still exist. Yesterday, for example, 30 Fanuc i industrial controllers started sending back RST tcp packets and refused to initiate new tcp connections until restarted. This behaviour occurred in a system which was stable for 2 years. Very likely an issue on one controller spread to the others by causing an error in a set of dlls from Japan. This occurred due to another set of the same DLLs connecting to 1 controller at the same time. Maybe. I have to talk to the manufacturer to get any further. Or disassemble their libraries. I have these sorts of ungoogleable problems all the time. I doubt industrial automation is the only field like this.

tonyarkles 9y ago
Being able to solve problems like that one result in you looking like a wizard btw. In case you haven't seen them, the SysInternals tools can serve as an interesting middle ground before popping the DLLs into an assembler. There's one (whose name escapes me) that is basically strace-for-Windows, and I've used it to successfully solve some pretty gnarly Win32 problems before.
- tjalfi 9y ago
  Are you thinking of Process Monitor?
  - tonyarkles 9y ago
    Just looked it up, and yes. Process Monitor and Process Explorer, depending on what kind of problem it is.
  - taneq 9y ago
    With a bit of web searching, that's what I found too.
jacquesm 9y ago
That sounds like an instance of the Thundering Herd problem.
- mercer 9y ago
  Thank's for making me look into that :). I'm half expecting a week or so of HN posts diving into this topic...
  - jacquesm 9y ago
    It can be pretty tricky to defend against this since it is mostly a client side issue. The best trick that I've found is to randomize the initial delay + exponential back-off for subsequent retries.
taneq 9y ago
Industrial automation is my field also, and yeah - you get that interesting mix of low-volume hardware, software written by electrical engineers, and messy real-world situations. You think you know what's going to happen but you're never quite sure... And when something does go wrong, it'll be a doozy.
- tonyarkles 9y ago
  > software written by electrical engineers I mentioned elsewhere in this thread that I did an EE/CS dual degree after having programmed for a long time beforehand. I really don't understand why, but some of the EE-only software I've worked with has been absolutely atrocious. One of my favourite moments in undergrad was in my Digital Communications class. I don't remember exactly what we were doing, but it involved Matlab and some kind of signal decoding. I handed in my assignment and got called in to have a one-on-one with the professor; he was certain that I had cheated and just hard-coded the answers to my assignment instead of actually writing the decoder. Why? Because my code ran instantaneously instead of sitting and grinding for 30+ seconds. What was my secret? I'd seen a couple of triply-nested for loops in the code and decided they could all be accomplished with a matrix multiply instead of doing all of that computation by hand. In the same class, a friend (also an EE/CS dual) and I came up with the dynamic programming solution to the Viterbi decoding problem as well, and again, got taken to task for having a solution that ran too fast. Sigh...
snerbles 9y ago
> Or disassemble their libraries. Be warned that FANUC is very litigious about that sort of thing. I can't go into specifics, but they've sued companies out of existence for less.

tonyarkles 9y ago

Being able to solve problems like that one result in you looking like a wizard btw. In case you haven't seen them, the SysInternals tools can serve as an interesting middle ground before popping the DLLs into an assembler. There's one (whose name escapes me) that is basically strace-for-Windows, and I've used it to successfully solve some pretty gnarly Win32 problems before.

tjalfi 9y ago
Are you thinking of Process Monitor?
- tonyarkles 9y ago
  Just looked it up, and yes. Process Monitor and Process Explorer, depending on what kind of problem it is.
- taneq 9y ago
  With a bit of web searching, that's what I found too.

jacquesm 9y ago

That sounds like an instance of the Thundering Herd problem.

mercer 9y ago
Thank's for making me look into that :). I'm half expecting a week or so of HN posts diving into this topic...
- jacquesm 9y ago
  It can be pretty tricky to defend against this since it is mostly a client side issue. The best trick that I've found is to randomize the initial delay + exponential back-off for subsequent retries.

taneq 9y ago

Industrial automation is my field also, and yeah - you get that interesting mix of low-volume hardware, software written by electrical engineers, and messy real-world situations. You think you know what's going to happen but you're never quite sure... And when something does go wrong, it'll be a doozy.

tonyarkles 9y ago
> software written by electrical engineers I mentioned elsewhere in this thread that I did an EE/CS dual degree after having programmed for a long time beforehand. I really don't understand why, but some of the EE-only software I've worked with has been absolutely atrocious. One of my favourite moments in undergrad was in my Digital Communications class. I don't remember exactly what we were doing, but it involved Matlab and some kind of signal decoding. I handed in my assignment and got called in to have a one-on-one with the professor; he was certain that I had cheated and just hard-coded the answers to my assignment instead of actually writing the decoder. Why? Because my code ran instantaneously instead of sitting and grinding for 30+ seconds. What was my secret? I'd seen a couple of triply-nested for loops in the code and decided they could all be accomplished with a matrix multiply instead of doing all of that computation by hand. In the same class, a friend (also an EE/CS dual) and I came up with the dynamic programming solution to the Viterbi decoding problem as well, and again, got taken to task for having a solution that ran too fast. Sigh...

snerbles 9y ago

> Or disassemble their libraries. Be warned that FANUC is very litigious about that sort of thing. I can't go into specifics, but they've sued companies out of existence for less.