5 ms·

Don't fix a production issue by 'hacking' in the solution live on production. Figure out what caused it and fix that. At least, that's what we do for our softwa

Don't fix a production issue by 'hacking' in the solution live on production. Figure out what caused it and fix that. At least, that's what we do for our software.

rurban 11y ago
Having worked as software dev for formula 1 and as hardware engineer ditto, I can verify this. You see all. You see people hacking a bugfix in one day on the F1 site, but this fix will not survive. Some shops restore from their backups nightly, you will certainly face a lot of meetings and reports to get your one-line fix back into production. Even on R&D problems, which mostly describes F1. This case reported here was on the assembly line, there are different rules. If a strange misproduction happens you certainly want to be able to evaluate the riscs properly. The analogy is a Heisenbug in the production compiler. How often does it happen, how expensive is the problem, how expensive is the fix. There's not one car on the risc, there's the whole line in question. The new robot? The SW or the HW? Wrong measurement, wrong planning, broken sensor, broken motor? Or a real Heisenbug, which appears in HW much more often than in SW. Magneto/electric cabling problem. Or sometimes even light. The cable guy bending an overlong fibre cable <30cm radius. A fibre with a bad cut at one end causing all sorts of crazy effects, and crazy hackish fixes. Missing CAN terminators (analogy to missing SCSI terminators on harddrive chains) in certain F1 cars. It might work, but I wouldn't want to sit in this thing. This new 140.000 rpm high-speed rotor behind my back, making the driver uncomfortable, who is starting making bad mistakes out of fear, and rather pulls this thing out. Relations to completely crazy effects from outside, such as a cleaning lady turning on a bad fan, causing a jitter or spike, having some effects elsewhere. A bug, a real bug, in the system, esp. in Asia or Australia. I see not much differences from SW and HW engineering. But the biggest difference I know about is the methology. Agile in car manifacturing or HW engineering at all only happens at Toyota, nowhere else. Properly planned waterfall, with excellent and high powered middle management is the norm. If in Asia, Europe or the US. There do exist some very small HW shops, with 1-4 engineers and 2 managers/sales people, and 2 support. There the development is of course engineering driven. Or there are some exceptions, such as Toyota or VW with crazy assembly optimizations, or e.g. Honda where their boss is one of the best engineers (they are lucky). But they eventually they ran out of luck with their funding and had to sell their very best test site, which was then taken over Daimler. That's why their are leading the F1 currently. Another difference is proper training. These good engineering companies train their people a lot. Maybe 10x more than in normal SW engineering.

Don't fix a production issue by 'hacking' in the solution live on production. Figure out what caused it and fix that. At least, that's what we do for our software.

rurban 11y ago

Having worked as software dev for formula 1 and as hardware engineer ditto, I can verify this. You see all. You see people hacking a bugfix in one day on the F1 site, but this fix will not survive. Some shops restore from their backups nightly, you will certainly face a lot of meetings and reports to get your one-line fix back into production. Even on R&D problems, which mostly describes F1. This case reported here was on the assembly line, there are different rules. If a strange misproduction happens you certainly want to be able to evaluate the riscs properly. The analogy is a Heisenbug in the production compiler. How often does it happen, how expensive is the problem, how expensive is the fix. There's not one car on the risc, there's the whole line in question. The new robot? The SW or the HW? Wrong measurement, wrong planning, broken sensor, broken motor? Or a real Heisenbug, which appears in HW much more often than in SW. Magneto/electric cabling problem. Or sometimes even light. The cable guy bending an overlong fibre cable <30cm radius. A fibre with a bad cut at one end causing all sorts of crazy effects, and crazy hackish fixes. Missing CAN terminators (analogy to missing SCSI terminators on harddrive chains) in certain F1 cars. It might work, but I wouldn't want to sit in this thing. This new 140.000 rpm high-speed rotor behind my back, making the driver uncomfortable, who is starting making bad mistakes out of fear, and rather pulls this thing out. Relations to completely crazy effects from outside, such as a cleaning lady turning on a bad fan, causing a jitter or spike, having some effects elsewhere. A bug, a real bug, in the system, esp. in Asia or Australia. I see not much differences from SW and HW engineering. But the biggest difference I know about is the methology. Agile in car manifacturing or HW engineering at all only happens at Toyota, nowhere else. Properly planned waterfall, with excellent and high powered middle management is the norm. If in Asia, Europe or the US. There do exist some very small HW shops, with 1-4 engineers and 2 managers/sales people, and 2 support. There the development is of course engineering driven. Or there are some exceptions, such as Toyota or VW with crazy assembly optimizations, or e.g. Honda where their boss is one of the best engineers (they are lucky). But they eventually they ran out of luck with their funding and had to sell their very best test site, which was then taken over Daimler. That's why their are leading the F1 currently. Another difference is proper training. These good engineering companies train their people a lot. Maybe 10x more than in normal SW engineering.