7 ms·
I frequently run into the same problem. I work with clients, and inevitably one of their employees asks something like "how did you get to be so good at Linux?
by tonyarkles 9y ago
I frequently run into the same problem. I work with clients, and inevitably one of their employees asks something like "how did you get to be so good at Linux? You figured out a problem we've been chasing for weeks!"
And I hate my answer, especially when the employee is around my age or older: I first installed Slackware from floppy disks when I was about 12 years old. I installed it because I heard that Linux had a great C compiler, and I was getting tired of the Power-C compiler that I'd been using for 2 years.
Like Rachel, I started on a Vic-20, around age 8. The computer was at my dad's house though, and I lived with my mom most of the week, so I would take out books from the library, write all of my code on loose leaf, and then fast & furiously type it in when I got over to dad's house to see if it worked. POKE 36879 indeed.
- taneq 9y agoI sometimes wonder if growing up before the internet and Wikipedia and Stack Overflow was actually a boon for my problem solving skills. I spent so many hours with nothing to go on but some books and my own persistent trial-and-error. If I'd done the same stuff these days I don't think I'd have learned half as much (although would I have done more, different things? Probably...)
- qznc 9y agoI believe the most important trait for debugging is stubbornness. When Stack Overflow and Google do not help, you start doing it yourself. You use tools like a debugger. If the tools do not help, you can look for other tools. If you cannot find them, you can develop your own. You can look at your code, the code of your operating system, and every library you use. You can trace and observe every instruction a CPU performs.
- cup-of-tea 9y agoYeah, but it's also really bad for sleeping patterns. Countless times I've started to address a bug in the evening only to find it spirals down into some obscure feature I didn't even know existed. After hours of thinking and learning more about the language/tools/computer than you thought there was you feel great, but it's 3am. In my experience there are only two ways to really learn about programming: teaching and debugging.
- tonyarkles 9y agoAnnnnnd... that's how I ended up solving a really stupid bug last night at 2am. I'll add curiosity to stubbornness too. I very much have to understand how a system is actually working before I'm willing to call it quits and call something successfully debugged. I've definitely worked with folks in the past who don't have that drive; they're more concerned about making it work than understanding why it does/doesn't work. My revelation last night was "deep in this (internal) library, a URL is being manipulated using a badly conceived regex, and that's why this URL is getting mangled". The curiosity to understand it on a deep level pretty much defined my academic career too. I'd already been programming (in some capacity) for about a decade by time I turned 18 and finished high school. Computer Science was a pretty obvious career path. But I ended up doing an EE/CS dual major, entirely because I wanted to understand the pieces below software as well. A few years after undergrad, I realized that a lot of people struggled to debug complex "cloud" systems, and went back to do an M.Sc. to study the problem more carefully. (And, per cup-of-tea's comment, I spent a fair bit of time during my M.Sc. teaching as well, including taking on a Spring & Summer session 200-level CS course as a sessional lecturer)
- jacquesm 9y ago> but it's 3am. Where do I sign? When I get really stuck on a bug it's sometimes taken weeks to solve in a never ending game of software dominoes. In very rare cases it ended up being a bug that you could only solve with a soldering iron and some creative rewiring of interrupts. Bugs can be crazily obscured. To the point where you wish you hadn't started but the sunk cost of time invested is such that there is no way back either. This is one of the reasons I was so happy to switch to Unix (and later Linux), the fact that in most cases the program crashes with a core file and access to post mortem debugging (and in many cases a nice stacktrace of how you got there) made short work of most of the bugs. Of course that means you now only have the hardest categories to deal with.
- checkdigit15 9y ago> To the point where you wish you hadn't started but the sunk cost of time invested is such that there is no way back either. "All causes shall give way. I am in code Stepped in so far that, should I wade no more, Returning were as tedious as go o'er."
- ryandrake 9y agoNot enough people reach for a debugger anymore and are actually willing to jump through code, set breakpoints, examine variables in memory. Not enough people dig down instruction by instruction and look at the registers. Debuggers and code monitors used to be your primary development tool. I’ve seen younger developers look at me as if I were some kind of sorceror when I merely step through a C function or dump the contents of RAM. This critical skill seems like a dying art. Isn’t debugging taught in school anymore?
- jacquesm 9y agoI actually rarely use a debugger. I used to but the speed with which current processors zip through code makes interactive debugging a lot less useful than it used to be, in very rare cases it is useful, but in the majority of the work I still do in the software world I've found that proper testing is what addresses the vast majority of the cases that would have hit the debugger in the past, and much faster too. Debuggers are still useful but I don't actually recall the last time when I hit a bug that required interactive debugging.
- tonyarkles 9y ago> Debuggers are still useful but I don't actually recall the last time when I hit a bug that required interactive debugging. Microcontrollers: As a side rant, I'm perpetually grouchy with the fact that Arduinos don't have debugger support. I was using AVRs well before the Arduino was a thing (still have my STK500 kicking around...) and I feel like it's a pretty big lost opportunity for hobbyists to not be able to do a deep dive into what is actually going on for when they want to "level up" their MCU skills. Printfs over serial are great, but they often dork with the timing enough to change the character of bugs; JTAG debugging does too, but you can often insert breakpoints with surgical precision to see what's happening without breaking the timing of what you're trying to debug.
- jacquesm 9y agoYes, in the embedded world I can see lots of applications for the debugger, that's a different world. Real time emulation made life a lot easier as well, compared to the old fashioned way (burn eprom. pray. power up.)
- AnIdiotOnTheNet 9y agoBeen there. Usually before I pull out the debugger on some third-party piece of trash product from the 90s (now owned and neglected by MS) I've come up with a way to work around the problem in 15 lines of powershell and a scheduled task. It ain't pretty, and not knowing what's wrong will eat at me, but it is preferable to wasting my life on problems that I don't really want to solve that badly.
- mercer 9y agoI've been reading the 'Hackers' book by Steven Levy, and the way it describes what makes hackers what they are is exactly this kind of tinkering and problem solving. One thing that's been fascinating me this past decade or so is that clearly programming and lots of other 'nerdy' stuff is quite mainstream now, and as a result the 'old-fashioned' concept of a hacker (or the 'hacker ethos' or whatever) seems to have become less of a specific subculture. But at the same time, I've been trying to do some 'code school' style teaching and I find that there's still a very distinct difference between the coders who mostly just want a job and see coding as a viable path (and even enjoy it, much of the time), and coders who have that same kind of 'hacker' mentality. It's hard to describe, because it's not just about skill or 'good code' or 'productive results'. I guess it's some degree of unusual passion or excitement about the whole thing. And definitely a tendency to just 'hack it' and coming up with sometimes horrifyingly creative solutions to the problem. For example, one guy I was teaching was incredibly messy in his code style, undisciplined, and clearly copy/pasting stuff from Stack Overflow on occasion. But when we talked about computers, programming, what kind of cool stuff he'd like to build, he was clearly different from the others. Where the others would sometimes diligently go through the Rails Tutorial chapter by chapter, with him sometimes the only thing that got him to actually learn was to give him something that 1) tickled his particular passions (games), and 2) was problematic enough that he'd be cursing at his computer until well after the 'official' study time. And while his solution rather predictably was often messy, different from what he'd been taught, and somewhat unconventional, he usually figured it out and there was almost always some 'cleverness' to it.
- caseymarquis 9y agoUngooglable problems still exist. Yesterday, for example, 30 Fanuc i industrial controllers started sending back RST tcp packets and refused to initiate new tcp connections until restarted. This behaviour occurred in a system which was stable for 2 years. Very likely an issue on one controller spread to the others by causing an error in a set of dlls from Japan. This occurred due to another set of the same DLLs connecting to 1 controller at the same time. Maybe. I have to talk to the manufacturer to get any further. Or disassemble their libraries. I have these sorts of ungoogleable problems all the time. I doubt industrial automation is the only field like this.
- tonyarkles 9y agoBeing able to solve problems like that one result in you looking like a wizard btw. In case you haven't seen them, the SysInternals tools can serve as an interesting middle ground before popping the DLLs into an assembler. There's one (whose name escapes me) that is basically strace-for-Windows, and I've used it to successfully solve some pretty gnarly Win32 problems before.
- tjalfi 9y agoAre you thinking of Process Monitor?
- tonyarkles 9y agoJust looked it up, and yes. Process Monitor and Process Explorer, depending on what kind of problem it is.
- taneq 9y agoWith a bit of web searching, that's what I found too.
- jacquesm 9y agoThat sounds like an instance of the Thundering Herd problem.
- sevensor 9y agoI think there's two really important points here, and they're only loosely connected to each other. 1. Youthful trial and error is really powerful. I had access to a 286 during my formative years, along with the manual for DOS 4.0. I sat down with that manual and typed in every command to see what it did, until I got to one (I think it was RECOVER?) that overwrote all 30 Mb of the hard drive with garbage files. Then I had to learn how to install an OS. After that, I stopped entering random commands, but fortunately I'd already discovered BASICA and EDLIN. 2. Being able to hold a program in your brain and write it down in a notebook is a huge advantage. I've done all of my best work with pen and paper.
- Yhippa 9y ago> Then I had to learn how to install an OS. I basically followed the same path you did. Once you have to wipe and reinstall you have gotten to a comfortable level with your systems and tend not to fear making mistakes.
- abakker 9y agoRe #2, I'm not a programmer, but as a writer I find the same thing to be true. I work better with a pen and legal pad than a keyboard. I've got this feeling that we have neural pathways built for typing and, subconsciously, we favor phrases, words, and sentences that are easy to type. I know I even have a tic where if I pause to think I'll insert extraneous commas. It's kind of like how musicians tend to improvise around licks and scales that they have practiced a lot. There's something to thinking about the idea of a song on paper that is more creative than unintentionally falling back on chords/forms that are easy to play. (Adam Neely discusses this very well here: https://www.youtube.com/watch?v=6c_LeIXrzAk https://www.youtube.com/watch?v=6c_LeIXrzAk)
- cortesoft 9y agoWouldn't the same be try of a pen and paper, though? We would be subconsciously favoring words that are easy to write? I have horrible handwriting and am a VERY slow writer (and my hand gets cramped after writing a few words). I know whenever I have to write something by hand, I tend to choose small words and short sentences. On the other hand, I am a very fast typist and can type faster than I can think in most cases, so I tend to be much more verbose. Yes, the medium we use matters.
- WalterGR 9y agoEvery generation wonders that about the next one. This is what the previous generation pondered: I sometimes wonder if growing up before pre-built computers and lots of books about computers was actually a boon for my problem solving skills. I spent so many hours with nothing to go on but some schematics and my own persistent trial-and-error. If I'd done the same stuff these days I don't think I'd have learned half as much (although would I have done more, different things? Probably...)
- mercer 9y agoThing is, as someone probably a generation apart, at least, I don't disagree. We all work within a context of abstractions, and in any given generation, poking and prodding at those abstractions and trying to understand what happens below it is immensely valuable and instructive. For one generation, below it might mean pen and pencil, and for the other it might mean looking into what jQuery actually does. Let's not forget that 'pen and pencil' was once upon a time an abstraction inaccessible to many too!
- jacobolus 9y agoSocrates complained that writing was ruining thinking and memory, IIRC.
- pm90 9y agoA determination to debug beyond just googling is ABSOLUTELY IMPORTANT for a software engineer. You are right in that it was probably better when an engineer would have to learn these skills the hard way; but let me offer myself as a counterpoint: I loved that Google/SO allowed me to get started and debug common problems right off the bat; it reduced the barrier to learning programming. Once I gained sufficient proficiency, I felt more confident to delve deeper into code and lower level specifics to debug problems. I guess what I'm saying is that the determined problem-solving mentality that were required before Google/SO are still important, but its possible to learn them even with Google/SO.
- taneq 9y agoThat's the untested (by me) contrapositive to my story. Now you don't have to spend untold hours trying to figure out why that goddamn COM call is crashing, but also, you don't have to waste hours on trivial technical crap when you could have spent those untold hours learning more about how to build complex systems. It ties into one of the best things about programming / software engineering / software architecture, which is that it's a self-scaling system. Everyone eventually finds the level at which it becomes 'too hard', so it's always a challenge.
- cbcoutinho 9y agoIs this the Peter Principle[0] of software engineering? [0] https://en.wikipedia.org/wiki/Peter_principle https://en.wikipedia.org/wiki/Peter_principle
- bitwize 9y ago> A determination to debug beyond just googling is ABSOLUTELY IMPORTANT for a software engineer. We're going to need you to time-box that fix. If you don't have a solution within a day, assign the Jira to Bob so he can take a look at it.
- tonyarkles 9y agoBut... Bob doesn't know his ass from a hole in the ground! It'll never get fixed! In my current gig, I guess I'm Bob. Whenever someone can't figure out the solution to a problem, it just gets punted over to me to figure out wtf someone did a couple of years ago that makes no sense but coincidentally worked for exactly the problem they had. /rant from last night.
- jaclaz 9y agoJFYI, C.S.Lewis: “Experience: that most brutal of teachers. But you learn, my God do you learn.”
- insulanus 9y agoI am so jealous. :) I started out on a similar computer, but once I worked my way through the basic manual that came with it, that was it. I couldn't get any more documentation until I got my hands on an XT many years later.
- tonyarkles 9y agoYeah, I really lucked out. The Vic was super old when I first got it, and we got an XT not too long later. Vic-Basic (3k of RAM) to XT (640k of RAM) was a huge jump. Then the owner of a local computer store I hung out at gave me a copy of Power-C for $20 I think. Not too long after, we got a Pentium 133 and I had a great box to dual-boot Linux on. It was a town of 35k people, and I just happened to luck out and meet the right ones at the right time. The owner of the computer store that gave me the C compiler ended up hiring me for summers and weekends through high school, and then had me back for the first couple years of summer during univeristy too.