5 ms·

Restoring YC's Xerox Alto day 9: tracing a crash through software and hardware

116 points by dwaxe 10y ago

S_A_P 10y ago
I need to remind myself to not read these for a while. I find myself disappointed when the post ends in a cliffhanger. I want to just binge read these after its all said and done. Really great reads!
digi_owl 10y ago
I find it hard enough to debug something when it is at the software level, and here we are looking at individual on/off signals between chips.
- jackhack 10y ago
  Isn't that the best? Hundreds-of-thousands of layers of abstraction, but if you dig deep enough, it all comes down to a switch being either on or off. That's it.
  - digi_owl 10y ago
    Ultimately leading to random acts of cosmic rays. https://blogs.oracle.com/ksplice/entry/attack_of_the_cosmic_rays1 https://blogs.oracle.com/ksplice/entry/attack_of_the_cosmic_...
  - agumonkey 10y ago
    Seen the fridge story yet ? https://news.ycombinator.com/item?id=12633119 https://news.ycombinator.com/item?id=12633119 Have a laugh Last year I saw a talk [1] where the guy told the story of a bug on web application. It was a scheme->js (based on bigloo) framework. Many servers were involved only one had the bug, some exception, digging led an arithmetic issue (some NaN or oo), then to a floating point issue, many wrong theories about low level architectures. Turns out it was a locale problem, changing the libc parsing of float (, vs .). Who said text was easy ? [1] was lucky and proud to grab a seat at C. Queinnec (of L.I.S.P) jubilee.
    - userbinator 10y ago
      Many servers were involved only one had the bug, This is where using multiple debuggers simultaneously is extremely helpful. Where the control flow diverges is where the problem is likely to be. The other skill that's valuable is binary search --- narrowing down the "window" between when everything is still OK and when something goes amiss, gets you exactly where the bug is.
    - digi_owl 10y ago
      Ugh, laptop power management is a mess and is getting worse by the day it seems. On that note, my laptop had developed this issue that it will no longer see the charger if i leave it plugged in across a sleep cycle. Only way i have found to temporarily remedy this is to unplug the charger at the wall, wait, and eventually it will momentarily think it has been plugged in before going back to an unplugged state.
      - agumonkey 10y ago
        Mainstream spoils everything.
  - userbinator 10y ago
    I don't think we're near that many layers of abstraction (yet --- and hopefully never?), but certainly more than ten. Indeed it's all about switching binary signals eventually. There's a great book about that too: https://www.amazon.com/Code-Language-Computer-Hardware-Software/dp/0735611319 https://www.amazon.com/Code-Language-Computer-Hardware-Softw...
    - jackhack 10y ago
      Funny, I was thinking of Petzold's book when I wrote that. +1 for mentioning it! I still say there are that many layers of abstraction, maybe more. Individual bits, ganged as words, with logic to treat as such, and ability to store and move along a bus. Then you add the ability to act on data: Control logic, arithmetic units, caches, and interpret data as instructions -- microcode running in the CPU, and processor instructions.Now move up to main memory, storage subsystems, networking, graphics (GPUs!!) etc. etc. etc. That gets you through the hardware. Now add software: BIOS, operating system, drivers, user space code, compilers, interpreters, it just goes on and on. And for every one of these giant groups of something, there are just so many logical groupings and abstractions. It's like individual cells in a body, forming organelles, forming organs, forming creatures. So many layers of organization. Amazing beauty. But always at the bottom of the pile, there is a switch.
    - kens 10y ago
      Well, let's see how many layers of abstraction the Alto has. 1) Software (a BCPL program). 2) Library / OS code. 3) Machine code. 4) Microcode. 5) Logic signals (TTL chips). 6) Analog signals (e.g. rise time and voltage of the logic signals). 7) Transistors inside the chips. 8) Quantum mechanics. (The last two levels are hidden to us.) If you're using Smalltalk, you could add another layer for the interpreter.
jackhack 10y ago
Now struggling to resist the urge to upgrade my sad old analog oscilloscope. Thanks, guys! Seriously, though, this is by far the most interesting series of articles I've read in at least 10 years. Digging in and understanding equipment from decades ago, where everything is potentially questionable is a heroic effort. It's like all the good parts of Radio Electronics & Byte magazine combined. A mystery who-done-it for nerds.
- gvb 10y ago
  Ending in a killer cliffhanger!
  - kens 10y ago
    For a bit of a spoiler: we've figured out that the rewiring on the control board converted it to work with a 3K RAM board. But we have a 1K RAM board which has a couple incompatible pins, grounding the signals we need. I don't know why our Alto would have incompatible boards unless some boards were swapped at some point. So we'll probably get a 3K RAM board and hopefully everything will work. If not, we may undo the rewiring.
    - linker3000 10y ago
      3K?? Steady on! My Acorn System 1 does fine with 1.125K Seems a bit reckless to me. PS: The articles have been a great read.
      - kens 10y ago
        Just to make sure everything is clear, the Alto has 256K words of 16-bit main memory. It also has 1K or 3K words of specialized 32-bit microcode memory, which is what I'm discussing above. Since you don't do much in microcode, 3K is a fair bit. (Also note that the microcode memory needs to be much faster (i.e. expensive) than main memory.) So the Alto and Acorn are in totally different worlds as far as memory. (Not to put down the Acorn.)
source99 10y ago
If you are enjoying these updates I can't say enough great things about the Book Hackers by Steven Levy - https://www.amazon.com/Hackers-Heroes-Computer-Revolution-Anniversary/dp/1449388396/ref=sr_1_3?ie=UTF8&qid=1476208619&sr=8-3&keywords=hackers https://www.amazon.com/Hackers-Heroes-Computer-Revolution-An...
- Dav3xor 10y ago
  His book Insanely Great has more info on PARC, although only through the lens of the development of the Macintosh.
david-given 10y ago
Is the Alto simple enough to simulate in toto on an FPGA, microcode and all? It sounds like it would be a fascinating system to play with.
- kens 10y ago
  If you want to play around with a simulated Alto, the ContrAlto simulator from the Living Computer Museum works very well. It is at http://www.livingcomputermuseum.org/Online-Systems/Press.aspx http://www.livingcomputermuseum.org/Online-Systems/Press.asp... I imagine a FPGA-based Alto would be straightforward since there's not a whole lot of hardware. The tricky part is the Alto is very tied to the hardware, so you'd also need to simulate the low-level display hardware, the disk interface, and so forth. There's no real abstraction layer. For instance, the microcode processes each word as it is read from disk, so the disk drive timing is important.
- rjsw 10y ago
  Yes, it would easily fit in a low cost FPGA.
bsder 10y ago
God, I miss being able to put clips directly on a chip.
ChuckMcM 10y ago
Aww fun fun. The offspring of SWAT was the the debugger in the D-systems. They had a 3 digit LED display on the front of the machine and when you crashed that display would show 918 (if I recall correctly). At which point you could connect over ethernet to the machine and be inside the debugger which felt a lot like your typical JTAG station does today. The D-machines also shared the fungible microcode and it was always very important to have the right version of the microcode for the version of the MESA compiler or you ended up with really crazy bugs. That said, you could iteratively design both the language code generator and the microcode to optimize for speed or code density or low interrupt latency, what ever. As a young engineer at Intel (my wife was an engineer at Xerox) I felt like that sort of architecture was 20 years ahead of its time, and eventually the nearly all hard coded instructions of microprocessors would be considered quaint. Watching these concepts come in and go out of favor several times, reading about the Alto feels like looking at the embryo of the future.
CamperBob2 10y ago
Thus, the cause of our crashes was these two wires that someone had added to the board! Could we simply cut these wires and have the system work correctly? We figured we should understand why the wires were there, rather than randomly ripping them out. Maybe our control board and CRAM board were incompatible? Maybe these wires were to support the Trident disk drive we aren't using? It was the end of the day by this point, so further investigation will wait until next time. Cool story that ultimately doesn't go anywhere, bummer.
lalalandland 10y ago
Old hardware is so simple it is possible to trace, understand and fix. What era computer will we not be able do this anymore ? For example if you have a multilayer PCB with damages it is very hard to fix. Also components become so small it is hard to physically exchange them. Will all newer than round 1987 hardware become nonworking museum pieces in a few years ?

116 points by dwaxe 10y ago

S_A_P 10y ago

I need to remind myself to not read these for a while. I find myself disappointed when the post ends in a cliffhanger. I want to just binge read these after its all said and done. Really great reads!

digi_owl 10y ago

I find it hard enough to debug something when it is at the software level, and here we are looking at individual on/off signals between chips.

jackhack 10y ago
Isn't that the best? Hundreds-of-thousands of layers of abstraction, but if you dig deep enough, it all comes down to a switch being either on or off. That's it.
- digi_owl 10y ago
  Ultimately leading to random acts of cosmic rays. https://blogs.oracle.com/ksplice/entry/attack_of_the_cosmic_rays1 https://blogs.oracle.com/ksplice/entry/attack_of_the_cosmic_...
- agumonkey 10y ago
  Seen the fridge story yet ? https://news.ycombinator.com/item?id=12633119 https://news.ycombinator.com/item?id=12633119 Have a laugh Last year I saw a talk [1] where the guy told the story of a bug on web application. It was a scheme->js (based on bigloo) framework. Many servers were involved only one had the bug, some exception, digging led an arithmetic issue (some NaN or oo), then to a floating point issue, many wrong theories about low level architectures. Turns out it was a locale problem, changing the libc parsing of float (, vs .). Who said text was easy ? [1] was lucky and proud to grab a seat at C. Queinnec (of L.I.S.P) jubilee.
  - userbinator 10y ago
    Many servers were involved only one had the bug, This is where using multiple debuggers simultaneously is extremely helpful. Where the control flow diverges is where the problem is likely to be. The other skill that's valuable is binary search --- narrowing down the "window" between when everything is still OK and when something goes amiss, gets you exactly where the bug is.
  - digi_owl 10y ago
    Ugh, laptop power management is a mess and is getting worse by the day it seems. On that note, my laptop had developed this issue that it will no longer see the charger if i leave it plugged in across a sleep cycle. Only way i have found to temporarily remedy this is to unplug the charger at the wall, wait, and eventually it will momentarily think it has been plugged in before going back to an unplugged state.
    - agumonkey 10y ago
      Mainstream spoils everything.
- userbinator 10y ago
  I don't think we're near that many layers of abstraction (yet --- and hopefully never?), but certainly more than ten. Indeed it's all about switching binary signals eventually. There's a great book about that too: https://www.amazon.com/Code-Language-Computer-Hardware-Software/dp/0735611319 https://www.amazon.com/Code-Language-Computer-Hardware-Softw...
  - jackhack 10y ago
    Funny, I was thinking of Petzold's book when I wrote that. +1 for mentioning it! I still say there are that many layers of abstraction, maybe more. Individual bits, ganged as words, with logic to treat as such, and ability to store and move along a bus. Then you add the ability to act on data: Control logic, arithmetic units, caches, and interpret data as instructions -- microcode running in the CPU, and processor instructions.Now move up to main memory, storage subsystems, networking, graphics (GPUs!!) etc. etc. etc. That gets you through the hardware. Now add software: BIOS, operating system, drivers, user space code, compilers, interpreters, it just goes on and on. And for every one of these giant groups of something, there are just so many logical groupings and abstractions. It's like individual cells in a body, forming organelles, forming organs, forming creatures. So many layers of organization. Amazing beauty. But always at the bottom of the pile, there is a switch.
  - kens 10y ago
    Well, let's see how many layers of abstraction the Alto has. 1) Software (a BCPL program). 2) Library / OS code. 3) Machine code. 4) Microcode. 5) Logic signals (TTL chips). 6) Analog signals (e.g. rise time and voltage of the logic signals). 7) Transistors inside the chips. 8) Quantum mechanics. (The last two levels are hidden to us.) If you're using Smalltalk, you could add another layer for the interpreter.

jackhack 10y ago

Now struggling to resist the urge to upgrade my sad old analog oscilloscope. Thanks, guys! Seriously, though, this is by far the most interesting series of articles I've read in at least 10 years. Digging in and understanding equipment from decades ago, where everything is potentially questionable is a heroic effort. It's like all the good parts of Radio Electronics & Byte magazine combined. A mystery who-done-it for nerds.

gvb 10y ago
Ending in a killer cliffhanger!
- kens 10y ago
  For a bit of a spoiler: we've figured out that the rewiring on the control board converted it to work with a 3K RAM board. But we have a 1K RAM board which has a couple incompatible pins, grounding the signals we need. I don't know why our Alto would have incompatible boards unless some boards were swapped at some point. So we'll probably get a 3K RAM board and hopefully everything will work. If not, we may undo the rewiring.
  - linker3000 10y ago
    3K?? Steady on! My Acorn System 1 does fine with 1.125K Seems a bit reckless to me. PS: The articles have been a great read.
    - kens 10y ago
      Just to make sure everything is clear, the Alto has 256K words of 16-bit main memory. It also has 1K or 3K words of specialized 32-bit microcode memory, which is what I'm discussing above. Since you don't do much in microcode, 3K is a fair bit. (Also note that the microcode memory needs to be much faster (i.e. expensive) than main memory.) So the Alto and Acorn are in totally different worlds as far as memory. (Not to put down the Acorn.)

source99 10y ago

If you are enjoying these updates I can't say enough great things about the Book Hackers by Steven Levy - https://www.amazon.com/Hackers-Heroes-Computer-Revolution-Anniversary/dp/1449388396/ref=sr_1_3?ie=UTF8&qid=1476208619&sr=8-3&keywords=hackers https://www.amazon.com/Hackers-Heroes-Computer-Revolution-An...

Dav3xor 10y ago
His book Insanely Great has more info on PARC, although only through the lens of the development of the Macintosh.

david-given 10y ago

Is the Alto simple enough to simulate in toto on an FPGA, microcode and all? It sounds like it would be a fascinating system to play with.

kens 10y ago
If you want to play around with a simulated Alto, the ContrAlto simulator from the Living Computer Museum works very well. It is at http://www.livingcomputermuseum.org/Online-Systems/Press.aspx http://www.livingcomputermuseum.org/Online-Systems/Press.asp... I imagine a FPGA-based Alto would be straightforward since there's not a whole lot of hardware. The tricky part is the Alto is very tied to the hardware, so you'd also need to simulate the low-level display hardware, the disk interface, and so forth. There's no real abstraction layer. For instance, the microcode processes each word as it is read from disk, so the disk drive timing is important.
rjsw 10y ago
Yes, it would easily fit in a low cost FPGA.

bsder 10y ago

God, I miss being able to put clips directly on a chip.

ChuckMcM 10y ago

Aww fun fun. The offspring of SWAT was the the debugger in the D-systems. They had a 3 digit LED display on the front of the machine and when you crashed that display would show 918 (if I recall correctly). At which point you could connect over ethernet to the machine and be inside the debugger which felt a lot like your typical JTAG station does today. The D-machines also shared the fungible microcode and it was always very important to have the right version of the microcode for the version of the MESA compiler or you ended up with really crazy bugs. That said, you could iteratively design both the language code generator and the microcode to optimize for speed or code density or low interrupt latency, what ever. As a young engineer at Intel (my wife was an engineer at Xerox) I felt like that sort of architecture was 20 years ahead of its time, and eventually the nearly all hard coded instructions of microprocessors would be considered quaint. Watching these concepts come in and go out of favor several times, reading about the Alto feels like looking at the embryo of the future.

CamperBob2 10y ago

Thus, the cause of our crashes was these two wires that someone had added to the board! Could we simply cut these wires and have the system work correctly? We figured we should understand why the wires were there, rather than randomly ripping them out. Maybe our control board and CRAM board were incompatible? Maybe these wires were to support the Trident disk drive we aren't using? It was the end of the day by this point, so further investigation will wait until next time. Cool story that ultimately doesn't go anywhere, bummer.

lalalandland 10y ago

Old hardware is so simple it is possible to trace, understand and fix. What era computer will we not be able do this anymore ? For example if you have a multilayer PCB with damages it is very hard to fix. Also components become so small it is hard to physically exchange them. Will all newer than round 1987 hardware become nonworking museum pieces in a few years ?