16 ms·

Sysadmin war story: the network ate my font

222 points by atsaloli 9y ago

lfx 9y ago
Love those stories. Read some similar ones in the past. First time when debugging - I'm looking into the cache.
- atsaloli 9y ago
  Thanks for the kind words!
sneak 9y ago
500 mile email comment in 3... 2... 1...
- athenot 9y ago
  I always enjoy re-reading it. http://www.ibiblio.org/harris/500milemail.html http://www.ibiblio.org/harris/500milemail.html
  - beautifulfreak 9y ago
    The Daily WTF has a trove of similar stories. https://thedailywtf.com https://thedailywtf.com
  - atsaloli 9y ago
    If you enjoy that, come and chat with us on the LOPSA mailing lists or on IRC. :) (LOPSA is the League of Professional System Administrators, Mr. Harris and I are both members, though my little story is nowhere near as awesome.) https://lopsa.org/Chat-maillists-and-more https://lopsa.org/Chat-maillists-and-more Plus we always welcome new members.
- pjc50 9y ago
  cf related war stories on HN https://news.ycombinator.com/item?id=4709438 https://news.ycombinator.com/item?id=4709438 My own personal strangest experience was having two autonegotiating 10/100baseT devices that wouldn't speak to each other but would via any other switch.
  - kakwa_ 9y ago
    Mine was about NFS. One or two years ago, I was tasked to solve the following issue: Installing an obsolete RHEL 4 on a brand new computer, with all the drivers issues that entails. Virtualization was not an option. The solution I chose was to "backport" the latest RHEL 5 kernel in the RHEL 4. A few weeks of works later repackaging the kernel, adapting the mkinitrd script and a lot of headaches around the install iso (it was actually the hardest part, with a lot of hacks around anaconda), I finally managed to have a working server. Then a few days latter, I was notified of a regression. There was a somewhat crazy application that was managing various configuration files on various devices of this particular infrastructure which stopped working. Digging in this application, I discovered that it was "pushing" the updated conf files through NFS, more exactly, it was notifying a service on the device targeted, and then this device mounted an NFS share on the RHEL 4 server, recovered the conf file, and unmounted the NFS share (I told you, "crazy"). The targeted devices were using RH 7 (RH, not RHEL, I'm talking about the one with a 2.4 kernel). Strangely enough, mounting the share by hand and doing an ls showed that the files were indeed present, and there was no permission issues. Reading the source code a little further and looking at quite old QT versions (the service on the device was QT based). I finally managed to find which line of code was not working: it was a simple readdir(). So I created a simple C program that just did the readdir and I finally managed to reproduce the bug, indeed, readdir was not listing the files present in the directory. But it was not helping me much... So I decided to do some network captures, everything looked OK. I did the same network captures with the old RHEL 4 kernel, and it looked exactly the same. After hours at steering at my screen with the 2 captures side by side, I finally spotted a subtle difference. The fileids (64 bits) was padded with zeros on the first 32 bits with the old kernel (ex: 0x0000000000000000A489097654456F97), and it was not the case with the new kernel (ex: 0xFC871902B9086456A489097654456F97). Basically, the old RH 7 was violating the NFS RFC by only handling 32 bits fileids (by the way, the RFC predates the RH 7 by 5 years). As it was impossible to upgrade the old RH 7, I finally backported the "bug" in my "RHEL 5 kernel on an RHEL 4" by adding a small two lines patch in the RHEL 5 kernel that ensured 32 bits fileids.
- kakwa_ 9y ago
  I discretely removed my comment about it. But, yes, still my favorite one ^^.
gumboshoes 9y ago
This sort of stuff happened all the time when I worked in tech support for ad agencies. Printing font problems seemed to be 99% of the job, some days. PDFs helped — but only of the person making them remembered to embed the fonts. Usually you would simply make an EPS (encapsulated Postscript file), which would work most of the time — as long as the receiving printer had sufficient memory for huge print files. Can't tell you how many thousands of times art directors who should've known better would just send Quark or PageMaker or (later) InDesign files without the fonts. In fact, there were (are?) preflighting programs designed just to solve this exact problem. They'd look at your files and determine: Are your fonts there? Are your images there? How will this print? etc., etc.
- semi-extrinsic 9y ago
  There are indeed PDF preflight tools to this day. E.g. on Linux you can run `pdffonts` and check that everything is embedded. I believe it's still very common, to the point that my local university's PhD thesis print delivery checklist includes it.
  - noir_lord 9y ago
    Scribus (DTP program for Linux I love) has it built right in and it's good. They also pride themselves on the quality of PDF output (and have full (iirc) support for PDF-A).
- qznc 9y ago
  There is nothing popular and free though. I built a simple one myself some time ago: https://bitbucket.org/qznc/vorflug https://bitbucket.org/qznc/vorflug The font check alone can be done in a few lines of Python: http://beza1e1.tuxen.de/articles/preflight.html http://beza1e1.tuxen.de/articles/preflight.html
apetresc 9y ago
I usually love these kinds of war stories, but this one was anticlimactic. This would have been the first instinct of anyone who's ever tried to open a PowerPoint they got over e-mail – "oh, one printer shows the wrong font? I bet the font's not bundled, and one machine just happens to already have it."
- atsaloli 9y ago
  Author here. I've opened a ton of PowerPoint presentations (God help me) but this was the first time I've run into this issue. :) I usually crawl around inside of Web services / infrastructures while troubleshooting.
- icebraining 9y ago
  I thought of that, but I have to say I wouldn't have thought about the printer's font cache. I didn't even know they had one. Then again, I try to avoid printers as much as I can :)
  - atsaloli 9y ago
    Same! You live, you learn. Thanks!
- ghostly_s 9y ago
  Yeah, I think it's interesting because this maybe isn't obvious when approaching with a network-guy mindset. A workstation guy or printer guy would probably check this first thing.
  - atsaloli 9y ago
    Indeed, my background was in administering Internet services (Web, mail, DNS, netnews, etc.) This was my first time troubleshooting a printing issue professionally. I was the only UNIX sysadmin in the shop and I was hired in as a "Network consultant" because they didn't have a slot for a Unix sysadmin -- didn't know what one was, even. But the (new) Director of Engineering specifically hired a sysadmin.
Top19 9y ago
I upvoted simply because the title was too good not to.
- atsaloli 9y ago
  Thanks!! The network ate the font is actually how the problem was reported. :-) That was a fun break from not breaking the HP-UX server running the 24/7 factory and from babysitting various Linux web app server issues at the colo. :-)
izzydata 9y ago
I love it when you are troubleshooting and question how it ever worked in the first place.
- KGIII 9y ago
  The average person is amazed when tech breaks. Those who understand it, are amazed it works at all.
- atsaloli 9y ago
  I know.. blows my mind sometimes.
- _cereal 9y ago
  Sounds like a schroedinbug: http://catb.org/jargon/html/S/schroedinbug.html http://catb.org/jargon/html/S/schroedinbug.html
  - lokedhs 9y ago
    If someone had not heard of that one before, it should not be confused with a Heisenbug, which is approximately the opposite thing. http://catb.org/jargon/html/H/heisenbug.html http://catb.org/jargon/html/H/heisenbug.html
- StavrosK 9y ago
  I've never felt so alone as when I'm worried that a think that shouldn't work, does.
- Too 9y ago
  The famous Six Stages of Debugging https://news.ycombinator.com/item?id=6477187 https://news.ycombinator.com/item?id=6477187 1. That can't happen. 2. That doesn't happen on my machine. 3. That shouldn't happen. 4. Why does that happen? 5. Oh, I see. 6. How did that ever work?
deleted 9y ago
[deleted]
AnimalMuppet 9y ago
> 508 Resource Limit Is Reached The network ate my webpage...
- atsaloli 9y ago
  Update: should be good now. The engineer logged in and killed a couple of stuck processes. In 2017, folks.
24gttghh 9y ago
Yeah? Try dealing with a printer that has to have a font file sent to it with an lp command from cron every 15 minutes or else checks wont be formatted correctly. AND there is a physical device (either USB on the printer or inline on ethernet in front of the printer) which has the MICR fonts stored. And then the printer you are migrating the MICR device to has decided it doesn't understand the PCL in the font file anymore. This is 2017 and I just dealt with this today.
- notgood 9y ago
  The engineering time dealing with that printer is probably more expensive than just buying a simpler/better one.
  - 24gttghh 9y ago
    Close. To be fair the printer is supposed to support PCL5, and this is the first time I've witnessed this happen.
    - kevin_thibedeau 9y ago
      If this is so mission critical why not get a proper Postscript printer with an onboard filesystem for font storage. It doesn't make much sense to run a business on consumer junk.
- 24gttghh 9y ago
  I should point out that a co-worker tried to convince people we could easily do it with a generated PDF and throw away the damn $500 MICR device but that was swiftly deemed 'impractical'.
- deleted 9y ago
  [deleted]
vehementi 9y ago
The darkest ring of hell would be tech support for wireless (bluetooth?) printers
- ianai 9y ago
  In a casino full of smokers and devices interfering with those signals.
cbhl 9y ago
This leads to an obvious question: "why did the programmers not always just embed the font?". And it turns out that the answer will probably be, "well, we asked the lawyers, and you need rights from the font owner to do so, and it makes files bigger, and most stuff is in whatever Microsoft Office uses as their default font this year anyway." And so we're torn between "yes, of course the designer who built the font should get paid" and "a sane default became a checkbox because of US copyright law".
- MBCook 9y ago
  Maybe because PDF has been around a long time and it used to be that embedding even a basic font would have been a MASSIVE increase in file size?
  - KekDemaga 9y ago
    Why don't PDFs just render the text and store it as some form of vector image file.
    - cbhl 9y ago
      That also takes more space. Understandably, when memory was counted in kilobytes and disks were counted in kilobytes, file formats were designed differently than they are today.
    - MBCook 9y ago
      Fonts can have different glyphs/hinting at different sizes and you don't know how big a PDF will be rendered.
    - kps 9y ago
      A font is some sort of vector image file.
      - yellowapple 9y ago
        Not always. Bitmap fonts are a thing.
joehan 9y ago
I just spent all day yesterday debugging why my web app's font-awesome/material design icons weren't showing on IE while they were showing on chrome, ff, and edge. Turned out that to be the corporate policy to disable font downloading on IE. https://www.stigviewer.com/stig/microsoft_internet_explorer_11/2017-03-01/finding/V-46505 https://www.stigviewer.com/stig/microsoft_internet_explorer_...
- 0x0 9y ago
  Ouch. To be fair, fonts really do pave the way for malware into your kernel - https://googleprojectzero.blogspot.no/2015/07/one-font-vulnerability-to-rule-them-all.html https://googleprojectzero.blogspot.no/2015/07/one-font-vulne...
bluedino 9y ago
We had a printer that did checks and kept the signature on a USB drive, at some point the printer started printing a distorted signature - it ended up using the BMP signature and not the PNG file. Changed out of the blue somehow... both images were fine but it had a bug in how it decoded the BMP files
yonatron 9y ago
It's "pored over", not "poured over".
- deleted 9y ago
  [deleted]
TempleOS 9y ago
TempleOS uses 640x480 VGA graphics mode not text mode. It draws the characters from an 8x8 font stored in the kernel. I prefer fixed-width fonts. If VGA init fails, it actually runs in text mode perhaps. ridiculous nigger monkeys have never seen a high caliber professional programmer. https://pchelpforum.net/t/updated-top-10-alternative-operating-systems.12821/ https://pchelpforum.net/t/updated-top-10-alternative-operati... brendan is a nigger buffoon whos grandiose ideas doom him. retard buffoon
dba7dba 9y ago
> Turns out the printer had a cache for fonts and was using the font cached from the earlier check image which included the font! Dang cache.
fit2rule 9y ago
I once (as a junior developer) had to write some software to format a massive text file for printing to an IBM line-printer (1403, model N1) .. this (1M) text file had to be checked a few times before it was printed, and that meant opening it up in an editor and verifying the data. I got pretty sick of opening this file after half a day, because it had tons and tons of CR/LF's and my editor at the time rendered these with strange characters on the screen .. and I didn't like that, as a junior guy, so I just replaced the LF's with 00's. For some reason, this just worked fine in my local environment, and I was able to validate the data in the file before sending it off to the spool for printing, later in the afternoon. About 3 minutes after I closed the job, the building got a fire alarm, and we all had to exit. Apparently there had been smoke detected in the operations room, where the printer was located, so the Halon systems went off, and we went into full-blown "Ops Reset" mode. After an hour of hanging around the parking lot, I was called in by the head honcho's in the Ops Room, sat down in front of the printer, and told to explain myself. Well, turns out, I was responsible. The lack of LF's in the text file meant that the printer was printing - as fast as it could - every single line on the very first character position .. and after a few minutes, the printer simply caught fire. Oh man, since that day (mid-80's), I've eschewed any job that requires me to deal with printers, and I've been anti-printer ever since. ;)

222 points by atsaloli 9y ago

lfx 9y ago

Love those stories. Read some similar ones in the past. First time when debugging - I'm looking into the cache.

atsaloli 9y ago
Thanks for the kind words!

sneak 9y ago

500 mile email comment in 3... 2... 1...

athenot 9y ago
I always enjoy re-reading it. http://www.ibiblio.org/harris/500milemail.html http://www.ibiblio.org/harris/500milemail.html
- beautifulfreak 9y ago
  The Daily WTF has a trove of similar stories. https://thedailywtf.com https://thedailywtf.com
- atsaloli 9y ago
  If you enjoy that, come and chat with us on the LOPSA mailing lists or on IRC. :) (LOPSA is the League of Professional System Administrators, Mr. Harris and I are both members, though my little story is nowhere near as awesome.) https://lopsa.org/Chat-maillists-and-more https://lopsa.org/Chat-maillists-and-more Plus we always welcome new members.
pjc50 9y ago
cf related war stories on HN https://news.ycombinator.com/item?id=4709438 https://news.ycombinator.com/item?id=4709438 My own personal strangest experience was having two autonegotiating 10/100baseT devices that wouldn't speak to each other but would via any other switch.
- kakwa_ 9y ago
  Mine was about NFS. One or two years ago, I was tasked to solve the following issue: Installing an obsolete RHEL 4 on a brand new computer, with all the drivers issues that entails. Virtualization was not an option. The solution I chose was to "backport" the latest RHEL 5 kernel in the RHEL 4. A few weeks of works later repackaging the kernel, adapting the mkinitrd script and a lot of headaches around the install iso (it was actually the hardest part, with a lot of hacks around anaconda), I finally managed to have a working server. Then a few days latter, I was notified of a regression. There was a somewhat crazy application that was managing various configuration files on various devices of this particular infrastructure which stopped working. Digging in this application, I discovered that it was "pushing" the updated conf files through NFS, more exactly, it was notifying a service on the device targeted, and then this device mounted an NFS share on the RHEL 4 server, recovered the conf file, and unmounted the NFS share (I told you, "crazy"). The targeted devices were using RH 7 (RH, not RHEL, I'm talking about the one with a 2.4 kernel). Strangely enough, mounting the share by hand and doing an ls showed that the files were indeed present, and there was no permission issues. Reading the source code a little further and looking at quite old QT versions (the service on the device was QT based). I finally managed to find which line of code was not working: it was a simple readdir(). So I created a simple C program that just did the readdir and I finally managed to reproduce the bug, indeed, readdir was not listing the files present in the directory. But it was not helping me much... So I decided to do some network captures, everything looked OK. I did the same network captures with the old RHEL 4 kernel, and it looked exactly the same. After hours at steering at my screen with the 2 captures side by side, I finally spotted a subtle difference. The fileids (64 bits) was padded with zeros on the first 32 bits with the old kernel (ex: 0x0000000000000000A489097654456F97), and it was not the case with the new kernel (ex: 0xFC871902B9086456A489097654456F97). Basically, the old RH 7 was violating the NFS RFC by only handling 32 bits fileids (by the way, the RFC predates the RH 7 by 5 years). As it was impossible to upgrade the old RH 7, I finally backported the "bug" in my "RHEL 5 kernel on an RHEL 4" by adding a small two lines patch in the RHEL 5 kernel that ensured 32 bits fileids.
kakwa_ 9y ago
I discretely removed my comment about it. But, yes, still my favorite one ^^.

gumboshoes 9y ago

This sort of stuff happened all the time when I worked in tech support for ad agencies. Printing font problems seemed to be 99% of the job, some days. PDFs helped — but only of the person making them remembered to embed the fonts. Usually you would simply make an EPS (encapsulated Postscript file), which would work most of the time — as long as the receiving printer had sufficient memory for huge print files. Can't tell you how many thousands of times art directors who should've known better would just send Quark or PageMaker or (later) InDesign files without the fonts. In fact, there were (are?) preflighting programs designed just to solve this exact problem. They'd look at your files and determine: Are your fonts there? Are your images there? How will this print? etc., etc.

semi-extrinsic 9y ago
There are indeed PDF preflight tools to this day. E.g. on Linux you can run `pdffonts` and check that everything is embedded. I believe it's still very common, to the point that my local university's PhD thesis print delivery checklist includes it.
- noir_lord 9y ago
  Scribus (DTP program for Linux I love) has it built right in and it's good. They also pride themselves on the quality of PDF output (and have full (iirc) support for PDF-A).
qznc 9y ago
There is nothing popular and free though. I built a simple one myself some time ago: https://bitbucket.org/qznc/vorflug https://bitbucket.org/qznc/vorflug The font check alone can be done in a few lines of Python: http://beza1e1.tuxen.de/articles/preflight.html http://beza1e1.tuxen.de/articles/preflight.html

apetresc 9y ago

I usually love these kinds of war stories, but this one was anticlimactic. This would have been the first instinct of anyone who's ever tried to open a PowerPoint they got over e-mail – "oh, one printer shows the wrong font? I bet the font's not bundled, and one machine just happens to already have it."

atsaloli 9y ago
Author here. I've opened a ton of PowerPoint presentations (God help me) but this was the first time I've run into this issue. :) I usually crawl around inside of Web services / infrastructures while troubleshooting.
icebraining 9y ago
I thought of that, but I have to say I wouldn't have thought about the printer's font cache. I didn't even know they had one. Then again, I try to avoid printers as much as I can :)
- atsaloli 9y ago
  Same! You live, you learn. Thanks!
ghostly_s 9y ago
Yeah, I think it's interesting because this maybe isn't obvious when approaching with a network-guy mindset. A workstation guy or printer guy would probably check this first thing.
- atsaloli 9y ago
  Indeed, my background was in administering Internet services (Web, mail, DNS, netnews, etc.) This was my first time troubleshooting a printing issue professionally. I was the only UNIX sysadmin in the shop and I was hired in as a "Network consultant" because they didn't have a slot for a Unix sysadmin -- didn't know what one was, even. But the (new) Director of Engineering specifically hired a sysadmin.

Top19 9y ago

I upvoted simply because the title was too good not to.

atsaloli 9y ago
Thanks!! The network ate the font is actually how the problem was reported. :-) That was a fun break from not breaking the HP-UX server running the 24/7 factory and from babysitting various Linux web app server issues at the colo. :-)

izzydata 9y ago

I love it when you are troubleshooting and question how it ever worked in the first place.

KGIII 9y ago
The average person is amazed when tech breaks. Those who understand it, are amazed it works at all.
atsaloli 9y ago
I know.. blows my mind sometimes.
_cereal 9y ago
Sounds like a schroedinbug: http://catb.org/jargon/html/S/schroedinbug.html http://catb.org/jargon/html/S/schroedinbug.html
- lokedhs 9y ago
  If someone had not heard of that one before, it should not be confused with a Heisenbug, which is approximately the opposite thing. http://catb.org/jargon/html/H/heisenbug.html http://catb.org/jargon/html/H/heisenbug.html
StavrosK 9y ago
I've never felt so alone as when I'm worried that a think that shouldn't work, does.
Too 9y ago
The famous Six Stages of Debugging https://news.ycombinator.com/item?id=6477187 https://news.ycombinator.com/item?id=6477187 1. That can't happen. 2. That doesn't happen on my machine. 3. That shouldn't happen. 4. Why does that happen? 5. Oh, I see. 6. How did that ever work?

deleted 9y ago

[deleted]

AnimalMuppet 9y ago

> 508 Resource Limit Is Reached The network ate my webpage...

atsaloli 9y ago
Update: should be good now. The engineer logged in and killed a couple of stuck processes. In 2017, folks.

24gttghh 9y ago

Yeah? Try dealing with a printer that has to have a font file sent to it with an lp command from cron every 15 minutes or else checks wont be formatted correctly. AND there is a physical device (either USB on the printer or inline on ethernet in front of the printer) which has the MICR fonts stored. And then the printer you are migrating the MICR device to has decided it doesn't understand the PCL in the font file anymore. This is 2017 and I just dealt with this today.

notgood 9y ago
The engineering time dealing with that printer is probably more expensive than just buying a simpler/better one.
- 24gttghh 9y ago
  Close. To be fair the printer is supposed to support PCL5, and this is the first time I've witnessed this happen.
  - kevin_thibedeau 9y ago
    If this is so mission critical why not get a proper Postscript printer with an onboard filesystem for font storage. It doesn't make much sense to run a business on consumer junk.
24gttghh 9y ago
I should point out that a co-worker tried to convince people we could easily do it with a generated PDF and throw away the damn $500 MICR device but that was swiftly deemed 'impractical'.
deleted 9y ago
[deleted]

vehementi 9y ago

The darkest ring of hell would be tech support for wireless (bluetooth?) printers

ianai 9y ago
In a casino full of smokers and devices interfering with those signals.

cbhl 9y ago

This leads to an obvious question: "why did the programmers not always just embed the font?". And it turns out that the answer will probably be, "well, we asked the lawyers, and you need rights from the font owner to do so, and it makes files bigger, and most stuff is in whatever Microsoft Office uses as their default font this year anyway." And so we're torn between "yes, of course the designer who built the font should get paid" and "a sane default became a checkbox because of US copyright law".

MBCook 9y ago
Maybe because PDF has been around a long time and it used to be that embedding even a basic font would have been a MASSIVE increase in file size?
- KekDemaga 9y ago
  Why don't PDFs just render the text and store it as some form of vector image file.
  - cbhl 9y ago
    That also takes more space. Understandably, when memory was counted in kilobytes and disks were counted in kilobytes, file formats were designed differently than they are today.
  - MBCook 9y ago
    Fonts can have different glyphs/hinting at different sizes and you don't know how big a PDF will be rendered.
  - kps 9y ago
    A font is some sort of vector image file.
    - yellowapple 9y ago
      Not always. Bitmap fonts are a thing.

joehan 9y ago

I just spent all day yesterday debugging why my web app's font-awesome/material design icons weren't showing on IE while they were showing on chrome, ff, and edge. Turned out that to be the corporate policy to disable font downloading on IE. https://www.stigviewer.com/stig/microsoft_internet_explorer_11/2017-03-01/finding/V-46505 https://www.stigviewer.com/stig/microsoft_internet_explorer_...

0x0 9y ago
Ouch. To be fair, fonts really do pave the way for malware into your kernel - https://googleprojectzero.blogspot.no/2015/07/one-font-vulnerability-to-rule-them-all.html https://googleprojectzero.blogspot.no/2015/07/one-font-vulne...

bluedino 9y ago

We had a printer that did checks and kept the signature on a USB drive, at some point the printer started printing a distorted signature - it ended up using the BMP signature and not the PNG file. Changed out of the blue somehow... both images were fine but it had a bug in how it decoded the BMP files

yonatron 9y ago

It's "pored over", not "poured over".

deleted 9y ago
[deleted]

TempleOS 9y ago

TempleOS uses 640x480 VGA graphics mode not text mode. It draws the characters from an 8x8 font stored in the kernel. I prefer fixed-width fonts. If VGA init fails, it actually runs in text mode perhaps. ridiculous nigger monkeys have never seen a high caliber professional programmer. https://pchelpforum.net/t/updated-top-10-alternative-operating-systems.12821/ https://pchelpforum.net/t/updated-top-10-alternative-operati... brendan is a nigger buffoon whos grandiose ideas doom him. retard buffoon

dba7dba 9y ago

> Turns out the printer had a cache for fonts and was using the font cached from the earlier check image which included the font! Dang cache.

fit2rule 9y ago

I once (as a junior developer) had to write some software to format a massive text file for printing to an IBM line-printer (1403, model N1) .. this (1M) text file had to be checked a few times before it was printed, and that meant opening it up in an editor and verifying the data. I got pretty sick of opening this file after half a day, because it had tons and tons of CR/LF's and my editor at the time rendered these with strange characters on the screen .. and I didn't like that, as a junior guy, so I just replaced the LF's with 00's. For some reason, this just worked fine in my local environment, and I was able to validate the data in the file before sending it off to the spool for printing, later in the afternoon. About 3 minutes after I closed the job, the building got a fire alarm, and we all had to exit. Apparently there had been smoke detected in the operations room, where the printer was located, so the Halon systems went off, and we went into full-blown "Ops Reset" mode. After an hour of hanging around the parking lot, I was called in by the head honcho's in the Ops Room, sat down in front of the printer, and told to explain myself. Well, turns out, I was responsible. The lack of LF's in the text file meant that the printer was printing - as fast as it could - every single line on the very first character position .. and after a few minutes, the printer simply caught fire. Oh man, since that day (mid-80's), I've eschewed any job that requires me to deal with printers, and I've been anti-printer ever since. ;)