5 ms·

Unix has a few tricks up its sleeve from when it wasn't the top dog. I hesitate to ever recommend perl, but pack [1] and unpack are pretty sweet for this kind o

by jfoutz 10y ago

Unix has a few tricks up its sleeve from when it wasn't the top dog. I hesitate to ever recommend perl, but pack [1] and unpack are pretty sweet for this kind of stuff. I only learned about them when a state sent me files in EBCDIC [2]. As with all things perl, you can convert from that to ASCII as a one liner. Or, rather, i helped someone much smarter than me do that, 20 years ago. [1]http://perldoc.perl.org/functions/pack.html http://perldoc.perl.org/functions/pack.html [2]https://en.wikipedia.org/wiki/EBCDIC https://en.wikipedia.org/wiki/EBCDIC

defined 10y ago
Please don't remind me. I had to convert the expat XML parser to compile on z/OS and work in EBCDIC, and found that round tripping between ASCII and EBCDIC was sometimes impossible because of the existence of not two, but THREE line terminator characters: CR, LF, and NL (0x85). Not to mention that you cannot test for uppercase or lowercase like the ASCII `ch >= 'A' && ch <= 'Z'` because they are not contiguous in EBCDIC. A good reason to use the C RTL.
- osullivj 10y ago
  I did the same to enable XML messages to flow over MQ between an RS/6000 based front office FX options system, and a back office S/390 system. IIRC there were six (!!) different EBCDIC codepages that could be in play. I had a code generator that would crank out C or Java bindings that could martial between the expat results and the COBOL data structure. 20 years ago now!
  - technofiend 10y ago
    Not sure if you remember but sendmail used to require a certain amount of m4 knowledge and hackery. Emboldened by that and reading the dragon book I was very impressed with myself when I wrote a COBOL parser in a mix of C, lex and YACC that automatically generated the needed 'C' structs and Sybase database layout to load data fed from a System/36. I made the data supplier put his code in the first part of the magtape, read and parsed that and then read the rest of the tape. These days I consider it more of a "what was I thinking" facepalm-worthy sort of thing but at the time I was very proud of it. The "what was I thinking" part is more about the fact that some poor bastard had to come along after me and support that mess.
    - osullivj 10y ago
      Likewise: I regard code generation as a red flag these days. The version skew issues when code generated off slightly different versions of messages are in play can be really nasty. CORBA suffered from that issue big time in the late 90s. And if your generated code uses mutexes in a misguided attempt to be "thread safe", all bets are off...
  - defined 10y ago
    Oh yeah, I remember the code pages. Joy. I didn't want to bloat my comment, so thanks for mentioning this!
- avar 10y ago
  This is how you can implement isUpper() on EBCDIC: https://github.com/Perl/perl5/blob/v5.24.0/handy.h#L1153 https://github.com/Perl/perl5/blob/v5.24.0/handy.h#L1153
  - WorldMaker 10y ago
    IIRC, you should be able to do it easier than that: EBCDIC is not contiguous because upper-case versus lower-case is a (shift) bit-flag.
- SmellTheGlove 10y ago
  > Not to mention that you cannot test for uppercase or lowercase like the ASCII `ch >= 'A' && ch <= 'Z'` because they are not contiguous in EBCDIC. A good reason to use the C RTL. Watch your sorting methods, too. I had a guy over here once totally confused why running his SAS job on the mainframe yielded a different result than the same code running on PC SAS against the same data.
cafard 10y ago
No kidding. Over 20 years ago, I used unpack to read minicomputer log files. It was vastly faster to FTP them over to a Sun Sparcstation and process them with Perl than to use the native log reader. It was also far more flexible.
kbenson 10y ago
> I hesitate to ever recommend perl ... Or, rather, i helped someone much smarter than me do that, 20 years ago. No need to hesitate. It's not half bad for a dynamic language if you keep a little discipline. 20 years ago the average Perl programmer was probably akin to the average PHP programmer from 10 years ago. That is, not very experienced, and with code that made that fairly obvious, even if it got the job done. With some of the more modern modules, you get something pretty swizzy[1]. :) 1: https://news.ycombinator.com/item?id=11633961 https://news.ycombinator.com/item?id=11633961

by jfoutz 10y ago

defined 10y ago

Please don't remind me. I had to convert the expat XML parser to compile on z/OS and work in EBCDIC, and found that round tripping between ASCII and EBCDIC was sometimes impossible because of the existence of not two, but THREE line terminator characters: CR, LF, and NL (0x85). Not to mention that you cannot test for uppercase or lowercase like the ASCII `ch >= 'A' && ch <= 'Z'` because they are not contiguous in EBCDIC. A good reason to use the C RTL.

osullivj 10y ago
I did the same to enable XML messages to flow over MQ between an RS/6000 based front office FX options system, and a back office S/390 system. IIRC there were six (!!) different EBCDIC codepages that could be in play. I had a code generator that would crank out C or Java bindings that could martial between the expat results and the COBOL data structure. 20 years ago now!
- technofiend 10y ago
  Not sure if you remember but sendmail used to require a certain amount of m4 knowledge and hackery. Emboldened by that and reading the dragon book I was very impressed with myself when I wrote a COBOL parser in a mix of C, lex and YACC that automatically generated the needed 'C' structs and Sybase database layout to load data fed from a System/36. I made the data supplier put his code in the first part of the magtape, read and parsed that and then read the rest of the tape. These days I consider it more of a "what was I thinking" facepalm-worthy sort of thing but at the time I was very proud of it. The "what was I thinking" part is more about the fact that some poor bastard had to come along after me and support that mess.
  - osullivj 10y ago
    Likewise: I regard code generation as a red flag these days. The version skew issues when code generated off slightly different versions of messages are in play can be really nasty. CORBA suffered from that issue big time in the late 90s. And if your generated code uses mutexes in a misguided attempt to be "thread safe", all bets are off...
- defined 10y ago
  Oh yeah, I remember the code pages. Joy. I didn't want to bloat my comment, so thanks for mentioning this!
avar 10y ago
This is how you can implement isUpper() on EBCDIC: https://github.com/Perl/perl5/blob/v5.24.0/handy.h#L1153 https://github.com/Perl/perl5/blob/v5.24.0/handy.h#L1153
- WorldMaker 10y ago
  IIRC, you should be able to do it easier than that: EBCDIC is not contiguous because upper-case versus lower-case is a (shift) bit-flag.
SmellTheGlove 10y ago
> Not to mention that you cannot test for uppercase or lowercase like the ASCII `ch >= 'A' && ch <= 'Z'` because they are not contiguous in EBCDIC. A good reason to use the C RTL. Watch your sorting methods, too. I had a guy over here once totally confused why running his SAS job on the mainframe yielded a different result than the same code running on PC SAS against the same data.

cafard 10y ago

No kidding. Over 20 years ago, I used unpack to read minicomputer log files. It was vastly faster to FTP them over to a Sun Sparcstation and process them with Perl than to use the native log reader. It was also far more flexible.

kbenson 10y ago

> I hesitate to ever recommend perl ... Or, rather, i helped someone much smarter than me do that, 20 years ago. No need to hesitate. It's not half bad for a dynamic language if you keep a little discipline. 20 years ago the average Perl programmer was probably akin to the average PHP programmer from 10 years ago. That is, not very experienced, and with code that made that fairly obvious, even if it got the job done. With some of the more modern modules, you get something pretty swizzy[1]. :) 1: https://news.ycombinator.com/item?id=11633961 https://news.ycombinator.com/item?id=11633961