4 ms·

You're blaming tools that don't support the full featureset of one of your platforms. VMS's ODS2 filesystem evidently has concept of files that are grouped in

by canucker2016 3y ago

You're blaming tools that don't support the full featureset of one of your platforms. VMS's ODS2 filesystem evidently has concept of files that are grouped in records. The ftp program that you use on VMS wasn't designed to support sending that info to a destination filesystem. Probably because the original Unix filesystem only supported the idea of one stream of bytes for a file. You'd need an FTP server on the destination side to support records seamlessly if you wanted full fidelity. That's why NTFS supports alternate data streams - the MacOS HFS filesystem supported data and resource forks for a file. An NTFS alternate datastream was where the resource fork was placed so the NTFS user would only see one file from the CLI and the GUI and the builtin copy tools would copy all streams. deleting such a file would remove both streams on NTFS.

chasil 3y ago
Actually, Multinet FTP does have a couple of switches for "VMS mode," but SFTP2 got a working solution in front of us with less reading of the manuals. POSIX was wise in having absolutely none of this. Would you rather Apple had won over this standard and we had resource forks everywhere? I understand that the largest use of alternate data streams at times in the past was malware, to evade virus scanners, which seems to be an anti-feature to me. A file should be a stream of bytes that can at least survive transfer over a socket (that is 8-bit clean), unlike this ODS-2 detritus.
- tedunangst 3y ago
  You know that e.g. linux supports extended attributes and they don't copy so easily over ftp either?
  - chasil 3y ago
    Yes, I know that ACLs, security contexts, and zfs/btrfs parameters aren't going to move. This would be know as "filesystem metadata," and the send/receive operations on zfs/btrfs would be the best bet to preserve them. However, my ODS-2 problem was not metadata, it was data that lacked esoteric formatting. I certainly don't need that for Oracle RMAN - an FTP binary transfer will always works.
    - tambourine_man 3y ago
      At some point you start wondering what's meta and what's data, after all.
      - chasil 3y ago
        Everything in the stat() structure is metadata. We could start there.
        tambourine_man 3y ago
        Why do we consider creation date to be metadata, for instance? In photos and audio files, it's so essencial that it is stored in the data stream, even though we call it metadata. It's a fuzzy line and everyone is probably a bit discontent to where it's drawn.
- canucker2016 3y ago
  I don't care who wins - I don't have a horse in this race at all. I'm sure the FTP devs were just trying to get something working - they weren't worried about supporting all the possibilities for the definition of "file". I'd guess the only people who really care are those affected (mainly VMS/Apple people who have to send/receive files via FTP). Well, I guess if I was trying to FTP a file that had data in an alternate data stream (ADS) from an NTFS drive, I'd care. But then I know that the ADS wouldn't be sent by FTP and package the file accordingly. I don't think POSIX designed anything - they just codified/standardized what was running in the wild. Wouldn't something like the VMS-equivalent of a tar archive of your files work better for round-tripping to a non-VMS computer? The largest use of alternate data streams is actually when a user downloads a file from the Internet via Internet Explorer/Chrome/Firefox on WinXP+. The browser creates an alternate data stream, Zone.Identifier, for the downloaded file so Windows will know that the file may be tainted. see https://guyrleech.wordpress.com/2018/08/06/where-did-that-download-come-from/ https://guyrleech.wordpress.com/2018/08/06/where-did-that-do...
MikeTheGreat 3y ago
Ok, now I'm curious :) Can I ask why a 'single binary stream' view of a file would cause problems when dealing with something from a 'file is actually a bunch of records' system? Like, if the FTP program sends an identical sequence of bits to the destination machine, how could the destination have problems deciphering the (identical) file? I don't have any experience with VMS, but I'm assuming that there's information in the file (somehow, somewhere) that tells you where the records start and stop (whether it's a terminating character or a set size of each record, etc) because otherwise how would VMS know where to find each record. Is this not true? Is there information about the file that's stored outside the file?
- chasil 3y ago
  I am guessing that there is some interior "magic" sequence, similar to the UNIX magic in the "file" utility, that inserts some kind of padding. That is a "wild assumptive guess."
- ale42 3y ago
  As far as I remember when I used VMS while at the high school, if you write a text file in VMS and then read it byte-by-byte as binary, well, there are no line separators at all. Each line is seemingly a record in the file. I don't know more about the details, but the information is not in the byte stream of the file, it's probably some metadata elsewhere in the filesystem. Other OSs don't work that way. An FTP transfer in ASCII mode will insert line breaks where appropriate while downloading from VMS, but other file types relying on records can't be reproduced outside VMS as simply.

chasil 3y ago

Actually, Multinet FTP does have a couple of switches for "VMS mode," but SFTP2 got a working solution in front of us with less reading of the manuals. POSIX was wise in having absolutely none of this. Would you rather Apple had won over this standard and we had resource forks everywhere? I understand that the largest use of alternate data streams at times in the past was malware, to evade virus scanners, which seems to be an anti-feature to me. A file should be a stream of bytes that can at least survive transfer over a socket (that is 8-bit clean), unlike this ODS-2 detritus.

tedunangst 3y ago
You know that e.g. linux supports extended attributes and they don't copy so easily over ftp either?
- chasil 3y ago
  Yes, I know that ACLs, security contexts, and zfs/btrfs parameters aren't going to move. This would be know as "filesystem metadata," and the send/receive operations on zfs/btrfs would be the best bet to preserve them. However, my ODS-2 problem was not metadata, it was data that lacked esoteric formatting. I certainly don't need that for Oracle RMAN - an FTP binary transfer will always works.
  - tambourine_man 3y ago
    At some point you start wondering what's meta and what's data, after all.
    - chasil 3y ago
      Everything in the stat() structure is metadata. We could start there.
      - tambourine_man 3y ago
        Why do we consider creation date to be metadata, for instance? In photos and audio files, it's so essencial that it is stored in the data stream, even though we call it metadata. It's a fuzzy line and everyone is probably a bit discontent to where it's drawn.
canucker2016 3y ago
I don't care who wins - I don't have a horse in this race at all. I'm sure the FTP devs were just trying to get something working - they weren't worried about supporting all the possibilities for the definition of "file". I'd guess the only people who really care are those affected (mainly VMS/Apple people who have to send/receive files via FTP). Well, I guess if I was trying to FTP a file that had data in an alternate data stream (ADS) from an NTFS drive, I'd care. But then I know that the ADS wouldn't be sent by FTP and package the file accordingly. I don't think POSIX designed anything - they just codified/standardized what was running in the wild. Wouldn't something like the VMS-equivalent of a tar archive of your files work better for round-tripping to a non-VMS computer? The largest use of alternate data streams is actually when a user downloads a file from the Internet via Internet Explorer/Chrome/Firefox on WinXP+. The browser creates an alternate data stream, Zone.Identifier, for the downloaded file so Windows will know that the file may be tainted. see https://guyrleech.wordpress.com/2018/08/06/where-did-that-download-come-from/ https://guyrleech.wordpress.com/2018/08/06/where-did-that-do...

MikeTheGreat 3y ago

Ok, now I'm curious :) Can I ask why a 'single binary stream' view of a file would cause problems when dealing with something from a 'file is actually a bunch of records' system? Like, if the FTP program sends an identical sequence of bits to the destination machine, how could the destination have problems deciphering the (identical) file? I don't have any experience with VMS, but I'm assuming that there's information in the file (somehow, somewhere) that tells you where the records start and stop (whether it's a terminating character or a set size of each record, etc) because otherwise how would VMS know where to find each record. Is this not true? Is there information about the file that's stored outside the file?

chasil 3y ago
I am guessing that there is some interior "magic" sequence, similar to the UNIX magic in the "file" utility, that inserts some kind of padding. That is a "wild assumptive guess."
ale42 3y ago
As far as I remember when I used VMS while at the high school, if you write a text file in VMS and then read it byte-by-byte as binary, well, there are no line separators at all. Each line is seemingly a record in the file. I don't know more about the details, but the information is not in the byte stream of the file, it's probably some metadata elsewhere in the filesystem. Other OSs don't work that way. An FTP transfer in ASCII mode will insert line breaks where appropriate while downloading from VMS, but other file types relying on records can't be reproduced outside VMS as simply.