3 ms·

I wonder what happens if you set the "Expiration Date" on a data set... Does it delete itself? Require special flags to access? Throw an error every time the da

I wonder what happens if you set the "Expiration Date" on a data set... Does it delete itself? Require special flags to access? Throw an error every time the data is accessed? Prevent writing? Can you imagine someone putting in an expiration date a decade in the future and it goes unnoticed until it hits? And presumably, like the other options, it can't be modified after you create the data set. What a truly frightening option to have on a server file system.

skissane 2y ago
> I wonder what happens if you set the "Expiration Date" on a data set... Does it delete itself? On a vanilla z/OS install, setting "Expiration Date" on a disk (DASD) data set does... nothing. It just acts as documentation. And it is normally an optional field which for disk data sets normally doesn't get set – although a minority of sites configure it to be mandatory. Now, if you enable DFSMShsm – which is an extra cost add-on which provides hierarchical storage management (silently moving rarely used files from disk to tape or cloud storage, and then moving them back on demand when an application attempts to read them) – well, by default DFSMShsm doesn't do anything differently, but then you set "SETSYS EXPIREDDATASETS(SCRATCH)", and it will delete ("scratch") expired disk datasets automatically in the background. Also, there is a distinction between SMS-managed and non-SMS-managed datasets. SMS (aka DFSMS, or to be more specific DFSMSdfp, originally DFP) started life as an add-on product to simplify some of the complexities of mainframe storage management, and eventually went from being an add-on to being included in the base OS. When you create a dataset, you can choose whether to put it under SMS management or not; if it is under SMS management, you have less control over it, because some decisions are based on storage policies configured by the storage administrator. In particular, SMS has a config setting OVRD_EXPDT which controls whether you are allowed to delete a dataset with an expiry date prior to its expiry date being reached. If you have OVRD_EXPDT(NO), then if an SMS-managed dataset has a future expiry date, your request to delete it will be ignored. Now, it is much more common to set the expiry date on tape datasets. For datasets on tape, there is both a volume expiry date set in the volume header, and dataset/file expiry dates set in the dataset/file headers. (People say that the correct mainframe term is "dataset" not "file"–that's true on disk, ignoring the Unix subsystem, but on tape, both "dataset" and "file" are actually correct.) There is another extra-cost add-on called DFSMSrmm. And if you use that, that will automatically set the tape volume expiry date based on the dataset expiry date (for multi-dataset tapes, DFSMSrmm can be configured to use either the latest expiry date of all the datasets/files, or else in "FIRSTFILE" mode, it just uses the expiry date of the first dataset/file on the tape.) As well as multi-dataset tape volumes, you can also have multi-volume tape datasets (one dataset spans multiple tape volumes), and DFSMSrmm has some more config options to control how volume expiry dates are handled in that scenario. And then there are third party packages you can use instead of DFSMSrmm, such as Broadcom CA-1 or Broadcom TLMS. Now, once your tape data set expiry date has been translated into a tape volume expiry date, that will go into the tape catalogue used by your tape library. And when the tape volume reaches its expiration date, the tape library management software may (again, depending on its configuration) mark it as a free tape, and then it may be overwritten by new data at any time. Or, maybe you are using a virtual tape library, where the tapes are actually files on a disk array, but the virtual tape library pretends to the mainframe to be an actual physical tape library – in which case the virtual tape library software may (again, configuration dependent) delete the file backing your tape volume once its expiry date is reached. > Can you imagine someone putting in an expiration date a decade in the future and it goes unnoticed until it hits? Since the expiry date is optional, it is commonly not actually set, except on temporary files, backup files, archival files, log files, etc. You can configure it to be mandatory or auto-populated, but you'd generally only do that for files/datasets whose names match certain patterns. Most sites, source code files, application binaries, configuration files, operating system files, etc, the expiry date is blank. > And presumably, like the other options, it can't be modified after you create the data set. Not true, it can be. Although there are configuration options which control whether you are allowed to do that or not. A common configuration is that you can increase the expiration date of one of your own datasets, but only a privileged user can reduce it (bringing the expiry date forward). > What a truly frightening option to have on a server file system. HSM systems exist on Linux/Unix/Windows too, and most HSM systems have a file expiration date feature. And you'll find similar features in tape management systems, record/content/document management systems, email archiving systems, e-discovery systems, etc, all of which exist on those platforms too. The difference is, that on mainframes and minicomputers (not just IBM), "expiry date" has commonly been a standard filesystem attribute – most commonly the system by default doesn't do anything with it, and you need add-on software to actually delete the expired files (if you really want to), but it is part of the core OS filesystem metadata. By contrast, on Linux/Unix/Windows, it isn't part of the core OS filesystem metadata, so even if you are using a HSM system and your file has an expiry date set, you'll need to use some proprietary API or non-standard extended attribute to get at it. That's the only real difference.
- vintagedave 2y ago
  This was fascinating. Thankyou for taking the time to write it. That said: > DFSMSrmm ... DFSMShsm ... DFSMSdfp If rmm, hsm, and dfp mean something, and IBM's documentation didn't explain the names to me. they are some of the most extraordinary acronyms, _mixed-case_ no less, I've ever seen.
  - wsh 2y ago
    DFSMS: Data Facility Storage Management Subsystem DFSMSdfp: Data Facility Product DFSMSdss: Data Set Services DFSMSrmm: Removable Media Manager DFSMShsm: Hierarchical Storage Manager DFSMStvs: Transactional VSAM Services DFSMSopt: Optimizer See ABCs of z/OS System Programming Volume 3 (SG24-6983), https://www.redbooks.ibm.com/redbooks/pdfs/sg246983.pdf https://www.redbooks.ibm.com/redbooks/pdfs/sg246983.pdf.
  - tiberious726 2y ago
    Welcome to IBM. Each new acronym is an extra fee, annually.
neverartful 2y ago
I can imagine where it would be really nice to have an expiration date on a filesystem. The behavior I would want to see for expired files would be: to show up on a report and maybe display in different color in directory listing. It could be a useful reminder to update an expiring certificate, expiring customer contract, etc.
cwbriscoe 2y ago
It is rarely used now days in my experience and I have never used a flat file that is over a decade old. Storage used to be expensive so this would ensure that old data got purged. Most flat files are useless after a month or less. Even if a mistake did happen, most MVS shops have pretty good backups. Also now days, data files that haven't been accessed for some period of time usually gets sent to tape/cartridge in some automated system like this: https://en.wikipedia.org/wiki/Tape_library#/media/File:StorageTek_Powderhorn_tape_library.jpg https://en.wikipedia.org/wiki/Tape_library#/media/File:Stora...

skissane 2y ago

> I wonder what happens if you set the "Expiration Date" on a data set... Does it delete itself? On a vanilla z/OS install, setting "Expiration Date" on a disk (DASD) data set does... nothing. It just acts as documentation. And it is normally an optional field which for disk data sets normally doesn't get set – although a minority of sites configure it to be mandatory. Now, if you enable DFSMShsm – which is an extra cost add-on which provides hierarchical storage management (silently moving rarely used files from disk to tape or cloud storage, and then moving them back on demand when an application attempts to read them) – well, by default DFSMShsm doesn't do anything differently, but then you set "SETSYS EXPIREDDATASETS(SCRATCH)", and it will delete ("scratch") expired disk datasets automatically in the background. Also, there is a distinction between SMS-managed and non-SMS-managed datasets. SMS (aka DFSMS, or to be more specific DFSMSdfp, originally DFP) started life as an add-on product to simplify some of the complexities of mainframe storage management, and eventually went from being an add-on to being included in the base OS. When you create a dataset, you can choose whether to put it under SMS management or not; if it is under SMS management, you have less control over it, because some decisions are based on storage policies configured by the storage administrator. In particular, SMS has a config setting OVRD_EXPDT which controls whether you are allowed to delete a dataset with an expiry date prior to its expiry date being reached. If you have OVRD_EXPDT(NO), then if an SMS-managed dataset has a future expiry date, your request to delete it will be ignored. Now, it is much more common to set the expiry date on tape datasets. For datasets on tape, there is both a volume expiry date set in the volume header, and dataset/file expiry dates set in the dataset/file headers. (People say that the correct mainframe term is "dataset" not "file"–that's true on disk, ignoring the Unix subsystem, but on tape, both "dataset" and "file" are actually correct.) There is another extra-cost add-on called DFSMSrmm. And if you use that, that will automatically set the tape volume expiry date based on the dataset expiry date (for multi-dataset tapes, DFSMSrmm can be configured to use either the latest expiry date of all the datasets/files, or else in "FIRSTFILE" mode, it just uses the expiry date of the first dataset/file on the tape.) As well as multi-dataset tape volumes, you can also have multi-volume tape datasets (one dataset spans multiple tape volumes), and DFSMSrmm has some more config options to control how volume expiry dates are handled in that scenario. And then there are third party packages you can use instead of DFSMSrmm, such as Broadcom CA-1 or Broadcom TLMS. Now, once your tape data set expiry date has been translated into a tape volume expiry date, that will go into the tape catalogue used by your tape library. And when the tape volume reaches its expiration date, the tape library management software may (again, depending on its configuration) mark it as a free tape, and then it may be overwritten by new data at any time. Or, maybe you are using a virtual tape library, where the tapes are actually files on a disk array, but the virtual tape library pretends to the mainframe to be an actual physical tape library – in which case the virtual tape library software may (again, configuration dependent) delete the file backing your tape volume once its expiry date is reached. > Can you imagine someone putting in an expiration date a decade in the future and it goes unnoticed until it hits? Since the expiry date is optional, it is commonly not actually set, except on temporary files, backup files, archival files, log files, etc. You can configure it to be mandatory or auto-populated, but you'd generally only do that for files/datasets whose names match certain patterns. Most sites, source code files, application binaries, configuration files, operating system files, etc, the expiry date is blank. > And presumably, like the other options, it can't be modified after you create the data set. Not true, it can be. Although there are configuration options which control whether you are allowed to do that or not. A common configuration is that you can increase the expiration date of one of your own datasets, but only a privileged user can reduce it (bringing the expiry date forward). > What a truly frightening option to have on a server file system. HSM systems exist on Linux/Unix/Windows too, and most HSM systems have a file expiration date feature. And you'll find similar features in tape management systems, record/content/document management systems, email archiving systems, e-discovery systems, etc, all of which exist on those platforms too. The difference is, that on mainframes and minicomputers (not just IBM), "expiry date" has commonly been a standard filesystem attribute – most commonly the system by default doesn't do anything with it, and you need add-on software to actually delete the expired files (if you really want to), but it is part of the core OS filesystem metadata. By contrast, on Linux/Unix/Windows, it isn't part of the core OS filesystem metadata, so even if you are using a HSM system and your file has an expiry date set, you'll need to use some proprietary API or non-standard extended attribute to get at it. That's the only real difference.

vintagedave 2y ago
This was fascinating. Thankyou for taking the time to write it. That said: > DFSMSrmm ... DFSMShsm ... DFSMSdfp If rmm, hsm, and dfp mean something, and IBM's documentation didn't explain the names to me. they are some of the most extraordinary acronyms, _mixed-case_ no less, I've ever seen.
- wsh 2y ago
  DFSMS: Data Facility Storage Management Subsystem DFSMSdfp: Data Facility Product DFSMSdss: Data Set Services DFSMSrmm: Removable Media Manager DFSMShsm: Hierarchical Storage Manager DFSMStvs: Transactional VSAM Services DFSMSopt: Optimizer See ABCs of z/OS System Programming Volume 3 (SG24-6983), https://www.redbooks.ibm.com/redbooks/pdfs/sg246983.pdf https://www.redbooks.ibm.com/redbooks/pdfs/sg246983.pdf.
- tiberious726 2y ago
  Welcome to IBM. Each new acronym is an extra fee, annually.

neverartful 2y ago

I can imagine where it would be really nice to have an expiration date on a filesystem. The behavior I would want to see for expired files would be: to show up on a report and maybe display in different color in directory listing. It could be a useful reminder to update an expiring certificate, expiring customer contract, etc.

cwbriscoe 2y ago

It is rarely used now days in my experience and I have never used a flat file that is over a decade old. Storage used to be expensive so this would ensure that old data got purged. Most flat files are useless after a month or less. Even if a mistake did happen, most MVS shops have pretty good backups. Also now days, data files that haven't been accessed for some period of time usually gets sent to tape/cartridge in some automated system like this: https://en.wikipedia.org/wiki/Tape_library#/media/File:StorageTek_Powderhorn_tape_library.jpg https://en.wikipedia.org/wiki/Tape_library#/media/File:Stora...