7 ms·

I work with a lot of tape in the oil and gas sector and one of the major issues we see is that industry wide file formats for data exchange are still based on o

I work with a lot of tape in the oil and gas sector and one of the major issues we see is that industry wide file formats for data exchange are still based on old tapes. These formats have fixed block sizes that haven't changed since the 1950's. Modern tapes have exponentially larger capacities and bandwidth but due to the use of these file formats we rarely see 20% of the theoretical throughput on a brand new tape drive and cartridge. The industry has no desire to change the file format so we are stuck in this situation where tape read/write times grow exponentially with every new tape generation.

kstrauser 5y ago
Can you give an example? What could change if we stopped optimizing for legacy stuff?
- dabiged 5y ago
  The main issue is that the standard was written back when file formats and tape formats were essentially the same thing. Data was read from tape, processed and written to a second tape. The memory of machine at the time were of the order of 1 block. Decoupling tape formats from file format effectively resolves the issue, that is: read the tape in 6k blocks, write the file to a modern file system. Set the tape block size to 10mb. Write the file back to tape as a tar. You get the bandwidth limit of the tape drive. The issue is almost all legacy industry software is still designed to read and write directly to tape objects so your tape isn't readable in all the software this data is designed for.
  - skissane 5y ago
    Couldn’t this be addressed with virtual tape devices? The application gets presented with something that is indistinguishable in behaviour from a physical tape drive, but actually backed by a file on SSD or hard disk. Then you can copy that file to a real tape device separately
    - newswasboring 5y ago
      I can't say for oil and gas but I encountered the same thing in semiconductor manufacturing. CMP machines still use tape formats even if the file is now written to different media. The challenge during transition was in proving reliability and transition costs. Sometimes letting engineers deal with the hassle is cheaper than letting them fix it.
    - dabiged 5y ago
      I agree. This would work, but you still need to get the data off the tape, and onto the SSD. The only way to do this is 60MB/sec.
lisper 5y ago
Might it be possible to embed the legacy format witbin a more modern format that gets written to the tape?
tyingq 5y ago
That's interesting. So the overhead to assemble the file is just so much that you can't stream output to the tape fast enough? That seems like something that could be overcome. I get that a better format would make it easier, but it seems like you should be able to, with some optimization or disk cache, stream the output fast enough to outpace the tape drive.
- dabiged 5y ago
  I am talking about bottlenecks on reading from the tape. Sending the data to /dev/null eliminates any write bottlenecks and we still max out at 60mb/sec on a 250mb/sec drive.
  - tyingq 5y ago
    Hrm. That is odd, as I assume these old formats would be something that compresses well, which gets higher read/write speeds. Edit: Hrm. I believe you can disable hardware compression on some LTO drives. Maybe that would help?
    - dabiged 5y ago
      The data is binary formatted floating point data, generally using the IBM floating point format, though IEEE floats are also allowed. Standard compression like gzip basically does nothing. All tape manufacturers claim their hardware based compression gives 2/2.5/3 times the tape capacity and it is all baloney. A 1TB tape holds 1TB of data. There have been multiple attempts over the years to implement all sorts of algorithms but they have all failed to gain traction. For example Wavelet based compression works fantastically well for some data (i.e. real signals with band limited frequencies), and catastrophically badly for others (earth models defined with piecewise functions containing non-differentiable points). Some people are happy with lossy compression, others are not. The only compression I have seen worth anything is in deep water, where there is literally no data in the water column (i.e. 500 consecutive zeroes in a 6000 byte block), gzip gives you the that section free.
macjohnmcc 5y ago
It helps that those tapes can be read in the future easily as well. HP at one point asked the company I worked for at the time if they could use their tape copy program with their new drives so that customers could copy older DAT tapes to a newer version so they didn't have to support older media formats in their newer drives. The newer media is typically on a new very fast tape drive and sometimes it is difficult to saturate those anyway and keep stopping and starting and having to reposition over and over.

kstrauser 5y ago

Can you give an example? What could change if we stopped optimizing for legacy stuff?

dabiged 5y ago
The main issue is that the standard was written back when file formats and tape formats were essentially the same thing. Data was read from tape, processed and written to a second tape. The memory of machine at the time were of the order of 1 block. Decoupling tape formats from file format effectively resolves the issue, that is: read the tape in 6k blocks, write the file to a modern file system. Set the tape block size to 10mb. Write the file back to tape as a tar. You get the bandwidth limit of the tape drive. The issue is almost all legacy industry software is still designed to read and write directly to tape objects so your tape isn't readable in all the software this data is designed for.
- skissane 5y ago
  Couldn’t this be addressed with virtual tape devices? The application gets presented with something that is indistinguishable in behaviour from a physical tape drive, but actually backed by a file on SSD or hard disk. Then you can copy that file to a real tape device separately
  - newswasboring 5y ago
    I can't say for oil and gas but I encountered the same thing in semiconductor manufacturing. CMP machines still use tape formats even if the file is now written to different media. The challenge during transition was in proving reliability and transition costs. Sometimes letting engineers deal with the hassle is cheaper than letting them fix it.
  - dabiged 5y ago
    I agree. This would work, but you still need to get the data off the tape, and onto the SSD. The only way to do this is 60MB/sec.

lisper 5y ago

Might it be possible to embed the legacy format witbin a more modern format that gets written to the tape?

tyingq 5y ago

That's interesting. So the overhead to assemble the file is just so much that you can't stream output to the tape fast enough? That seems like something that could be overcome. I get that a better format would make it easier, but it seems like you should be able to, with some optimization or disk cache, stream the output fast enough to outpace the tape drive.

dabiged 5y ago
I am talking about bottlenecks on reading from the tape. Sending the data to /dev/null eliminates any write bottlenecks and we still max out at 60mb/sec on a 250mb/sec drive.
- tyingq 5y ago
  Hrm. That is odd, as I assume these old formats would be something that compresses well, which gets higher read/write speeds. Edit: Hrm. I believe you can disable hardware compression on some LTO drives. Maybe that would help?
  - dabiged 5y ago
    The data is binary formatted floating point data, generally using the IBM floating point format, though IEEE floats are also allowed. Standard compression like gzip basically does nothing. All tape manufacturers claim their hardware based compression gives 2/2.5/3 times the tape capacity and it is all baloney. A 1TB tape holds 1TB of data. There have been multiple attempts over the years to implement all sorts of algorithms but they have all failed to gain traction. For example Wavelet based compression works fantastically well for some data (i.e. real signals with band limited frequencies), and catastrophically badly for others (earth models defined with piecewise functions containing non-differentiable points). Some people are happy with lossy compression, others are not. The only compression I have seen worth anything is in deep water, where there is literally no data in the water column (i.e. 500 consecutive zeroes in a 6000 byte block), gzip gives you the that section free.

macjohnmcc 5y ago

It helps that those tapes can be read in the future easily as well. HP at one point asked the company I worked for at the time if they could use their tape copy program with their new drives so that customers could copy older DAT tapes to a newer version so they didn't have to support older media formats in their newer drives. The newer media is typically on a new very fast tape drive and sometimes it is difficult to saturate those anyway and keep stopping and starting and having to reposition over and over.