5 ms·

I am talking about bottlenecks on reading from the tape. Sending the data to /dev/null eliminates any write bottlenecks and we still max out at 60mb/sec on a 25

• tyingq 5y ago
  Hrm. That is odd, as I assume these old formats would be something that compresses well, which gets higher read/write speeds. Edit: Hrm. I believe you can disable hardware compression on some LTO drives. Maybe that would help?
  • dabiged 5y ago
    The data is binary formatted floating point data, generally using the IBM floating point format, though IEEE floats are also allowed. Standard compression like gzip basically does nothing. All tape manufacturers claim their hardware based compression gives 2/2.5/3 times the tape capacity and it is all baloney. A 1TB tape holds 1TB of data. There have been multiple attempts over the years to implement all sorts of algorithms but they have all failed to gain traction. For example Wavelet based compression works fantastically well for some data (i.e. real signals with band limited frequencies), and catastrophically badly for others (earth models defined with piecewise functions containing non-differentiable points). Some people are happy with lossy compression, others are not. The only compression I have seen worth anything is in deep water, where there is literally no data in the water column (i.e. 500 consecutive zeroes in a 6000 byte block), gzip gives you the that section free.