3 ms·

If you have to regularly transfer large amounts of data over a network, it might be worth looking into a wan optimization product like Riverbed's Steelhead, Sil

by joshAg 12y ago

If you have to regularly transfer large amounts of data over a network, it might be worth looking into a wan optimization product like Riverbed's Steelhead, Silverpeak's VX/NX lines, or Bluecoat Mach 5, or one of the other vendors' solutions. Yeah, you could try and roll it yourself, since really it just comes down to compressing and deduplicating what you send over the wire, but doing that well and also making it simple to use is not a trivial problem. Why reinvent the wheel badly?

epistasis 12y ago
There's not many situations where these types of products help, in my experience. Especially for the type of data that's going to be transferred between UCI and the Broad. Enterprise compute data, cached webpages, etc., may have a good amount of deduplication capacity. But for actual "data," being measurements etc, these products will achieve nothing. The data itself almost never has any duplicated chunks, and if there are petabytes of data, it's almost certainly stored in some sort of compressed format already.
- semi-extrinsic 12y ago
  We had to explain this repeatedly to several vendors the last time we were buying a small-ish (30 TB) file server. They seemed very skeptical of this concept that we were storing lots of data in compressed binary formats.
  - epistasis 12y ago
    I think it's one of those situations where for most vendor's customers, buying more hardware is far cheaper than hiring smart programmers. But for academic situations, there's a surplus of clever programmers with low wages, and not nearly enough money for hardware. So in "enterprise" the solution is to shove everything into SQL databases and just buy a ton more compute and disk to manage the extra inefficiencies, whereas academic situations have not had that luxury. As data science progresses, the amount of enterprisey large data situations will also decrease, I think.
- joshAg 12y ago
  Yeah, they definitely won't do well over non-compressible non-repetitive data, but they can help for situations where the data isn't compressed at rest or where data is repetitive. Like I said, for many technical people you can roll your own and get reasonably close, but not everyone or every application that deals with large amounts of data fits that mold.

by joshAg 12y ago

epistasis 12y ago

There's not many situations where these types of products help, in my experience. Especially for the type of data that's going to be transferred between UCI and the Broad. Enterprise compute data, cached webpages, etc., may have a good amount of deduplication capacity. But for actual "data," being measurements etc, these products will achieve nothing. The data itself almost never has any duplicated chunks, and if there are petabytes of data, it's almost certainly stored in some sort of compressed format already.

semi-extrinsic 12y ago
We had to explain this repeatedly to several vendors the last time we were buying a small-ish (30 TB) file server. They seemed very skeptical of this concept that we were storing lots of data in compressed binary formats.
- epistasis 12y ago
  I think it's one of those situations where for most vendor's customers, buying more hardware is far cheaper than hiring smart programmers. But for academic situations, there's a surplus of clever programmers with low wages, and not nearly enough money for hardware. So in "enterprise" the solution is to shove everything into SQL databases and just buy a ton more compute and disk to manage the extra inefficiencies, whereas academic situations have not had that luxury. As data science progresses, the amount of enterprisey large data situations will also decrease, I think.
joshAg 12y ago
Yeah, they definitely won't do well over non-compressible non-repetitive data, but they can help for situations where the data isn't compressed at rest or where data is repetitive. Like I said, for many technical people you can roll your own and get reasonably close, but not everyone or every application that deals with large amounts of data fits that mold.