3 ms·
Am I missing something? That doesn't seem all that odd, but perhaps that's because I'm familiar with IP,TCP and UDP and I've written something to read PNG head
by kbenson 10y ago
Am I missing something? That doesn't seem all that odd, but perhaps that's because I'm familiar with IP,TCP and UDP and I've written something to read PNG headers before. Interchange file formats often include variable length blocks, which a part of the header defining the block length. That first part of the record is just a very small header per item, which defines the type of item and the size. Look into the deflate algorithm, or tar file format, or any number of on-disk storage formats and you'll find they all (well, definitely most) do this, because it's the most efficient way.
Fixed length records are less common tor interchange formats, but likely you use them every day anyway. That's what databases often use, and the reason is because it makes it very efficient to index intro the structure and get whole records (know your data is sorted? Binary search is possible and easy then). Sometimes it's just the indices that are stored this way (they essentially have to be), but you can get fairly efficient table access out of some engines without indexing everything if all the records are fixed and the engine can determine that.
If you generally don't program in a low level language, this is generally abstracted away by some library that is written in that language. People do't usually write PNG and JPEG libraries in pure Python or Ruby (or at least, they don't expect them to be used much in production), they write a shim that wraps libpng or libjpeg.
- tyingq 10y ago>Am I missing something? The interesting part is that the fixed and variable record formats are first class things on the mainframe. So, something like DB2 on a mainframe can use system supplied functionality (VSAM) as their storage engine. As opposed to unix, where higher level databases like MySql, CockroachDB, etc either roll their own (InnoDB) or use some 3rd party offering like RocksDB, LevelDB, etc. VSAM isn't just one thing either...it supports k/v indexing, or indexing via relative byte address, or indexing via record number, etc. So, basically, when you talk with mainframe people about interchanging data, they don't tend to consider that you might actually have to write some code to parse what they are sending you. They tend to assume you already have utilties that understand these things. It's not an interchange format...it's the native format for them. I suppose the answer seemed novel because it's speaking with very specific "official sounding" terminology about something that's usually ad-hoc negotiated by project in the unix world.
- kbenson 10y agoSo I guess it's fair to say the difference is that the mainframes have a standardized format for record creation and consumption in the OS, sort of like DB2 being included in the kernel? That is nice, and it does explain why it might be confusing, even if I think it doesn't necessarily get you much over using a third party library (unless there are other benefits I'm not considering).
- tyingq 10y agoIt's less of a difference now, and somewhat unrelated to the specific topic, but... A big historical difference with mainframes and data was the architecture around I/O. They always had separate processors to offload I/O, and I/O was always asynchronous. And things like VSAM were highly tuned to take advantage of that. That's why mainframes continued to outpace Linux/X86 for some types of workloads...even after X86 performance far outpaced the main processors in a mainframe. I believe that advantage is completely gone now, but mostly via brute force vs elegance. Commodity hardware is just so fast now.
- kbenson 10y agoSure, I wasn't trying to indicate there was no reason or benefit to mainframes, just to summarize the situation to make sure I understood it correctly. It does make sense to have an integrated library for advanced file access if you have dedicated IO hardware. That prevents a lot of misconfiguration of libraries that might try unsuccessfully use that system, if they even support it at all (i.e. OpenSSL and crypto hardware such as dedicated AES hardware as in the Via mini-ITX platforms of yesteryear).
- BillWoodger 10y agoCorrect about the I/O. You can let the space-bar auto-repeat 1919 times, for instance (nearest equivalent to circling the mouse) and the CPU cost is... zero. When, exactly, do you think that the X86 surpassed the Mainframe processors, and in what particular way? Current generation (expect a new one this year) is 5Ghz (actually slower than the previous) and has lots of stuff. A fully-loaded box has a theoretical throughput of 30bn (yes, billion) RESTful transactions per day. And if that isn't enough power, you can hang another 31 boxes onto it and treat them as one.