4 ms·

> Numbers uses a proprietary, compressed binary format to store its tables. This format is comprised of a zip file containing images, as well as Snappy-compress

> Numbers uses a proprietary, compressed binary format to store its tables. This format is comprised of a zip file containing images, as well as Snappy-compressed Protobuf .iwa files containing metadata, text, and all other definitions used in the spreadsheet. I've seen this elsewhere in the wild. E.g. HDFS namenode file system image. But it seems like such a strange choice for a spreadsheet. Why not throw it into sqlite?

jolux 5y ago
The format is probably one step above serializing internal data structures and writing them out to disk and they haven't bothered to specialize it in the framework they use to develop these apps.
adolph 5y ago
My guess would be that SQLite is great for record/rows but not so good for bulk (de)serialization, as the sibling comment notes. A better flexible format nowadays might be Apache Arrow for the task at hand. Given Joel Spolsky’s stories of weird Excel corner cases the Numbers team had their hands full making redundant software somehow different and yet remain grounded in something akin to printed sheets of paper. https://www.joelonsoftware.com/2006/06/16/my-first-billg-review/ https://www.joelonsoftware.com/2006/06/16/my-first-billg-rev...
- nerdponx 5y ago
  If you don't work with spreadsheets often, you'd be surprised at how often spreadsheets deviate from the strict row/column format. SQLite actually sounds like a bad idea, unless you use it as a key-value store for cells.
- fnord123 5y ago
  sqlite is great for bulk (de)serialization. But it helps if you can setup the queries to pull all the relevant data at once. If you have a deep formula hierarchy it might be difficult to do this naively and get performance. If you're loading an entire sheet, I guarantee sqlite will be faster than deserializing protobuf.
dagmx 5y ago
It makes more sense if you consider the file format for iWorks used Document Packages ( https://developer.apple.com/library/archive/documentation/CoreFoundation/Conceptual/CFBundles/DocumentPackages/DocumentPackages.html https://developer.apple.com/library/archive/documentation/Co... ) So this single file setup is actually a conversion of how a multi file directory structure would have been. The choice of a document package makes sense when you see how even a spreadsheet might embed media etc... But synchronization applications like Box fail to handle packages even today so moving to a single container file allows keeping a lot of the old parsing code intact while achieving better support for things that prefer single files.
- Telemakhos 5y ago
  > even a spreadsheet might embed media This is not the only thing wrong in the world, but it is one of the things wrong in the world.
  - deleted 5y ago
    [deleted]
  - AlotOfReading 5y ago
    That includes things like graphs and charts, which seem pretty reasonable to have in a spreadsheet.
    - gspr 5y ago
      But surely those can be generated from the spreadsheet data?
      - xxpor 5y ago
        How long has the file format been around? That may not have been reasonable 20 years ago.
  - dagmx 5y ago
    It lets you create presentation materials with your dataset without exporting to a keynote etc.. https://images.app.goo.gl/eigVnbYKQXpyV5JP6 https://images.app.goo.gl/eigVnbYKQXpyV5JP6

jolux 5y ago

The format is probably one step above serializing internal data structures and writing them out to disk and they haven't bothered to specialize it in the framework they use to develop these apps.

adolph 5y ago

My guess would be that SQLite is great for record/rows but not so good for bulk (de)serialization, as the sibling comment notes. A better flexible format nowadays might be Apache Arrow for the task at hand. Given Joel Spolsky’s stories of weird Excel corner cases the Numbers team had their hands full making redundant software somehow different and yet remain grounded in something akin to printed sheets of paper. https://www.joelonsoftware.com/2006/06/16/my-first-billg-review/ https://www.joelonsoftware.com/2006/06/16/my-first-billg-rev...

nerdponx 5y ago
If you don't work with spreadsheets often, you'd be surprised at how often spreadsheets deviate from the strict row/column format. SQLite actually sounds like a bad idea, unless you use it as a key-value store for cells.
fnord123 5y ago
sqlite is great for bulk (de)serialization. But it helps if you can setup the queries to pull all the relevant data at once. If you have a deep formula hierarchy it might be difficult to do this naively and get performance. If you're loading an entire sheet, I guarantee sqlite will be faster than deserializing protobuf.

dagmx 5y ago

It makes more sense if you consider the file format for iWorks used Document Packages ( https://developer.apple.com/library/archive/documentation/CoreFoundation/Conceptual/CFBundles/DocumentPackages/DocumentPackages.html https://developer.apple.com/library/archive/documentation/Co... ) So this single file setup is actually a conversion of how a multi file directory structure would have been. The choice of a document package makes sense when you see how even a spreadsheet might embed media etc... But synchronization applications like Box fail to handle packages even today so moving to a single container file allows keeping a lot of the old parsing code intact while achieving better support for things that prefer single files.

Telemakhos 5y ago
> even a spreadsheet might embed media This is not the only thing wrong in the world, but it is one of the things wrong in the world.
- deleted 5y ago
  [deleted]
- AlotOfReading 5y ago
  That includes things like graphs and charts, which seem pretty reasonable to have in a spreadsheet.
  - gspr 5y ago
    But surely those can be generated from the spreadsheet data?
    - xxpor 5y ago
      How long has the file format been around? That may not have been reasonable 20 years ago.
- dagmx 5y ago
  It lets you create presentation materials with your dataset without exporting to a keynote etc.. https://images.app.goo.gl/eigVnbYKQXpyV5JP6 https://images.app.goo.gl/eigVnbYKQXpyV5JP6