4 ms·

Writing your own methods to retrieve arbitrary chunks of data from a monolithic file would be a lot of work. Using map tiles as the example, how do you easily r

Writing your own methods to retrieve arbitrary chunks of data from a monolithic file would be a lot of work. Using map tiles as the example, how do you easily retrieve just the tiles in a specific region? Ok, how about all tiles that have the “hasLand” attribute set? Or the “containsCoastline” attribute? You’d end up rewriting your own version of a database.

akira2501 9y ago
> You’d end up rewriting your own version of a database. This is the hell we just crawled out of in one of our codebases.. it's not so much the access for a single process, but when you want to share the data in the file, that's when it seriously starts to get messy and breaking through otherwise clean abstraction layers and/or forcing recompile on compatibility breaking changes.
- adventured 9y ago
  If you don't mind answering, out of sheer curiosity how was the data stored in the individual file/s?
  - akira2501 9y ago
    Straight array of structs. The complication was that there were several of them, they were stored in bus-attached MRAM which was accessed directly (e.g. mmap of /dev/ram) by several independent processes. The idea came from previous generations of the product which generally ran on microcontrollers coupled with custom FPGA designs, so, not entirely unwarranted.. but completely incorrect for a newer product with much greater computing power than their previous devices.
inetknght 9y ago
> Writing your own methods to retrieve arbitrary chunks of data from a monolithic file would be a lot of work. Oh goodness have we come a long way if interacting with a file is harder than interacting with a SQL database. (it really depends on what language and/or library(ies) you use and for what purpose)
- function_seven 9y ago
  As far as I know, querying a database has always been easier than querying a custom file format. It's the whole reason databases exist in the first place. And using one off the shelf is clearly easier than writing your own.
  - tetromino_ 9y ago
    Not easier, but vastly more flexible. Sure, it's completely trivial to read a file containing an array of C structs of metadata and offsets into a blob of concatenated PNGs: just mmap the thing, and your main data structure is sitting right there in memory. But problems start arising when you need to update the structure of that metadata. Add some more fields. Some more structured, multi-value fields, referring to multiple entities in your increasingly complex data structure. Entities which must be guaranteed to exist. And perform modifications concurrently, from multiple processes. And have the file not get corrupted when something kills a writer process in the middle of writing... Databases are slow and overkill for the simple case, but they certainly make adding unexpected requested features much easier. And it can be expected that you will get unexpected feature requests.
- imron 9y ago
  > Oh goodness have we come a long way if interacting with a file is harder than interacting with a SQL database. It's not interacting with a file, it's interacting with a file that contains other files of varying sizes that need to be accessed randomly with good performance.
  - diroussel 9y ago
    If you only want to read then both zip and tar are good for this purpose. It's commonly used and known to work well. There are many formats that are really just a renamed zip file. Like jar, and epub. File entries don't have to be compressed so you can mmap. If you want to do random writes then more consideration is needed.
    - 9y ago

akira2501 9y ago

> You’d end up rewriting your own version of a database. This is the hell we just crawled out of in one of our codebases.. it's not so much the access for a single process, but when you want to share the data in the file, that's when it seriously starts to get messy and breaking through otherwise clean abstraction layers and/or forcing recompile on compatibility breaking changes.

adventured 9y ago
If you don't mind answering, out of sheer curiosity how was the data stored in the individual file/s?
- akira2501 9y ago
  Straight array of structs. The complication was that there were several of them, they were stored in bus-attached MRAM which was accessed directly (e.g. mmap of /dev/ram) by several independent processes. The idea came from previous generations of the product which generally ran on microcontrollers coupled with custom FPGA designs, so, not entirely unwarranted.. but completely incorrect for a newer product with much greater computing power than their previous devices.

inetknght 9y ago

> Writing your own methods to retrieve arbitrary chunks of data from a monolithic file would be a lot of work. Oh goodness have we come a long way if interacting with a file is harder than interacting with a SQL database. (it really depends on what language and/or library(ies) you use and for what purpose)

function_seven 9y ago
As far as I know, querying a database has always been easier than querying a custom file format. It's the whole reason databases exist in the first place. And using one off the shelf is clearly easier than writing your own.
- tetromino_ 9y ago
  Not easier, but vastly more flexible. Sure, it's completely trivial to read a file containing an array of C structs of metadata and offsets into a blob of concatenated PNGs: just mmap the thing, and your main data structure is sitting right there in memory. But problems start arising when you need to update the structure of that metadata. Add some more fields. Some more structured, multi-value fields, referring to multiple entities in your increasingly complex data structure. Entities which must be guaranteed to exist. And perform modifications concurrently, from multiple processes. And have the file not get corrupted when something kills a writer process in the middle of writing... Databases are slow and overkill for the simple case, but they certainly make adding unexpected requested features much easier. And it can be expected that you will get unexpected feature requests.
imron 9y ago
> Oh goodness have we come a long way if interacting with a file is harder than interacting with a SQL database. It's not interacting with a file, it's interacting with a file that contains other files of varying sizes that need to be accessed randomly with good performance.
- diroussel 9y ago
  If you only want to read then both zip and tar are good for this purpose. It's commonly used and known to work well. There are many formats that are really just a renamed zip file. Like jar, and epub. File entries don't have to be compressed so you can mmap. If you want to do random writes then more consideration is needed.
  - 9y ago