4 ms·

For an app I'm building I need to store each day ~1M json files 10kb each, which I then upload as parquet to S3. What is the alternative to using the filesystem

by lairv 4y ago

For an app I'm building I need to store each day ~1M json files 10kb each, which I then upload as parquet to S3. What is the alternative to using the filesystem here ? Should I put them in a database ?

Someone 4y ago
Using the file system, but combining multiple json files into a single file might be an option. - Open ‘current.json’ for append, - write the new json, - check file size, - if it’s over a threshold (say 100MB): - rename file (use UUID or timestamp) Your tooling would have to support that, of course (writing each json in a single line might be needed)
- lairv 4y ago
  Would this get messed up if I write concurrently in the same file ?
  - Someone 4y ago
    Of course, but that can be prevented.
fIREpOK 4y ago
If you insist on 10kb files, you could use sub directories to split them up.