4 ms·
If data needs to be processed, an option would be to use MongoDB for collecting the data in bulk and later decide how you need to structure it for your needs.
by mrtksn 9y ago
If data needs to be processed, an option would be to use MongoDB for collecting the data in bulk and later decide how you need to structure it for your needs.
When it comes to "read" the data, you read it only from processed database that can be anything.
Since you can do your processing completely independently of your web server, your are not necessarily pushing your computational load from the DB to the web server.
- emodendroket 9y agoIt that point it's harder for me to understand why I'm not just writing out JSON files or something.
- mrtksn 9y agoYou definitely can but MongoDB provides convenience over storing and managing JSON files on you filesystem by your own efforts.
- emodendroket 9y agoSure, but it is also another service to keep running. Depends on what scale you're operating at.
- mrtksn 9y agoAny RDBMS is just a toolset to manage files on some filesystem. If you are dealing with some config files you can spare yourself the trouble and just load them from the filesystem. NoSQL or classic RDBMS, these tools give you ways to manage your data on the filesystem(or in the memory?). You can spend time to write a code that processes files in a folder or you can use the software that does it for you. You can write a code that relates two files in a folder or you can use a software that already can do it and you can just ask that software for the outcome.
- emodendroket 9y agoI don't really have to write that code though; what language doesn't already have file-manipulation libraries?
- mrtksn 9y agoWell I guess some people wonder why people are using ready software when you can achieve the exact same thing with a text editor and a compiler.
- emodendroket 9y agoThat's not the same thing at all; you're suggesting an approach that probably takes more code and definitely takes more setup.
- mrtksn 9y agoYou are underestimating the complexities of file management
- emodendroket 9y agoAgain, depends on scope and scale. If you have a handful of items to process it's not a big deal.
- deleted 9y ago[deleted]
- lowbloodsugar 9y agoNo, I really can't think of any situation where installing, managing, updating, maintaining, and crying over a mongodb cluster would be easier for a flat-file datastore than say, a file system, or S3. In fact if one is on AWS, then nothing beats just dumping them in S3 and processing them as necessary. If I'm a startup and I really wanted some querying then PostgreSQL looks great, with Amazon handling the maint and backups.
- bdcravens 9y agoYou can even query flatfiles in s3 using SQL via AWS Athena.
- flavio81 9y ago> If data needs to be processed, an option would be to use MongoDB for collecting the data in bulk and later decide how you need to structure it for your needs. OR, i can open a file stream, serialize my data to JSON, entity by entity, and dump all to a file. The good old file.
- mrtksn 9y agoreplied to the same question here: https://news.ycombinator.com/item?id=14805812 https://news.ycombinator.com/item?id=14805812
- threeseed 9y agoBecause we want to do partial updates, searches, indexing etc. Your position can be applied to all databases. Why not abandon them all and just use CSV ?
- bdcravens 9y agoYou could always do that, dump the data to s3, and use Athena :-) https://aws.amazon.com/athena/ https://aws.amazon.com/athena/
- pritambaral 9y agoAnd then be locked-in to Amazon. From ubiquitous files to ... a single service provider.
- aidos 9y agoThen again, if you're going to push into Postgres you might as well store the temp unstructured records there too
- mrtksn 9y agoWell, that's also an option :) But I believe this is a relatively new option. Personally, I never got into the hype with the NoSQL and for me the only use is as a convenient API for storing JSON files during prototyping so that I can later decide how my data should be structured. Definitely tech debt for the future but fast iterated design that needs the database stuff fixed has more chance for success than the rigid design with the perfect data architecture.