3 ms·
If you want high performance and simple why not use flat files, bash, grep (maybe parallel), cut, awk, wc, uniq, etc. You can get very far with these and if yo
by codeonfire 11y ago
If you want high performance and simple why not use flat files, bash, grep (maybe parallel), cut, awk, wc, uniq, etc. You can get very far with these and if you have a fast local disk you cat get huge read rates. A few GB can be scanned in a matter of seconds. Awk can be used to write your queries. I don't understand what you are trying to do, but if it can be done with a SQL database and doesn't involve lots of joins then it can be done with a delimited text file. If you don't have a lot of disk space you can also work with gzipped files, zcat, zgrep, etc. I would not even consider distributed solutions or nosql until I had at least 100GB of data (more like 1TB of data). I would not consider any sort of SQL database unless I had a lot of complex joins.
- cactusface 11y agoThis is definitely the most straightforward way to go. It is almost certainly not the way he will go. In particular Awk is just great for this job and very easy to learn.