3 ms·
You have 250GB of "raw" data stored in CSV format. The parsed version of this data in memory is likely to be a fraction of the on-disk size. A `long` or `double
by giantrobot 7y ago
You have 250GB of "raw" data stored in CSV format. The parsed version of this data in memory is likely to be a fraction of the on-disk size. A `long` or `double` only take up eight bytes in memory but 10-20 bytes on disk stored as ASCII in a CSV file. Even if your raw data was 250GB you could store it in memory mapped files. A fast SSD can easily hit a gigabyte per second sequential read speed, far faster than your typical network.
Segmenting your raw data and using memory mapped files will let you work with large data sets without needing huge amounts of RAM. From there it's a question of your single system's processing speed and IO capacity. This is only necessary if your processing needs random access to the entire dataset.
If your CSV data is more like a streaming data source, you're processing each record as it's read in, you can just stream it in through `stdin`. At 1GB/s you're looking at five minutes or so to process your 250GB of raw data. A SATA SSD might take twenty minutes to stream that raw data.
- inetknght 7y ago> A fast SSD can easily hit a gigabyte per second sequential read speed, far faster than your typical network. It's important to note that often your disks aren't directly attached to your compute. That's frequently the case in (particularly cheap) cloud instances.
- fock 7y agoso, a hundred of the cheapest cloud instances cost you what? 1000 USD per month? You can easily get a deskside workstation for 10000 USD with a nice 2TB NVME-SSD, a 32-Core Threadripper and 128GB of memory... Utilization might be lower of course, but even if it's only utilized 8h/day this seems like a bargain for any solid business. For any startup aiming for exponential growth by burning through cash and having a 15month "half-life" this is not gonna work out though
- giantrobot 7y agoThat's the thing, you don't necessarily need a bunch of cloud instances to process data. If you must be doing everything in "the cloud" every service has dedicated instances with fast attached storage available. You can spin up one long enough to rip through your data instead of trying to distribute it over hundreds of workers. It's also a domain where you can buy an off-the-shelf desktop for a few hundred dollars to do the work. That's the thrust of this whole thread, because scalable "cloud" systems exist and look cheap people obsess about throwing more instances at problems. Modern commodity systems are ridiculously powerful and far more capable than people tend to assume. Even "the cloud" gets underestimated because people look at the low end cheap instances and assume they need to spin up hundreds of those when one beefy image for a short duration could do the same work.
- masklinn 7y ago> A fast SSD can easily hit a gigabyte per second sequential read speed, far faster than your typical network. Nit: 1GB/s is ok, not even solid let alone fast. A fast SSD pretty much saturates 4x3.0 links (which explains why they universally tend to cap out at 3.5GB/s). In fact there are now a few PCIe4 SSDs (e.g. Corsair's MP600) which close in on 5GB/s.
- giantrobot 7y agoI got the 1GB/s from my laptop I was writing the comment on. The internal NVME drive tops out at 1.5GB/s. I consider that fast but as you point out there's drives that make mine look slow.