7 ms·

Can you expand on this? I have some pretty massive compute loads that need to be scaled onto a cluster with 100+ workers for most computations. This is after I

Can you expand on this? I have some pretty massive compute loads that need to be scaled onto a cluster with 100+ workers for most computations. This is after I use a library called dask that graphically does its own mapreduce optimisation inside its modules. This is all for a relatively small 250GB raw data file that I keep in a csv (and need to convert to SQL at some point). Are you saying this can be optimised to fit inside a single 10 core server in terms of compute loads?

sfifs 7y ago
Don't know why you're being downvoted but I'll assume your question is genuine. You use a cluster when your data and compute requirements are large and parallel enough that the tax paid on network latency trumps the 10-20X speedup you get on SSD and 1000X speedup you get from just keeping data in RAM. 250 Gigs is tiny enough that you could probably much get better performance running on high memory instance in AWS or GCP. You'll generally have to write your own multiprocessing code though which is fairly simple - your existing library may also be able to support it. I once actually ran this kind of workload on just my laptop using a compiled language that performed better than pyspark on a cluster.
- hnews_account_1 7y ago
  I'd love to keep it in RAM if I could. The problem is, the library I'm familiar with (pandas) typically seems to take more memory than the original csv file once it loads it onto memory. I know this is due to bad data types but in certain cases, I cannot get around those. However, even if I could load it all into memory at once, and assuming it takes 200 gb, I'm still using a master's student access to a cluster. So I get preempted like it's nobody's business. Hence why I prefer a smaller memory footprint even if I take up cpus at variable rates through a single execution. I did try to write my own multiprocessing code for this, but the operations are sometimes too complicated (like groupby) for me to rewrite everything from the ground up. If I'm not reliant on serial data communication between processes (like you'd need to sort a column), I can get it done pretty easily. In fact, I wrote my data cleaning code with this and cleaned up the entire file in half an hour because single chunks didn't rely on others. However, if you have some idea of how to run these computational loads in parallel in python or any other language on single compute instances (like the size of a laptop's memory of 16 gb), I'd really love to see it. Thanks.
  - giantrobot 7y ago
    Numpy supports memory mapping `ndarrays` which can back a DataFrame in pandas. This lets you access a dataset far larger than will fit in RAM as if it lived in RAM. Provided it's on fast SSD storage you'll have speedy access to the data and can process huge chunks at once.
    - hnews_account_1 7y ago
      Can you provide a link to this please? My current knowledge is that all numpy data lives in memory, and pandas itself has a feature to fragment any data into iterables so I can read upto my memory limit. I cannot use this feature due to the serial nature of some of the operations that I alluded to (I'd have to almost rewrite the entire library for some of these complicated operations like groupby and sorting). I do have fast SSD storage because it's on the scratch drive of a cluster and from what I've seen it can do ~300-400 MB/s easily. I haven't had a chance to test more than that since I'm mostly memory constrained in much of my testing. My current attempt is to push this data into a pure database handling system like SQL so that I can query it. But like I said, I work with a less-than-stellar set of tools and I have to literally set up a postgres server from ground up to write to it. Which shouldn't be a big deal except when it's on a non-root user and I have to keep remapping dependencies (took 5-6 hours to set it up on the instance I have access to). My other option was to write the entire 250 GB to a sqllite database using the sqlalchemy library in Python, but that seems to fail whether I do it with parallel writes, or serial writes. In both cases, it fails after I create ~64-70 tables.
      - westurner 7y ago
        Dask groupby example: https://examples.dask.org/dataframes/02-groupby.html https://examples.dask.org/dataframes/02-groupby.html > Generally speaking, Dask.dataframe groupby-aggregations are roughly same performance as Pandas groupby-aggregations, just more scalable. The dask.distributed scheduler can also run on one high-RAM instance (with threads or processes) https://docs.dask.org/en/latest/setup.html https://docs.dask.org/en/latest/setup.html Pandas docs > Ecosystem > Out-of-core: https://pandas.pydata.org/pandas-docs/stable/ecosystem.html#out-of-core https://pandas.pydata.org/pandas-docs/stable/ecosystem.html#... Reading from Parquet into Apache Arrow is much faster than CSV because the data can just be directly loaded into RAM. https://ursalabs.org/blog/2019-10-columnar-perf/ https://ursalabs.org/blog/2019-10-columnar-perf/ If you have GPU instances, cuDF has a Pandas-like API on top of Apache Arrow. https://github.com/rapidsai/cudf https://github.com/rapidsai/cudf > Built based on the Apache Arrow columnar memory format, cuDF is a GPU DataFrame library for loading, joining, aggregating, filtering, and otherwise manipulating data. > cuDF provides a pandas-like API that will be familiar to data engineers & data scientists, so they can use it to easily accelerate their workflows without going into the details of CUDA programming. Dask-ML makes scalable scikit-learn, XGBoost, TensorFlow really easy. https://dask-ml.readthedocs.io/en/latest/ https://dask-ml.readthedocs.io/en/latest/ ... re: the OT: While it's possible to write C++ code that's really fast, it's generally inflexible, expensive to develop, and dangerous for devs with experience in their respective domains of experience to write. Much saner to put a Python API on top and optimize that during compilation. There are a few C++ frameworks in the top quartile of the TechEmpower framework benchmarks. https://www.techempower.com/benchmarks/ https://www.techempower.com/benchmarks/ Hardware/hosting is relatively cheap. Developers and memory vulnerabilities aren't.
- rumanator 7y ago
  > You use a cluster when your data and compute requirements are large and parallel enough that the tax paid on network latency trumps the 10-20X speedup you get on SSD and 1000X speedup you get from just keeping data in RAM. Nowadays there's another reason to use clusters: to autoscale your expenditure wrt workload. A little inefficiency might be acceptable if you don't have to pay for a huge beefed up server idling at, say, 30%.
- jstrong 7y ago
  note: doing anything on a 250gb file in python will require a lot more ram than 250gb. generally my expectation is I will need 10x the ram as the size of the file when using pandas, for when I accidentally do something that triggers pathological behavior.
godelski 7y ago
I'm doing my PhD in HPC (part of my work is in situ stuff). One of the biggest problems is actually IO. But honestly, 250GB isn't that big. I haven't used dask, but I know people that do. I just avoid python for anything I need performance for, C++ is just always going to be faster. Biased: you might want to look into DOE libraries. For IO I suggest ADIOS2 [0]. There's python bindings too. One of the biggest things you can do is use a different storage type like BP (ADIOS) or hdf5. These are readable but binary. But to really determine how to speed up your problem you have to know where the bottleneck is. Is it IO or compute? With 100 workers (threads or nodes?) you aren't highly parallelized. I mean that could be a single node if it's threads. [0] https://github.com/ornladios/ADIOS2 https://github.com/ornladios/ADIOS2
bagels 7y ago
Not who you asked, but, it's hard to say without knowing exactly what computation is being done, or how much of the time is spent on IO. If you organize that 250GB in ram the right way (cache coherency, right container types), and spend a lot of effort doing analysis of algorithm selection, you might be surprised how much you can get done on a single (large) machine with enough cores.
- Narishma 7y ago
  Cache locality, not coherency. I'm curious as to why this seems like a common mistake (I've seen it a few times already in this comment section).
tkyjonathan 7y ago
Not sure which DB you are using, but you can load the csv file into the DB directly on a single thread using something like LOAD DATA INFILE. If you have some good indexes and do some push-down work (give the database aggregation tasks to do instead of your python code), you should probably be more than fine. For a 250Gb file.. should be ok.. maybe add some partitioning too.
- hnews_account_1 7y ago
  I'm open to using any db that I can query over some engine with a python implementation. So any SQL db should be fine. However, I don't know how to convert a csv to an SQL directly. Is the command you mentioned part of some SQL server package? Sounds like it's exactly what I need.
  - nwallin 7y ago
    pandas can read from a CSV file and then write to SQL. Even if you don't go the SQL route, you'd probably gain significant benefits by working with HDF instead of CSV. https://pandas.pydata.org/pandas-docs/version/0.22/generated/pandas.read_csv.html#pandas.read_csv https://pandas.pydata.org/pandas-docs/version/0.22/generated... https://pandas.pydata.org/pandas-docs/version/0.22/io.html#writing-dataframes https://pandas.pydata.org/pandas-docs/version/0.22/io.html#w...
pmlnr 7y ago
250GB data is tiny. There are laptops with 128GB RAM, let alone servers. Obligatory read: https://aadrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html https://aadrake.com/command-line-tools-can-be-235x-faster-th...
- hnews_account_1 7y ago
  Yes, but I'm heavily constrained because my access to the cluster I'm using is very low level and I get pre-empted quite a bit in my tasks. I'm probably over stretching between the amount of data I need to handle and my severe lack of skills (I'm a quant in training).
inetknght 7y ago
> Are you saying this can be optimised to fit inside a single 10 core server in terms of compute loads? I'm currently employed to write software which does DNA analysis. DNA is known for being Big Data. Some applications are very compute intensive and others are very data-relation intensive. The very compute intensive applications process about 1GB of data in about 30 minutes on a 32 core Xeon 6xxx with 32GB of RAM assigned to it. The data-relation intensive application processes about 350GB of data in 15 minutes on a single core of the same CPU but with 700GB of RAM assigned to it. Both are heavily optimized but in different ways. So without knowing more about what you're doing with that data, it's hard to say.
- vagab0nd 7y ago
  That's very interesting. Do you mind sharing some more details on what those computations are? Where can I learn more about these DNA analysis use cases?
  - inetknght 7y ago
    I've found that Wikipedia has really good high-level information for a lot of the subject matter. For use cases, the business sells direct-to-consumer DNA tests. I've worked on several of the analysis products' softwares. The compute-heavy workload calculates edit distance [0] of short paired-end sequenced DNA [1] vs the human genome [2]. There is open source software to manipulate the FASTA/FASTQ [3] and SAM files [4] and run the calculations [5]. The aligned file is processed in a couple of minutes to genetic variation report [6] which is used for some of the analysis products that were purchased. One popular product will give you a haplogroup [7] which basically tells you where you are in a genetic tree. The relationship estimator uses a different sequencing technology and basically consumes a CSV file from the sequencer's manufacturer. It uses a proprietary algorithm to calculate centimorgans [8]. That then gives relationship estimates between you and other people who've purchased the product. [0]: https://en.wikipedia.org/wiki/Needleman%E2%80%93Wunsch_algorithm https://en.wikipedia.org/wiki/Needleman%E2%80%93Wunsch_algor... [1]: https://en.wikipedia.org/wiki/Next-generation_sequencing https://en.wikipedia.org/wiki/Next-generation_sequencing [2]: https://en.wikipedia.org/wiki/Reference_genome https://en.wikipedia.org/wiki/Reference_genome [3]: https://en.wikipedia.org/wiki/FASTQ_format https://en.wikipedia.org/wiki/FASTQ_format [4]: https://github.com/samtools/hts-specs https://github.com/samtools/hts-specs [5]: https://en.wikipedia.org/wiki/List_of_sequence_alignment_software https://en.wikipedia.org/wiki/List_of_sequence_alignment_sof... [6]: https://en.wikipedia.org/wiki/Variant_Call_Format https://en.wikipedia.org/wiki/Variant_Call_Format [7]: https://en.wikipedia.org/wiki/Haplogroup https://en.wikipedia.org/wiki/Haplogroup [8]: https://en.wikipedia.org/wiki/Centimorgan https://en.wikipedia.org/wiki/Centimorgan
giantrobot 7y ago
You have 250GB of "raw" data stored in CSV format. The parsed version of this data in memory is likely to be a fraction of the on-disk size. A `long` or `double` only take up eight bytes in memory but 10-20 bytes on disk stored as ASCII in a CSV file. Even if your raw data was 250GB you could store it in memory mapped files. A fast SSD can easily hit a gigabyte per second sequential read speed, far faster than your typical network. Segmenting your raw data and using memory mapped files will let you work with large data sets without needing huge amounts of RAM. From there it's a question of your single system's processing speed and IO capacity. This is only necessary if your processing needs random access to the entire dataset. If your CSV data is more like a streaming data source, you're processing each record as it's read in, you can just stream it in through `stdin`. At 1GB/s you're looking at five minutes or so to process your 250GB of raw data. A SATA SSD might take twenty minutes to stream that raw data.
- inetknght 7y ago
  > A fast SSD can easily hit a gigabyte per second sequential read speed, far faster than your typical network. It's important to note that often your disks aren't directly attached to your compute. That's frequently the case in (particularly cheap) cloud instances.
  - fock 7y ago
    so, a hundred of the cheapest cloud instances cost you what? 1000 USD per month? You can easily get a deskside workstation for 10000 USD with a nice 2TB NVME-SSD, a 32-Core Threadripper and 128GB of memory... Utilization might be lower of course, but even if it's only utilized 8h/day this seems like a bargain for any solid business. For any startup aiming for exponential growth by burning through cash and having a 15month "half-life" this is not gonna work out though
  - giantrobot 7y ago
    That's the thing, you don't necessarily need a bunch of cloud instances to process data. If you must be doing everything in "the cloud" every service has dedicated instances with fast attached storage available. You can spin up one long enough to rip through your data instead of trying to distribute it over hundreds of workers. It's also a domain where you can buy an off-the-shelf desktop for a few hundred dollars to do the work. That's the thrust of this whole thread, because scalable "cloud" systems exist and look cheap people obsess about throwing more instances at problems. Modern commodity systems are ridiculously powerful and far more capable than people tend to assume. Even "the cloud" gets underestimated because people look at the low end cheap instances and assume they need to spin up hundreds of those when one beefy image for a short duration could do the same work.
roel_v 7y ago
If there are floating point numbers in those csvs, I sped up a system like that 10x just by writing a custom (Java equivalent of) atof() that didn't do variable decimal separators and scientific notation. That's not even counting the improvements in I/O speed from the size reduction. Any system that works from CSV's is going to be slow. I don't know what sort of computations you're doing of course, but I did all the work on a laptop, couldn't be bothered to scale it out after the improvements I made in the first pass. How much of your code spends its time in I/O (including conversions) vs actually calculating?
vchak1 7y ago
Need to understand your domain better, but in many cases, the 250GB csv can be compressed down quite effectively using a columnar representation. And the columns can (potentially) be processed using simd/gpu based approaches to where a single server would outrun a cluster. Food for thought..

sfifs 7y ago

Don't know why you're being downvoted but I'll assume your question is genuine. You use a cluster when your data and compute requirements are large and parallel enough that the tax paid on network latency trumps the 10-20X speedup you get on SSD and 1000X speedup you get from just keeping data in RAM. 250 Gigs is tiny enough that you could probably much get better performance running on high memory instance in AWS or GCP. You'll generally have to write your own multiprocessing code though which is fairly simple - your existing library may also be able to support it. I once actually ran this kind of workload on just my laptop using a compiled language that performed better than pyspark on a cluster.

hnews_account_1 7y ago
I'd love to keep it in RAM if I could. The problem is, the library I'm familiar with (pandas) typically seems to take more memory than the original csv file once it loads it onto memory. I know this is due to bad data types but in certain cases, I cannot get around those. However, even if I could load it all into memory at once, and assuming it takes 200 gb, I'm still using a master's student access to a cluster. So I get preempted like it's nobody's business. Hence why I prefer a smaller memory footprint even if I take up cpus at variable rates through a single execution. I did try to write my own multiprocessing code for this, but the operations are sometimes too complicated (like groupby) for me to rewrite everything from the ground up. If I'm not reliant on serial data communication between processes (like you'd need to sort a column), I can get it done pretty easily. In fact, I wrote my data cleaning code with this and cleaned up the entire file in half an hour because single chunks didn't rely on others. However, if you have some idea of how to run these computational loads in parallel in python or any other language on single compute instances (like the size of a laptop's memory of 16 gb), I'd really love to see it. Thanks.
- giantrobot 7y ago
  Numpy supports memory mapping `ndarrays` which can back a DataFrame in pandas. This lets you access a dataset far larger than will fit in RAM as if it lived in RAM. Provided it's on fast SSD storage you'll have speedy access to the data and can process huge chunks at once.
  - hnews_account_1 7y ago
    Can you provide a link to this please? My current knowledge is that all numpy data lives in memory, and pandas itself has a feature to fragment any data into iterables so I can read upto my memory limit. I cannot use this feature due to the serial nature of some of the operations that I alluded to (I'd have to almost rewrite the entire library for some of these complicated operations like groupby and sorting). I do have fast SSD storage because it's on the scratch drive of a cluster and from what I've seen it can do ~300-400 MB/s easily. I haven't had a chance to test more than that since I'm mostly memory constrained in much of my testing. My current attempt is to push this data into a pure database handling system like SQL so that I can query it. But like I said, I work with a less-than-stellar set of tools and I have to literally set up a postgres server from ground up to write to it. Which shouldn't be a big deal except when it's on a non-root user and I have to keep remapping dependencies (took 5-6 hours to set it up on the instance I have access to). My other option was to write the entire 250 GB to a sqllite database using the sqlalchemy library in Python, but that seems to fail whether I do it with parallel writes, or serial writes. In both cases, it fails after I create ~64-70 tables.
    - westurner 7y ago
      Dask groupby example: https://examples.dask.org/dataframes/02-groupby.html https://examples.dask.org/dataframes/02-groupby.html > Generally speaking, Dask.dataframe groupby-aggregations are roughly same performance as Pandas groupby-aggregations, just more scalable. The dask.distributed scheduler can also run on one high-RAM instance (with threads or processes) https://docs.dask.org/en/latest/setup.html https://docs.dask.org/en/latest/setup.html Pandas docs > Ecosystem > Out-of-core: https://pandas.pydata.org/pandas-docs/stable/ecosystem.html#out-of-core https://pandas.pydata.org/pandas-docs/stable/ecosystem.html#... Reading from Parquet into Apache Arrow is much faster than CSV because the data can just be directly loaded into RAM. https://ursalabs.org/blog/2019-10-columnar-perf/ https://ursalabs.org/blog/2019-10-columnar-perf/ If you have GPU instances, cuDF has a Pandas-like API on top of Apache Arrow. https://github.com/rapidsai/cudf https://github.com/rapidsai/cudf > Built based on the Apache Arrow columnar memory format, cuDF is a GPU DataFrame library for loading, joining, aggregating, filtering, and otherwise manipulating data. > cuDF provides a pandas-like API that will be familiar to data engineers & data scientists, so they can use it to easily accelerate their workflows without going into the details of CUDA programming. Dask-ML makes scalable scikit-learn, XGBoost, TensorFlow really easy. https://dask-ml.readthedocs.io/en/latest/ https://dask-ml.readthedocs.io/en/latest/ ... re: the OT: While it's possible to write C++ code that's really fast, it's generally inflexible, expensive to develop, and dangerous for devs with experience in their respective domains of experience to write. Much saner to put a Python API on top and optimize that during compilation. There are a few C++ frameworks in the top quartile of the TechEmpower framework benchmarks. https://www.techempower.com/benchmarks/ https://www.techempower.com/benchmarks/ Hardware/hosting is relatively cheap. Developers and memory vulnerabilities aren't.
rumanator 7y ago
> You use a cluster when your data and compute requirements are large and parallel enough that the tax paid on network latency trumps the 10-20X speedup you get on SSD and 1000X speedup you get from just keeping data in RAM. Nowadays there's another reason to use clusters: to autoscale your expenditure wrt workload. A little inefficiency might be acceptable if you don't have to pay for a huge beefed up server idling at, say, 30%.
jstrong 7y ago
note: doing anything on a 250gb file in python will require a lot more ram than 250gb. generally my expectation is I will need 10x the ram as the size of the file when using pandas, for when I accidentally do something that triggers pathological behavior.

godelski 7y ago

I'm doing my PhD in HPC (part of my work is in situ stuff). One of the biggest problems is actually IO. But honestly, 250GB isn't that big. I haven't used dask, but I know people that do. I just avoid python for anything I need performance for, C++ is just always going to be faster. Biased: you might want to look into DOE libraries. For IO I suggest ADIOS2 [0]. There's python bindings too. One of the biggest things you can do is use a different storage type like BP (ADIOS) or hdf5. These are readable but binary. But to really determine how to speed up your problem you have to know where the bottleneck is. Is it IO or compute? With 100 workers (threads or nodes?) you aren't highly parallelized. I mean that could be a single node if it's threads. [0] https://github.com/ornladios/ADIOS2 https://github.com/ornladios/ADIOS2

bagels 7y ago

Not who you asked, but, it's hard to say without knowing exactly what computation is being done, or how much of the time is spent on IO. If you organize that 250GB in ram the right way (cache coherency, right container types), and spend a lot of effort doing analysis of algorithm selection, you might be surprised how much you can get done on a single (large) machine with enough cores.

Narishma 7y ago
Cache locality, not coherency. I'm curious as to why this seems like a common mistake (I've seen it a few times already in this comment section).

tkyjonathan 7y ago

Not sure which DB you are using, but you can load the csv file into the DB directly on a single thread using something like LOAD DATA INFILE. If you have some good indexes and do some push-down work (give the database aggregation tasks to do instead of your python code), you should probably be more than fine. For a 250Gb file.. should be ok.. maybe add some partitioning too.

hnews_account_1 7y ago
I'm open to using any db that I can query over some engine with a python implementation. So any SQL db should be fine. However, I don't know how to convert a csv to an SQL directly. Is the command you mentioned part of some SQL server package? Sounds like it's exactly what I need.
- nwallin 7y ago
  pandas can read from a CSV file and then write to SQL. Even if you don't go the SQL route, you'd probably gain significant benefits by working with HDF instead of CSV. https://pandas.pydata.org/pandas-docs/version/0.22/generated/pandas.read_csv.html#pandas.read_csv https://pandas.pydata.org/pandas-docs/version/0.22/generated... https://pandas.pydata.org/pandas-docs/version/0.22/io.html#writing-dataframes https://pandas.pydata.org/pandas-docs/version/0.22/io.html#w...

pmlnr 7y ago

250GB data is tiny. There are laptops with 128GB RAM, let alone servers. Obligatory read: https://aadrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html https://aadrake.com/command-line-tools-can-be-235x-faster-th...

hnews_account_1 7y ago
Yes, but I'm heavily constrained because my access to the cluster I'm using is very low level and I get pre-empted quite a bit in my tasks. I'm probably over stretching between the amount of data I need to handle and my severe lack of skills (I'm a quant in training).

inetknght 7y ago

> Are you saying this can be optimised to fit inside a single 10 core server in terms of compute loads? I'm currently employed to write software which does DNA analysis. DNA is known for being Big Data. Some applications are very compute intensive and others are very data-relation intensive. The very compute intensive applications process about 1GB of data in about 30 minutes on a 32 core Xeon 6xxx with 32GB of RAM assigned to it. The data-relation intensive application processes about 350GB of data in 15 minutes on a single core of the same CPU but with 700GB of RAM assigned to it. Both are heavily optimized but in different ways. So without knowing more about what you're doing with that data, it's hard to say.

vagab0nd 7y ago
That's very interesting. Do you mind sharing some more details on what those computations are? Where can I learn more about these DNA analysis use cases?
- inetknght 7y ago
  I've found that Wikipedia has really good high-level information for a lot of the subject matter. For use cases, the business sells direct-to-consumer DNA tests. I've worked on several of the analysis products' softwares. The compute-heavy workload calculates edit distance [0] of short paired-end sequenced DNA [1] vs the human genome [2]. There is open source software to manipulate the FASTA/FASTQ [3] and SAM files [4] and run the calculations [5]. The aligned file is processed in a couple of minutes to genetic variation report [6] which is used for some of the analysis products that were purchased. One popular product will give you a haplogroup [7] which basically tells you where you are in a genetic tree. The relationship estimator uses a different sequencing technology and basically consumes a CSV file from the sequencer's manufacturer. It uses a proprietary algorithm to calculate centimorgans [8]. That then gives relationship estimates between you and other people who've purchased the product. [0]: https://en.wikipedia.org/wiki/Needleman%E2%80%93Wunsch_algorithm https://en.wikipedia.org/wiki/Needleman%E2%80%93Wunsch_algor... [1]: https://en.wikipedia.org/wiki/Next-generation_sequencing https://en.wikipedia.org/wiki/Next-generation_sequencing [2]: https://en.wikipedia.org/wiki/Reference_genome https://en.wikipedia.org/wiki/Reference_genome [3]: https://en.wikipedia.org/wiki/FASTQ_format https://en.wikipedia.org/wiki/FASTQ_format [4]: https://github.com/samtools/hts-specs https://github.com/samtools/hts-specs [5]: https://en.wikipedia.org/wiki/List_of_sequence_alignment_software https://en.wikipedia.org/wiki/List_of_sequence_alignment_sof... [6]: https://en.wikipedia.org/wiki/Variant_Call_Format https://en.wikipedia.org/wiki/Variant_Call_Format [7]: https://en.wikipedia.org/wiki/Haplogroup https://en.wikipedia.org/wiki/Haplogroup [8]: https://en.wikipedia.org/wiki/Centimorgan https://en.wikipedia.org/wiki/Centimorgan

giantrobot 7y ago

You have 250GB of "raw" data stored in CSV format. The parsed version of this data in memory is likely to be a fraction of the on-disk size. A `long` or `double` only take up eight bytes in memory but 10-20 bytes on disk stored as ASCII in a CSV file. Even if your raw data was 250GB you could store it in memory mapped files. A fast SSD can easily hit a gigabyte per second sequential read speed, far faster than your typical network. Segmenting your raw data and using memory mapped files will let you work with large data sets without needing huge amounts of RAM. From there it's a question of your single system's processing speed and IO capacity. This is only necessary if your processing needs random access to the entire dataset. If your CSV data is more like a streaming data source, you're processing each record as it's read in, you can just stream it in through `stdin`. At 1GB/s you're looking at five minutes or so to process your 250GB of raw data. A SATA SSD might take twenty minutes to stream that raw data.

inetknght 7y ago
> A fast SSD can easily hit a gigabyte per second sequential read speed, far faster than your typical network. It's important to note that often your disks aren't directly attached to your compute. That's frequently the case in (particularly cheap) cloud instances.
- fock 7y ago
  so, a hundred of the cheapest cloud instances cost you what? 1000 USD per month? You can easily get a deskside workstation for 10000 USD with a nice 2TB NVME-SSD, a 32-Core Threadripper and 128GB of memory... Utilization might be lower of course, but even if it's only utilized 8h/day this seems like a bargain for any solid business. For any startup aiming for exponential growth by burning through cash and having a 15month "half-life" this is not gonna work out though
- giantrobot 7y ago
  That's the thing, you don't necessarily need a bunch of cloud instances to process data. If you must be doing everything in "the cloud" every service has dedicated instances with fast attached storage available. You can spin up one long enough to rip through your data instead of trying to distribute it over hundreds of workers. It's also a domain where you can buy an off-the-shelf desktop for a few hundred dollars to do the work. That's the thrust of this whole thread, because scalable "cloud" systems exist and look cheap people obsess about throwing more instances at problems. Modern commodity systems are ridiculously powerful and far more capable than people tend to assume. Even "the cloud" gets underestimated because people look at the low end cheap instances and assume they need to spin up hundreds of those when one beefy image for a short duration could do the same work.

roel_v 7y ago

If there are floating point numbers in those csvs, I sped up a system like that 10x just by writing a custom (Java equivalent of) atof() that didn't do variable decimal separators and scientific notation. That's not even counting the improvements in I/O speed from the size reduction. Any system that works from CSV's is going to be slow. I don't know what sort of computations you're doing of course, but I did all the work on a laptop, couldn't be bothered to scale it out after the improvements I made in the first pass. How much of your code spends its time in I/O (including conversions) vs actually calculating?

vchak1 7y ago

Need to understand your domain better, but in many cases, the 250GB csv can be compressed down quite effectively using a columnar representation. And the columns can (potentially) be processed using simd/gpu based approaches to where a single server would outrun a cluster. Food for thought..