20 ms·

CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

177 points by nemoniac 6y ago

qaq 6y ago
Julia is a hidden gem Once ecosystem for web dev matures Julia will be the killer lang for building web apps
- dmos62 6y ago
  > Julia will be the killer lang for building web apps That would be fun, but Julia's community aren't web devs. Julia spawned around very specific needs of scientific computing, which is characterised by a short-running daemon or a script-type interpreter. A web server is a long-running process. Not knowledgeable enough about Julia to tell how it lends itself to server uses, but heard hearsay that it's problematic.
  - ViralBShah 6y ago
    See Jacob Quinn's (karbarcca) other package: https://github.com/JuliaWeb/HTTP.jl https://github.com/JuliaWeb/HTTP.jl It is quite common for scientists and engineers to want to present the findings of their work or share them with others who are non-coders. Thus you get tools like Jupyter, Shiny, Pluto, etc. There are several long running Julia processes in real applications. While there's no doubt that there is work to be done, the set of these applications is not the null set either.
  - langitbiru 6y ago
    Seconded this. If you looked at the Julia website (https://julialang.org https://julialang.org), in the Ecosystem section, they listed these things: "Visualization, General Purpose, Data Science, Machine Learning, Scientific Domains, Parallel Computing." They already positioned themselves in scientific computing battleground. Web development (or desktop programming) is not their focus (for better or worse).
    - qaq 6y ago
      Julia is very capable general purpose lang and web libs are maturing pretty fast. It does not have to be the focus of core team for it to happen.
  - comicjk 6y ago
    I don't think you can define scientific computing in a way that excludes quantum mechanics, molecular dynamics, fluid dynamics, Monte Carlo simulations... There are lots of long-running applications. We don't want them to be long-running, but it's not unusual for a run to take weeks and not unheard-of for it to take months.
- akerro 6y ago
  Rust is already killing old web app building technologies!
  - qaq 6y ago
    Julia is a good bit easier to pick up than Rust. I like Rust btw. but say at my job chances of us adopting Rust would be close to 0. Julia is really easy to pick up so would be much easier sell once ecosystem is mature.
- crispyambulance 6y ago
  > Once ecosystem for web dev matures Julia will be the killer... Looking back at history, javascript was put together in haste, almost like an afterthought. Even the name was a stupid marketing tack-on. Yet it stuck and kept going even when people became sick of it. It has reinvented itself several times over, and in spite of the emergence of multiple superior technologies that were specifically targeted at web-development, it still stands at the top of hill. I guess it would be properly ironic that something designed for mathematics would take down javascript for web-dev, but I doubt it. It would be amusing though.
- deathtrader666 6y ago
  Genie.jl exists for web apps.
  - qaq 6y ago
    In no way did I mean to imply there are no packages, I just noted they are not yet as mature as compared to web frameworks/packages in other langs.
mushufasa 6y ago
this is neat. that said, to me, this is like comparing car startup times. even if a tesla starts faster than a toyota, they're both fast enough to not notice 99.999% of the time. This isn't a factor in a purchasing decision -- i've never even thought of that before contriving this metaphor
- lxe 6y ago
  Maybe not a deciding factor, but definitely a factor in a purchasing decision, among many others.
CJefferson 6y ago
EDIT: Be sure to read the reply from the author of the package. I do think Julia has promise, but it is still early days. In the last 2 months there have been problems with: * Throwing rows away: https://github.com/JuliaData/CSV.jl/issues/720 https://github.com/JuliaData/CSV.jl/issues/720 * Incorrect float parsing: https://github.com/JuliaData/CSV.jl/issues/714 https://github.com/JuliaData/CSV.jl/issues/714 At there are unfixed issues to do with parsing large files ( https://github.com/JuliaData/CSV.jl/issues/749 https://github.com/JuliaData/CSV.jl/issues/749 , https://github.com/JuliaData/CSV.jl/issues/734 https://github.com/JuliaData/CSV.jl/issues/734 ). While Julia has become much more stable than it used to be, it is still in many places an evolving system and you (in my limited experience) should be prepared to hit issues.
- fermienrico 6y ago
  With all due respect to all the people working on Julia, I see a stark difference in rigor between say Python's PEP / RFCs and how Julia's core developers operate. The former is a strict formal process with a lot of thought given to a particular PEP proposal, whereas the latter seems like an informal chaotic hashing/bikeshedding between a few key people in Julia community. I've contributed to Julia and people are really nice but IMO it is missing structure, discipline, hardline regimented approach of confronting difficult problems, etc. One feels like hearing a supreme court case and the other a street brawl to decide upon important issues.
  - xyzzy_plugh 6y ago
    > Python's PEP / RFCs Is this process actually fundamentally better? There are so many PEPs that are up to the reader to interpret. There are so many PEPs that contradict each other (just read some of the early PEPs). The whole thing feels basically meaningless, and toothless at this point. I think Go's proposal process nails balancing formalities and discussion/bikeshedding. They seem to be very productive at making the language better, thinking critically and avoiding regrets, rather than pandering to the very vocal and often misaligned minority of the community.
    - 6y ago
mcrad 6y ago
> The very first task in any data analysis workflow is simply reading the data, and this absolutely must be done quickly. Lost me here. For what use-case of data analysis workflow does saving a minute or two make a difference? When will you tech bros learn to separate web dev from data science?
- deleted 6y ago
  [deleted]
- gameswithgo 6y ago
  I can think of many cases where that is useful. It is important to remember not all workflows and needs are the same as yours. I don’t really understand tour web dev comment either. Web devs are notoriously not interested in performance.
  - fermienrico 6y ago
    Then it goes both ways right? I think we need to also say "For most people Python is fine, if you need that extra speed, try Julia" instead of blanket praises of Julia as true competitor to Python (It doesn't even come close IMO).
  - mcrad 6y ago
    Maybe I don't understand web dev; only meant that when you have many users besides yourself (analyzing data) then performance matters more. It's important to remember, wtf?
- karbarcca 6y ago
  It can make huge amounts of difference in a production system; where I work, we process terabytes of csv data every day; saving minutes per file can add up to enormous differences in CPU cost/time for a production system running 24/7. I agree that for a data scientist doing exploratory analysis locally on their computer, it doesn't make nearly as much a difference (also because they're usually not working on crazy large files). The performance work in the CSV.jl package (that the article is about) was very much geared towards these kinds of production scenarios.
deleted 6y ago
[deleted]
mrfusion 6y ago
I’d really think all these languages would have the bottleneck of disk read speed. So how can Julia be that much faster? Are the others really that inefficient?
- dagw 6y ago
  Without having looked at it, it might also be a case that the other libraries spend more time handling more obscure corner cases and doing more robust error handling.
- mynegation 6y ago
  Because it is not a bottleneck. Modern SSDs can read data at more than 500 megabytes per second, which, on a single thread gives you just about 6-8 clock cycles to process a single byte. That’s not a lot of CPU instructions so it will take quite a number of threads to handle this bandwidth.
  - jdietrich 6y ago
    The SSD in my current workstation tops out at 5GB/s sequential reads and 680kIOPS. That's a modestly high-end consumer M.2 drive, not enterprise exotica. SSDs are really, really fast.
    - xiaodai 6y ago
      Yeah Intel optane and the like will be even faster.
- karbarcca 6y ago
  The biggest difference in these benchmarks comes down to how multiple threads are leveraged; (disclaimer: primary CSV.jl author here). In CSV.jl, it was relatively straightforward to add multithreaded parsing support; we chunk up the file, find row starts, then spawn a threaded task to process each chunk. In data.table (fread), they have a constraint of R having a global String intern store; so it fundamentally restricts the multithreading capabilities by requiring strings to be interned sequentially. In pandas, there are similar nontrivialities jumping between the C++ source and python object world that adds a lot of complexity (and probably explains why no one has made the effort to do multithreaded parsing). It's interesting to note, however, that the apache arrow project (pyarrow package in Python), has a multithreaded csv parser integrated with the project. It provides similar performance to CSV.jl because it was built from the ground up to process chunks on multiple threads into "arrow tables". Unfortunately, it's tied very specifically to the arrow project, so pandas doesn't benefit from its work! It's one of my favorite features about Julia that CSV.jl automatically integrates with other "data table formats"; i.e. ODBC.jl, SQLite.jl, MySQL.jl, Arrow.jl, DataFrames.jl, etc. They can all leverage CSV.jl as "their csv parser" because it's seamlessly integrated via well-established "table" apis.
bonoboTP 6y ago
Maybe I'm a cranky old man, but I think there's immense value in the fact that Python's (scientific and non-scientific) ecosystem has matured and has become standard over the last 1-2 decades. Starting again from the ground up in a shiny new language (even if the language is perhaps slightly better) is a waste of so much effort. Transitioning from Python 2 to 3 was already a nightmare. I know there's some interoperability between Julia and Python, but a lot of stuff is not interoperable, and there are annoying differences in array indexing etc. Why can't people just settle on some tech, let it mature, and enjoy reaching higher and higher in capability? Why do we have to always tear things down and start from scratch? This causes a lot of artificial bit rot, there's a never ending treadmill of just porting well-developed concepts and mature code to slightly changed representations. It's the busywork of tech. Does the scientific community also want to follow web dev in their insane churn of frameworks and libraries where last week's best practice is laughable ancient stone-age primitiveness today?
- Datenstrom 6y ago
  There is an immense value in most of the ecosystem converging on Python over the last decade and especially with leaving MATLAB behind in the dust. I don't think decade long transitions from old tech to new tech is a bad time scale or even close to that of the churn of web technologies. Julia offers enough of an improvement over Python to warrant a switch over the next 5-10 years and leave Python behind in the same way. I believe Julia offers the same level of improvement over Python as Python does over MATLAB. I also believe that it has drawn too much from MATLAB only for the sake of being familiar to that group and not the computer science and software engineering communities though such as the one based indexing you mentioned. Unfortunately that ship has already sailed and we will likely have to wait another 10 years for the next one to hopefully fix those problems.
  - ScottPJones 6y ago
    Back 5.5 years ago, I used to complain about the 1 based indexing and the column-major structure of matrices in Julia (both like Fortran), however, those issues have been solved by OffsetArrays and PermutedDimsArrays, giving far more flexibility that is possible in most other languages. It's silly to keep bringing up the issue of one based indexing, when you can use any integer as the base, just like Fortran 90 (so you can index by -5..5, for example). For some things, 0 based does make things easier, sure, but you can do that easily in Julia (and more!)
varbhat 6y ago
Well, Julia is compiled .Python and R are both interpreted. I think that simdjson is fastest json parser out there( Parsing gigabytes of JSON per second ). https://github.com/simdjson/simdjson https://github.com/simdjson/simdjson So,if I use Python binding for simdjson , then parsing of json in python must be very much faster than the fastest one mentioned in the above post. So,at the end,it depends on implementation not only on language.
- bjoli 6y ago
  Python's CSV parser is in C. I don't think a native python CSV parser would be nearly as fast.
  - ViralBShah 6y ago
    You can write slow programs in all languages. There are many cases where Julia's language features enable higher performance than in comparable C libraries. Have a look at Steve Johnson's keynote from JuliaCon 2019: https://www.youtube.com/watch?v=mSgXWpvQEHE https://www.youtube.com/watch?v=mSgXWpvQEHE
    - bjoli 6y ago
      Pythons CSV reader is actually decently fast. It is just that speed wasnt a priority. I dislike the whole "faster than C" comparisons. Almost nothing is. If you want speed you chose C. If you want something that is plenty fast with a much better speed-to-effort ratio, Julia is a strong contender.
      - johnisgood 6y ago
        Exactly. Nothing is faster than C that is also high-level. If it is, then the C version is not equivalent to the code with what you are comparing. Or are there any examples where this is not the case? If you want performance in your language, you typically write those parts in C, and if it is in C and still not fast enough, you typically go for inline assembly.
fizixer 6y ago
This is great. I'm definitely going to keep Julia in mind the next time my languages self-review comes up, in about 2030. Could even happen as early as 2029.
- xiaodai 6y ago
  If only learn Julia in 2029 you will miss out on top notch productivity for another 9 years! Please reconsider for Ur own sake. If not that then for your whisky.
cozzyd 6y ago
If your CSV file is so large parsing it takes a nontrivial amount of time, why didn't you use a binary format?
- ebg13 6y ago
  And if someone else uses CSV? The people processing data are almost never the people who generated the data.
  - cozzyd 6y ago
    Probably still worth it to convert to a binary format, assuming it's read more than once or twice and is bigger than 10 MB or so.
willvarfar 6y ago
Don’t get me started on Java performance! Java made things fast at the macro level by making threading possible. Remember trying to do any kind of portable multi core in C/C++ in 2000? It was no surprise JVM languages took over and became what we’ve built Big Data on (Hadoop, Hive, spark, etc) But they leave so much on the table for micro performance! Take a CSV or JSON parser for example - likely spends all its time extracting string objects! Now the Big Data community mostly doesn’t notice; for example, the Parquet code is about 5x slower than the Orc code, apparently, but we all adopt Parquet anyhow. I’ve been going through my java codebases doing the basic stuff like “let’s write a simple fast json parser that doesn’t make a lot of garbage” and you can easily half the number of boxes you need etc. sometimes order of magnitude improvements by writing java code that is more like C. But there’s this level you cannot reach. You look longingly at the raw mechanical sympathy C enabled. Perhaps Rust is the future? I expect we’ll just keep using more cores than necessary though.
- karbarcca 6y ago
  Take a deep look at Julia! Adding multithreaded parsing capabilities to CSV.jl was really a joy; basically just chunking up the file and spawning threaded tasks to process each chunk with the existing parsing code. My favorite favorite thing about developing in Julia is the ability to write "high-level" type code and usually get decent performance, BUT THEN have the ability to fine tune for that extra boost of performance. The code inspection tools (see lowered, typed, LLVM, or native IR code levels) and super nice Task-based multithreading really make performance tuning a delight.
  - willvarfar 6y ago
    (Some of the comments in https://news.ycombinator.com/item?id=24736559 https://news.ycombinator.com/item?id=24736559 talk about new lines inside fields, making multi-threading difficult) (My quick search to find that recent story on HN made me find this too https://news.ycombinator.com/item?id=24740527 https://news.ycombinator.com/item?id=24740527)
deleted 6y ago
[deleted]
RobinL 6y ago
If you want to read csvs fast in Python, you should consider using the Apache Arrow[0] csv reader. Depending on your number of CPU cores it can be 10x-20x as fast as the native pandas reader. [1] More broadly, because Arrow is cross platform it can give you similar performance in many languages. And once the dataframe is in memory, you can share it between languages with no need for serialisation and deserialisation. [0] https://arrow.apache.org/ https://arrow.apache.org/ [1] https://youtu.be/fyj4FyH3XdU?t=1036 https://youtu.be/fyj4FyH3XdU?t=1036
- karbarcca 6y ago
  I agree; if I needed to parse CSVs in python and could utilize the arrow format, I would definitely use pyarrow. I actually recently finished support for reading/writing the arrow format in Julia (https://github.com/JuliaData/Arrow.jl https://github.com/JuliaData/Arrow.jl), and it's automatically integrated with the CSV.jl package; so you can do `Arrow.write("data.arrow", CSV.File("data.csv"))` and convert a csv file to arrow format directly. I'm very bullish on arrow as a standard binary data format for the future.
- boxed 6y ago
  Pandas is pretty slow and since it loads into memory it can be totally infeasible for even relatively small data sets. The csv module it what one should compare it to imo.
  - RobinL 6y ago
    Apache Arrow reads csvs into memory in Arrow format, not pandas format. They are independent libraries that do different things. In Arrow, it is possible to read a csv in batches, obviating memory problems. See 'incremental reading' - http://arrow.apache.org/docs/python/csv.html http://arrow.apache.org/docs/python/csv.html
klhugo 6y ago
Here is the thing made me a happier person this year: I realized when CSVs take too long to load, I should not be using them in the first place! I love SQLite now <3
- karbarcca 6y ago
  It's always a transcendental experience to move data from CSV into any more structured/flexible format!
  - klhugo 6y ago
    And fast! I got easily 3 orders of magnitude speedup. I used to load CSVs with pandas and do typical filtering operations... oh boy, some of the stuff would take a hole morning, not to mention the memory usage. I do all that in minutes now.
    - benhurmarcel 6y ago
      Sorry if that sounds stupid, but then when you get a csv file you write it to an SQLite file before doing computations?
dmw_ng 6y ago
There is a ton of conflation in this post, yesterdays post, and the comments. * Comparing parallel implementations (Julia) to single thread (Python csv module). This is still quite relevant because in places like Python, it is impossible to parallelize the csv module without essentially updating its native implementation. That's not true in other places, where some simple wrapper code may be all required to get that "10x faster" number. * Comparing concrete object decoding (Python csv module) to array decoding (PyArrow). Producing concrete objects representing every row (e.g. tuples of strings) will always be slower because of the pressure it puts on the heap and memory. Storing 10 million rows with 10 cols of 8-byte floats in an array might only generate 800 MB of memory bandwidth, the equivalent concrete list-of-floats PyObjects would come out at 6160 MB bandwidth and a ton of CPU burned in the allocator. There are use cases where either representation is preferred, and using e.g. PyArrow's parser then simply iterating the result as concrete objects, worst case the result will be slower than directly decoding to PyObject to begin with. * Lumping floating point decoding (a problem with large performance-correctness tradeoffs) in with CSV parsing. It's hard to decode floats both quickly and precisely, it's also impractical to describe in the context of a comparison of CSV parsers which language/implementation might decode floats better and why that is better.
- spockz 6y ago
  So what about using Flatbuffers[1] to avoid memory allocation of the data and only have the structure allocated additionally? [1]: https://google.github.io/flatbuffers/ https://google.github.io/flatbuffers/
  - secondcoming 6y ago
    Binary will always beat the pants off human-readable formats. I doubt Excel can export to Flatbuffer, Protobuf, MessagePack, Parquet, etc. So CSV is pretty much here to stay, even though correctly parsing it is more difficult that it seems. To gain even from parallelism, make all your rows a fixed size. Arbitrary length fields aren't your friend if you want max speed.
- liuliu 6y ago
  For 1), it is simple, but not trivial to parallelize csv parsing correctly, hence the yesterday’s post. Otherwise, agree on 2 and 3. Object wrapping, number decoding can be slow on the whole “file to dataframe object” path, but trivially parallelizable.
echelon 6y ago
I think there's a strong chance that Swift or Rust take a lot of Python's data science cake. Both of them are extremely fast, concrete, and have lots of investment being poured into numerics and fitting into the Python/ML ecosystem. I don't think Julia or R are going to steal this away. In fact, I think Julia is a major turn off to engineers with some of the bizarre choices they made (eg. 1-based indexing to appease math-centric backgrounds). If you don't excite engineers, you're not going to get as many library and optimization contributions. Julia feels like it came too late in the game and the newer entrants are making quick strides in the mathematics space. I could be totally wrong. This is just my take, and I have no stake in this fight other than I want a good, safe, highly-compatible, and fast ecosystem to work with.
- ddragon 6y ago
  Not sure what you mean about being a turn-off to engineers. I'm an electronic engineer currently working on production data engineering pipelines (which has no relation to the engineering in my degree) and I quite like the language. 1-based index is such a small "problem" compared to actual problems you encounter when developing software that is not even worth commenting in this context. I work on a polyglot company, and parts of the pipeline have different demands. For low latency and fast development speed we use Elixir, for high throughput we use Scala, for data analytics in batch jobs we use Python. As I see, Julia is a language that is more concise and faster than Python for the number crunching I do, and has the potential to match Scala's throughput all while having multithreading/distributed that could rival the erlang VM (not in latency, but in speed and easy to use) and Scala's Akka eventually. So as an engineer I'm excited about the potential (a single language that you can develop as fast as python that could compete with some of the top tiers in different areas of data engineering), even if it's not yet. It needs to complete the multithreading (structured parallelism so you can safely monitor and restart threads to help with reliability and with no risk of leaking, and have some library that works at the level of abstraction as the OTP/Akka), it needs native support for the infra-structure (Kafka, Prometheus) and a solid support for the web (and both of those last two can be done right now with Julia 1.5). For now it's an amazing language for exploratory data analysis and research, but I can't wait for the moment that I can safely recommend as the backend of my company's data infra-structure.
ku-man 6y ago
I'm glad Julia does exist. It's great to have a free and open-source programming language focused on scientific computing. What I dislike from the Julia community is the unhealthy fixation they have with Python and R. Virtually, no comment or blog-post is written without mentioning how slow, inefficient, inappropriate, inelegant is Python or R. Somehow, the Julia community convinced itself that best way to attract more users is to constantly put down Python and R. A sad state of affairs definetely.
- bachmeier 6y ago
  Completely agree. I was turned off by two things when Julia started getting hyped. The first was the benchmarks they reported, which were, shall I say, carefully selected. This is a perfect example. As someone else pointed out, R has much faster ways to read csv files than what they used. The thing is, most of the time it doesn't make any difference at all. The other was the nonstop attacks on other languages, many of which represented the ignorance of the critic (not unrelated to the first point) rather than deep insight.
  - StefanKarpinski 6y ago
    For the historical record, when we picked those benchmarks [1] we were terrible at them. Like 10,000x slower than C. We picked them because they were representative of the kinds of things that we felt a language should be good at: iteration, recursion, shuffling data in arrays, floating-point operations, complex number arithmetic, string operations, small matrix operations and large matrix operations. Then we worked on the language until it was no worse than 2x slower than C. So claiming that we cherry-picked those benchmarks because they made Julia look good simply isn't historically accurate: we picked them because they made Julia look bad, and then we improved Julia until it didn't look bad anymore. [1] https://julialang.org/benchmarks/ https://julialang.org/benchmarks/ The only faster way to read CSV files in R that I'm aware of is vroom, which is faster because it's lazy: it doesn't parse until you actually access the data. According to vroom's own benchmarks, if you access all the data, it is slower than data.table, which is compared here. So yes, if you want to load a CSV file and not access most of the data, then vroom is faster; if you want to access all of the data (which seems pretty typical), it is not faster. Regarding "nonstop attacks on other languages", honestly, we don't care and would be happy to stop talking about other languages. However, there is a constant barrage of people making comparisons to other languages and demanding that we justify Julia's existence. Like it is literally offensive to people that Julia exists — "why didn't you just use Python/R/Matlab?!??!" I cannot express how continual this is. So many people have asked to see these exact benchmarks comparing CSV.jl to read.table and Pandas' csv reader. So here are the benchmarks. If you see a mildly worded benchmark comparison showing that one CSV library is faster than some others as "an attack", then you are too good for the internet.
burntsushi 6y ago
I posted this comment on lobste.rs when this came up:[1] It kind of looks like Julia’s CSV parser is cheating: https://github.com/JuliaData/CSV.jl/blob/9f6ef108d195f85daa535d23d398253a7ca52e20/src/detection.jl#L304-L309 https://github.com/JuliaData/CSV.jl/blob/9f6ef108d195f85daa5... It’s doing parallel parsing, but I’m pretty sure their technique won’t work for all inputs. Namely, they try to hop around the CSV data and chunk it up, and then parse each chunk in a separate thread AIUI. But you can’t do this in general because of quoting. If you read the code around where I linked, you can see they try to be a bit speculative and avoid common failures (“now we read the next 5 rows and see if we get the right # of columns”), but that isn’t going to be universally correct. It might be a fair trade off to make, since CSV data that fails there is probably quite rare. But either I’m misunderstanding their optimization or they aren’t being transparent about it. I don’t see this downside anywhere in the README or the benchmark article. [1] - https://lobste.rs/s/zksa0f/loading_csv_file_at_speed_limit_nvme#c_y2fkbj https://lobste.rs/s/zksa0f/loading_csv_file_at_speed_limit_n...
- Sukera 6y ago
  I wouldn't call that cheating, how else would you multithread this? You basically have to chunk it up and try to find a boundary and from what I can tell, they try to find out if they're in a quoted block. If they are, they find the end of the quoted block (or EOF, which shouldn't happen => broken file) or ask the user to report the bug. This is more like failsaving, not cheating.
  - lifthrasiir 6y ago
    > [...] how else would you multithread this? Use a fast "frontier" thread to determine quotes and split works at the safe boundary. I'm personally okay with the speculative execution as long as its result is correct and I wouldn't consider it cheating. But it is concerning that the benchmark doesn't list all the caveats (as all proper benchmarks should do, like [1], which is incidentally written by who you are replying to). I can't even determine whether the machine is hitting I/O bound or not from the OP! [1] https://blog.burntsushi.net/ripgrep/ https://blog.burntsushi.net/ripgrep/
  - 6y ago
ggrothendieck 6y ago
The vroom R package is likely the fastest R package in read speed, not the ones used in this article. vroom was 13.1x faster than R's fread/data.table based on the read performance benchmark in https://cran.r-project.org/web/packages/vroom/vignettes/benchmarks.html https://cran.r-project.org/web/packages/vroom/vignettes/benc... so may be similar or faster in read speed vs julia.
- nojito 6y ago
  vroom doesn't really read files initially though. Which is why the comparison isn't 1:1.
- xiaodai 6y ago
  Also Vroom never Worked For mE
- StefanKarpinski 6y ago
  Vroom is lazy though: it doesn’t parse until you access the data. That’s a very different approach, which is why it’s no direct comparison. You’ll note that data.table, which is compared in the article, is faster than vroom’s default configuration when it comes to whole table scans, which would be the appropriate comparison. If the data is mostly numeric, then data.table is faster for all configurations. That said, if you don’t plan to access most of the data in a data set, then vrooms lazy approach can be as big win.
jiggunjer 6y ago
I'm still looking for a nice way to read password protected xlsx files in R....
ryansmccoy 6y ago
I'm pretty sure you can add multiprocessing parsing of csv with Python and Pandas. I think I've done it before.
vmchale 6y ago
Wonder how it performs compared to the Jd csv reader
nnm 6y ago
> The tools used for benchmarking were BenchmarkTools.jl for Julia, microbenchmark for R, and timeit for Python. Is this a meaningful comparison? These benchmarks repeatedly call the function (with same input file) many times and then take the average of the time spent. However, in real world, we only read a specific file ONCE -- if we call the csv reader several times, it is almost always for different files.
- oizin 6y ago
  I think this is an important point, if you look at the warmup run on [1] the Julia CSV readers are slower than many of the R and python packages . It makes the claims of 10-20x faster quite disengenuous in my opinion. A lot of data science work will only read the file on once per session and what these benchmarks actually suggest for that sort of work is not reflected in the title. [1] https://www.queryverse.org/benchmarks/ https://www.queryverse.org/benchmarks/
- StefanKarpinski 6y ago
  There are two potential issues that this might disregard: 1. cold file cache 2. JIT compile time The cold/hot file cache issue affects all languages equally, so it doesn’t invalidate the comparison. It is possible that CSV file is not in cache and a system's disk I/O is slow enough that it becomes a bottleneck, which would make all parsers equally slow. However, this is not very realistic because CSV files—especially large enough ones where you care about speed—are almost always compressed—So you're not going to be bottlenecked on disk read. Assuming that compressed disk read + decompression is fast enough to keep the data flow high, you're back to CSV parsing being the bottleneck. The JIT compile cache only affects Julia. The reason is it not included in the benchmark results is because it is a small, fixed overhead: it is only paid on the first CSV file read (no matter the size), and if you read a larger data set, the compile time does not increase. Since the point of benchmarks is typically to project from a smaller case how long it would take to do even larger tasks, you don't want to include small, fixed overheads. For some concrete numbers, I just timed reading a tiny CSV file on my MacBook Pro 2018 and the first read, including compile time took 4 seconds. The second read took 0.000347 seconds. So that's the fixed overhead we're talking about here for reading a CSV file: about 4 seconds on the very first CSV file you read. People can be the judge of whether that's a showstopper for them or not.
fnord123 6y ago
> The tools used for benchmarking were BenchmarkTools.jl for Julia, microbenchmark for R, and timeit for Python. Using different tools to measure performance between platforms? > https://github.com/JuliaCI/BenchmarkTools.jl/blob/master/src/execution.jl#L287 https://github.com/JuliaCI/BenchmarkTools.jl/blob/master/src... Using a warmup to get the slow JIT within the same galaxy as the C implementations and not mentioning it in the post?
glangdale 6y ago
Frankly, all these implementations are disappointingly slow. Daniel Lemire and I wrote simdjson to use SIMD to read JSON quickly - CSV is strictly easier than that. I made a start on simdcsv but got bored, but the same principles would apply. IMO this task should be doable at 1-2GB/s on a single core; it's not something that should really need multiple cores.
- liuliu 6y ago
  Agreed. But doing SIMD optimizations could quickly become 2-3 weeks project for me just for one platform. Adding NEON would be another 1 or 2 weeks. Need much more dedication I guess. Your work on simdjson is awesome! Thanks for saving the world so much unnecessary power consumptions :)
  - glangdale 6y ago
    Thanks for the kind words. I suppose this confirms my general suspicion that principles of simdjson need to be lifted out into a more general tool that allows people that don't live and breathe SIMD (strange that such people exist :-) ) to still get the benefit of these techniques for formats that aren't JSON.
purzelrakete 6y ago
I built a pagerank implementation in Julia many years ago. It was operating on an adjacency list with half a billion edges, and I had to resort to writing a custom reader to get reasonable read speeds over the csv. In case anyone is interested: https://github.com/purzelrakete/Pagerank.jl/blob/070a193826f6ff7c2fe11fb5556f21d039b82e26/src/io.jl#L47 https://github.com/purzelrakete/Pagerank.jl/blob/070a193826f...
mr_toad 6y ago
From where? I couldn’t see anywhere in the article that specifies where the data is stored.
xiaodai 6y ago
Julia gets alot of hate nowadays that's for sure https://twitter.com/danluu/status/1315407713430458368?s=19 https://twitter.com/danluu/status/1315407713430458368?s=19
4gotunameagain 6y ago
The problem with CSV files are the endless edge cases. What I'll typically do for larger ones is that I'll write a program that does the reading and the rudimentary processing in C, and then saves the data as a binary .npy file in /dev/shm (or ImDisk if I'm on windows) and load that up in Python. Massage the data more if necessary and finally do the plotting. I found that this helps, hope it helps you too

177 points by nemoniac 6y ago

qaq 6y ago

Julia is a hidden gem Once ecosystem for web dev matures Julia will be the killer lang for building web apps

dmos62 6y ago
> Julia will be the killer lang for building web apps That would be fun, but Julia's community aren't web devs. Julia spawned around very specific needs of scientific computing, which is characterised by a short-running daemon or a script-type interpreter. A web server is a long-running process. Not knowledgeable enough about Julia to tell how it lends itself to server uses, but heard hearsay that it's problematic.
- ViralBShah 6y ago
  See Jacob Quinn's (karbarcca) other package: https://github.com/JuliaWeb/HTTP.jl https://github.com/JuliaWeb/HTTP.jl It is quite common for scientists and engineers to want to present the findings of their work or share them with others who are non-coders. Thus you get tools like Jupyter, Shiny, Pluto, etc. There are several long running Julia processes in real applications. While there's no doubt that there is work to be done, the set of these applications is not the null set either.
- langitbiru 6y ago
  Seconded this. If you looked at the Julia website (https://julialang.org https://julialang.org), in the Ecosystem section, they listed these things: "Visualization, General Purpose, Data Science, Machine Learning, Scientific Domains, Parallel Computing." They already positioned themselves in scientific computing battleground. Web development (or desktop programming) is not their focus (for better or worse).
  - qaq 6y ago
    Julia is very capable general purpose lang and web libs are maturing pretty fast. It does not have to be the focus of core team for it to happen.
- comicjk 6y ago
  I don't think you can define scientific computing in a way that excludes quantum mechanics, molecular dynamics, fluid dynamics, Monte Carlo simulations... There are lots of long-running applications. We don't want them to be long-running, but it's not unusual for a run to take weeks and not unheard-of for it to take months.
akerro 6y ago
Rust is already killing old web app building technologies!
- qaq 6y ago
  Julia is a good bit easier to pick up than Rust. I like Rust btw. but say at my job chances of us adopting Rust would be close to 0. Julia is really easy to pick up so would be much easier sell once ecosystem is mature.
crispyambulance 6y ago
> Once ecosystem for web dev matures Julia will be the killer... Looking back at history, javascript was put together in haste, almost like an afterthought. Even the name was a stupid marketing tack-on. Yet it stuck and kept going even when people became sick of it. It has reinvented itself several times over, and in spite of the emergence of multiple superior technologies that were specifically targeted at web-development, it still stands at the top of hill. I guess it would be properly ironic that something designed for mathematics would take down javascript for web-dev, but I doubt it. It would be amusing though.
deathtrader666 6y ago
Genie.jl exists for web apps.
- qaq 6y ago
  In no way did I mean to imply there are no packages, I just noted they are not yet as mature as compared to web frameworks/packages in other langs.

mushufasa 6y ago

this is neat. that said, to me, this is like comparing car startup times. even if a tesla starts faster than a toyota, they're both fast enough to not notice 99.999% of the time. This isn't a factor in a purchasing decision -- i've never even thought of that before contriving this metaphor

lxe 6y ago
Maybe not a deciding factor, but definitely a factor in a purchasing decision, among many others.

CJefferson 6y ago

EDIT: Be sure to read the reply from the author of the package. I do think Julia has promise, but it is still early days. In the last 2 months there have been problems with: * Throwing rows away: https://github.com/JuliaData/CSV.jl/issues/720 https://github.com/JuliaData/CSV.jl/issues/720 * Incorrect float parsing: https://github.com/JuliaData/CSV.jl/issues/714 https://github.com/JuliaData/CSV.jl/issues/714 At there are unfixed issues to do with parsing large files ( https://github.com/JuliaData/CSV.jl/issues/749 https://github.com/JuliaData/CSV.jl/issues/749 , https://github.com/JuliaData/CSV.jl/issues/734 https://github.com/JuliaData/CSV.jl/issues/734 ). While Julia has become much more stable than it used to be, it is still in many places an evolving system and you (in my limited experience) should be prepared to hit issues.

fermienrico 6y ago
With all due respect to all the people working on Julia, I see a stark difference in rigor between say Python's PEP / RFCs and how Julia's core developers operate. The former is a strict formal process with a lot of thought given to a particular PEP proposal, whereas the latter seems like an informal chaotic hashing/bikeshedding between a few key people in Julia community. I've contributed to Julia and people are really nice but IMO it is missing structure, discipline, hardline regimented approach of confronting difficult problems, etc. One feels like hearing a supreme court case and the other a street brawl to decide upon important issues.
- xyzzy_plugh 6y ago
  > Python's PEP / RFCs Is this process actually fundamentally better? There are so many PEPs that are up to the reader to interpret. There are so many PEPs that contradict each other (just read some of the early PEPs). The whole thing feels basically meaningless, and toothless at this point. I think Go's proposal process nails balancing formalities and discussion/bikeshedding. They seem to be very productive at making the language better, thinking critically and avoiding regrets, rather than pandering to the very vocal and often misaligned minority of the community.
  - 6y ago

mcrad 6y ago

> The very first task in any data analysis workflow is simply reading the data, and this absolutely must be done quickly. Lost me here. For what use-case of data analysis workflow does saving a minute or two make a difference? When will you tech bros learn to separate web dev from data science?

deleted 6y ago
[deleted]
gameswithgo 6y ago
I can think of many cases where that is useful. It is important to remember not all workflows and needs are the same as yours. I don’t really understand tour web dev comment either. Web devs are notoriously not interested in performance.
- fermienrico 6y ago
  Then it goes both ways right? I think we need to also say "For most people Python is fine, if you need that extra speed, try Julia" instead of blanket praises of Julia as true competitor to Python (It doesn't even come close IMO).
- mcrad 6y ago
  Maybe I don't understand web dev; only meant that when you have many users besides yourself (analyzing data) then performance matters more. It's important to remember, wtf?
karbarcca 6y ago
It can make huge amounts of difference in a production system; where I work, we process terabytes of csv data every day; saving minutes per file can add up to enormous differences in CPU cost/time for a production system running 24/7. I agree that for a data scientist doing exploratory analysis locally on their computer, it doesn't make nearly as much a difference (also because they're usually not working on crazy large files). The performance work in the CSV.jl package (that the article is about) was very much geared towards these kinds of production scenarios.

deleted 6y ago

[deleted]

mrfusion 6y ago

I’d really think all these languages would have the bottleneck of disk read speed. So how can Julia be that much faster? Are the others really that inefficient?

dagw 6y ago
Without having looked at it, it might also be a case that the other libraries spend more time handling more obscure corner cases and doing more robust error handling.
mynegation 6y ago
Because it is not a bottleneck. Modern SSDs can read data at more than 500 megabytes per second, which, on a single thread gives you just about 6-8 clock cycles to process a single byte. That’s not a lot of CPU instructions so it will take quite a number of threads to handle this bandwidth.
- jdietrich 6y ago
  The SSD in my current workstation tops out at 5GB/s sequential reads and 680kIOPS. That's a modestly high-end consumer M.2 drive, not enterprise exotica. SSDs are really, really fast.
  - xiaodai 6y ago
    Yeah Intel optane and the like will be even faster.
karbarcca 6y ago
The biggest difference in these benchmarks comes down to how multiple threads are leveraged; (disclaimer: primary CSV.jl author here). In CSV.jl, it was relatively straightforward to add multithreaded parsing support; we chunk up the file, find row starts, then spawn a threaded task to process each chunk. In data.table (fread), they have a constraint of R having a global String intern store; so it fundamentally restricts the multithreading capabilities by requiring strings to be interned sequentially. In pandas, there are similar nontrivialities jumping between the C++ source and python object world that adds a lot of complexity (and probably explains why no one has made the effort to do multithreaded parsing). It's interesting to note, however, that the apache arrow project (pyarrow package in Python), has a multithreaded csv parser integrated with the project. It provides similar performance to CSV.jl because it was built from the ground up to process chunks on multiple threads into "arrow tables". Unfortunately, it's tied very specifically to the arrow project, so pandas doesn't benefit from its work! It's one of my favorite features about Julia that CSV.jl automatically integrates with other "data table formats"; i.e. ODBC.jl, SQLite.jl, MySQL.jl, Arrow.jl, DataFrames.jl, etc. They can all leverage CSV.jl as "their csv parser" because it's seamlessly integrated via well-established "table" apis.

bonoboTP 6y ago

Maybe I'm a cranky old man, but I think there's immense value in the fact that Python's (scientific and non-scientific) ecosystem has matured and has become standard over the last 1-2 decades. Starting again from the ground up in a shiny new language (even if the language is perhaps slightly better) is a waste of so much effort. Transitioning from Python 2 to 3 was already a nightmare. I know there's some interoperability between Julia and Python, but a lot of stuff is not interoperable, and there are annoying differences in array indexing etc. Why can't people just settle on some tech, let it mature, and enjoy reaching higher and higher in capability? Why do we have to always tear things down and start from scratch? This causes a lot of artificial bit rot, there's a never ending treadmill of just porting well-developed concepts and mature code to slightly changed representations. It's the busywork of tech. Does the scientific community also want to follow web dev in their insane churn of frameworks and libraries where last week's best practice is laughable ancient stone-age primitiveness today?

Datenstrom 6y ago
There is an immense value in most of the ecosystem converging on Python over the last decade and especially with leaving MATLAB behind in the dust. I don't think decade long transitions from old tech to new tech is a bad time scale or even close to that of the churn of web technologies. Julia offers enough of an improvement over Python to warrant a switch over the next 5-10 years and leave Python behind in the same way. I believe Julia offers the same level of improvement over Python as Python does over MATLAB. I also believe that it has drawn too much from MATLAB only for the sake of being familiar to that group and not the computer science and software engineering communities though such as the one based indexing you mentioned. Unfortunately that ship has already sailed and we will likely have to wait another 10 years for the next one to hopefully fix those problems.
- ScottPJones 6y ago
  Back 5.5 years ago, I used to complain about the 1 based indexing and the column-major structure of matrices in Julia (both like Fortran), however, those issues have been solved by OffsetArrays and PermutedDimsArrays, giving far more flexibility that is possible in most other languages. It's silly to keep bringing up the issue of one based indexing, when you can use any integer as the base, just like Fortran 90 (so you can index by -5..5, for example). For some things, 0 based does make things easier, sure, but you can do that easily in Julia (and more!)

varbhat 6y ago

Well, Julia is compiled .Python and R are both interpreted. I think that simdjson is fastest json parser out there( Parsing gigabytes of JSON per second ). https://github.com/simdjson/simdjson https://github.com/simdjson/simdjson So,if I use Python binding for simdjson , then parsing of json in python must be very much faster than the fastest one mentioned in the above post. So,at the end,it depends on implementation not only on language.

bjoli 6y ago
Python's CSV parser is in C. I don't think a native python CSV parser would be nearly as fast.
- ViralBShah 6y ago
  You can write slow programs in all languages. There are many cases where Julia's language features enable higher performance than in comparable C libraries. Have a look at Steve Johnson's keynote from JuliaCon 2019: https://www.youtube.com/watch?v=mSgXWpvQEHE https://www.youtube.com/watch?v=mSgXWpvQEHE
  - bjoli 6y ago
    Pythons CSV reader is actually decently fast. It is just that speed wasnt a priority. I dislike the whole "faster than C" comparisons. Almost nothing is. If you want speed you chose C. If you want something that is plenty fast with a much better speed-to-effort ratio, Julia is a strong contender.
    - johnisgood 6y ago
      Exactly. Nothing is faster than C that is also high-level. If it is, then the C version is not equivalent to the code with what you are comparing. Or are there any examples where this is not the case? If you want performance in your language, you typically write those parts in C, and if it is in C and still not fast enough, you typically go for inline assembly.

fizixer 6y ago

This is great. I'm definitely going to keep Julia in mind the next time my languages self-review comes up, in about 2030. Could even happen as early as 2029.

xiaodai 6y ago
If only learn Julia in 2029 you will miss out on top notch productivity for another 9 years! Please reconsider for Ur own sake. If not that then for your whisky.

cozzyd 6y ago

If your CSV file is so large parsing it takes a nontrivial amount of time, why didn't you use a binary format?

ebg13 6y ago
And if someone else uses CSV? The people processing data are almost never the people who generated the data.
- cozzyd 6y ago
  Probably still worth it to convert to a binary format, assuming it's read more than once or twice and is bigger than 10 MB or so.

willvarfar 6y ago

Don’t get me started on Java performance! Java made things fast at the macro level by making threading possible. Remember trying to do any kind of portable multi core in C/C++ in 2000? It was no surprise JVM languages took over and became what we’ve built Big Data on (Hadoop, Hive, spark, etc) But they leave so much on the table for micro performance! Take a CSV or JSON parser for example - likely spends all its time extracting string objects! Now the Big Data community mostly doesn’t notice; for example, the Parquet code is about 5x slower than the Orc code, apparently, but we all adopt Parquet anyhow. I’ve been going through my java codebases doing the basic stuff like “let’s write a simple fast json parser that doesn’t make a lot of garbage” and you can easily half the number of boxes you need etc. sometimes order of magnitude improvements by writing java code that is more like C. But there’s this level you cannot reach. You look longingly at the raw mechanical sympathy C enabled. Perhaps Rust is the future? I expect we’ll just keep using more cores than necessary though.

karbarcca 6y ago
Take a deep look at Julia! Adding multithreaded parsing capabilities to CSV.jl was really a joy; basically just chunking up the file and spawning threaded tasks to process each chunk with the existing parsing code. My favorite favorite thing about developing in Julia is the ability to write "high-level" type code and usually get decent performance, BUT THEN have the ability to fine tune for that extra boost of performance. The code inspection tools (see lowered, typed, LLVM, or native IR code levels) and super nice Task-based multithreading really make performance tuning a delight.
- willvarfar 6y ago
  (Some of the comments in https://news.ycombinator.com/item?id=24736559 https://news.ycombinator.com/item?id=24736559 talk about new lines inside fields, making multi-threading difficult) (My quick search to find that recent story on HN made me find this too https://news.ycombinator.com/item?id=24740527 https://news.ycombinator.com/item?id=24740527)

deleted 6y ago

[deleted]

RobinL 6y ago

If you want to read csvs fast in Python, you should consider using the Apache Arrow[0] csv reader. Depending on your number of CPU cores it can be 10x-20x as fast as the native pandas reader. [1] More broadly, because Arrow is cross platform it can give you similar performance in many languages. And once the dataframe is in memory, you can share it between languages with no need for serialisation and deserialisation. [0] https://arrow.apache.org/ https://arrow.apache.org/ [1] https://youtu.be/fyj4FyH3XdU?t=1036 https://youtu.be/fyj4FyH3XdU?t=1036

karbarcca 6y ago
I agree; if I needed to parse CSVs in python and could utilize the arrow format, I would definitely use pyarrow. I actually recently finished support for reading/writing the arrow format in Julia (https://github.com/JuliaData/Arrow.jl https://github.com/JuliaData/Arrow.jl), and it's automatically integrated with the CSV.jl package; so you can do `Arrow.write("data.arrow", CSV.File("data.csv"))` and convert a csv file to arrow format directly. I'm very bullish on arrow as a standard binary data format for the future.
boxed 6y ago
Pandas is pretty slow and since it loads into memory it can be totally infeasible for even relatively small data sets. The csv module it what one should compare it to imo.
- RobinL 6y ago
  Apache Arrow reads csvs into memory in Arrow format, not pandas format. They are independent libraries that do different things. In Arrow, it is possible to read a csv in batches, obviating memory problems. See 'incremental reading' - http://arrow.apache.org/docs/python/csv.html http://arrow.apache.org/docs/python/csv.html

klhugo 6y ago

Here is the thing made me a happier person this year: I realized when CSVs take too long to load, I should not be using them in the first place! I love SQLite now <3

karbarcca 6y ago
It's always a transcendental experience to move data from CSV into any more structured/flexible format!
- klhugo 6y ago
  And fast! I got easily 3 orders of magnitude speedup. I used to load CSVs with pandas and do typical filtering operations... oh boy, some of the stuff would take a hole morning, not to mention the memory usage. I do all that in minutes now.
  - benhurmarcel 6y ago
    Sorry if that sounds stupid, but then when you get a csv file you write it to an SQLite file before doing computations?

dmw_ng 6y ago

There is a ton of conflation in this post, yesterdays post, and the comments. * Comparing parallel implementations (Julia) to single thread (Python csv module). This is still quite relevant because in places like Python, it is impossible to parallelize the csv module without essentially updating its native implementation. That's not true in other places, where some simple wrapper code may be all required to get that "10x faster" number. * Comparing concrete object decoding (Python csv module) to array decoding (PyArrow). Producing concrete objects representing every row (e.g. tuples of strings) will always be slower because of the pressure it puts on the heap and memory. Storing 10 million rows with 10 cols of 8-byte floats in an array might only generate 800 MB of memory bandwidth, the equivalent concrete list-of-floats PyObjects would come out at 6160 MB bandwidth and a ton of CPU burned in the allocator. There are use cases where either representation is preferred, and using e.g. PyArrow's parser then simply iterating the result as concrete objects, worst case the result will be slower than directly decoding to PyObject to begin with. * Lumping floating point decoding (a problem with large performance-correctness tradeoffs) in with CSV parsing. It's hard to decode floats both quickly and precisely, it's also impractical to describe in the context of a comparison of CSV parsers which language/implementation might decode floats better and why that is better.

spockz 6y ago
So what about using Flatbuffers[1] to avoid memory allocation of the data and only have the structure allocated additionally? [1]: https://google.github.io/flatbuffers/ https://google.github.io/flatbuffers/
- secondcoming 6y ago
  Binary will always beat the pants off human-readable formats. I doubt Excel can export to Flatbuffer, Protobuf, MessagePack, Parquet, etc. So CSV is pretty much here to stay, even though correctly parsing it is more difficult that it seems. To gain even from parallelism, make all your rows a fixed size. Arbitrary length fields aren't your friend if you want max speed.
liuliu 6y ago
For 1), it is simple, but not trivial to parallelize csv parsing correctly, hence the yesterday’s post. Otherwise, agree on 2 and 3. Object wrapping, number decoding can be slow on the whole “file to dataframe object” path, but trivially parallelizable.

echelon 6y ago

I think there's a strong chance that Swift or Rust take a lot of Python's data science cake. Both of them are extremely fast, concrete, and have lots of investment being poured into numerics and fitting into the Python/ML ecosystem. I don't think Julia or R are going to steal this away. In fact, I think Julia is a major turn off to engineers with some of the bizarre choices they made (eg. 1-based indexing to appease math-centric backgrounds). If you don't excite engineers, you're not going to get as many library and optimization contributions. Julia feels like it came too late in the game and the newer entrants are making quick strides in the mathematics space. I could be totally wrong. This is just my take, and I have no stake in this fight other than I want a good, safe, highly-compatible, and fast ecosystem to work with.

ddragon 6y ago
Not sure what you mean about being a turn-off to engineers. I'm an electronic engineer currently working on production data engineering pipelines (which has no relation to the engineering in my degree) and I quite like the language. 1-based index is such a small "problem" compared to actual problems you encounter when developing software that is not even worth commenting in this context. I work on a polyglot company, and parts of the pipeline have different demands. For low latency and fast development speed we use Elixir, for high throughput we use Scala, for data analytics in batch jobs we use Python. As I see, Julia is a language that is more concise and faster than Python for the number crunching I do, and has the potential to match Scala's throughput all while having multithreading/distributed that could rival the erlang VM (not in latency, but in speed and easy to use) and Scala's Akka eventually. So as an engineer I'm excited about the potential (a single language that you can develop as fast as python that could compete with some of the top tiers in different areas of data engineering), even if it's not yet. It needs to complete the multithreading (structured parallelism so you can safely monitor and restart threads to help with reliability and with no risk of leaking, and have some library that works at the level of abstraction as the OTP/Akka), it needs native support for the infra-structure (Kafka, Prometheus) and a solid support for the web (and both of those last two can be done right now with Julia 1.5). For now it's an amazing language for exploratory data analysis and research, but I can't wait for the moment that I can safely recommend as the backend of my company's data infra-structure.

ku-man 6y ago

I'm glad Julia does exist. It's great to have a free and open-source programming language focused on scientific computing. What I dislike from the Julia community is the unhealthy fixation they have with Python and R. Virtually, no comment or blog-post is written without mentioning how slow, inefficient, inappropriate, inelegant is Python or R. Somehow, the Julia community convinced itself that best way to attract more users is to constantly put down Python and R. A sad state of affairs definetely.

bachmeier 6y ago
Completely agree. I was turned off by two things when Julia started getting hyped. The first was the benchmarks they reported, which were, shall I say, carefully selected. This is a perfect example. As someone else pointed out, R has much faster ways to read csv files than what they used. The thing is, most of the time it doesn't make any difference at all. The other was the nonstop attacks on other languages, many of which represented the ignorance of the critic (not unrelated to the first point) rather than deep insight.
- StefanKarpinski 6y ago
  For the historical record, when we picked those benchmarks [1] we were terrible at them. Like 10,000x slower than C. We picked them because they were representative of the kinds of things that we felt a language should be good at: iteration, recursion, shuffling data in arrays, floating-point operations, complex number arithmetic, string operations, small matrix operations and large matrix operations. Then we worked on the language until it was no worse than 2x slower than C. So claiming that we cherry-picked those benchmarks because they made Julia look good simply isn't historically accurate: we picked them because they made Julia look bad, and then we improved Julia until it didn't look bad anymore. [1] https://julialang.org/benchmarks/ https://julialang.org/benchmarks/ The only faster way to read CSV files in R that I'm aware of is vroom, which is faster because it's lazy: it doesn't parse until you actually access the data. According to vroom's own benchmarks, if you access all the data, it is slower than data.table, which is compared here. So yes, if you want to load a CSV file and not access most of the data, then vroom is faster; if you want to access all of the data (which seems pretty typical), it is not faster. Regarding "nonstop attacks on other languages", honestly, we don't care and would be happy to stop talking about other languages. However, there is a constant barrage of people making comparisons to other languages and demanding that we justify Julia's existence. Like it is literally offensive to people that Julia exists — "why didn't you just use Python/R/Matlab?!??!" I cannot express how continual this is. So many people have asked to see these exact benchmarks comparing CSV.jl to read.table and Pandas' csv reader. So here are the benchmarks. If you see a mildly worded benchmark comparison showing that one CSV library is faster than some others as "an attack", then you are too good for the internet.

burntsushi 6y ago

I posted this comment on lobste.rs when this came up:[1] It kind of looks like Julia’s CSV parser is cheating: https://github.com/JuliaData/CSV.jl/blob/9f6ef108d195f85daa535d23d398253a7ca52e20/src/detection.jl#L304-L309 https://github.com/JuliaData/CSV.jl/blob/9f6ef108d195f85daa5... It’s doing parallel parsing, but I’m pretty sure their technique won’t work for all inputs. Namely, they try to hop around the CSV data and chunk it up, and then parse each chunk in a separate thread AIUI. But you can’t do this in general because of quoting. If you read the code around where I linked, you can see they try to be a bit speculative and avoid common failures (“now we read the next 5 rows and see if we get the right # of columns”), but that isn’t going to be universally correct. It might be a fair trade off to make, since CSV data that fails there is probably quite rare. But either I’m misunderstanding their optimization or they aren’t being transparent about it. I don’t see this downside anywhere in the README or the benchmark article. [1] - https://lobste.rs/s/zksa0f/loading_csv_file_at_speed_limit_nvme#c_y2fkbj https://lobste.rs/s/zksa0f/loading_csv_file_at_speed_limit_n...

Sukera 6y ago
I wouldn't call that cheating, how else would you multithread this? You basically have to chunk it up and try to find a boundary and from what I can tell, they try to find out if they're in a quoted block. If they are, they find the end of the quoted block (or EOF, which shouldn't happen => broken file) or ask the user to report the bug. This is more like failsaving, not cheating.
- lifthrasiir 6y ago
  > [...] how else would you multithread this? Use a fast "frontier" thread to determine quotes and split works at the safe boundary. I'm personally okay with the speculative execution as long as its result is correct and I wouldn't consider it cheating. But it is concerning that the benchmark doesn't list all the caveats (as all proper benchmarks should do, like [1], which is incidentally written by who you are replying to). I can't even determine whether the machine is hitting I/O bound or not from the OP! [1] https://blog.burntsushi.net/ripgrep/ https://blog.burntsushi.net/ripgrep/
- 6y ago

ggrothendieck 6y ago

The vroom R package is likely the fastest R package in read speed, not the ones used in this article. vroom was 13.1x faster than R's fread/data.table based on the read performance benchmark in https://cran.r-project.org/web/packages/vroom/vignettes/benchmarks.html https://cran.r-project.org/web/packages/vroom/vignettes/benc... so may be similar or faster in read speed vs julia.

nojito 6y ago
vroom doesn't really read files initially though. Which is why the comparison isn't 1:1.
xiaodai 6y ago
Also Vroom never Worked For mE
StefanKarpinski 6y ago
Vroom is lazy though: it doesn’t parse until you access the data. That’s a very different approach, which is why it’s no direct comparison. You’ll note that data.table, which is compared in the article, is faster than vroom’s default configuration when it comes to whole table scans, which would be the appropriate comparison. If the data is mostly numeric, then data.table is faster for all configurations. That said, if you don’t plan to access most of the data in a data set, then vrooms lazy approach can be as big win.

jiggunjer 6y ago

I'm still looking for a nice way to read password protected xlsx files in R....

ryansmccoy 6y ago

I'm pretty sure you can add multiprocessing parsing of csv with Python and Pandas. I think I've done it before.

vmchale 6y ago

Wonder how it performs compared to the Jd csv reader

nnm 6y ago

> The tools used for benchmarking were BenchmarkTools.jl for Julia, microbenchmark for R, and timeit for Python. Is this a meaningful comparison? These benchmarks repeatedly call the function (with same input file) many times and then take the average of the time spent. However, in real world, we only read a specific file ONCE -- if we call the csv reader several times, it is almost always for different files.

oizin 6y ago
I think this is an important point, if you look at the warmup run on [1] the Julia CSV readers are slower than many of the R and python packages . It makes the claims of 10-20x faster quite disengenuous in my opinion. A lot of data science work will only read the file on once per session and what these benchmarks actually suggest for that sort of work is not reflected in the title. [1] https://www.queryverse.org/benchmarks/ https://www.queryverse.org/benchmarks/
StefanKarpinski 6y ago
There are two potential issues that this might disregard: 1. cold file cache 2. JIT compile time The cold/hot file cache issue affects all languages equally, so it doesn’t invalidate the comparison. It is possible that CSV file is not in cache and a system's disk I/O is slow enough that it becomes a bottleneck, which would make all parsers equally slow. However, this is not very realistic because CSV files—especially large enough ones where you care about speed—are almost always compressed—So you're not going to be bottlenecked on disk read. Assuming that compressed disk read + decompression is fast enough to keep the data flow high, you're back to CSV parsing being the bottleneck. The JIT compile cache only affects Julia. The reason is it not included in the benchmark results is because it is a small, fixed overhead: it is only paid on the first CSV file read (no matter the size), and if you read a larger data set, the compile time does not increase. Since the point of benchmarks is typically to project from a smaller case how long it would take to do even larger tasks, you don't want to include small, fixed overheads. For some concrete numbers, I just timed reading a tiny CSV file on my MacBook Pro 2018 and the first read, including compile time took 4 seconds. The second read took 0.000347 seconds. So that's the fixed overhead we're talking about here for reading a CSV file: about 4 seconds on the very first CSV file you read. People can be the judge of whether that's a showstopper for them or not.

fnord123 6y ago

> The tools used for benchmarking were BenchmarkTools.jl for Julia, microbenchmark for R, and timeit for Python. Using different tools to measure performance between platforms? > https://github.com/JuliaCI/BenchmarkTools.jl/blob/master/src/execution.jl#L287 https://github.com/JuliaCI/BenchmarkTools.jl/blob/master/src... Using a warmup to get the slow JIT within the same galaxy as the C implementations and not mentioning it in the post?

glangdale 6y ago

Frankly, all these implementations are disappointingly slow. Daniel Lemire and I wrote simdjson to use SIMD to read JSON quickly - CSV is strictly easier than that. I made a start on simdcsv but got bored, but the same principles would apply. IMO this task should be doable at 1-2GB/s on a single core; it's not something that should really need multiple cores.

liuliu 6y ago
Agreed. But doing SIMD optimizations could quickly become 2-3 weeks project for me just for one platform. Adding NEON would be another 1 or 2 weeks. Need much more dedication I guess. Your work on simdjson is awesome! Thanks for saving the world so much unnecessary power consumptions :)
- glangdale 6y ago
  Thanks for the kind words. I suppose this confirms my general suspicion that principles of simdjson need to be lifted out into a more general tool that allows people that don't live and breathe SIMD (strange that such people exist :-) ) to still get the benefit of these techniques for formats that aren't JSON.

purzelrakete 6y ago

I built a pagerank implementation in Julia many years ago. It was operating on an adjacency list with half a billion edges, and I had to resort to writing a custom reader to get reasonable read speeds over the csv. In case anyone is interested: https://github.com/purzelrakete/Pagerank.jl/blob/070a193826f6ff7c2fe11fb5556f21d039b82e26/src/io.jl#L47 https://github.com/purzelrakete/Pagerank.jl/blob/070a193826f...

mr_toad 6y ago

From where? I couldn’t see anywhere in the article that specifies where the data is stored.

xiaodai 6y ago

Julia gets alot of hate nowadays that's for sure https://twitter.com/danluu/status/1315407713430458368?s=19 https://twitter.com/danluu/status/1315407713430458368?s=19

4gotunameagain 6y ago

The problem with CSV files are the endless edge cases. What I'll typically do for larger ones is that I'll write a program that does the reading and the rudimentary processing in C, and then saves the data as a binary .npy file in /dev/shm (or ImDisk if I'm on windows) and load that up in Python. Massage the data more if necessary and finally do the plotting. I found that this helps, hope it helps you too