4 ms·

EDIT: Be sure to read the reply from the author of the package. I do think Julia has promise, but it is still early days. In the last 2 months there have been

EDIT: Be sure to read the reply from the author of the package. I do think Julia has promise, but it is still early days. In the last 2 months there have been problems with: * Throwing rows away: https://github.com/JuliaData/CSV.jl/issues/720 https://github.com/JuliaData/CSV.jl/issues/720 * Incorrect float parsing: https://github.com/JuliaData/CSV.jl/issues/714 https://github.com/JuliaData/CSV.jl/issues/714 At there are unfixed issues to do with parsing large files ( https://github.com/JuliaData/CSV.jl/issues/749 https://github.com/JuliaData/CSV.jl/issues/749 , https://github.com/JuliaData/CSV.jl/issues/734 https://github.com/JuliaData/CSV.jl/issues/734 ). While Julia has become much more stable than it used to be, it is still in many places an evolving system and you (in my limited experience) should be prepared to hit issues.

fermienrico 6y ago
With all due respect to all the people working on Julia, I see a stark difference in rigor between say Python's PEP / RFCs and how Julia's core developers operate. The former is a strict formal process with a lot of thought given to a particular PEP proposal, whereas the latter seems like an informal chaotic hashing/bikeshedding between a few key people in Julia community. I've contributed to Julia and people are really nice but IMO it is missing structure, discipline, hardline regimented approach of confronting difficult problems, etc. One feels like hearing a supreme court case and the other a street brawl to decide upon important issues.
- xyzzy_plugh 6y ago
  > Python's PEP / RFCs Is this process actually fundamentally better? There are so many PEPs that are up to the reader to interpret. There are so many PEPs that contradict each other (just read some of the early PEPs). The whole thing feels basically meaningless, and toothless at this point. I think Go's proposal process nails balancing formalities and discussion/bikeshedding. They seem to be very productive at making the language better, thinking critically and avoiding regrets, rather than pandering to the very vocal and often misaligned minority of the community.
  - oefrha 6y ago
    I can’t even tell if this is a bad faith attack or genuine cluelessness. No, language features, standard library modules, infrastructure implementations, standard APIs, development and community processes, etc. etc. aren’t “up to the reader to interpret”, aren’t “basically meaningless”, aren’t “toothless”, especially not “at this point”. Informational PEPs are informational. python-committers also aren’t a misaligned minority of the community. If you feel committers don’t serve your interest, step up and start contributing.
  - fermienrico 6y ago
    Can you provide some meaningless and toothless examples? Which PEPs disagree with which ones? I think PEPs, RFCs, and formal processes are designed to keep structure and discipline. It may sound like unnecessary beauracracy but you see this in PostgreSQL development process as well. Vast number of critical pieces of software is developed by this process - from Linux to proprietary software in large companies such as Apple, Morgan Stanley and many many more.
- karbarcca 6y ago
  One advantage I've noticed at least (having been a contributor to the Julia language itself), is that it at least makes contributing to the language very approachable. Granted you see lower quality proposals from time to time, but in general, I even question whether I myself would have gotten involved or been brave enough to propose language features if I would have had to do such a rigorous, formal PEP writeup. In some respects, you're restricting the pool of people who actually contribute ideas to the language. I also have appreciated the "meritocratic" approach of Julia language features; if someone can show that an idea/approach/algorithm is fundamentally faster, more flexible, etc. it's generally been accepted, regardless if the proposer is a first-time contributor or not. That also makes contributing feel approachable and very fair, as opposed to just trying to please a single core developer or playing some politics game.
  - oefrha 6y ago
    > if someone can show that an idea/approach/algorithm is fundamentally faster, more flexible, etc. it's generally been accepted, regardless if the proposer is a first-time contributor or not. If you can submit a measurably faster csv implementation to cpython (without breaking existing code of course) I can guarantee it will be welcome whether you’re a first time contributor or not. Not sure about your definition of “more flexible”, if that entails breaking existing code then it may rightfully meet resistance. Not sure why you’re implying other languages’ proposal processes are about pleasing a single core developer or playing some politics game. Sounds like either you have a lot more experience than I do and hence have been exposed to politics games or anything else non-meritocratic (that I haven’t sensed in CPython development, at least), or you just don’t have as much experience contributing to other languages. (I'm a bit cranky today. Sorry if I sound aggressive.)
    - woadwarrior01 6y ago
      > If you can submit a measurably faster csv implementation to cpython (without breaking existing code of course) I can guarantee it will be welcome whether you’re a first time contributor or not. I wouldn't be so sure. Search for `site:bugs.python.org "maintenance overhead"` on your favourite search engine.
- throwaway894345 6y ago
  On the other hand, many important things move at a grinding pace in Python. Packaging and performance are two major issues for just about everyone I’ve spoken with who uses Python in a serious capacity, and these have been notorious problems for decades with no end in sight. Most of this seems to boil down to an unwillingness to encourage the ecosystem towards a narrower packaging format (“sorry, downloading a package and running setup.py just to determine the dependency set is insane and we’re going to deprecate that going forward”) or a narrower C extension interface that would be compatible with more aggressive optimizations. There’s value in stability, but if the governance model can’t pass these kinds of slow, steady changes for paramount issues, then perhaps new languages shouldn’t look to Python as a model for a well-governed language. I would suggest Rust—I’m not a fanboy by any means, but Rust is a large, rapidly growing project that is unparalleled with respect to its rate of improvement as far as I’m concerned. I know recommending Rust for things is a cliche (esp “rewrite it in Rust”), but I would defy anyone to find a better governed language.
  - fermienrico 6y ago
    I find Rust really difficult to get into :( Python, you're right has a bunch of problems with datetime and packages, etc. but when it comes to productivity, it is insane how amazing Python is. Also being able to hack internals of python library and "batteries included" philosophy is why it is the most popular language in the world now overtaking Java.
    - throwaway894345 6y ago
      Rust does have a steep learning curve and Python is a very productive language... until you are the one responsible for builds and deploys, or performance, or concurrency, or quality, or etc. I think you’ll find that Go hits a nice sweet spit between all of those competing concerns even if it isn’t the best at any of them (in general; however, probably not for data science in particular).
      - fermienrico 6y ago
        Thanks, I am willing to try both Rust and Go.
karbarcca 6y ago
As the primary author of CSV.jl, I can help clarify on the posted issues: * #720: ended up not being an issue at all, but a misconfigured environment * #714: there was indeed a corner case when automatically detecting float values where the float started with '-' sign and only had a trailing decimal (e.g. '-123.'). Not super common, but indeed a bug * #749, #734 are related to a new "beta" feature (CSV.Chunks) which allows iterating over large CSV files in batches. I've been trying to track down the issues 2 people have reported, but haven't been able to reproduce on the same large file. Once we iron out some of those issues, we'll mark the feature as "out of beta". I agree that Julia packages in general are still evolving and you might run into issues, but at the same time, I strongly believe it's reached a similar maturity in most ways with other language package ecosystems. For example, I use a lot of Javascript/Python frameworks/packages and at least for me, I tend to run into corner case bugs/issues as often as I do for the most common/mature Julia packages. Compared with, say, data.table in R, or pandas in python, one of the things I enjoy most about Julia packages is that they're almost exclusively written in pure Julia. Having had to dive into data.table/pandas source C/C++ + language binding glue code is a huge pain when trying to track down bugs, so I feel like my knowledge of Julia "goes further" in that if I run into a package bug, it's relatively much easier to track down what's going on and even submit a pull request to fix!
- CJefferson 6y ago
  I should have been a bit careful with this message, and mentioning issues that might be in beta. I don't want to specifically pick on csv.jl, but my experience of Julia has been every time I've worked on a significant program, we've hit an issue, either in core or a common library, certainly compared to Python (I avoid Javascript) I agree that Julia is progressing quickly, but I think a lot of people (certainly myself) got burnt back in the earlier days of Julia when things were much less stable and changing rapidly and the language was (in many blogs) sold as ready for use.
  - karbarcca 6y ago
    No worries. The CSV.jl package has just been around for 5 years now, has some 700 issues opened/resolved, has built up a pretty extensive test suite, and is used in production by a number of companies, so I just wanted to hopefully clarify its level of maturity :D I can definitely understand getting burnt in Julia early days (I was around back then as well), but since 1.0, the package ecosystem has matured quite a bit IMO; most of the really popular packages are very stable and work similarly to popular packages in other languages.

fermienrico 6y ago

With all due respect to all the people working on Julia, I see a stark difference in rigor between say Python's PEP / RFCs and how Julia's core developers operate. The former is a strict formal process with a lot of thought given to a particular PEP proposal, whereas the latter seems like an informal chaotic hashing/bikeshedding between a few key people in Julia community. I've contributed to Julia and people are really nice but IMO it is missing structure, discipline, hardline regimented approach of confronting difficult problems, etc. One feels like hearing a supreme court case and the other a street brawl to decide upon important issues.

xyzzy_plugh 6y ago
> Python's PEP / RFCs Is this process actually fundamentally better? There are so many PEPs that are up to the reader to interpret. There are so many PEPs that contradict each other (just read some of the early PEPs). The whole thing feels basically meaningless, and toothless at this point. I think Go's proposal process nails balancing formalities and discussion/bikeshedding. They seem to be very productive at making the language better, thinking critically and avoiding regrets, rather than pandering to the very vocal and often misaligned minority of the community.
- oefrha 6y ago
  I can’t even tell if this is a bad faith attack or genuine cluelessness. No, language features, standard library modules, infrastructure implementations, standard APIs, development and community processes, etc. etc. aren’t “up to the reader to interpret”, aren’t “basically meaningless”, aren’t “toothless”, especially not “at this point”. Informational PEPs are informational. python-committers also aren’t a misaligned minority of the community. If you feel committers don’t serve your interest, step up and start contributing.
- fermienrico 6y ago
  Can you provide some meaningless and toothless examples? Which PEPs disagree with which ones? I think PEPs, RFCs, and formal processes are designed to keep structure and discipline. It may sound like unnecessary beauracracy but you see this in PostgreSQL development process as well. Vast number of critical pieces of software is developed by this process - from Linux to proprietary software in large companies such as Apple, Morgan Stanley and many many more.
karbarcca 6y ago
One advantage I've noticed at least (having been a contributor to the Julia language itself), is that it at least makes contributing to the language very approachable. Granted you see lower quality proposals from time to time, but in general, I even question whether I myself would have gotten involved or been brave enough to propose language features if I would have had to do such a rigorous, formal PEP writeup. In some respects, you're restricting the pool of people who actually contribute ideas to the language. I also have appreciated the "meritocratic" approach of Julia language features; if someone can show that an idea/approach/algorithm is fundamentally faster, more flexible, etc. it's generally been accepted, regardless if the proposer is a first-time contributor or not. That also makes contributing feel approachable and very fair, as opposed to just trying to please a single core developer or playing some politics game.
- oefrha 6y ago
  > if someone can show that an idea/approach/algorithm is fundamentally faster, more flexible, etc. it's generally been accepted, regardless if the proposer is a first-time contributor or not. If you can submit a measurably faster csv implementation to cpython (without breaking existing code of course) I can guarantee it will be welcome whether you’re a first time contributor or not. Not sure about your definition of “more flexible”, if that entails breaking existing code then it may rightfully meet resistance. Not sure why you’re implying other languages’ proposal processes are about pleasing a single core developer or playing some politics game. Sounds like either you have a lot more experience than I do and hence have been exposed to politics games or anything else non-meritocratic (that I haven’t sensed in CPython development, at least), or you just don’t have as much experience contributing to other languages. (I'm a bit cranky today. Sorry if I sound aggressive.)
  - woadwarrior01 6y ago
    > If you can submit a measurably faster csv implementation to cpython (without breaking existing code of course) I can guarantee it will be welcome whether you’re a first time contributor or not. I wouldn't be so sure. Search for `site:bugs.python.org "maintenance overhead"` on your favourite search engine.
throwaway894345 6y ago
On the other hand, many important things move at a grinding pace in Python. Packaging and performance are two major issues for just about everyone I’ve spoken with who uses Python in a serious capacity, and these have been notorious problems for decades with no end in sight. Most of this seems to boil down to an unwillingness to encourage the ecosystem towards a narrower packaging format (“sorry, downloading a package and running setup.py just to determine the dependency set is insane and we’re going to deprecate that going forward”) or a narrower C extension interface that would be compatible with more aggressive optimizations. There’s value in stability, but if the governance model can’t pass these kinds of slow, steady changes for paramount issues, then perhaps new languages shouldn’t look to Python as a model for a well-governed language. I would suggest Rust—I’m not a fanboy by any means, but Rust is a large, rapidly growing project that is unparalleled with respect to its rate of improvement as far as I’m concerned. I know recommending Rust for things is a cliche (esp “rewrite it in Rust”), but I would defy anyone to find a better governed language.
- fermienrico 6y ago
  I find Rust really difficult to get into :( Python, you're right has a bunch of problems with datetime and packages, etc. but when it comes to productivity, it is insane how amazing Python is. Also being able to hack internals of python library and "batteries included" philosophy is why it is the most popular language in the world now overtaking Java.
  - throwaway894345 6y ago
    Rust does have a steep learning curve and Python is a very productive language... until you are the one responsible for builds and deploys, or performance, or concurrency, or quality, or etc. I think you’ll find that Go hits a nice sweet spit between all of those competing concerns even if it isn’t the best at any of them (in general; however, probably not for data science in particular).
    - fermienrico 6y ago
      Thanks, I am willing to try both Rust and Go.

karbarcca 6y ago

As the primary author of CSV.jl, I can help clarify on the posted issues: * #720: ended up not being an issue at all, but a misconfigured environment * #714: there was indeed a corner case when automatically detecting float values where the float started with '-' sign and only had a trailing decimal (e.g. '-123.'). Not super common, but indeed a bug * #749, #734 are related to a new "beta" feature (CSV.Chunks) which allows iterating over large CSV files in batches. I've been trying to track down the issues 2 people have reported, but haven't been able to reproduce on the same large file. Once we iron out some of those issues, we'll mark the feature as "out of beta". I agree that Julia packages in general are still evolving and you might run into issues, but at the same time, I strongly believe it's reached a similar maturity in most ways with other language package ecosystems. For example, I use a lot of Javascript/Python frameworks/packages and at least for me, I tend to run into corner case bugs/issues as often as I do for the most common/mature Julia packages. Compared with, say, data.table in R, or pandas in python, one of the things I enjoy most about Julia packages is that they're almost exclusively written in pure Julia. Having had to dive into data.table/pandas source C/C++ + language binding glue code is a huge pain when trying to track down bugs, so I feel like my knowledge of Julia "goes further" in that if I run into a package bug, it's relatively much easier to track down what's going on and even submit a pull request to fix!

CJefferson 6y ago
I should have been a bit careful with this message, and mentioning issues that might be in beta. I don't want to specifically pick on csv.jl, but my experience of Julia has been every time I've worked on a significant program, we've hit an issue, either in core or a common library, certainly compared to Python (I avoid Javascript) I agree that Julia is progressing quickly, but I think a lot of people (certainly myself) got burnt back in the earlier days of Julia when things were much less stable and changing rapidly and the language was (in many blogs) sold as ready for use.
- karbarcca 6y ago
  No worries. The CSV.jl package has just been around for 5 years now, has some 700 issues opened/resolved, has built up a pretty extensive test suite, and is used in production by a number of companies, so I just wanted to hopefully clarify its level of maturity :D I can definitely understand getting burnt in Julia early days (I was around back then as well), but since 1.0, the package ecosystem has matured quite a bit IMO; most of the really popular packages are very stable and work similarly to popular packages in other languages.