5 ms·

Companies with Python code are probably better off keeping their working, tested code than switching to an incompatible interpreter and set of libraries which a

by jzwinck 10y ago

Companies with Python code are probably better off keeping their working, tested code than switching to an incompatible interpreter and set of libraries which among other things will print "b'Hello',b'World'" into their mission critical CSV files. Yes, the built in csv module really does that in Python 3.

aidos 10y ago
But people that have that csv issue have deeper lurker encoding issues they're probably not aware of. Sure, don't change the stuff that works and cause yourself unnecessary pain, but don't blame python 3 for your misencoded data.
- jzwinck 10y ago
  Oh we are perfectly well aware of our encoding issues. In NumPy, one of the most important libraries in Python 2 or 3, strings always take one byte per character. And it's not going to change. So the built in csv module needs to support a reasonable behavior. Which it does not. If a company tries Python 3 and discovers basic things like CSV produce utter gibberish, they would do well to opt out. And they do--in droves. My data is not misencoded, you see. It's just misunderstood.
  - guitarbill 10y ago
    > So the built in csv module needs to support a reasonable behavior. Which it does not. It could be argued that the csv module's behaviour is reasonable, and NumPy's isn't. (I'm not 100% sure about all the details of this issue) Hopefully, NumPy will change it's behaviour to match Python 3, but if not you could still use the NumPy CSV routines like `loadtxt` or `genfromtxt` [0]. So then this becomes a documentation change to add some warnings to both modules. > they would do well to opt out. And they do--in droves. This is simply not true. They would do well to handle strings properly and so avoid bugs in future - something Python 3 actively encourages, and Python 2 obscures. And while I can't speak for every company, our metrics show that our Python 3 code has far less customer issues than Python 2, Perl, or Ruby. Now that's business value. (Edit: I mean it's hard to make the comparison - the Perl code is e.g. older - but we're writing code now, and when the interns add new stuff to the Python 3 codebase, it breaks less. All of them are still actively developed, and the Ruby one is about as old as the Python 3 one). [0] https://docs.scipy.org/doc/numpy/reference/generated/numpy.loadtxt.html https://docs.scipy.org/doc/numpy/reference/generated/numpy.l...
    - jimnotgym 10y ago
      I moved to Python and got back into coding specifically to work with csv files. And from the first time I used code copy and pasted from SO or wherever to import and export CSV's with Python 3 I have not found that I am plagued by unreliable behavior and mischievous encoding. Yes there seems to be a type conversion necessary with NumPy, but it simply is not true that Python 3 has an endemic problem with CSV files! In fact, the reason I chose Python was because I was able to dive so quickly into real problems like this with no problems whatsoever. This is a minor issue for someone porting from 2 to 3, it is not a problem with 3
    - ianamartin 10y ago
      I say this as someone who uses the latest version of Python available in every new project or script. Text encoding issues are absolute garbage in Python 3.x I fucking hate the way that csv module works with text encodings. As soon as I can figure out a reliable way to take latin-1 and save it as UTF-8 without breaking everything, I will try to shoehorn in a PR. Right now, it's fucking awful. My ETL pipeline hates it, I hate it, my boss hates it, and my internal constituents hate it. Because it sucks. A file I can read in one encoding and write as another should be readable with the encoding I wrote it in. That is not currently the case with the latest version of Python. And it makes me hate the world.
      - Sir_Cmpwn 10y ago
        with open('some latin-1 file', 'rb) as f: text = f.read().decode('latin-1') with open('some utf8 file', 'wb') as f: f.write(text.encode('utf-8')) Python 3's string encoding support is super good. I've said it before and I'll say it again: if you use bytes as a string you are Doing It Wrong. If you use bytes as a string you are Doing It Wrong. If you use bytes as a string you are Doing It Wrong.
        ianamartin 10y ago
        And what happens when you don't get a choice about what strings you are digesting?
  - vurpo 10y ago
    >In NumPy, one of the most important libraries in Python 2 or 3, strings always take one byte per character. So I won't be able to use NumPy with either of my two native languages. That sounds like a bit of a shortcoming for the majority of the world.
    - aidos 10y ago
      Numpy isn't really used like that though. It's for numerical computation. There might be cases for putting text in there but you can always keep it locally and map it to an int that you use in numpy for mapping (I do that in places).
crdoconnor 10y ago
Companies using the built in csv module would be better off moving to something like csvparser. Python's a superlative language but it has a pretty terrible set of included libraries. urllib2 isn't the only library with a superior alternative on pypi. Pretty much all of them do.
ubernostrum 10y ago
Yes, the built in csv module really does that in Python 3. If you pass it bytes, yes, it does. If you pass it strings, no, it doesn't. If what you pass to the built-in CSV writer is not a string, the CSV writer will call str() to get a string representation it can write out. The string representation of a bytes object includes the 'b' prefix. Meanwhile, you discovered your bug: you were treating bytes as text, which is likely to blow up on you sooner or later, and thanks to how Python now handles text, it blew up on you immediately as a way to remind you not to treat bytes as text. What you probably think you want is for the CSV writer to realize it got a bytes object and, instead of calling str(), call its decode() method to get text it can write. But that is once again a dangerous operation, and sort of the whole point of Python 3's text changes is it won't let you get away with that stuff anymore.
- CJefferson 10y ago
  I had this problem. It doesn't "blow up". If it blew up and retired, I would have seen the problem. The problem, like several python 2/3 incompatibilities, is that Python 3 merrily did something different, without telling anyone, until eventually we track down what has changed. I spent quite a while on this very bug myself, and it, along with others, persuaded me to switch to a different language (serious I know, but I was just getting annoyed with python's general loose dynamic nature, in combination with the python 2/3 changes.)
  - bb611 10y ago
    It's unreasonable to say "merrily did something different, without telling anyone" when fixing the string implementation was a significant reason for creating backwards-incompatible Python3 in the first place. It's not a bug, it's a fix for an architectural error in Python 2, and it was quite well announced at the time: https://docs.python.org/3.0/whatsnew/3.0.html https://docs.python.org/3.0/whatsnew/3.0.html
    - int_19h 10y ago
      Fixing the string mess in Py2 was a good thing. But the fact that the same code now silently does the always wrong thing in Py3 wrt CSV is clearly a bug. Actually, the design defect here is calling str() on everything, and assuming that the output is sensible for CSV. It may be a decent rule of thumb, but it clearly does not apply to bytes. Given the likelihood that someone might mistakenly use bytes as a string (for example, because they're porting a legacy Py2 codebase), this should be a hard error, immediately reported as such, and not just a silent behavior change.
singularity2001 10y ago
"b'Hello',b'World'" this (and 1001 similar)

by jzwinck 10y ago

aidos 10y ago

But people that have that csv issue have deeper lurker encoding issues they're probably not aware of. Sure, don't change the stuff that works and cause yourself unnecessary pain, but don't blame python 3 for your misencoded data.

jzwinck 10y ago
Oh we are perfectly well aware of our encoding issues. In NumPy, one of the most important libraries in Python 2 or 3, strings always take one byte per character. And it's not going to change. So the built in csv module needs to support a reasonable behavior. Which it does not. If a company tries Python 3 and discovers basic things like CSV produce utter gibberish, they would do well to opt out. And they do--in droves. My data is not misencoded, you see. It's just misunderstood.
- guitarbill 10y ago
  > So the built in csv module needs to support a reasonable behavior. Which it does not. It could be argued that the csv module's behaviour is reasonable, and NumPy's isn't. (I'm not 100% sure about all the details of this issue) Hopefully, NumPy will change it's behaviour to match Python 3, but if not you could still use the NumPy CSV routines like `loadtxt` or `genfromtxt` [0]. So then this becomes a documentation change to add some warnings to both modules. > they would do well to opt out. And they do--in droves. This is simply not true. They would do well to handle strings properly and so avoid bugs in future - something Python 3 actively encourages, and Python 2 obscures. And while I can't speak for every company, our metrics show that our Python 3 code has far less customer issues than Python 2, Perl, or Ruby. Now that's business value. (Edit: I mean it's hard to make the comparison - the Perl code is e.g. older - but we're writing code now, and when the interns add new stuff to the Python 3 codebase, it breaks less. All of them are still actively developed, and the Ruby one is about as old as the Python 3 one). [0] https://docs.scipy.org/doc/numpy/reference/generated/numpy.loadtxt.html https://docs.scipy.org/doc/numpy/reference/generated/numpy.l...
  - jimnotgym 10y ago
    I moved to Python and got back into coding specifically to work with csv files. And from the first time I used code copy and pasted from SO or wherever to import and export CSV's with Python 3 I have not found that I am plagued by unreliable behavior and mischievous encoding. Yes there seems to be a type conversion necessary with NumPy, but it simply is not true that Python 3 has an endemic problem with CSV files! In fact, the reason I chose Python was because I was able to dive so quickly into real problems like this with no problems whatsoever. This is a minor issue for someone porting from 2 to 3, it is not a problem with 3
  - ianamartin 10y ago
    I say this as someone who uses the latest version of Python available in every new project or script. Text encoding issues are absolute garbage in Python 3.x I fucking hate the way that csv module works with text encodings. As soon as I can figure out a reliable way to take latin-1 and save it as UTF-8 without breaking everything, I will try to shoehorn in a PR. Right now, it's fucking awful. My ETL pipeline hates it, I hate it, my boss hates it, and my internal constituents hate it. Because it sucks. A file I can read in one encoding and write as another should be readable with the encoding I wrote it in. That is not currently the case with the latest version of Python. And it makes me hate the world.
    - Sir_Cmpwn 10y ago
      with open('some latin-1 file', 'rb) as f: text = f.read().decode('latin-1') with open('some utf8 file', 'wb') as f: f.write(text.encode('utf-8')) Python 3's string encoding support is super good. I've said it before and I'll say it again: if you use bytes as a string you are Doing It Wrong. If you use bytes as a string you are Doing It Wrong. If you use bytes as a string you are Doing It Wrong.
      - ianamartin 10y ago
        And what happens when you don't get a choice about what strings you are digesting?
- vurpo 10y ago
  >In NumPy, one of the most important libraries in Python 2 or 3, strings always take one byte per character. So I won't be able to use NumPy with either of my two native languages. That sounds like a bit of a shortcoming for the majority of the world.
  - aidos 10y ago
    Numpy isn't really used like that though. It's for numerical computation. There might be cases for putting text in there but you can always keep it locally and map it to an int that you use in numpy for mapping (I do that in places).

crdoconnor 10y ago

Companies using the built in csv module would be better off moving to something like csvparser. Python's a superlative language but it has a pretty terrible set of included libraries. urllib2 isn't the only library with a superior alternative on pypi. Pretty much all of them do.

ubernostrum 10y ago

Yes, the built in csv module really does that in Python 3. If you pass it bytes, yes, it does. If you pass it strings, no, it doesn't. If what you pass to the built-in CSV writer is not a string, the CSV writer will call str() to get a string representation it can write out. The string representation of a bytes object includes the 'b' prefix. Meanwhile, you discovered your bug: you were treating bytes as text, which is likely to blow up on you sooner or later, and thanks to how Python now handles text, it blew up on you immediately as a way to remind you not to treat bytes as text. What you probably think you want is for the CSV writer to realize it got a bytes object and, instead of calling str(), call its decode() method to get text it can write. But that is once again a dangerous operation, and sort of the whole point of Python 3's text changes is it won't let you get away with that stuff anymore.

CJefferson 10y ago
I had this problem. It doesn't "blow up". If it blew up and retired, I would have seen the problem. The problem, like several python 2/3 incompatibilities, is that Python 3 merrily did something different, without telling anyone, until eventually we track down what has changed. I spent quite a while on this very bug myself, and it, along with others, persuaded me to switch to a different language (serious I know, but I was just getting annoyed with python's general loose dynamic nature, in combination with the python 2/3 changes.)
- bb611 10y ago
  It's unreasonable to say "merrily did something different, without telling anyone" when fixing the string implementation was a significant reason for creating backwards-incompatible Python3 in the first place. It's not a bug, it's a fix for an architectural error in Python 2, and it was quite well announced at the time: https://docs.python.org/3.0/whatsnew/3.0.html https://docs.python.org/3.0/whatsnew/3.0.html
  - int_19h 10y ago
    Fixing the string mess in Py2 was a good thing. But the fact that the same code now silently does the always wrong thing in Py3 wrt CSV is clearly a bug. Actually, the design defect here is calling str() on everything, and assuming that the output is sensible for CSV. It may be a decent rule of thumb, but it clearly does not apply to bytes. Given the likelihood that someone might mistakenly use bytes as a string (for example, because they're porting a legacy Py2 codebase), this should be a hard error, immediately reported as such, and not just a silent behavior change.

singularity2001 10y ago

"b'Hello',b'World'" this (and 1001 similar)