4 ms·

Yes, the built in csv module really does that in Python 3. If you pass it bytes, yes, it does. If you pass it strings, no, it doesn't. If what you pass to the

Yes, the built in csv module really does that in Python 3. If you pass it bytes, yes, it does. If you pass it strings, no, it doesn't. If what you pass to the built-in CSV writer is not a string, the CSV writer will call str() to get a string representation it can write out. The string representation of a bytes object includes the 'b' prefix. Meanwhile, you discovered your bug: you were treating bytes as text, which is likely to blow up on you sooner or later, and thanks to how Python now handles text, it blew up on you immediately as a way to remind you not to treat bytes as text. What you probably think you want is for the CSV writer to realize it got a bytes object and, instead of calling str(), call its decode() method to get text it can write. But that is once again a dangerous operation, and sort of the whole point of Python 3's text changes is it won't let you get away with that stuff anymore.

CJefferson 10y ago
I had this problem. It doesn't "blow up". If it blew up and retired, I would have seen the problem. The problem, like several python 2/3 incompatibilities, is that Python 3 merrily did something different, without telling anyone, until eventually we track down what has changed. I spent quite a while on this very bug myself, and it, along with others, persuaded me to switch to a different language (serious I know, but I was just getting annoyed with python's general loose dynamic nature, in combination with the python 2/3 changes.)
- bb611 10y ago
  It's unreasonable to say "merrily did something different, without telling anyone" when fixing the string implementation was a significant reason for creating backwards-incompatible Python3 in the first place. It's not a bug, it's a fix for an architectural error in Python 2, and it was quite well announced at the time: https://docs.python.org/3.0/whatsnew/3.0.html https://docs.python.org/3.0/whatsnew/3.0.html
  - int_19h 10y ago
    Fixing the string mess in Py2 was a good thing. But the fact that the same code now silently does the always wrong thing in Py3 wrt CSV is clearly a bug. Actually, the design defect here is calling str() on everything, and assuming that the output is sensible for CSV. It may be a decent rule of thumb, but it clearly does not apply to bytes. Given the likelihood that someone might mistakenly use bytes as a string (for example, because they're porting a legacy Py2 codebase), this should be a hard error, immediately reported as such, and not just a silent behavior change.
    - ubernostrum 10y ago
      Except if you make an exception for bytes, what about other types that might get passed into a CSV writer, whose __str__ is something "wrong" for CSV purposes? Do they also get auto-detected? Do we add a new __csv__() method just for when outputting to CSV (since it might not be "wrong" for other output formats)? Or do we ditch str()-ifying altogether, but then add back in a bunch of special cases for numeric types and other things where str() is "the right thing"? Or do we say "CSV outputs strings, whatever is the string representation of what you passed in is what gets written out", and trust people to figure out when they're working with something that has a "wrong" string representation for their use case? Because remember: the whole underlying cause of this was treating a dangerously non-string value as a string. Those bytes objects should have been decoded to strings long before reaching the CSV writer. Python 3 does raise more and louder exceptions when you pass bytes to things that expect strings, but the CSV writer isn't a thing that expects strings; it expects things that have a string representation, and several common use cases get much more difficult if you change that to force every user to explicitly do throwaway casts to string in the name of protecting people who keep insisting on writing dangerous "I'll treat bytes as string until it breaks, and then complain that the language did the wrong thing, not me" code.
      - int_19h 10y ago
        `bytes` is plainly special case for historical reasons here - it's something that is not a string, but that so many people assume to be a string. So yeah, I would be fine with making an exception for it (and providing some kind of option to disable that exception, for that incredibly rare case where someone really does need b"foo" in their CSV output). And then in 5 years, flip the default of that switch, and deprecate it. In another 5, remove it entirely. Also, note that raising an error in this case is not placating the people who insist on using bytes as strings. Quite the opposite - it very loudly and unambiguously tells them that they're wrong, and how exactly they're wrong.

CJefferson 10y ago

I had this problem. It doesn't "blow up". If it blew up and retired, I would have seen the problem. The problem, like several python 2/3 incompatibilities, is that Python 3 merrily did something different, without telling anyone, until eventually we track down what has changed. I spent quite a while on this very bug myself, and it, along with others, persuaded me to switch to a different language (serious I know, but I was just getting annoyed with python's general loose dynamic nature, in combination with the python 2/3 changes.)

bb611 10y ago
It's unreasonable to say "merrily did something different, without telling anyone" when fixing the string implementation was a significant reason for creating backwards-incompatible Python3 in the first place. It's not a bug, it's a fix for an architectural error in Python 2, and it was quite well announced at the time: https://docs.python.org/3.0/whatsnew/3.0.html https://docs.python.org/3.0/whatsnew/3.0.html
- int_19h 10y ago
  Fixing the string mess in Py2 was a good thing. But the fact that the same code now silently does the always wrong thing in Py3 wrt CSV is clearly a bug. Actually, the design defect here is calling str() on everything, and assuming that the output is sensible for CSV. It may be a decent rule of thumb, but it clearly does not apply to bytes. Given the likelihood that someone might mistakenly use bytes as a string (for example, because they're porting a legacy Py2 codebase), this should be a hard error, immediately reported as such, and not just a silent behavior change.
  - ubernostrum 10y ago
    Except if you make an exception for bytes, what about other types that might get passed into a CSV writer, whose __str__ is something "wrong" for CSV purposes? Do they also get auto-detected? Do we add a new __csv__() method just for when outputting to CSV (since it might not be "wrong" for other output formats)? Or do we ditch str()-ifying altogether, but then add back in a bunch of special cases for numeric types and other things where str() is "the right thing"? Or do we say "CSV outputs strings, whatever is the string representation of what you passed in is what gets written out", and trust people to figure out when they're working with something that has a "wrong" string representation for their use case? Because remember: the whole underlying cause of this was treating a dangerously non-string value as a string. Those bytes objects should have been decoded to strings long before reaching the CSV writer. Python 3 does raise more and louder exceptions when you pass bytes to things that expect strings, but the CSV writer isn't a thing that expects strings; it expects things that have a string representation, and several common use cases get much more difficult if you change that to force every user to explicitly do throwaway casts to string in the name of protecting people who keep insisting on writing dangerous "I'll treat bytes as string until it breaks, and then complain that the language did the wrong thing, not me" code.
    - int_19h 10y ago
      `bytes` is plainly special case for historical reasons here - it's something that is not a string, but that so many people assume to be a string. So yeah, I would be fine with making an exception for it (and providing some kind of option to disable that exception, for that incredibly rare case where someone really does need b"foo" in their CSV output). And then in 5 years, flip the default of that switch, and deprecate it. In another 5, remove it entirely. Also, note that raising an error in this case is not placating the people who insist on using bytes as strings. Quite the opposite - it very loudly and unambiguously tells them that they're wrong, and how exactly they're wrong.