4 ms·

What is a reasonable use for a null character in a string? My first instinct is that strings with nulls in them should absolutely be rejected.

jagged-chisel 2y ago
Not everything needs to be a C-string (null-terminated array/sequence of characters.) We are advanced enough with our understanding of Things that we can include metadata along with a chunk of bytes to indicate “this is a ‘string’ and it’s q bytes long and can have any value you want in there.” That said, I’m with you. And if someone wants nulls inside their “strings” then they probably want blobs.
larsnystrom 2y ago
There are two kinds of programmers: Those who think of strings as text, and those who think of strings as a sequence of bytes. The second group doesn’t care about the special case where a byte is all zeroes.
- mattashii 2y ago
  In that second case the string is better represented as "bytea", which has most (but not all) of the features of the "text" type.
  - pdimitar 2y ago
    I agree with your take, it's just that many programmers want to easily jump from "byte array" to "string in XYZ encoding". I personally prefer byte arrays for unsafe data and to do deserialization in application code.
    - rlupi 2y ago
      In other words, considering we are talking about string and unicode... There are two types of programmers, those that are wrong and those that are very wrong
      - pdimitar 2y ago
        lol. :) Funny but not entirely true. I had cases when we had to urgently store a firehose of data and figure out the right string encoding later. Just dumping the strings with uncertain encoding in `bytea` columns helped us there. Plus for some fields it helps with auditability f.ex. when you get raw binary-encoded telemetry from devices in the field, you should store their raw payloads _and_ the parsed data structures that you got from them. Being this paranoid has saved my neck a few times. The secret is to accept you are not without fault and take measures to be able to correct yourself in the future.
        emmelaich 2y ago
        Indeed, one system I dealt with used char instead of blob. The text as stored was riddled with U+FFFE (unicode unknown character).
lelanthran 2y ago
Yup. It's a huge red flag when a datatype intended to be used for representation of written human language is abused to store something that has no glyph recognisable in any human language. There's a lot to complain about with nul-terminated strings, but not being able to store arbitrary bytes ain't one of them.
tlarkworthy 2y ago
That your JSON deserializer accepted them.

jagged-chisel 2y ago

Not everything needs to be a C-string (null-terminated array/sequence of characters.) We are advanced enough with our understanding of Things that we can include metadata along with a chunk of bytes to indicate “this is a ‘string’ and it’s q bytes long and can have any value you want in there.” That said, I’m with you. And if someone wants nulls inside their “strings” then they probably want blobs.

larsnystrom 2y ago

There are two kinds of programmers: Those who think of strings as text, and those who think of strings as a sequence of bytes. The second group doesn’t care about the special case where a byte is all zeroes.

mattashii 2y ago
In that second case the string is better represented as "bytea", which has most (but not all) of the features of the "text" type.
- pdimitar 2y ago
  I agree with your take, it's just that many programmers want to easily jump from "byte array" to "string in XYZ encoding". I personally prefer byte arrays for unsafe data and to do deserialization in application code.
  - rlupi 2y ago
    In other words, considering we are talking about string and unicode... There are two types of programmers, those that are wrong and those that are very wrong
    - pdimitar 2y ago
      lol. :) Funny but not entirely true. I had cases when we had to urgently store a firehose of data and figure out the right string encoding later. Just dumping the strings with uncertain encoding in `bytea` columns helped us there. Plus for some fields it helps with auditability f.ex. when you get raw binary-encoded telemetry from devices in the field, you should store their raw payloads _and_ the parsed data structures that you got from them. Being this paranoid has saved my neck a few times. The secret is to accept you are not without fault and take measures to be able to correct yourself in the future.
      - emmelaich 2y ago
        Indeed, one system I dealt with used char instead of blob. The text as stored was riddled with U+FFFE (unicode unknown character).

lelanthran 2y ago

Yup. It's a huge red flag when a datatype intended to be used for representation of written human language is abused to store something that has no glyph recognisable in any human language. There's a lot to complain about with nul-terminated strings, but not being able to store arbitrary bytes ain't one of them.

tlarkworthy 2y ago

That your JSON deserializer accepted them.