3 ms·

> Also it's less likely to legitimately occur in something like a name, so there are many users of CSVs who can say they will never need to consider data contai

> Also it's less likely to legitimately occur in something like a name, so there are many users of CSVs who can say they will never need to consider data containing a record separator, and they will be right more often than those who never consider data containing a comma. No, they'll be right exactly as often, 0% of the time. But their mistake will show up less frequently, causing more problems when it does. As soon as it's possible for some of your data to come from someone else's dataset, you're guaranteed to have to accommodate record separators within your data as well as within the metadata. You're better off using a system that plans for this inevitability than one that pretends it can't happen at all.

penteract 2y ago
> No, they'll be right exactly as often, 0% of the time. > But their mistake will show up less frequently, causing more problems when it does. Enough people use CSVs (and have limited, small-scale use-cases) that I'd be willing to bet "less frequently" means never for at least 1% of people who use CSVs. I don't know whether the chance of no problems is worth the increased difficulty of problems that do occur - considering that balance feels a bit silly because if you're aware there could be a problem in a context where you could choose between commas and unit separators, you could just add validation or escaping.
- thaumasiotes 2y ago
  > considering that balance feels a bit silly because if you're aware there could be a problem in a context where you could choose between commas and record separators, you could just add validation or escaping. As soon as you have validation or escaping, having a record separator character loses its entire purpose. The existence of the character is predicated on the idea that you don't have to do that, and that idea is false. That's why the character is never used. It's a conceptual mistake that was accidentally enshrined in a series of encoding standards that had enough free space to accommodate it.
  - penteract 2y ago
    > As soon as you have validation or escaping, having a record separator character loses its entire purpose. The existence of the character is predicated on the idea that you don't have to do that, and that idea is false. I disagree with this - the data needs to be stored somehow, and while other characters (like comma) can be used, having a dedicated character can help - for example if the data might legitimately contain commas or newlines but not unit separators or record separators, then escaping isn't needed if you use unit/record separators (although validation is still necessary).
    - Symbiote 2y ago
      I agree. TSV is widely used, but lacks a way to escape the tab and new line characterss. RS-V is the same, but allows including tabs and new lines in records.
  - Dylan16807 2y ago
    > As soon as you have validation or escaping, having a record separator character loses its entire purpose. Not true. Validation is easier than escaping.

penteract 2y ago

> No, they'll be right exactly as often, 0% of the time. > But their mistake will show up less frequently, causing more problems when it does. Enough people use CSVs (and have limited, small-scale use-cases) that I'd be willing to bet "less frequently" means never for at least 1% of people who use CSVs. I don't know whether the chance of no problems is worth the increased difficulty of problems that do occur - considering that balance feels a bit silly because if you're aware there could be a problem in a context where you could choose between commas and unit separators, you could just add validation or escaping.

thaumasiotes 2y ago
> considering that balance feels a bit silly because if you're aware there could be a problem in a context where you could choose between commas and record separators, you could just add validation or escaping. As soon as you have validation or escaping, having a record separator character loses its entire purpose. The existence of the character is predicated on the idea that you don't have to do that, and that idea is false. That's why the character is never used. It's a conceptual mistake that was accidentally enshrined in a series of encoding standards that had enough free space to accommodate it.
- penteract 2y ago
  > As soon as you have validation or escaping, having a record separator character loses its entire purpose. The existence of the character is predicated on the idea that you don't have to do that, and that idea is false. I disagree with this - the data needs to be stored somehow, and while other characters (like comma) can be used, having a dedicated character can help - for example if the data might legitimately contain commas or newlines but not unit separators or record separators, then escaping isn't needed if you use unit/record separators (although validation is still necessary).
  - Symbiote 2y ago
    I agree. TSV is widely used, but lacks a way to escape the tab and new line characterss. RS-V is the same, but allows including tabs and new lines in records.
- Dylan16807 2y ago
  > As soon as you have validation or escaping, having a record separator character loses its entire purpose. Not true. Validation is easier than escaping.