4 ms·

Agree. Not saying csv doesn’t have its issues, but I don’t think the author made a convincing argument. A lot of the issues the author brought up didn’t sound

by gxs 3y ago

Agree. Not saying csv doesn’t have its issues, but I don’t think the author made a convincing argument. A lot of the issues the author brought up didn’t sound that bad and/or it sounds like he never looked at the source data first. If you’re doing work with large datasets, I think it’s a good practice to at least go and look at the source data briefly to see what to expect. This will give you a good idea of the format it outputs, data types, some domain context, etc. or some combination thereof and I don’t think it even takes that long. Also, it reminds me of the arguments against excel in a way. Most people know what a csv is, more or less how to open it, and don’t need too much context when discussing the file. Someone will quickly understand if you tell them the file isn’t delimited properly right away. These are pros that shouldn’t be taken for granted. Again, I’m not saying csv doesnt have issues or that there aren’t better alternatives, simply that I didn’t find this particular argument convincing.

mrgoldenbrown 3y ago
IME most people don't know that using Excel to open and save a csv will silently mangle data. In our application leading zeros are significant, so we constantly get screwed by people trying to do quick manual edits and breaking the data. If we're lucky it breaks so badly the import fails. It's worse when the mangling results in structurally valid but wrong data.
- gxs 3y ago
  I think what you’re saying is accurate, but it’s also important to be practical about stuff. These are pretty well know excel limitations by now. And really, anyone using excel who is somehow not aware of that limitation is probably not someone yet experienced enough to be working on a larger and/or mission critical dataset to begin with. Are there exceptions? Sure. You might be tempted to cite the example of the incident where this happened to some biologists not too long ago, but mistakes happen. I’ve seen people make mistakes building android or iPhone using the right (TM) tools. What is the exact number of mistakes where you make the decision to jump to a new format? I’m not sure. This does happen eventually, but the author didn’t make a strong case here imo.
likis 3y ago
But the point is that you don't have to look at the source data if you have an actual specification and defined format, right?

by gxs 3y ago

mrgoldenbrown 3y ago

IME most people don't know that using Excel to open and save a csv will silently mangle data. In our application leading zeros are significant, so we constantly get screwed by people trying to do quick manual edits and breaking the data. If we're lucky it breaks so badly the import fails. It's worse when the mangling results in structurally valid but wrong data.

gxs 3y ago
I think what you’re saying is accurate, but it’s also important to be practical about stuff. These are pretty well know excel limitations by now. And really, anyone using excel who is somehow not aware of that limitation is probably not someone yet experienced enough to be working on a larger and/or mission critical dataset to begin with. Are there exceptions? Sure. You might be tempted to cite the example of the incident where this happened to some biologists not too long ago, but mistakes happen. I’ve seen people make mistakes building android or iPhone using the right (TM) tools. What is the exact number of mistakes where you make the decision to jump to a new format? I’m not sure. This does happen eventually, but the author didn’t make a strong case here imo.

likis 3y ago

But the point is that you don't have to look at the source data if you have an actual specification and defined format, right?