4 ms·

There are plenty of values in a purpose built CSV parser. Several important features, depending on the CSVs you'll be working with are: 1. Sometimes you want t

There are plenty of values in a purpose built CSV parser. Several important features, depending on the CSVs you'll be working with are: 1. Sometimes you want to actually write files, in which case you'll want help escaping quoting. 2. Shortcuts for using header information to provide more convenient access into a particular column, rather than always doing it by index. 3. Conveniently slurp in only parts of a file, or only particular columns in a file, without loading everything into memory.

moron4hire 12y ago
All things a proper understanding of regex and a minimal understanding of streaming file IO can cover. The whole "if you think regex is the solution to your problem, now you have two problems" thing has gotten out of hand. Regex is not that hard.
- newobj 12y ago
  So, what is the regex to parse csv?
- Tyr42 12y ago
  It's simpler to handle quotes and backslashes escaping commas with a custom parser. And then there's the domain knowledge baked into the lib. Does your regex solution produce excel compatible csv files when you have leading zeros? That's important to some people.
- moron4hire 12y ago
  I think people who are downvoting me don't understand how ludicrously retarded most people who output CSV are. CSV is not RFC4180. It's whatever bullshit text file your client has handed you and convinced your project manager is your problem to parse, not their problem to generate even remotely correctly. There is no CSV library capable of handling "CSV". Every time someone asks you for it, you better kick and scream or expect to do a custom job.
  - Too 12y ago
    The csv library of python has handled every csv i've ever thrown on it, csv is "standardized" enough for that. Just set two things, the delimiter, the escape quote method and be done with it. The output is a list of dictionaries with the column headers as keys, very elegant. The best part is that using the same input you did to read a file you can use to save/modify the file and be sure it will look the same when your client re-opens it. A regexp would take twice the time to write and wouldn't give you half of those features, and it would probably fail at escaping sooner or later, for the same reason as xml can't be parsed with regexps.
    - tutuca 12y ago
      While I agree with you, I'm having a lot of trouble handling unicode and windows' latin-1 encodings...
  - mercurial 12y ago
    Well, if you look at, eg, Python's CSV parsing library, it's been more than enough to cover my needs so far, and handles different CSV flavours. It is much nicer and less error-prone to use than using regexps.
  - smcl 12y ago
    You are getting downvoted to hell but I can see what you're meaning. Generally if someone hands you a CSV file there is no guarantee that something mental isn't happening as there is no "CSV" standard. So you're saying that when your task is "process the client's CSV file" you might not necessarily be able to rely on a library handling it correctly, and that you should prepare to get your hands dirty (perhaps hacking together something with a regex or two).
  - ghshephard 12y ago
    I think people are being a fit unfair downvoting you right now (I bumped you up) - but I also disagree with you. When I'm working with a file that purports to be CSV/TSV, with python, I reach out to the CSV module, specify the dialect that created it - and instantly get all the power of being able to identify refer to all the fields, rows without having to otherwise worry about parsing them. Is it 100% bulletproof - definitely not - but, then again, I'm not writing a life safety system. And I've also never had the Python CSV parser break on any reasonable file I've sent it. I'm truly thankful for robust CSV/TSV parsers. Throwaway code like this just works - in particular handles parsing the column headers to automatically build the dict for me. sitesFN=['gateway.tsv','relay.tsv'] dsites={} for fn in sitesFN: f=open(fn,'r') reader=csv.DictReader(f,dialect='excel-tab') for row in reader: dsites[row['NIC_Serial_No']]=row['Device_Name'] Perhaps what you are trying to say that people are failing to hear, is that you can't rely on a CSV parser to, a priori, handle all possible files that purport to be "TSV/CSV" - in that I agree with you, and that you will always need to examine the file and determine if the built in parser will handle it. But - What if it turns out the standard library CSV parser handles the "CSV" file just fine - in that case it seems to make a lot of sense to use it, rather than taking the time in writing your own (along with the bugs that come from re-writing anything). And, speaking just for myself - again, I've never seen a CSV/TSV file that the python didn't handle just fine - not to say they aren't out there - you just have to go out of your way to create them.

moron4hire 12y ago

All things a proper understanding of regex and a minimal understanding of streaming file IO can cover. The whole "if you think regex is the solution to your problem, now you have two problems" thing has gotten out of hand. Regex is not that hard.

newobj 12y ago
So, what is the regex to parse csv?
Tyr42 12y ago
It's simpler to handle quotes and backslashes escaping commas with a custom parser. And then there's the domain knowledge baked into the lib. Does your regex solution produce excel compatible csv files when you have leading zeros? That's important to some people.
moron4hire 12y ago
I think people who are downvoting me don't understand how ludicrously retarded most people who output CSV are. CSV is not RFC4180. It's whatever bullshit text file your client has handed you and convinced your project manager is your problem to parse, not their problem to generate even remotely correctly. There is no CSV library capable of handling "CSV". Every time someone asks you for it, you better kick and scream or expect to do a custom job.
- Too 12y ago
  The csv library of python has handled every csv i've ever thrown on it, csv is "standardized" enough for that. Just set two things, the delimiter, the escape quote method and be done with it. The output is a list of dictionaries with the column headers as keys, very elegant. The best part is that using the same input you did to read a file you can use to save/modify the file and be sure it will look the same when your client re-opens it. A regexp would take twice the time to write and wouldn't give you half of those features, and it would probably fail at escaping sooner or later, for the same reason as xml can't be parsed with regexps.
  - tutuca 12y ago
    While I agree with you, I'm having a lot of trouble handling unicode and windows' latin-1 encodings...
- mercurial 12y ago
  Well, if you look at, eg, Python's CSV parsing library, it's been more than enough to cover my needs so far, and handles different CSV flavours. It is much nicer and less error-prone to use than using regexps.
- smcl 12y ago
  You are getting downvoted to hell but I can see what you're meaning. Generally if someone hands you a CSV file there is no guarantee that something mental isn't happening as there is no "CSV" standard. So you're saying that when your task is "process the client's CSV file" you might not necessarily be able to rely on a library handling it correctly, and that you should prepare to get your hands dirty (perhaps hacking together something with a regex or two).
- ghshephard 12y ago
  I think people are being a fit unfair downvoting you right now (I bumped you up) - but I also disagree with you. When I'm working with a file that purports to be CSV/TSV, with python, I reach out to the CSV module, specify the dialect that created it - and instantly get all the power of being able to identify refer to all the fields, rows without having to otherwise worry about parsing them. Is it 100% bulletproof - definitely not - but, then again, I'm not writing a life safety system. And I've also never had the Python CSV parser break on any reasonable file I've sent it. I'm truly thankful for robust CSV/TSV parsers. Throwaway code like this just works - in particular handles parsing the column headers to automatically build the dict for me. sitesFN=['gateway.tsv','relay.tsv'] dsites={} for fn in sitesFN: f=open(fn,'r') reader=csv.DictReader(f,dialect='excel-tab') for row in reader: dsites[row['NIC_Serial_No']]=row['Device_Name'] Perhaps what you are trying to say that people are failing to hear, is that you can't rely on a CSV parser to, a priori, handle all possible files that purport to be "TSV/CSV" - in that I agree with you, and that you will always need to examine the file and determine if the built in parser will handle it. But - What if it turns out the standard library CSV parser handles the "CSV" file just fine - in that case it seems to make a lot of sense to use it, rather than taking the time in writing your own (along with the bugs that come from re-writing anything). And, speaking just for myself - again, I've never seen a CSV/TSV file that the python didn't handle just fine - not to say they aren't out there - you just have to go out of your way to create them.