4 ms·

>obvious stuff like XML or JSON parsing! There is one case when those wheels can be worth (partially) reinventing: if you only need them to parse some files w

• yo-code-sucks 11y ago
  Nope nope nope. Your not a special flower, you cannot do it better, at least not on my dime.
  • xamuel 11y ago
    Did you even read what I wrote? Of course I can't do a better XML or JSON parser. But if what you need parsed is ten thousand lines of <key1>value1</key1> then I'm pretty sure I can write something a lot more efficient and portable, with better error reporting, than W3CBBQ, and I can do it in 10 minutes. That said, I'm very flattered you created a throwaway account just to respond to me.
    • MaulingMonkey 11y ago
      > then I'm pretty sure I can write something a lot more efficient Sure. > and portable ...it's hard to be "a lot more portable" than a simple C library that compiles out of the box on all your preferred compilers and platforms. Care to elaborate how you end up being "a lot more portable"? > with better error reporting Plenty of room to do your own error reporting with off the shelf parsers. > and I can do it in 10 minutes The bug reports, on the other hand, will last a lifetime. Because that format will change, your underlying assumptions will be invalidated, and your parser will eventually grow into one of the 4+ fully featured redundant XML parsers in your project, each with their own set of weird assumptions, inconsistencies, and "supported bugs" (for backwards compatibility), and all of them will need to be maintained. At least until someone takes the time to make a breaking change to the data format, and rip them out to replace them with a proper off-the-shelf JSON parser that has orders of magnitudes more eyeballs fixing the corner cases. Then you'll only have 3+. > A full XML parser comes with a whole ton of bloat you don't need for that! I'll take minor binary bloat over major maintenance bloat any day. And you're wrong: It turns out I actually need a lot of that "bloat" in the next revision. > Plus, if there's ever a problem with those files, the outsourced XML parser will tend to give the end-user absolutely useless error messages. If you have a blindly auto-generated XML-to-binary-format reflection based converter, sure. That's not a good reason to write off "off the shelf" XML parsers outright though, when there's plenty of ways to use them that will give you perfectly fine error messages. To keep with your theme of generalizations: in-house XML parsers tend to give the end-user no error messages, which is far worse ;)
      • MichaelGG 11y ago
        I've personally seen this in two separate projects that decided they needed their own XML parser. In one case, they simply didn't know better. And proceeded to get the basics wrong, like empty elements. In the other case, they used XML for no good reason, then, gasp, decided parsing 30KB XML files all over the place (thousands a second, deeply integrated into all parts of the code) wasn't very quick. So they copied libxml's source, then hacked it up, breaking things like character encoding. Oh, and they added an XML pre-processor while they were at it, to do file includes and variable substitution. Thus it's not even compatible with XML, yet they still pay all the overhead. Awesome.
    • kyllo 11y ago
      My employer has a history of using CDFs (Comma Delimited Files) for lots of things. They're just like CSV, only not. It works great right up until someone wants to know if they can store data with internal commas in it like "Acme, Inc." Then we have to reinvent CSV anyway.
    • crdoconnor 11y ago
      >then I'm pretty sure I can write something a lot more efficient and portable, with better error reporting I would give pretty good odds that you couldn't. Good json/XML parsers (you did pick a good one, right?) are usually heavily optimized in ways you wouldn't even think of and cover more edge cases than you can imagine because heavy usage in production has picked them up. You are not, as he said, a special flower.
      • kybernetikos 11y ago
        > (you did pick a good one, right?) So, the immediate costs to use are that you have to research a bunch of parsers to see which fit your needs and are 'good'. Then you have to write the code to use them, which depending on their interface may itself add a significant number of lines of code with potential for their own bugs (given a simple enough example, the integration code can be larger than the code you would have written). Then you have to include not just the library, but all its dependencies in your codebase. You're then committed to keep track of the library and all its dependencies, and updating them if and when dangerous bugs are discovered. This means that your code now has a reason to change or rot (API changes from upstream) that has nothing to do with you or any feature or bug you've written. If your requirement is later found out to be slightly unusual and not exactly covered by the library, you're in for a world of pain working around that too (e.g. in the example given, this probably isn't a valid xml file because it doesn't have a single surrounding tag and there may be other nonstandard issues in the input). The amazing thing is that this actually is worth it in many cases, but people often underestimate the cost. Sure, the poster you're replying to isn't a special flower that can write a better parser for whatever format, but he's saying that in this case he doesn't need one. A number of other commenters have pointed out that requirements change and the generic parser will be a better choice than evolving your homegrown simple code into a poorly thought out monster. This is true, but it's also premature optimisation. The right time to pull in those libraries is on the first non trivial change, not at the beginning.
        • crdoconnor 11y ago
          >So, the immediate costs to use are that you have to research a bunch of parsers to see which fit your needs and are 'good'. Yep. This is known as an investment. It keeps paying dividends every time you use it. >Then you have to write the code to use them, which depending on their interface may itself add a significant number of lines of code with potential for their own bugs (given a simple enough example, the integration code can be larger than the code you would have written). Only if you're a bad programmer, and/or you used a bad library. >Then you have to include not just the library, but all its dependencies in your codebase. For me this means adding one line to a file called "requirements.txt". E.g. python-cjson==1.1.0. The setup scripts take care of the rest. Trivial. >You're then committed to keep track of the library and all its dependencies, and updating them if and when dangerous bugs are discovered. Yep. Once a month or so I upgrade all of my libraries, run the tests and see what breaks. Mostly nothing breaks and the amount of thinking I have to do to take care of this problem is usually zero. >This means that your code now has a reason to change or rot (API changes from upstream) that has nothing to do with you or any feature or bug you've written. While true, stuff like cjson just doesn't change its APIs. In practice, while this is a problem for some libraries, it's one that is relatively minor. The reverse is more usually true: upgrade everything and you get bugfixes for free. Good library designers do not change their APIs often but they do fix bugs frequently. > If your requirement is later found out to be slightly unusual and not exactly covered by the library, you're in for a world of pain working around that too (e.g. in the example given, this probably isn't a valid xml file because it doesn't have a single surrounding tag). Hence "choose a good library". For xml that means choose one that can deal with invalid XML because invalid XML is a problem you're just going to have to deal with eventually if you use XML. I use libxml2. It's good. It's fast and it handles invalid XML well. I recommend it. Better yet, don't use XML coz it's crap (not always avoidable, I know). >The amazing thing is that this actually is worth it in many cases, but people often underestimate the cost. The cost is higher if you choose poor libraries. Choosing a good library is more art than science, but if you do it right you will drop your costs like a rock. It also helps to pick a big ecosystem + language that has good libraries. >Sure, the poster you're replying to isn't a special flower that can write a better parser for whatever format, but he's saying that in this case he doesn't need one. A number of other commenters have pointed out that requirements change and the generic parser will be a better choice than evolving your homegrown simple code into a poorly thought out monster. This is true, but it's also premature optimisation. This is a misuse of the term premature optimization. You can prematurely optimize for speed. You cannot prematurely optimize to eliminate not invented here syndrome, code duplication or tight coupling. >The right time to pull in those libraries is on the first non trivial change, not at the beginning. No, the right time is at the beginning. If you need to parse JSON use a json parsing library. If you need to parse XML, use an XML parsing library. If you're worried google what people have to say about it. Do they say that it sucks? It's glacially slow under the wrong conditions? If you write loosely coupled and cohesive code as well, you can vastly reduce the costs of choosing a bad library later on.
    • yo-code-sucks 11y ago
      This isn't a fucking throw away, I created it because your comment showed so much immaturity as a developer that I almost threw up in my mouth. Geniuses do not recreate shit, they piece other cool shit together into a solution that works good enough. Also not all Xml libs are bad, but yeah the default .Net one is, what your really needing is s StreamReader instead of loading the whole damn thing. So yeah not a throwaway and I promise to rile you up a lot! This site has become a fucking circle-jerk lately.
• chriswarbo 11y ago
  This is often a slippery slope. Assumptions tend to erode over time, eg. I've seen code assuming ASCII-only input fall over due to people copy/pasting from Word (which automagically turns hyphens, quotes, etc. into non-ASCII). Also, whenever there's a lot of data being looped over, it can be very tempting to add new features to the existing loop, rather than adding a new pass (inefficient) or trying to collate the results after they've been spread out (complicated & error-prone). This can turn a simple parser into a core piece of business logic, and of course it's then only a matter of time before custom, XML/JSON-incompatible 'directives' start creeping in to control that business logic.
• wvenable 11y ago
  I guess one cannot pick a non-controversial example. I recently had to deal with someone generating XML by string concat and it worked for years until recently. Something with international characters suddenly started causing problems. The solution was simple; I replaced the code with a proper XML library and it just worked. I didn't even bother to figure out what the underlying issue was. Worrying about the bloat is premature optimization. If your process is actually too slow and parsing is the bottleneck then by all means rip out the XML parser library and roll your own.