6 ms·

ESR announces ForgePlucker: solving the data-jail problems of OSS hosting sites

27 points by Luyt 17y ago

jgrahamc 17y ago
It's amusing to actually look at the source code of this. It's a single file, here: http://svn.gna.org/viewcvs/forgeplucker/trunk/bugplucker.py?rev=7&view=markup http://svn.gna.org/viewcvs/forgeplucker/trunk/bugplucker.py?... Reading it, it looks like a total hack job by a poor programmer. For example, HTML parsing it done by a bunch of regular expressions. Which include stuff like # Yes, Berlios generated \r<BR> sequences with no \n text = text.replace("\r<BR>", "\r\n") # And Berlios generated doubled </TD>s text = text.replace("</TD></TD>", "</TD>") All the pain of maintaining these little cases could have been removed by simply running the page being walked through an actual HTML parser that produces a DOM tree. Similarly the function dehtmlize contains a limit set of HTML attributes that it will convert (e.g. it does not convert  ). Also, then you get stuff like: # First, strip out all attributes for easier parsing text = re.sub('<TR[^>]+>', '<TR>', text, re.I) text = re.sub('<TD[^>]+>', '<TD>', text, re.I) text = re.sub('<tr[^>]+>', '<TR>', text, re.I) text = re.sub('<td[^>]+>', '<TD>', text, re.I) Why have you got four expressions there? They are all doing case insensitive matching yet there are upper and lowercase versions of each. Hmm. The rest of the code is really pretty crappy. He could have just read the thing into lxml and have done XPath queries to extract data. It would have had the advantage of making clear what parts of the page he was extracting and made maintenance easy.
- esr 17y ago
  It's a proof-of-concept. Don't get hung up on the parsing details, I expect I'm going to have to rewrite it at least twice.
  - esr 17y ago
    I've written something relevent here: "Structure Is Not Meaning". http://esr.ibiblio.org/?p=1387 http://esr.ibiblio.org/?p=1387
    - jgrahamc 17y ago
      Your reply doesn't address why you do the things I highlighted in my original comment. For example, in the middle of your generic table parser you have code specific to Berlios making it non-generic. That's there because you haven't actually parsed the HTML you are trying to hack around it with regexps. In your post you say: So instead, I walk through the page looking for anything that looks like a hotlink wrapping literal text of the form #[0-9]. Actually, that oversimplifies; I look for a hotlink with a specific pattern of URL in a hotlink that I know points into the site bugtracker. I don't have any problem with you using regexps to identify the particular fragments you are looking for, my criticism is that you use them for HTML parsing. For example, in your code you use a regexp like this <A HREF="/bugs/\?func=detailbug&bug_id=([0-9]+)&group_id=%s"> This mixes structure and meaning to use your terms. You assume that there's an HREF attribute after the A. Your code is brittle when it comes to a change in the HTML (e.g. suppose the page author adds a CLASS= attribute). This would not happen if you parsed the HTML into a DOM tree and then did queries against it. You could quickly extract all the <A> tags in the path with a //A query (or even all those that have an HREF) and get the actual HREF robustly. Or you could not use XPath and use a parser that does calls backs with robust lists of attributes. Doing that would be both robust against changes in page structure, and robust against changes in the attributes or placement of attributes in the <A>. PS Is your name calling really necessary? In your post you refer to me as a 'snotty kid' and a 'fool' PPS Also you say: If I did what my unwise critic says is right, I’d take a parse tree of the entire page and then write a path query down to the table cell and the embedded link. I never said that, I said that you could use XPath queries to extract the data. I never implied that you use a complete rooted path to get where you want in the document. You've twisted what I actually said to fit into your 'structure and meaning' blog post.
      - frig 17y ago
        This is bazaar-style development in action: do a crap initial job and troll up helpful comments from people like you.
        jgrahamc 17y ago
- fnid 17y ago
  I think you are being a little over-critical. There are always better ways to do everything. This contributor was doing something that had never been done before -- by himself. Isn't the whole point of open source to show the code and let community involvement make it better? Since you have a lot of experience with HTML parsing, it'd be better to contribute your experience to the project than to bash someone for not doing something you know well as well as you could do it.
  - astine 17y ago
    ESR is a well known hacker who has maintained several large-scale OSS projects and done lots of coding for several decades. He literally wrote the book about OSS. If ESR's code was markedly bad, it would be news.
    - jgrahamc 17y ago
      Which large scale OSS projects?
      - astine 17y ago
        Fetchmail is the most famous one. http://fetchmail.berlios.de/ http://fetchmail.berlios.de/
        gonzo 17y ago
        gpsd is another, but esr seems to have trouble understanding parts of it. CML2 is yet another, though it failed to gain acceptance by the kernel group. the GNU ncurses library was maintained by Mr. Raymond for a while. fetchmail wasn't an original idea, it came out of UC Berkeley's "popclient". ESR has also famously over-claimed his experience (several times).
        astine 17y ago
        I'm aware. fnid just seemed to be under the impression that ESR was a newbie that was genuinely looking for help from more experienced hackers. I was just showing that he (ESR) is a noted (notorious) figure that should know better than to make noobish mistakes in code.
- _csoo 17y ago
  Reading it, it looks like a total hack job by a poor programmer. Unsurprising, most programmers are poor.
  - gonzo 17y ago
    especially those who aren't self-confident enough to say, "Oh, you're right!"
- deleted 17y ago
  [deleted]
_csoo 17y ago
Jesus fuck, the first comments I see here are complaints about the code. What about the concept? Is it any good? Is it worth working on?
- cdibona 17y ago
  Yes and no... for most of the sites syncing svn from site to site is pretty simple and usable, even if it can take a bit of time. For bugs, code.google.com has a export mechanism and you can kind of gin one up for sf.net via its feeds (I've not done it, so ymmv). For wikis, we (google) store wiki content in the vc, so that's easy and so we're pretty happy with the ability of people to pull info out of code.google.com. I think it is a non-trivial problem to take bugs from any bug tracking system to any other different one. There are too many customizations and changes even within single projects to make it a simple task. I think of taking a bug from a Demetrius project to , say, jira and that strikes me as being a poor match from a data perspective. Backing up your data off of the hosting site makes a lot of sense to me, which if Eric can make his program do, then that would be more immediately useful than host to host migration. In my experience, people tend to stick with hosts once they have an established project there.
9turningmirrors 17y ago
"Currently there is one area director, for the Open Source Awards, John Graham-Cummings" from: http://catb.org/~esr/roles.html http://catb.org/~esr/roles.html google result for "Eric raymond" "John Graham-Cumming" - http://bit.ly/3eCOSZ http://bit.ly/3eCOSZ not that I'm trying to prove anything ...... rofl

27 points by Luyt 17y ago

It's amusing to actually look at the source code of this. It's a single file, here: http://svn.gna.org/viewcvs/forgeplucker/trunk/bugplucker.py?rev=7&view=markup http://svn.gna.org/viewcvs/forgeplucker/trunk/bugplucker.py?... Reading it, it looks like a total hack job by a poor programmer. For example, HTML parsing it done by a bunch of regular expressions. Which include stuff like # Yes, Berlios generated \r<BR> sequences with no \n text = text.replace("\r<BR>", "\r\n") # And Berlios generated doubled </TD>s text = text.replace("</TD></TD>", "</TD>") All the pain of maintaining these little cases could have been removed by simply running the page being walked through an actual HTML parser that produces a DOM tree. Similarly the function dehtmlize contains a limit set of HTML attributes that it will convert (e.g. it does not convert  ). Also, then you get stuff like: # First, strip out all attributes for easier parsing text = re.sub('<TR[^>]+>', '<TR>', text, re.I) text = re.sub('<TD[^>]+>', '<TD>', text, re.I) text = re.sub('<tr[^>]+>', '<TR>', text, re.I) text = re.sub('<td[^>]+>', '<TD>', text, re.I) Why have you got four expressions there? They are all doing case insensitive matching yet there are upper and lowercase versions of each. Hmm. The rest of the code is really pretty crappy. He could have just read the thing into lxml and have done XPath queries to extract data. It would have had the advantage of making clear what parts of the page he was extracting and made maintenance easy.

esr 17y ago
It's a proof-of-concept. Don't get hung up on the parsing details, I expect I'm going to have to rewrite it at least twice.
- esr 17y ago
  I've written something relevent here: "Structure Is Not Meaning". http://esr.ibiblio.org/?p=1387 http://esr.ibiblio.org/?p=1387
  - jgrahamc 17y ago
    Your reply doesn't address why you do the things I highlighted in my original comment. For example, in the middle of your generic table parser you have code specific to Berlios making it non-generic. That's there because you haven't actually parsed the HTML you are trying to hack around it with regexps. In your post you say: So instead, I walk through the page looking for anything that looks like a hotlink wrapping literal text of the form #[0-9]. Actually, that oversimplifies; I look for a hotlink with a specific pattern of URL in a hotlink that I know points into the site bugtracker. I don't have any problem with you using regexps to identify the particular fragments you are looking for, my criticism is that you use them for HTML parsing. For example, in your code you use a regexp like this <A HREF="/bugs/\?func=detailbug&bug_id=([0-9]+)&group_id=%s"> This mixes structure and meaning to use your terms. You assume that there's an HREF attribute after the A. Your code is brittle when it comes to a change in the HTML (e.g. suppose the page author adds a CLASS= attribute). This would not happen if you parsed the HTML into a DOM tree and then did queries against it. You could quickly extract all the <A> tags in the path with a //A query (or even all those that have an HREF) and get the actual HREF robustly. Or you could not use XPath and use a parser that does calls backs with robust lists of attributes. Doing that would be both robust against changes in page structure, and robust against changes in the attributes or placement of attributes in the <A>. PS Is your name calling really necessary? In your post you refer to me as a 'snotty kid' and a 'fool' PPS Also you say: If I did what my unwise critic says is right, I’d take a parse tree of the entire page and then write a path query down to the table cell and the embedded link. I never said that, I said that you could use XPath queries to extract the data. I never implied that you use a complete rooted path to get where you want in the document. You've twisted what I actually said to fit into your 'structure and meaning' blog post.
    - frig 17y ago
      This is bazaar-style development in action: do a crap initial job and troll up helpful comments from people like you.
      - jgrahamc 17y ago
fnid 17y ago
I think you are being a little over-critical. There are always better ways to do everything. This contributor was doing something that had never been done before -- by himself. Isn't the whole point of open source to show the code and let community involvement make it better? Since you have a lot of experience with HTML parsing, it'd be better to contribute your experience to the project than to bash someone for not doing something you know well as well as you could do it.
- astine 17y ago
  ESR is a well known hacker who has maintained several large-scale OSS projects and done lots of coding for several decades. He literally wrote the book about OSS. If ESR's code was markedly bad, it would be news.
  - jgrahamc 17y ago
    Which large scale OSS projects?
    - astine 17y ago
      Fetchmail is the most famous one. http://fetchmail.berlios.de/ http://fetchmail.berlios.de/
      - gonzo 17y ago
        gpsd is another, but esr seems to have trouble understanding parts of it. CML2 is yet another, though it failed to gain acceptance by the kernel group. the GNU ncurses library was maintained by Mr. Raymond for a while. fetchmail wasn't an original idea, it came out of UC Berkeley's "popclient". ESR has also famously over-claimed his experience (several times).
        astine 17y ago
        I'm aware. fnid just seemed to be under the impression that ESR was a newbie that was genuinely looking for help from more experienced hackers. I was just showing that he (ESR) is a noted (notorious) figure that should know better than to make noobish mistakes in code.
_csoo 17y ago
Reading it, it looks like a total hack job by a poor programmer. Unsurprising, most programmers are poor.
- gonzo 17y ago
  especially those who aren't self-confident enough to say, "Oh, you're right!"
deleted 17y ago
[deleted]

_csoo 17y ago

Jesus fuck, the first comments I see here are complaints about the code. What about the concept? Is it any good? Is it worth working on?

cdibona 17y ago
Yes and no... for most of the sites syncing svn from site to site is pretty simple and usable, even if it can take a bit of time. For bugs, code.google.com has a export mechanism and you can kind of gin one up for sf.net via its feeds (I've not done it, so ymmv). For wikis, we (google) store wiki content in the vc, so that's easy and so we're pretty happy with the ability of people to pull info out of code.google.com. I think it is a non-trivial problem to take bugs from any bug tracking system to any other different one. There are too many customizations and changes even within single projects to make it a simple task. I think of taking a bug from a Demetrius project to , say, jira and that strikes me as being a poor match from a data perspective. Backing up your data off of the hosting site makes a lot of sense to me, which if Eric can make his program do, then that would be more immediately useful than host to host migration. In my experience, people tend to stick with hosts once they have an established project there.

9turningmirrors 17y ago

"Currently there is one area director, for the Open Source Awards, John Graham-Cummings" from: http://catb.org/~esr/roles.html http://catb.org/~esr/roles.html google result for "Eric raymond" "John Graham-Cumming" - http://bit.ly/3eCOSZ http://bit.ly/3eCOSZ not that I'm trying to prove anything ...... rofl