4 ms·

Reinstate hystry's Hacker News

18 points by pchristensen 19y ago

hystry.com lets me read Hacker News comments in chronological order instead of having to go back to each story (http://hystry.com/newsyc/follow/). I found it invaluable and I was a lot more involved in the community when I could follow the conversations easier. It died a couple days ago, apparently its IP address was banned from crawling Hacker News. pg, could you please let them back in? Please, pretty please? EDIT: or at least change the Comments page to include link and title for the story each comment pertains to?

ojbyrne 19y ago
How is that different from the "comments" link? Other than you don't have to login.
pchristensen 19y ago
The comments page shows all of the comments in order, but hystry shows the link to the article next to each comment, lets you filter out comments from specific stories, and view all the parents of a comment. It's hard to get a feel for it now since the feed got shut down, but I could browse through the last 100 comments in just a few minutes, and quickly get a feel for which stories people were talking about.
- pg 19y ago
  It was really pounding the server. Crawling for a search index is ok, but retrieving pages to dynamically update stuff generates a lot of requests. If you can tell me what you think is missing from News, I can try to add it.
  - pchristensen 19y ago
    Until ojbyrne mentioned it, I had never really looked at the comments page. It's almost right, but it would be much easier to follow the conversation if each comment was tagged with the story it refers to. Right now the comment threads are disjointed from each other (that's the price of a chronological list) but they're also disjointed from their source article. On hystry, it's 2 columns, with the left column being the title and link and the right being the comment. This makes it easier to mentally sort the comments into conversations. Like I said, most of it is already in the Comments page, which I now use instead of hystry. You might think of another way to do it that also works well. Thanks, pg!
    - pg 19y ago
      Is the comments page better now?
      - dood 19y ago
        It's a useful addition, but it messes up the visual flow, making it hard to quickly scan the comments page. How about putting the title at the end of the line with the rest of the metadata. E.g. 3 points by pg 8 minutes ago | link | parent | source: Reinstate hystry's Hacker News A 'source' or 'root' link would be useful when navigating comments also, to save having to click through each parent.
        pg 19y ago
        Yeah, you're right. How about now? Kind of interesting modifying software on the fly in response to user input. I wonder how often this happens.
        deleted 19y ago
        [deleted]
        dood 19y ago
        Kind of interesting modifying software on the fly in response to user input. I wonder how often this happens Funnily enough, I was just thinking that. New version works well for me.
        carpal 19y ago
        Awesome. Can you add the same thing to each user's comments page, too?
        pg 19y ago
        Ok.
        carpal 19y ago
  - imsteve 19y ago
    What would be great for kartik's app is a live "push" style feed of all the new content on hacker news (possibly including votes too). This way, the load on the site would be minimal. This would also let a lot of other people scratch their own itch to try out outlandish new ideas. Few of us newsyc readers are even going to bother thinking up ideas for your site because we realize that there's a low chance of them being implemented. You just couldn't implement them all if you wanted to. If you provide this feed though, it would allow people to get the instant gratification of coding up their own ideas and, I think, a lot more innovation would occur. I don't think you're trying to get rich by selling this site so there's not much to lose. Brand dilution likewise seems unlikely in this case. Any other downsides?
    - pchristensen 19y ago
      Great idea - I actually used and participated in Hacker News a lot more since I found the hystry page.
    - pg 19y ago
      This would also let a lot of other people scratch their own itch to try out outlandish new ideas. There's an even better way to do that, which is to let users write little Arc programs to control the way pages are generated. That is the eventual plan.
      - akkartik 19y ago
        Great idea. I always knew my crawler was a bit of a hack and would be eventually obsoleted. Until we get there, though, may my crawler be let back in? Like I said in email I promise to only restart it after I embed controls over how many URLs I get per hour deeply inside it, and to log my crawling activity and keep an eye on it in future. The reason I was crawling every few minutes was to get notified of updates to conversations as they happen, before the 'herd' moves on to new feeding grounds. But I can make it crawl even hourly and it'll still allow me to stay kinda abreast of the conversation here.
      - kirubakaran 19y ago
        Wow that would be awesome! And if everyone can see and vote on each other's little Arc program, the best one(s) will win.
  - chengmi 19y ago
    Why don't URLs in Ask YC posts appear as links? e.g. http://hystry.com/newsyc/follow/ http://hystry.com/newsyc/follow/ in pchristensen's submission?
    - pg 19y ago
      To discourage the trick mentioned in the sixth paragraph: http://ycombinator.com/newsguidelines.html http://ycombinator.com/newsguidelines.html
  - bootload 19y ago
    "... It was really pounding the server. Crawling for a search index is ok, but retrieving pages to dynamically update stuff generates a lot of requests. ..." I hit the RSS feed every 15m and have so for months. I call the RSS file, extract the necessary data from this file, then re-hit user pages once [0] to get their contact info. Some of the things I looked at to try and reduce hits is to check some basic HTTP options [1]: - 'Last-Modified/If-Modified-Since' to only take, if updated - 'ETag/If-None-Match' checking admin defined time to reload - 'gzip' to reduce size of downloads But none of the above are implemented (probably for good reason - complicated & not really required) checking the HTTP spec implemented is pretty bare. I only found server status. I could also check the the robot.txt file, but like I said I only download the RSS file and user pages. A quick check for "robot.txt" reveals none. So you could automate/enforce crawling rules using a combination of HTTP and/or robot.txt files. But this does require the client to correctly implement them. Having said that increasing the RSS feed from 25 to 30/35 entries to cover the front page [2] is about all I can really think of. [0] Have to check this. Mainly get contact info & cache if found. [1] The basics are found here: http://www.diveintopython.org/http_web_services/http_features.html http://www.diveintopython.org/http_web_services/http_feature... [2] Having only 25 entries means you might be tempted to grab the front page (and maybe the second page) instead of RSS because you miss the bottom 5 entries.
dpapathanasiou 19y ago
Are you sure YC is blocking hystry? The hystry blog makes no mention of anything like that. It might be that Kartik just stopped running it, for whatever reason. Note that the last blog entry was back in November, so maybe he's just moved on to something else, e.g.
- pchristensen 19y ago
  I found this out by emailing him yesterday. I thought it would be better if I made the case for reinstating it by telling pg the value it creates for me while using his site. Kartik had nothing to do with this public request.
henning 19y ago
Did hystry get permission from PG to crawl the server? Did they agree upon a certain rate at which hystry would access hacker news?
- pchristensen 19y ago
  I'm not saying hystry has any right, I'm just saying it was valuable to me. Like I mentioned in my other comments, the Comments page does almost everything hystry does, it's just missing a little story context.
- cratuki 19y ago
  Should they need to? They should be free to scrape and represent, and ycom should be free to block them if they choose. When you pursue anti-rights relating to the ownership of information distribution you come to disgusting solutions like current copyright models.
akkartik 19y ago
Wow, this is awesome. I've managed to make something at least somebody wants. It's a start. I didn't see this until now because I can't seem to go back to reading Hacker News without the ability to bounce between my interface and the original. I'm sure hystry'll be back once PG's given us a timeout :) And it's a well-deserved timeout. My crawler process seems to have gone rogue and I wasn't keeping an eye on it. Definitely my mistake.

18 points by pchristensen 19y ago

ojbyrne 19y ago

How is that different from the "comments" link? Other than you don't have to login.

pchristensen 19y ago

The comments page shows all of the comments in order, but hystry shows the link to the article next to each comment, lets you filter out comments from specific stories, and view all the parents of a comment. It's hard to get a feel for it now since the feed got shut down, but I could browse through the last 100 comments in just a few minutes, and quickly get a feel for which stories people were talking about.

pg 19y ago
It was really pounding the server. Crawling for a search index is ok, but retrieving pages to dynamically update stuff generates a lot of requests. If you can tell me what you think is missing from News, I can try to add it.
- pchristensen 19y ago
  Until ojbyrne mentioned it, I had never really looked at the comments page. It's almost right, but it would be much easier to follow the conversation if each comment was tagged with the story it refers to. Right now the comment threads are disjointed from each other (that's the price of a chronological list) but they're also disjointed from their source article. On hystry, it's 2 columns, with the left column being the title and link and the right being the comment. This makes it easier to mentally sort the comments into conversations. Like I said, most of it is already in the Comments page, which I now use instead of hystry. You might think of another way to do it that also works well. Thanks, pg!
  - pg 19y ago
    Is the comments page better now?
    - dood 19y ago
      It's a useful addition, but it messes up the visual flow, making it hard to quickly scan the comments page. How about putting the title at the end of the line with the rest of the metadata. E.g. 3 points by pg 8 minutes ago | link | parent | source: Reinstate hystry's Hacker News A 'source' or 'root' link would be useful when navigating comments also, to save having to click through each parent.
      - pg 19y ago
        Yeah, you're right. How about now? Kind of interesting modifying software on the fly in response to user input. I wonder how often this happens.
        deleted 19y ago
        [deleted]
        dood 19y ago
        Kind of interesting modifying software on the fly in response to user input. I wonder how often this happens Funnily enough, I was just thinking that. New version works well for me.
        carpal 19y ago
        Awesome. Can you add the same thing to each user's comments page, too?
        pg 19y ago
        Ok.
        carpal 19y ago
- imsteve 19y ago
  What would be great for kartik's app is a live "push" style feed of all the new content on hacker news (possibly including votes too). This way, the load on the site would be minimal. This would also let a lot of other people scratch their own itch to try out outlandish new ideas. Few of us newsyc readers are even going to bother thinking up ideas for your site because we realize that there's a low chance of them being implemented. You just couldn't implement them all if you wanted to. If you provide this feed though, it would allow people to get the instant gratification of coding up their own ideas and, I think, a lot more innovation would occur. I don't think you're trying to get rich by selling this site so there's not much to lose. Brand dilution likewise seems unlikely in this case. Any other downsides?
  - pchristensen 19y ago
    Great idea - I actually used and participated in Hacker News a lot more since I found the hystry page.
  - pg 19y ago
    This would also let a lot of other people scratch their own itch to try out outlandish new ideas. There's an even better way to do that, which is to let users write little Arc programs to control the way pages are generated. That is the eventual plan.
    - akkartik 19y ago
      Great idea. I always knew my crawler was a bit of a hack and would be eventually obsoleted. Until we get there, though, may my crawler be let back in? Like I said in email I promise to only restart it after I embed controls over how many URLs I get per hour deeply inside it, and to log my crawling activity and keep an eye on it in future. The reason I was crawling every few minutes was to get notified of updates to conversations as they happen, before the 'herd' moves on to new feeding grounds. But I can make it crawl even hourly and it'll still allow me to stay kinda abreast of the conversation here.
    - kirubakaran 19y ago
      Wow that would be awesome! And if everyone can see and vote on each other's little Arc program, the best one(s) will win.
- chengmi 19y ago
  Why don't URLs in Ask YC posts appear as links? e.g. http://hystry.com/newsyc/follow/ http://hystry.com/newsyc/follow/ in pchristensen's submission?
  - pg 19y ago
    To discourage the trick mentioned in the sixth paragraph: http://ycombinator.com/newsguidelines.html http://ycombinator.com/newsguidelines.html
- bootload 19y ago
  "... It was really pounding the server. Crawling for a search index is ok, but retrieving pages to dynamically update stuff generates a lot of requests. ..." I hit the RSS feed every 15m and have so for months. I call the RSS file, extract the necessary data from this file, then re-hit user pages once [0] to get their contact info. Some of the things I looked at to try and reduce hits is to check some basic HTTP options [1]: - 'Last-Modified/If-Modified-Since' to only take, if updated - 'ETag/If-None-Match' checking admin defined time to reload - 'gzip' to reduce size of downloads But none of the above are implemented (probably for good reason - complicated & not really required) checking the HTTP spec implemented is pretty bare. I only found server status. I could also check the the robot.txt file, but like I said I only download the RSS file and user pages. A quick check for "robot.txt" reveals none. So you could automate/enforce crawling rules using a combination of HTTP and/or robot.txt files. But this does require the client to correctly implement them. Having said that increasing the RSS feed from 25 to 30/35 entries to cover the front page [2] is about all I can really think of. [0] Have to check this. Mainly get contact info & cache if found. [1] The basics are found here: http://www.diveintopython.org/http_web_services/http_features.html http://www.diveintopython.org/http_web_services/http_feature... [2] Having only 25 entries means you might be tempted to grab the front page (and maybe the second page) instead of RSS because you miss the bottom 5 entries.

dpapathanasiou 19y ago

Are you sure YC is blocking hystry? The hystry blog makes no mention of anything like that. It might be that Kartik just stopped running it, for whatever reason. Note that the last blog entry was back in November, so maybe he's just moved on to something else, e.g.

pchristensen 19y ago
I found this out by emailing him yesterday. I thought it would be better if I made the case for reinstating it by telling pg the value it creates for me while using his site. Kartik had nothing to do with this public request.

henning 19y ago

Did hystry get permission from PG to crawl the server? Did they agree upon a certain rate at which hystry would access hacker news?

pchristensen 19y ago
I'm not saying hystry has any right, I'm just saying it was valuable to me. Like I mentioned in my other comments, the Comments page does almost everything hystry does, it's just missing a little story context.
cratuki 19y ago
Should they need to? They should be free to scrape and represent, and ycom should be free to block them if they choose. When you pursue anti-rights relating to the ownership of information distribution you come to disgusting solutions like current copyright models.

akkartik 19y ago

Wow, this is awesome. I've managed to make something at least somebody wants. It's a start. I didn't see this until now because I can't seem to go back to reading Hacker News without the ability to bounce between my interface and the original. I'm sure hystry'll be back once PG's given us a timeout :) And it's a well-deserved timeout. My crawler process seems to have gone rogue and I wasn't keeping an eye on it. Definitely my mistake.