11 ms·

Show HN: Hacker News Title Edit Tracker

366 points by petercooper 7y ago

petercooper 7y ago
I built this several months ago as I was interested to see how the titles of posts evolve/change on HN. It turns out lots of titles are edited every day (in both subjectively and objectively good and bad ways!) and I've found it interesting to see how titles have evolved. The whole thing is automated and is built around a Ruby script that pulls down the titles on the front page at a frequent interval. Any titles for specific story IDs that change get tracked and rendered out to a static HTML page hosted on Netlify. It runs by itself without incident so far. This has been on Show HN before but I was kindly invited by dang to repost it.
- cjamesd 7y ago
  Please tell me you plan on going meta and editing the title of this post.
  - dang 7y ago
    Good idea. I put "test" up there. When will it show in the list? Edit: there it is. Title reverted now.
- paulgb 7y ago
  Cool! Since you have the infrastructure already, it would be neat to track URL changes as well. Probably less frequent, but I bet they happen at least daily (e.g. if a submission is flagged as blogspam.)
  - petercooper 7y ago
    Yes, that would be possible, though I think it should rendered to a totally separate page to avoid confusion. I didn't realize URLs were edited here, so it was off my radar till now :)
- carrozo 7y ago
  Nicely done! It would be fantastic to see this embedded between post and comments on every piece here!
- blondin 7y ago
  love simple but useful apps like this! how's the hosting with netlify?
  - petercooper 7y ago
    I'm very happy with Netlify for numerous things. In this case, Netlify is just running a build script that converts the JSON I create at http://s3.amazonaws.com/jeanfromeastenders/current.json http://s3.amazonaws.com/jeanfromeastenders/current.json into the final HTML page.
- sarosh 7y ago
  Any plans to release the code? Which scraper? Chrome headless?
  - iudqnolq 7y ago
    Since there's no interactivity or logins or anything complex curl would probably do the job.
  - petercooper 7y ago
    It's not very good, very basic and naive (but works!) https://gist.github.com/peterc/b325cdea7561528c506f073f477412fa https://gist.github.com/peterc/b325cdea7561528c506f073f47741... The actual turning the JSON into HTML takes place on Netlify. If you wish to render the JSON for your own benefit, the current version is always at http://s3.amazonaws.com/jeanfromeastenders/current.json http://s3.amazonaws.com/jeanfromeastenders/current.json
- JoshMcguigan 7y ago
  > Any titles for specific story IDs that change get tracked.. Can you talk more about the data infrastructure?
- mft_ 7y ago
  OOI, where does the Ruby script run?
  - petercooper 7y ago
    Just on a random VPS I have via a cronjob. Ideally I'd run it as a Lambda job and then I wouldn't have any infrastructure at all. I just didn't get round to it after prototyping.
- samstave 7y ago
  This would be really good to apply to watching spin on various verticals. Such as entertainment and financial articles... Recall during the banking crisis of 2008 there was the article stating that the EU was about 16 TRILLION in debt crisis, but then that was too alarmist and they didnt. Want that getting out, so they edited the title, but they forgot to change the UrL which had the original title in it.
  - Scoundreller 7y ago
    Oh god yes. I hate seeing that an article was « updated ». Well what the heck changed since I read it yesterday?!?!? And this is a well-regarded national newspaper in Canada.
  - dang 7y ago
    News outlets do that all the time. NYT in particular. Often they don't change the URLs, and I've always assumed that's for technical reasons—either there's a system that's treats URLs as immutable, or it's copied through multiple systems that the people who change the title can't edit, or some such thing.
    - CPLX 7y ago
      Perhaps you’ll appreciate this: https://mobile.twitter.com/nyt_diff https://mobile.twitter.com/nyt_diff
- bambax 7y ago
  I just posted a very similar thing implemented as a browser extension (HNtitles[0]) so that it shows the original text in the page without any effort for the user. (I had written a first version of this some time ago[1] as a bookmaklet, but a browser extension is simpler to use.) [0] https://news.ycombinator.com/item?id=21621411 https://news.ycombinator.com/item?id=21621411 [1]: https://news.ycombinator.com/item?id=6573188 https://news.ycombinator.com/item?id=6573188
  - petercooper 7y ago
    I think it works well as an extension for people who want to keep an eye on this sort of thing. I only wanted a casual, occasional look at the edits, but HNtitles looks ideal for the power user - good work.
    - bambax 7y ago
      Thanks. The list you are making is very good, it's a destination in itself. As said elsewhere, the extension helps to avoid clicking on the same story twice when the title changed. (I will add url monitoring too.)
iamwil 7y ago
What is the rule for titles being edited? Is it just no editoralization in the titles?
- tptacek 7y ago
  See the guidelines. The titles should be the original article title, unless that title is clickbaity, in which case either a less baity subhed can be used or a carefully written neutral headline. The big no-no is extracting some interesting fact from inside the article and using it as the title; submissions are community property, and the person who submits them isn't entitled to decide what the most important angle in the article is.
julienreszka 7y ago
This is great! Transparency, accountability, predictability etc should be more prevalent in the web.
carrozo 7y ago
This would be interesting for a number of news sites too.
- vageli 7y ago
  This is one for the new york times [0]. [0]: https://mobile.twitter.com/nyt_diff https://mobile.twitter.com/nyt_diff
- petercooper 7y ago
  Someone I know runs something like this for news content. See https://www.newssniffer.co.uk/ https://www.newssniffer.co.uk/
dang 7y ago
That is such a great UI. I feel like the list should be longer. Can you do that without it costing more? Not all of those edits are by mods, of course. Some are made by submitters (edit: as the site points out!). Also, some are because we switched the URL and thus to the title of the new article (example: https://news.ycombinator.com/item?id=21616157 https://news.ycombinator.com/item?id=21616157). Those look weird if you assume they're moderation edits. It looks like the list is sorted by reverse ID, which means articles that were submitted earlier are lower down on the page. But sometimes we re-up those (https://news.ycombinator.com/item?id=11662380 https://news.ycombinator.com/item?id=11662380), so from a front-page perspective some 'newer' stories are below older ones.
- octosphere 7y ago
  > I feel like the list should be longer Isn't there a small window of time to edit a submission before it gets committed permanently to HN? I imagine the list would be bigger if the grace period lasted much longer.
  - dang 7y ago
    Yes, but moderators sometimes edit titles and do other moderatory things after that limit has expired.
- petercooper 7y ago
  Can you do that without it costing more? Yes, I just increase a LIMIT in an SQL query. I'll try that now. I didn't want the page to look too cumbersome.
yellow_lead 7y ago
This shows how much work goes into moderation here. Almost all titles seem more accurate and less clickbaity.
jcims 7y ago
It would be interesting to let submitters provide multiple titles and a/b test them from the outset.
- bonoboTP 7y ago
  The entire point is to make titles less clickbaity. A/B testing them to get more clicks exactly defeats the entire purpose why mods edit titles.
  - jcims 7y ago
    Yes but the vast majority of folks on the Internet, including HN, don't get to directly experience the effect that headlines have on engagement. I never said it was a good idea, just interesting. :)
dorkwood 7y ago
This sequence provided some comedy: > 21:00 Trends in the San Francisco poop crisis > 21:15 Trends in the San Francisco (dog) poop crisis > 22:25 Trends in the San Francisco (mostly dog) poop crisis
- dang 7y ago
  https://news.ycombinator.com/item?id=21610152 https://news.ycombinator.com/item?id=21610152
  - quickthrower2 7y ago
    Title was updated because of some primary research on the data set. Thats kinda cool.
codetrotter 7y ago
What frequency of sending GET requests to the servers of HN is an acceptable rate for a bot? I tried to look for an answer on this but didn't find any. In the past I got the IP of my server banned from accessing HN for sending too many requests in too short of a time span. I found the unban interface that you provide, lowered the request rate of my crawler and tried again but was still sending too many requests in a limited amount of time and got the IP of my server banned again. If I recall correctly, I got it unbanned a third time and lowered the request rate even more but then got banned again and then I think it would not allow automatic unbanning. Don't remember if I just gave up at that point or if I sent an email about it, or if I just waited some amount of time if there was a statement about how long I would have to wait before being able to use the unban interface for the IP of my server again. Anyway, an official answer about the acceptable request rate would be nice. Perhaps put it in the FAQ? Also, if people doing automated GET requests were to create a unique UA string for their scrapers that include a way for HN staff to get in touch, like for example (but with actual names of bot and site) Examplebot/0.95 (+http://www.example.com/bot.html) where the page on that URL would list an e-mail address for getting in touch, as well as having a statement about how to verify that a given crawler belongs to that service, would that help in not getting server IP banned automatically?
- dang 7y ago
  Once per 30 seconds. That's in our robots.txt: https://news.ycombinator.com/robots.txt https://news.ycombinator.com/robots.txt. We've been working for a long time on (edit: what we expect to be) some serious performance improvements that might allow us to relax that limit. For now though, HN's process still runs on a single core and we don't have much performance to spare. If you need more than that, you should use the Firebase-based API (https://github.com/HackerNews/API https://github.com/HackerNews/API). The public dataset is also available as a Google BigQuery table: https://bigquery.cloud.google.com/dataset/bigquery-public-data:hacker_news https://bigquery.cloud.google.com/dataset/bigquery-public-da.... Edit: since this subthread is not really on topic I detached it from https://news.ycombinator.com/item?id=21617478 https://news.ycombinator.com/item?id=21617478.
  - stoops 7y ago
deleted 7y ago
[deleted]
- deleted 7y ago
  [deleted]
edoceo 7y ago
Could you also match to the source article/post title?
Karunamon 7y ago
Some of these edits seem really questionable when laid out like this. Pew Research: 2.2% of Americans produce 97% of political tweets ↓ Small share of U.S. adults produce majority of tweets on national politics Why remove the exact figures? Former Apple chip executives found company to take on Intel, AMD ↓ Three of Apple and Google’s former star chip designers launch NUVIA Isn't "star designers" more subjective than "executives"? 1.2B people exposed in data leak includes personal info, LinkedIn, Facebook ↓ Personal and social information of 1B people discovered in data leak Why make the headline less informative? Data leaks happen regularly. Data leaks from Facebook and LinkedIn has different implications than a leak from LexisNexis or a random blog. Cloudflare open-sources Flan Scan, a network vulnerability scanner ↓ Flan Scan: Lightweight Network Vulnerability Scanner Again, why remove info? The fact that CloudFlare is behind this is more interesting than yet another random tool. Mozilla: “Dear Facebook: Stop cross site tracking by default” ↓ Dear Facebook: Stop cross site tracking by default Same complaint. This distinguishes a random person making a random gripe from freakin' Mozilla who has the control to make Facebook's tracking more difficult. ---- Every single one of these headlines actually are less informative or less interesting (in general, of lower quality) than their original submissions. They actually served to make HN less informative. WTF? That gripe aside, most of the edits are useful (typo fixing, adding dates, and such). These just leave me scratching my head.
- zamadatix 7y ago
  Most of these edits seem to be to make the title match the title of the article instead of being editoralized. From the guidelines: "Please don't do things to make titles stand out, like using uppercase or exclamation points, or saying how great an article is. It's implicit in submitting something that you think it's important. Please submit the original source. If a post reports on something found on another site, submit the latter. If you submit a link to a video or pdf, please warn us by appending [video] or [pdf] to the title. If the title includes the name of the site, please take it out, because the site name will be displayed after the link. If the title begins with a number or number + gratuitous adjective, we'd appreciate it if you'd crop it. E.g. translate "10 Ways To Do X" to "How To Do X," and "14 Amazing Ys" to "Ys." Exception: when the number is meaningful, e.g. "The 5 Platonic Solids." Otherwise please use the original title, unless it is misleading or linkbait; don't editorialize."
bonoboTP 7y ago
I am very thankful for this policy. The HN titles are essentially always better and more honest representations of the article contents. Keep up the good work!
kovan_music 7y ago
KovanKovan
Kinnard 7y ago
This is really cool! It'd be great to have a tool that tracks all editing/curation/censorship on hacker news.
- Kinnard 7y ago
  Or you know . . . we could just throw it onto the blockchain . . . ;)
dredmorbius 7y ago
As someone who frequently submits title change requests to the HN mod address (hn@ycombinator.com), this is an interesting and insightful tool. No, none of my recent requests (the most recent is a few days ago) are shown. Which suggests I'm not too much of an annoyance, yet. HN's mods (dang and sctb) are amazingly responsive and tolerant. I try to make their job easier in requests by keeping those short and clear. Others may find this useful or have futher suggestions. Generally: - Action in the subject: title, clickbait, spam, link disintermediation (pointing to a primary rather than secondary source), self-promotion (I've landed on "one-note flute" to describe this), and behavioural issues. Also occasionally vouches (for flagged posts/comments), or best-of nominations (there's a curated list put out by HN monthly). - Followed by the title-as-presently standing. Should make identifying the post easier. - Link to post (or comment) as first line. - Often a link to the submitted article. - A suggested change or revert. Often these are clear, sometimes not. My view is that submitting both a "this needs changing" and "here's my suggestion", along with a possible rationale (usually a subhead, lede line, occasionally a good overview line from the article) makes the editors' job easier. These are often accepted, sometimes with a slight change, sometimes as is. I generally don't follow up with a thanks or further acknowledgement, but usually do note when the request isn't accepted that I'm OK with it. There are times I've differed with views (usually tech-politics intersections). I really wish HN could discuss such topics better than it does, though that also seems to be ... somewhat ... improving with time. Discussion on HN is almost always superior to other venues I frequent online. Response times are generally a few minutes to hours, longer during off-hours. Rarely, less-critical issues may take a few days to generate a response. But there's very nearly always one, which I appreciate.
kevlar1818 7y ago
This is great! I have a feature request: One thing that worries me about HN is that I've had a _link_ changed by a moderator in the past. And I don't mean removing query junk or changing between mobile/non-mobile sites -- I mean changing the link to an entirely different site. Even worse, by the time I noticed, I couldn't edit or delete my post (which I wanted to do because I disagreed with the new link). I was essentially forced to post content I didn't want to post! I think this is crosses the line from curation to impersonation.
- dang 7y ago
  What was the link? We change URLs every day, for lots of different reasons—for example if one article is mostly just copying from another source, or if users suggest a better URL. When we do that we nearly always post something saying what we did, and including the previous link: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=by%3Adang%20changed%20url&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que... https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=by%3Adang%20switched%20htttp&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
  - kevlar1818 7y ago
    It was over a year ago I think. I can do some digging, but not right now.
    - dang 7y ago
      If you find it, please email hn@ycombinator.com so we can take a look. If we got something wrong, we're definitely interested in learning from that.
      - ignoramous 7y ago
        One such instance: https://news.ycombinator.com/item?id=21378471 https://news.ycombinator.com/item?id=21378471 I remember commenting on a different article but the one that it points to now is a different article: https://news.ycombinator.com/item?id=21384151 https://news.ycombinator.com/item?id=21384151
kylebenzle 7y ago
20:40 Is the San Francisco Poop Crisis Out of Control? ↓ 21:00 Trends in the San Francisco poop crisis ↓ 21:15 Trends in the San Francisco (dog) poop crisis ↓ 22:25 Trends in the San Francisco (mostly dog) poop crisis
- dang 7y ago
  It's missing one now though.
OrgNet 7y ago
you should also track the comments moving from thread to thread (when an admin moves comments from one thread to anoter), comment shadow banning, post popularity based on upvote counts (lots of times, post get to the front page with only a few votes), or maybe even track how fast posts get deranked
- gerikson 7y ago
  HN has an API: https://github.com/HackerNews/API https://github.com/HackerNews/API I'm using it to track common items here and on Lobste.rs (and Proggit): http://gerikson.com/hnlo/ http://gerikson.com/hnlo/ Here's the endpoint for the latest 500 submissions: https://hacker-news.firebaseio.com/v0/newstories.json https://hacker-news.firebaseio.com/v0/newstories.json here's the one for the current top stories: https://hacker-news.firebaseio.com/v0/topstories.json https://hacker-news.firebaseio.com/v0/topstories.json It's actually quite nice to work with. I don't know how to keep track of comments moving from thread to thread, because that's not a metric I'm interested in, but it should be possible to track somehow.
mft_ 7y ago
Thanks - really interesting. Take this one, for example: nvidia Drops Support for CUDA on macOS --> CUDA 10.2 is the last release to support macOS --> CUDA Toolkit Release Notes The first two titles are quite interesting to me, as a macOS user and a general follower of the tech space (and I'd note neither are sensationalised, or click-bait, from what I can see). The last...? I'm not going to click on that in a million years, as I don't work with CUDA. I'm not totally clear what the moderators' motivations always are, but might it be true that in some cases, maybe they're prioritising strict accuracy over interest, or discoverability? And as a result, their actions are actually diminishing the value of HN as a discussion forum?
- kemyd 7y ago
  You have the answer here: https://news.ycombinator.com/item?id=21617317 https://news.ycombinator.com/item?id=21617317 "[...] The big no-no is extracting some interesting fact from inside the article and using it as the title"
  - Tenoke 7y ago
    Ironically, that's exactly what dang often does, including today.
    - dang 7y ago
      Often? That doesn't seem true to me. Can you give examples?
  - mft_ 7y ago
    Thanks - makes sense from a ‘they’re following the rules perspective’, but less so from a ‘enable discovery and subsequent interesting discussion’ perspective. I wonder whether a subtitle giving an editorialised “so what” would help... but that would of course mean twice as much to moderate, and twice as much for people to complain about...
  - egeozcan 7y ago
    Interesting, I had forgotten about this rule. Doesn't that encourage blogspam links instead of the source with context?
tjoff 7y ago
One thing I dislike about title edits, even when it is for the better, is that it makes it hard to know if you already have read the article. Popular topics often get multiple submissions and trigger related topics. When the title changes it is hard to know whether it is a new link with new/different information or whether it is the same one that I've already read. In my opinion once something hits the front page it is too late to edit it (other than minor things such as adding year or [pdf/video]).
- bambax 7y ago
  Exactly! That was my motivation too when writing a similar tool as a browser extension [0]. So for each title you can see the original text. [0] hntitles.adgent.com
- joncrane 7y ago
  My browser (FF) displays the link in a slightly lighter grey if I've visited it. To clean up my feed, I hide topics that I am "done with." That is, I gave it the amount of attention I'm willing to give it. This keeps my feed updating quicker as new items enter the stack at the bottom to replace the ones I've hidden. I started doing this because it mimics the behavior of one of reddit's settings, which is to hide all topics I've voted on. I found the feature so convenient that I started using the hiding behavior here in the same way.
Fnoord 7y ago
I remember there being an extension for Firefox which showed editorialized (news) site titles and content. I forgot the name though. Does anyone have any recommendations for this feature?
ocdtrekkie 7y ago
I'd love to see the domain added to the end of these, like they are on HN itself. For example, one might wonder why NVIDIA CUDA Toolkit is shorted to CUDA Toolkit... unless you see that (nvidia.com) is after it anyways. It might add some context to some of the edits.
dang 7y ago
Arguably a bug: some of the titles listed have subsequent edits that don't show up on the page. Examples: https://news.ycombinator.com/item?id=21619671 https://news.ycombinator.com/item?id=21619671 https://news.ycombinator.com/item?id=21609572 https://news.ycombinator.com/item?id=21609572 Does the script stop checking them at some point?
- usr1106 7y ago
  I understood the script only checks the front page. Could the change have happened when the article was not on the front page? Or are all articles that have ever been on the front page supposed to be checked "forever"? I have no idea how often / how quickly an article could oscillate between rank 30 and 31 for example.
phailhaus 7y ago
Have you considered adding a "sort by amount of change" option? It would be very interesting to see why larger changes happen compared to smaller ones.

366 points by petercooper 7y ago

I built this several months ago as I was interested to see how the titles of posts evolve/change on HN. It turns out lots of titles are edited every day (in both subjectively and objectively good and bad ways!) and I've found it interesting to see how titles have evolved. The whole thing is automated and is built around a Ruby script that pulls down the titles on the front page at a frequent interval. Any titles for specific story IDs that change get tracked and rendered out to a static HTML page hosted on Netlify. It runs by itself without incident so far. This has been on Show HN before but I was kindly invited by dang to repost it.

cjamesd 7y ago
Please tell me you plan on going meta and editing the title of this post.
- dang 7y ago
  Good idea. I put "test" up there. When will it show in the list? Edit: there it is. Title reverted now.
paulgb 7y ago
Cool! Since you have the infrastructure already, it would be neat to track URL changes as well. Probably less frequent, but I bet they happen at least daily (e.g. if a submission is flagged as blogspam.)
- petercooper 7y ago
  Yes, that would be possible, though I think it should rendered to a totally separate page to avoid confusion. I didn't realize URLs were edited here, so it was off my radar till now :)
carrozo 7y ago
Nicely done! It would be fantastic to see this embedded between post and comments on every piece here!
blondin 7y ago
love simple but useful apps like this! how's the hosting with netlify?
- petercooper 7y ago
  I'm very happy with Netlify for numerous things. In this case, Netlify is just running a build script that converts the JSON I create at http://s3.amazonaws.com/jeanfromeastenders/current.json http://s3.amazonaws.com/jeanfromeastenders/current.json into the final HTML page.
sarosh 7y ago
Any plans to release the code? Which scraper? Chrome headless?
- iudqnolq 7y ago
  Since there's no interactivity or logins or anything complex curl would probably do the job.
- petercooper 7y ago
  It's not very good, very basic and naive (but works!) https://gist.github.com/peterc/b325cdea7561528c506f073f477412fa https://gist.github.com/peterc/b325cdea7561528c506f073f47741... The actual turning the JSON into HTML takes place on Netlify. If you wish to render the JSON for your own benefit, the current version is always at http://s3.amazonaws.com/jeanfromeastenders/current.json http://s3.amazonaws.com/jeanfromeastenders/current.json
JoshMcguigan 7y ago
> Any titles for specific story IDs that change get tracked.. Can you talk more about the data infrastructure?
mft_ 7y ago
OOI, where does the Ruby script run?
- petercooper 7y ago
  Just on a random VPS I have via a cronjob. Ideally I'd run it as a Lambda job and then I wouldn't have any infrastructure at all. I just didn't get round to it after prototyping.
samstave 7y ago
This would be really good to apply to watching spin on various verticals. Such as entertainment and financial articles... Recall during the banking crisis of 2008 there was the article stating that the EU was about 16 TRILLION in debt crisis, but then that was too alarmist and they didnt. Want that getting out, so they edited the title, but they forgot to change the UrL which had the original title in it.
- Scoundreller 7y ago
  Oh god yes. I hate seeing that an article was « updated ». Well what the heck changed since I read it yesterday?!?!? And this is a well-regarded national newspaper in Canada.
- dang 7y ago
  News outlets do that all the time. NYT in particular. Often they don't change the URLs, and I've always assumed that's for technical reasons—either there's a system that's treats URLs as immutable, or it's copied through multiple systems that the people who change the title can't edit, or some such thing.
  - CPLX 7y ago
    Perhaps you’ll appreciate this: https://mobile.twitter.com/nyt_diff https://mobile.twitter.com/nyt_diff
bambax 7y ago
I just posted a very similar thing implemented as a browser extension (HNtitles[0]) so that it shows the original text in the page without any effort for the user. (I had written a first version of this some time ago[1] as a bookmaklet, but a browser extension is simpler to use.) [0] https://news.ycombinator.com/item?id=21621411 https://news.ycombinator.com/item?id=21621411 [1]: https://news.ycombinator.com/item?id=6573188 https://news.ycombinator.com/item?id=6573188
- petercooper 7y ago
  I think it works well as an extension for people who want to keep an eye on this sort of thing. I only wanted a casual, occasional look at the edits, but HNtitles looks ideal for the power user - good work.
  - bambax 7y ago
    Thanks. The list you are making is very good, it's a destination in itself. As said elsewhere, the extension helps to avoid clicking on the same story twice when the title changed. (I will add url monitoring too.)

iamwil 7y ago

What is the rule for titles being edited? Is it just no editoralization in the titles?

tptacek 7y ago
See the guidelines. The titles should be the original article title, unless that title is clickbaity, in which case either a less baity subhed can be used or a carefully written neutral headline. The big no-no is extracting some interesting fact from inside the article and using it as the title; submissions are community property, and the person who submits them isn't entitled to decide what the most important angle in the article is.

julienreszka 7y ago

This is great! Transparency, accountability, predictability etc should be more prevalent in the web.

carrozo 7y ago

This would be interesting for a number of news sites too.

vageli 7y ago
This is one for the new york times [0]. [0]: https://mobile.twitter.com/nyt_diff https://mobile.twitter.com/nyt_diff
petercooper 7y ago
Someone I know runs something like this for news content. See https://www.newssniffer.co.uk/ https://www.newssniffer.co.uk/

dang 7y ago

That is such a great UI. I feel like the list should be longer. Can you do that without it costing more? Not all of those edits are by mods, of course. Some are made by submitters (edit: as the site points out!). Also, some are because we switched the URL and thus to the title of the new article (example: https://news.ycombinator.com/item?id=21616157 https://news.ycombinator.com/item?id=21616157). Those look weird if you assume they're moderation edits. It looks like the list is sorted by reverse ID, which means articles that were submitted earlier are lower down on the page. But sometimes we re-up those (https://news.ycombinator.com/item?id=11662380 https://news.ycombinator.com/item?id=11662380), so from a front-page perspective some 'newer' stories are below older ones.

octosphere 7y ago
> I feel like the list should be longer Isn't there a small window of time to edit a submission before it gets committed permanently to HN? I imagine the list would be bigger if the grace period lasted much longer.
- dang 7y ago
  Yes, but moderators sometimes edit titles and do other moderatory things after that limit has expired.
petercooper 7y ago
Can you do that without it costing more? Yes, I just increase a LIMIT in an SQL query. I'll try that now. I didn't want the page to look too cumbersome.

yellow_lead 7y ago

This shows how much work goes into moderation here. Almost all titles seem more accurate and less clickbaity.

jcims 7y ago

It would be interesting to let submitters provide multiple titles and a/b test them from the outset.

bonoboTP 7y ago
The entire point is to make titles less clickbaity. A/B testing them to get more clicks exactly defeats the entire purpose why mods edit titles.
- jcims 7y ago
  Yes but the vast majority of folks on the Internet, including HN, don't get to directly experience the effect that headlines have on engagement. I never said it was a good idea, just interesting. :)

dorkwood 7y ago

This sequence provided some comedy: > 21:00 Trends in the San Francisco poop crisis > 21:15 Trends in the San Francisco (dog) poop crisis > 22:25 Trends in the San Francisco (mostly dog) poop crisis

dang 7y ago
https://news.ycombinator.com/item?id=21610152 https://news.ycombinator.com/item?id=21610152
- quickthrower2 7y ago
  Title was updated because of some primary research on the data set. Thats kinda cool.

codetrotter 7y ago

What frequency of sending GET requests to the servers of HN is an acceptable rate for a bot? I tried to look for an answer on this but didn't find any. In the past I got the IP of my server banned from accessing HN for sending too many requests in too short of a time span. I found the unban interface that you provide, lowered the request rate of my crawler and tried again but was still sending too many requests in a limited amount of time and got the IP of my server banned again. If I recall correctly, I got it unbanned a third time and lowered the request rate even more but then got banned again and then I think it would not allow automatic unbanning. Don't remember if I just gave up at that point or if I sent an email about it, or if I just waited some amount of time if there was a statement about how long I would have to wait before being able to use the unban interface for the IP of my server again. Anyway, an official answer about the acceptable request rate would be nice. Perhaps put it in the FAQ? Also, if people doing automated GET requests were to create a unique UA string for their scrapers that include a way for HN staff to get in touch, like for example (but with actual names of bot and site) Examplebot/0.95 (+http://www.example.com/bot.html) where the page on that URL would list an e-mail address for getting in touch, as well as having a statement about how to verify that a given crawler belongs to that service, would that help in not getting server IP banned automatically?

dang 7y ago
Once per 30 seconds. That's in our robots.txt: https://news.ycombinator.com/robots.txt https://news.ycombinator.com/robots.txt. We've been working for a long time on (edit: what we expect to be) some serious performance improvements that might allow us to relax that limit. For now though, HN's process still runs on a single core and we don't have much performance to spare. If you need more than that, you should use the Firebase-based API (https://github.com/HackerNews/API https://github.com/HackerNews/API). The public dataset is also available as a Google BigQuery table: https://bigquery.cloud.google.com/dataset/bigquery-public-data:hacker_news https://bigquery.cloud.google.com/dataset/bigquery-public-da.... Edit: since this subthread is not really on topic I detached it from https://news.ycombinator.com/item?id=21617478 https://news.ycombinator.com/item?id=21617478.
- stoops 7y ago

deleted 7y ago

[deleted]

deleted 7y ago
[deleted]

edoceo 7y ago

Could you also match to the source article/post title?

Karunamon 7y ago

Some of these edits seem really questionable when laid out like this. Pew Research: 2.2% of Americans produce 97% of political tweets ↓ Small share of U.S. adults produce majority of tweets on national politics Why remove the exact figures? Former Apple chip executives found company to take on Intel, AMD ↓ Three of Apple and Google’s former star chip designers launch NUVIA Isn't "star designers" more subjective than "executives"? 1.2B people exposed in data leak includes personal info, LinkedIn, Facebook ↓ Personal and social information of 1B people discovered in data leak Why make the headline less informative? Data leaks happen regularly. Data leaks from Facebook and LinkedIn has different implications than a leak from LexisNexis or a random blog. Cloudflare open-sources Flan Scan, a network vulnerability scanner ↓ Flan Scan: Lightweight Network Vulnerability Scanner Again, why remove info? The fact that CloudFlare is behind this is more interesting than yet another random tool. Mozilla: “Dear Facebook: Stop cross site tracking by default” ↓ Dear Facebook: Stop cross site tracking by default Same complaint. This distinguishes a random person making a random gripe from freakin' Mozilla who has the control to make Facebook's tracking more difficult. ---- Every single one of these headlines actually are less informative or less interesting (in general, of lower quality) than their original submissions. They actually served to make HN less informative. WTF? That gripe aside, most of the edits are useful (typo fixing, adding dates, and such). These just leave me scratching my head.

zamadatix 7y ago
Most of these edits seem to be to make the title match the title of the article instead of being editoralized. From the guidelines: "Please don't do things to make titles stand out, like using uppercase or exclamation points, or saying how great an article is. It's implicit in submitting something that you think it's important. Please submit the original source. If a post reports on something found on another site, submit the latter. If you submit a link to a video or pdf, please warn us by appending [video] or [pdf] to the title. If the title includes the name of the site, please take it out, because the site name will be displayed after the link. If the title begins with a number or number + gratuitous adjective, we'd appreciate it if you'd crop it. E.g. translate "10 Ways To Do X" to "How To Do X," and "14 Amazing Ys" to "Ys." Exception: when the number is meaningful, e.g. "The 5 Platonic Solids." Otherwise please use the original title, unless it is misleading or linkbait; don't editorialize."

bonoboTP 7y ago

I am very thankful for this policy. The HN titles are essentially always better and more honest representations of the article contents. Keep up the good work!

kovan_music 7y ago

KovanKovan

Kinnard 7y ago

This is really cool! It'd be great to have a tool that tracks all editing/curation/censorship on hacker news.

Kinnard 7y ago
Or you know . . . we could just throw it onto the blockchain . . . ;)

dredmorbius 7y ago

As someone who frequently submits title change requests to the HN mod address (hn@ycombinator.com), this is an interesting and insightful tool. No, none of my recent requests (the most recent is a few days ago) are shown. Which suggests I'm not too much of an annoyance, yet. HN's mods (dang and sctb) are amazingly responsive and tolerant. I try to make their job easier in requests by keeping those short and clear. Others may find this useful or have futher suggestions. Generally: - Action in the subject: title, clickbait, spam, link disintermediation (pointing to a primary rather than secondary source), self-promotion (I've landed on "one-note flute" to describe this), and behavioural issues. Also occasionally vouches (for flagged posts/comments), or best-of nominations (there's a curated list put out by HN monthly). - Followed by the title-as-presently standing. Should make identifying the post easier. - Link to post (or comment) as first line. - Often a link to the submitted article. - A suggested change or revert. Often these are clear, sometimes not. My view is that submitting both a "this needs changing" and "here's my suggestion", along with a possible rationale (usually a subhead, lede line, occasionally a good overview line from the article) makes the editors' job easier. These are often accepted, sometimes with a slight change, sometimes as is. I generally don't follow up with a thanks or further acknowledgement, but usually do note when the request isn't accepted that I'm OK with it. There are times I've differed with views (usually tech-politics intersections). I really wish HN could discuss such topics better than it does, though that also seems to be ... somewhat ... improving with time. Discussion on HN is almost always superior to other venues I frequent online. Response times are generally a few minutes to hours, longer during off-hours. Rarely, less-critical issues may take a few days to generate a response. But there's very nearly always one, which I appreciate.

kevlar1818 7y ago

This is great! I have a feature request: One thing that worries me about HN is that I've had a _link_ changed by a moderator in the past. And I don't mean removing query junk or changing between mobile/non-mobile sites -- I mean changing the link to an entirely different site. Even worse, by the time I noticed, I couldn't edit or delete my post (which I wanted to do because I disagreed with the new link). I was essentially forced to post content I didn't want to post! I think this is crosses the line from curation to impersonation.

dang 7y ago
What was the link? We change URLs every day, for lots of different reasons—for example if one article is mostly just copying from another source, or if users suggest a better URL. When we do that we nearly always post something saying what we did, and including the previous link: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=by%3Adang%20changed%20url&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que... https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=by%3Adang%20switched%20htttp&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
- kevlar1818 7y ago
  It was over a year ago I think. I can do some digging, but not right now.
  - dang 7y ago
    If you find it, please email hn@ycombinator.com so we can take a look. If we got something wrong, we're definitely interested in learning from that.
    - ignoramous 7y ago
      One such instance: https://news.ycombinator.com/item?id=21378471 https://news.ycombinator.com/item?id=21378471 I remember commenting on a different article but the one that it points to now is a different article: https://news.ycombinator.com/item?id=21384151 https://news.ycombinator.com/item?id=21384151

kylebenzle 7y ago

20:40 Is the San Francisco Poop Crisis Out of Control? ↓ 21:00 Trends in the San Francisco poop crisis ↓ 21:15 Trends in the San Francisco (dog) poop crisis ↓ 22:25 Trends in the San Francisco (mostly dog) poop crisis

dang 7y ago
It's missing one now though.

OrgNet 7y ago

you should also track the comments moving from thread to thread (when an admin moves comments from one thread to anoter), comment shadow banning, post popularity based on upvote counts (lots of times, post get to the front page with only a few votes), or maybe even track how fast posts get deranked

gerikson 7y ago
HN has an API: https://github.com/HackerNews/API https://github.com/HackerNews/API I'm using it to track common items here and on Lobste.rs (and Proggit): http://gerikson.com/hnlo/ http://gerikson.com/hnlo/ Here's the endpoint for the latest 500 submissions: https://hacker-news.firebaseio.com/v0/newstories.json https://hacker-news.firebaseio.com/v0/newstories.json here's the one for the current top stories: https://hacker-news.firebaseio.com/v0/topstories.json https://hacker-news.firebaseio.com/v0/topstories.json It's actually quite nice to work with. I don't know how to keep track of comments moving from thread to thread, because that's not a metric I'm interested in, but it should be possible to track somehow.

mft_ 7y ago

Thanks - really interesting. Take this one, for example: nvidia Drops Support for CUDA on macOS --> CUDA 10.2 is the last release to support macOS --> CUDA Toolkit Release Notes The first two titles are quite interesting to me, as a macOS user and a general follower of the tech space (and I'd note neither are sensationalised, or click-bait, from what I can see). The last...? I'm not going to click on that in a million years, as I don't work with CUDA. I'm not totally clear what the moderators' motivations always are, but might it be true that in some cases, maybe they're prioritising strict accuracy over interest, or discoverability? And as a result, their actions are actually diminishing the value of HN as a discussion forum?

kemyd 7y ago
You have the answer here: https://news.ycombinator.com/item?id=21617317 https://news.ycombinator.com/item?id=21617317 "[...] The big no-no is extracting some interesting fact from inside the article and using it as the title"
- Tenoke 7y ago
  Ironically, that's exactly what dang often does, including today.
  - dang 7y ago
    Often? That doesn't seem true to me. Can you give examples?
- mft_ 7y ago
  Thanks - makes sense from a ‘they’re following the rules perspective’, but less so from a ‘enable discovery and subsequent interesting discussion’ perspective. I wonder whether a subtitle giving an editorialised “so what” would help... but that would of course mean twice as much to moderate, and twice as much for people to complain about...
- egeozcan 7y ago
  Interesting, I had forgotten about this rule. Doesn't that encourage blogspam links instead of the source with context?

tjoff 7y ago

One thing I dislike about title edits, even when it is for the better, is that it makes it hard to know if you already have read the article. Popular topics often get multiple submissions and trigger related topics. When the title changes it is hard to know whether it is a new link with new/different information or whether it is the same one that I've already read. In my opinion once something hits the front page it is too late to edit it (other than minor things such as adding year or [pdf/video]).

bambax 7y ago
Exactly! That was my motivation too when writing a similar tool as a browser extension [0]. So for each title you can see the original text. [0] hntitles.adgent.com
joncrane 7y ago
My browser (FF) displays the link in a slightly lighter grey if I've visited it. To clean up my feed, I hide topics that I am "done with." That is, I gave it the amount of attention I'm willing to give it. This keeps my feed updating quicker as new items enter the stack at the bottom to replace the ones I've hidden. I started doing this because it mimics the behavior of one of reddit's settings, which is to hide all topics I've voted on. I found the feature so convenient that I started using the hiding behavior here in the same way.

Fnoord 7y ago

I remember there being an extension for Firefox which showed editorialized (news) site titles and content. I forgot the name though. Does anyone have any recommendations for this feature?

ocdtrekkie 7y ago

I'd love to see the domain added to the end of these, like they are on HN itself. For example, one might wonder why NVIDIA CUDA Toolkit is shorted to CUDA Toolkit... unless you see that (nvidia.com) is after it anyways. It might add some context to some of the edits.

dang 7y ago

Arguably a bug: some of the titles listed have subsequent edits that don't show up on the page. Examples: https://news.ycombinator.com/item?id=21619671 https://news.ycombinator.com/item?id=21619671 https://news.ycombinator.com/item?id=21609572 https://news.ycombinator.com/item?id=21609572 Does the script stop checking them at some point?

usr1106 7y ago
I understood the script only checks the front page. Could the change have happened when the article was not on the front page? Or are all articles that have ever been on the front page supposed to be checked "forever"? I have no idea how often / how quickly an article could oscillate between rank 30 and 31 for example.

phailhaus 7y ago

Have you considered adding a "sort by amount of change" option? It would be very interesting to see why larger changes happen compared to smaller ones.