5 ms·

I wonder if they'll let us search flagged and dead posts and comments. I remember reading some insightful exchanges back in the day that got flagged because of

I wonder if they'll let us search flagged and dead posts and comments. I remember reading some insightful exchanges back in the day that got flagged because of being a controversial topic that other users didn't like. No way to find them now, even knowing some keywords and approximate month and year.

yorwba 2y ago
They're using the firebase API, https://github.com/devflowinc/trieve-hn-discovery/tree/main/hackernews-ingest https://github.com/devflowinc/trieve-hn-discovery/tree/main/... so no showdead, I think.
greenchair 2y ago
contributes to the echo chamber
dang 2y ago
We had to exclude [dead] and eventually even just [flagged] posts from the public API because many third-party clients and sites were displaying them as if they were regular posts. For the ever-fragile HN ecosystem, that is catastrophic. We would get angry emails saying "how can your expletive site possibly condone such expletive expletive comments as <link>" ... and then it would turn out that <link> was a post by some account that had been banned for years. It's fine if users turn 'showdead' on in their profile to read everything—just please remember that by doing that, you're subscribing to various bottoms of various barrels. But it's definitely not ok when people browse HN with some app we have nothing to do with, run into horrible things, understandably are outraged and then forever have their view of HN imprinted. IMO this issue is existential for HN. We've spent years and so much energy trying to find a balance between internet openness and human decency, a task which oscillates between barely-possible and simply-doomed, so the idea that anybody anywhere sees anything labeled "Hacker News" that pours all the toxic waste back into the commons is physically painful to me. Much as I dislike the idea of restricting anyone's curiosity about the entire corpus of what gets posted, I don't see what choice we have.
- spasibot 2y ago
  Thank you dang, I very much appreciate you taking the time to explain. I was going to suggest maybe the public API could have a "showdead" flag too but I guess that too easily enables the problem you're trying to prevent? As in an enterprising app developer could turn the "showdead" tap to "yes" with every request and then the waste gushes out once more.
- dredmorbius 2y ago
  I can appreciate that concern and see it even with flagging / dead / killed posts and submissions. I've had my own concerns about HN's moderation, both excesses and insufficiencies. When I've done occasional polls about what people's issues are about HN I'm very often pointed to comments which now show as flagged. I'd found a few which hadn't been flagged and forwarded those to dang, who (admittedly long after the fact) flagged them. As dang's said many time, moderators don't see everything, most moderation is by members, and mods step in relatively rarely. Based on Whaly's 2021 analysis and looking at dang's own comment post history (via Algolia), HN nets roughly 4 million comments/year and 400k submissions, with about 150k active members. Over his ten years as moderator dang's averaged about 20 comments per day, though there's a great deal more moderation occurring (some automated, some member-based, some manual but not noted with comments, which tend to be reserved for established accounts). My read is that HN mostly tends toward its stated goals and, frankly, good-netizen behaviour. It does have a pronounced status-quo bias, though it seems to be self-aware on this point. I've a few further concerns I'm still thinking through. The problem with an overly-open archive is that this makes possible misconstrued assertions about what HN does or doesn't tolerate. An open-access archive and third-party apps which don't reflect moderation actions, say, a third-party app which explicitly only showed flagged, killed, and/or dead posts, comments, and users, would paint a distinctly different picture of HN, and one which would greatly harm the reputation of the site. There are some ... possible ways around this. HN uses sequentially-numbered IDs for posts and comments (both are treated the same so far as I can tell). UserIDs seem to have an internal representation which is similar (I've seen, for example, names which change over time), but the internal representation doesn't seem to be publicly exposed. If you want to find my own content you'd do it with "UserID=dredmorbius" and not by some numeric identifier. But the numeric content ID means that a determined scraper could walk (sequentially or randomly) through the entire database, pull out every post and comment, and then glue those back together. That's somewhat north of 40 million items presently.... (There are benefits to using sparse, random / arbitrary UUIDs for systems.)
dredmorbius 2y ago
It's possible to favourite such items. They'll not be visible unless you're logged in, but at least you can go through your fave list and find them. Or, of course, bookmark them yourself for later reference. I've run into this issue myself, though dang's reply (still being edited as I write this) does hit on some valid points.
zX41ZdbW 2y ago
Here is the list of recent flagged/dead stories, ordered by rating, that they managed to accumulate: https://play.clickhouse.com/play?user=play#U0VMRUNUIGlkLCB0aW1lLCB0eXBlLCB0aXRsZSwgdXJsIEZST00gaGFja2VybmV3c19oaXN0b3J5IFdIRVJFIHR5cGUgPSAnc3RvcnknIEFORCB0aXRsZSAhPSAnJyBBTkQgaWQgSU4gKFNFTEVDVCBpZCBGUk9NIGhhY2tlcm5ld3NfaGlzdG9yeSBXSEVSRSB0eXBlID0gJ3N0b3J5JyBBTkQgZGVhZCBBTkQgdGltZSA+PSAnMjAyNC0wMS0wMScpIE9SREVSIEJZIHNjb3JlIERFU0MgTElNSVQgMSBCWSBpZA== https://play.clickhouse.com/play?user=play#U0VMRUNUIGlkLCB0a... Here is the list of recent flagged/dead stories: https://play.clickhouse.com/play?user=play#U0VMRUNUIGlkLCB0aW1lLCB0eXBlLCB0aXRsZSwgdXJsIEZST00gaGFja2VybmV3c19oaXN0b3J5IFdIRVJFIHR5cGUgPSAnc3RvcnknIEFORCB0aXRsZSAhPSAnJyBBTkQgaWQgSU4gKFNFTEVDVCBpZCBGUk9NIGhhY2tlcm5ld3NfaGlzdG9yeSBXSEVSRSB0eXBlID0gJ3N0b3J5JyBBTkQgZGVhZCBBTkQgdGltZSA+PSAnMjAyNC0wMS0wMScpIE9SREVSIEJZIHRpbWUgREVTQw== https://play.clickhouse.com/play?user=play#U0VMRUNUIGlkLCB0a...
- yamrzou 2y ago
  Is that playground open source?
  - zX41ZdbW 2y ago
    Yes, the backend is https://github.com/ClickHouse/ClickHouse/ https://github.com/ClickHouse/ClickHouse/ It has no front-end (everything is served by the database itself). The HTML page: https://github.com/ClickHouse/ClickHouse/blob/master/programs/server/play.html https://github.com/ClickHouse/ClickHouse/blob/master/program... (or press "view source" in the browser). The dataset: https://github.com/ClickHouse/ClickHouse/issues/29693 https://github.com/ClickHouse/ClickHouse/issues/29693

yorwba 2y ago

They're using the firebase API, https://github.com/devflowinc/trieve-hn-discovery/tree/main/hackernews-ingest https://github.com/devflowinc/trieve-hn-discovery/tree/main/... so no showdead, I think.

greenchair 2y ago

contributes to the echo chamber

dang 2y ago

We had to exclude [dead] and eventually even just [flagged] posts from the public API because many third-party clients and sites were displaying them as if they were regular posts. For the ever-fragile HN ecosystem, that is catastrophic. We would get angry emails saying "how can your expletive site possibly condone such expletive expletive comments as <link>" ... and then it would turn out that <link> was a post by some account that had been banned for years. It's fine if users turn 'showdead' on in their profile to read everything—just please remember that by doing that, you're subscribing to various bottoms of various barrels. But it's definitely not ok when people browse HN with some app we have nothing to do with, run into horrible things, understandably are outraged and then forever have their view of HN imprinted. IMO this issue is existential for HN. We've spent years and so much energy trying to find a balance between internet openness and human decency, a task which oscillates between barely-possible and simply-doomed, so the idea that anybody anywhere sees anything labeled "Hacker News" that pours all the toxic waste back into the commons is physically painful to me. Much as I dislike the idea of restricting anyone's curiosity about the entire corpus of what gets posted, I don't see what choice we have.

spasibot 2y ago
Thank you dang, I very much appreciate you taking the time to explain. I was going to suggest maybe the public API could have a "showdead" flag too but I guess that too easily enables the problem you're trying to prevent? As in an enterprising app developer could turn the "showdead" tap to "yes" with every request and then the waste gushes out once more.
dredmorbius 2y ago
I can appreciate that concern and see it even with flagging / dead / killed posts and submissions. I've had my own concerns about HN's moderation, both excesses and insufficiencies. When I've done occasional polls about what people's issues are about HN I'm very often pointed to comments which now show as flagged. I'd found a few which hadn't been flagged and forwarded those to dang, who (admittedly long after the fact) flagged them. As dang's said many time, moderators don't see everything, most moderation is by members, and mods step in relatively rarely. Based on Whaly's 2021 analysis and looking at dang's own comment post history (via Algolia), HN nets roughly 4 million comments/year and 400k submissions, with about 150k active members. Over his ten years as moderator dang's averaged about 20 comments per day, though there's a great deal more moderation occurring (some automated, some member-based, some manual but not noted with comments, which tend to be reserved for established accounts). My read is that HN mostly tends toward its stated goals and, frankly, good-netizen behaviour. It does have a pronounced status-quo bias, though it seems to be self-aware on this point. I've a few further concerns I'm still thinking through. The problem with an overly-open archive is that this makes possible misconstrued assertions about what HN does or doesn't tolerate. An open-access archive and third-party apps which don't reflect moderation actions, say, a third-party app which explicitly only showed flagged, killed, and/or dead posts, comments, and users, would paint a distinctly different picture of HN, and one which would greatly harm the reputation of the site. There are some ... possible ways around this. HN uses sequentially-numbered IDs for posts and comments (both are treated the same so far as I can tell). UserIDs seem to have an internal representation which is similar (I've seen, for example, names which change over time), but the internal representation doesn't seem to be publicly exposed. If you want to find my own content you'd do it with "UserID=dredmorbius" and not by some numeric identifier. But the numeric content ID means that a determined scraper could walk (sequentially or randomly) through the entire database, pull out every post and comment, and then glue those back together. That's somewhat north of 40 million items presently.... (There are benefits to using sparse, random / arbitrary UUIDs for systems.)

dredmorbius 2y ago

It's possible to favourite such items. They'll not be visible unless you're logged in, but at least you can go through your fave list and find them. Or, of course, bookmark them yourself for later reference. I've run into this issue myself, though dang's reply (still being edited as I write this) does hit on some valid points.

zX41ZdbW 2y ago

Here is the list of recent flagged/dead stories, ordered by rating, that they managed to accumulate: https://play.clickhouse.com/play?user=play#U0VMRUNUIGlkLCB0aW1lLCB0eXBlLCB0aXRsZSwgdXJsIEZST00gaGFja2VybmV3c19oaXN0b3J5IFdIRVJFIHR5cGUgPSAnc3RvcnknIEFORCB0aXRsZSAhPSAnJyBBTkQgaWQgSU4gKFNFTEVDVCBpZCBGUk9NIGhhY2tlcm5ld3NfaGlzdG9yeSBXSEVSRSB0eXBlID0gJ3N0b3J5JyBBTkQgZGVhZCBBTkQgdGltZSA+PSAnMjAyNC0wMS0wMScpIE9SREVSIEJZIHNjb3JlIERFU0MgTElNSVQgMSBCWSBpZA== https://play.clickhouse.com/play?user=play#U0VMRUNUIGlkLCB0a... Here is the list of recent flagged/dead stories: https://play.clickhouse.com/play?user=play#U0VMRUNUIGlkLCB0aW1lLCB0eXBlLCB0aXRsZSwgdXJsIEZST00gaGFja2VybmV3c19oaXN0b3J5IFdIRVJFIHR5cGUgPSAnc3RvcnknIEFORCB0aXRsZSAhPSAnJyBBTkQgaWQgSU4gKFNFTEVDVCBpZCBGUk9NIGhhY2tlcm5ld3NfaGlzdG9yeSBXSEVSRSB0eXBlID0gJ3N0b3J5JyBBTkQgZGVhZCBBTkQgdGltZSA+PSAnMjAyNC0wMS0wMScpIE9SREVSIEJZIHRpbWUgREVTQw== https://play.clickhouse.com/play?user=play#U0VMRUNUIGlkLCB0a...

yamrzou 2y ago
Is that playground open source?
- zX41ZdbW 2y ago
  Yes, the backend is https://github.com/ClickHouse/ClickHouse/ https://github.com/ClickHouse/ClickHouse/ It has no front-end (everything is served by the database itself). The HTML page: https://github.com/ClickHouse/ClickHouse/blob/master/programs/server/play.html https://github.com/ClickHouse/ClickHouse/blob/master/program... (or press "view source" in the browser). The dataset: https://github.com/ClickHouse/ClickHouse/issues/29693 https://github.com/ClickHouse/ClickHouse/issues/29693