20 ms·
You Wouldn't Download a Hacker News
- jakegmaths 1y agoYour query for Java will include all instances of JavaScript as well, so you're over representing Java.
- jasonthorsness 1y agoAh right… maybe even more unexpected then to see a decline
- deleted 1y ago[deleted]
- cs02rm0 1y agoI'm not so sure, while Java's never looked better to me, it does "feel" to me to be in significant decline in terms of what people are asking for on LinkedIn. I'd imagine these days typescript or node might be taking over some of what would have hit on javascript.
- cess11 1y agoRecruiting Java developers is easy mode, there are rather large consultancies and similar suppliers that will sell or rent them to you in bulk so you don't need to nag with adverts to the same extent as with pythonistas and rubyists and TypeScript. But there is likely some decline for Java. I'd bet Elixir and Erlang have been nibbling away on the JVM space for quite some time, they make it pretty comfortable to build the kind of systems you'd otherwise use a JVM-JMS-Wildfly/JBoss rig for. Oracle doesn't help, they take zero issue with being widely perceived as nasty and it takes a bit of courage and knowledge to manage to avoid getting a call from them at your inconvenience.
- patates 1y agoSpeaking as someone who ended up in the corporate Java world somewhat accidentally (wasn't deep in the ecosystem before): even the most invested Java shops seem wary of Oracle's influence now. Questioning Oracle tech, if not outright planning an exit strategy, feels like the default stance.
- cess11 1y agoMost such places probably have some trauma related to Oracle now. Someone spun up the wrong JVM by accident and within hours salespeople were on the phone with some middle manager about how they would like to pay for it, that kind of thing. Or just the issue of injecting their surveillance trojans everywhere and knowing they're there, that's pretty off-putting in itself. Which is a pity, once you learn to submit to and tolerate Maven it's generally a very productive and for the most part convenient language and 'ecosystem'. It's like Debian, even if you fuck up badly there is likely a documented way to fix it. And there are good libraries for pretty much anything one could want to do.
- karel-3d 1y agoNew Java looks actually good, but most of the Java actual ecosystem is stuck in the past.... and you will mostly work within the existing ecosystem
- smcin 1y agoa) Does your query for 'JS' return instances of 'JSON'? b) The ultimate hard search topic for is 'R' / 'R language'. Check if you think you index it corectly. Or related terms like RStudio, Posit, [R]Shiny, tidyverse, data.table, Hadleyverse...
- smarnach 1y agoSimilarly, the Rust query will include "trust", "antitrust", "frustration" and a bunch of other words
- matsemann 1y agoReminded me about Scunthorpe problem https://en.wikipedia.org/wiki/Scunthorpe_problem https://en.wikipedia.org/wiki/Scunthorpe_problem
- sph 1y agoA guerilla marketing plan for a new language is to call it a common one word syllable, so that it appears much more prominent than it really is on badly-done popularity contests. Call it "Go", for example. (Necessary disclaimer for the irony-impaired: this is a joke and an attempt at being witty.)
- InDubioProRubio 1y agoYou also wouldn't acronym hijack overload to boost mental presence in gamers LOL
- setopt 1y agoLet’s make a language called “A” in that case. (I mean C was fine, so why not one letter?)
- TZubiri 1y agoOr call it the name of a popular song to appeal to the youngins. I present to you "Gangam C"
- deleted 1y ago[deleted]
- brian-armstrong 1y agoAmusingly, the chart shows Rust's popularity starting from before its release. The rust hype crowd is so exuberant, they began before the language even existed!
- ashish01 1y agoI wrote one a while back https://github.com/ashish01/hn-data-dumps https://github.com/ashish01/hn-data-dumps and it was a lot of fun. One thing which will be cool to implement is that more recent items will update more over time making any recent downloaded items more stale than older ones.
- jasonthorsness 1y agoYeah I’m really happy HN offers an API like this instead of locking things down like a bunch of other sites… I used a function based on the age for staleness, it considers things stale after a minute or two initially and immutable after about two weeks old. // DefaultStaleIf marks stale at 60 seconds after creation, then frequently for the first few days after an item is // created, then quickly tapers after the first week to never again mark stale items more than a few weeks old. const DefaultStaleIf = "(:now-refreshed)>" + "(60.0*(log2(max(0.0,((:now-Time)/60.0))+1.0)+pow(((:now-Time)/(24.0*60.0*60.0)),3)))" https://github.com/jasonthorsness/unlurker/blob/main/hn/core/item_file_cache.go https://github.com/jasonthorsness/unlurker/blob/main/hn/core...
- flakiness 1y agoI have done something similar. I cheated to use BigQuery dataset (which somehow keeps getting updated) and export the data to parquet, download it and query it using duckdb.
- minimaxir 1y agoThat's not cheating, that's just pragmatic.
- AbstractH24 1y agoWhat a pragmatic way to rationalize most cheating
- matsemann 1y agoOne thing I'm curious about, but I guess not visible in any way, is random stats about my own user/usage of the site. What's my upvote/downvote ratio? Are there users I constantly upvote/downvote? Who is liking/hating my comments the most? And some I guessed could be scrapable: Which days/times are I the most active (like the github green grid thingy)? How's my activity changed over the years?
- minimaxir 1y agoThe only vote data that is visible via any HN API is the scores on submissions. Day/Hour activity maps for a given user are relatively trivial to do in a single query, but only public submission/comment data could be used to infer it.
- ryandrake 1y agoToo bad! I’ve always sort of wanted to be able to query things like what were my most upvoted and downvoted comments, how often are my comments flagged, and so on.
- saagarjha 1y agoI did this once by scraping the site (very slowly, to be nice). It’s not that hard since the HTML is pretty consistent.
- nottorp 1y ago> Are there users I constantly upvote/downvote? Hmm. Personally I never look at user names when I comment on something. It's too easy to go from "i agree/disagree with this piece of info" to "i like/dislike this guy"...
- matsemann 1y agoSame, which is why it would be cool to see. Perhaps there are people I both upvote and downvote?
- 1y ago
- pier25 1y agowould love to see the graph of React, Vue, Angular, and Svelte
- Too 1y agoAnd Nextjs
- andrewshadura 1y agoFunny nobody's mentioned "correct horse battery staple" in the comments yet…
- hsbauauvhabzb 1y agoIs the raw dataset available anywhere? I really don’t like the HN search function, and grepping through the data would be handy.
- Havoc 1y agoIt’s on firebase/bigquery to avoid people doing what OP did If you click the api link bottom of page it’ll explain.
- jasonthorsness 1y agoI used the API! It only takes a few hours to download your own copy with the tool I used https://github.com/jasonthorsness/unlurker https://github.com/jasonthorsness/unlurker I had to CTRL-C and resume a few times when it stalled; it might be a bug in my tool
- xnx 1y agoIs there any advantage to making all these requests instead of using Clickhouse o BigQuery?
- jasonthorsness 1y agoProbably not :P. I made the client for another project, https://hn.unlurker.com https://hn.unlurker.com, and then just jumped straight to using it to download the whole thing instead of searching for an already available full data set.
- Havoc 1y agoMy mistake - apologies. Had misunderstood what you did
- 9rx 1y ago> The Rise Of Rust Shouldn't that be The Fall Of Rust? According to this, it saw the most attention during the years before it was created!
- stefs 1y agoplease do not use stacked charts! i think it's close to impossible to not to distort the readers impression because a) it's very hard to gauge the height of a certain data point in the noise and b) they're implying a dependency where there _probably_ is none.
- dguest 1y agoHow do you feel about stacked plots on a logarithmic y axis? Some physics experiments do this all the time [1] but I find them pretty uninitiative. [1]: https://atlas.web.cern.ch/Atlas/GROUPS/PHYSICS/PUBNOTES/ATL-PHYS-PUB-2024-001/fig_03a.png https://atlas.web.cern.ch/Atlas/GROUPS/PHYSICS/PUBNOTES/ATL-...
- lblume 1y agoWhat is this even supposed to represent? The entire justification I could give for stacked bars is that you could permute the sub-bars and obtain comparable results. Do the bars still represent additive terms? Multiplicative constants? As a non-physicist I would have no idea on how to interpret this.
- dguest 1y agoIt's a histogram. Each color is a different simulated physical process: they can all happen in particle collisions, so the sum of all of them should add up to the data the experiment takes. The data isn't shown here because it hasn't been taken yet: this is an extrapolation to a future dataset. And the dotted lines are some hypothetical signal. The area occupied by each color is basically meaningless, though, because of the logarithmic y-scale. It always looks like there's way more of whatever you put on the bottom. And obviously you can grow it without bound: if you move the lower y-limit to 1e-20 you'll have the whole plot dominated by whatever is on the bottom. For the record I think it's a terrible convention, it just somehow became standard in some fields.
- seabass 1y agoMy first thought as well! The author of uPlot has a good demo illustrating their pitfalls https://leeoniya.github.io/uPlot/demos/stacked-series.html https://leeoniya.github.io/uPlot/demos/stacked-series.html
- tacker2000 1y agoYea, i also get the feeling that these rust evangelists get more annoying every day ;p
- userbinator 1y agoI had a 20 GiB JSON file of everything that has ever happened on Hacker News I'm actually surprised at that volume, given this is a text-only site. Humans have managed to post over 20 billion bytes of text to it over the 18 years that HN existed? That averages to over 2MB per day, or around 7.5KB/s.
- deleted 1y ago[deleted]
- sph 1y ago2 MB per day doesn't sound like a lot. The amount of posts probably has increased exponentially over the years, especially after the Reddit fiasco when we had our latest, and biggest neverending September. Also, I bet a decent amount of that is not from humans. /newest is full of bot spam.
- samplatt 1y agoPlus the JSON structure metadata, which for the average comment is going to add, what, 10%?
- deleted 1y ago[deleted]
- kevincox 1y agoI suspect it is closer to 100% increase for the average comment. If the average comment is a few senteces and the metadata has id, parent id, author, timestamp and a vote count that can add up pretty fast.
- FabHK 1y agoAround one book every 12 hours.
- xnx 1y ago20 GB JSON is surprising to me. I have an sqlite file of all HN data that is 20 GB, it would be much larger as JSON.
- montebicyclelo 1y agoThere's also two DBs I know of that have an updated Hacker News table for running analytics on without needing to download it first. - BigQuery, (requires Google Cloud account, querying will be free tier I'd guess) — `bigquery-public-data.hacker_news.full` - ClickHouse, no signup needed, can run queries in browser directly, [1] [1] https://play.clickhouse.com/play?user=play#U0VMRUNUICogRlJPTSBoYWNrZXJuZXdzX2hpc3RvcnkgV0hFUkUgbG93ZXIodGV4dCkgTElLRSAnJXB5dGhvbiUnIE9SREVSIEJZIHRpbWUgREVTQyBMSU1JVCAxMA== https://play.clickhouse.com/play?user=play#U0VMRUNUICogRlJPT...
- xnx 1y agoThe ClickHouse resource is amazing. It even has history! I had already done my own exercise of downloading all the JSON before discovering the Clickhouse HN DBs.
- kordlessagain 1y agoIt even finds your comment 'clickhouse': https://play.clickhouse.com/play?user=play#U0VMRUNUICogRlJPTSBoYWNrZXJuZXdzX2hpc3RvcnkgV0hFUkUgbG93ZXIodGV4dCkgTElLRSAnJWNsaWNraG91c2UlJyBPUkRFUiBCWSB0aW1lIERFU0MgTElNSVQgMTA= https://play.clickhouse.com/play?user=play#U0VMRUNUICogRlJPT...
- ZeWaka 1y agoand now yours :)
- k0ns0l 1y ago+1
- bambax 1y ago> Now that I have a local download of all Hacker News content, I can train hundreds of LLM-based bots on it and run them as contributors, slowly and inevitably replacing all human text with the output of a chinese room oscillator perpetually echoing and recycling the past. The author said this in jest, but I fear someone, someday, will try this; I hope it never happens but if it does, could we stop it?
- drcongo 1y agoThe internet is going to become like William Basinski's Disintegration Loops, regurgitating itself with worse fidelity until it's all just unintelligible noise.
- ahoka 1y agoProbably already happening.
- nashashmi 1y agoWe LLMs only output the average response of humanity because we can only give results that are confirmed by multiple sources. On the contrary, many of HN’s comments are quite unique insights that run contrary to the average popular thought. If this is ever to be emulated by an LLM, we would give only gibberish answers. If we had a filter to that gibberish to only permit answers that are reasonable and sensible, our answers would be boring and still be gibberish. In order for our answers to be precise, accurate and unique, we must use something other than LLMs.
- miki123211 1y agoHow do you know it isn't already happening? With long and substantive comments, sure, you can usually tell, though much less so now than a year or two ago. With short, 1 to 2 sentence comments though? I think LLMs are good enough to pass as humans by now.
- Joker_vD 1y agoBut what if LLMs will start leaving constructive and helpful comments? I personally would feel like xkcd [0], but others may disagree. [0] https://xkcd.com/810/ https://xkcd.com/810/
- SilverBirch 1y agoWhat is the netiquette of downloading HN? Do you ping Dang and ask him before you blow up his servers? Or do you just assume at this point that every billion dollar tech company is doing this many times over so you probably won't even be noticed?
- euroderf 1y agoNot to mention three-letter agencies, incidentally attaching real names to HN monikers ?
- krapp 1y agoHN has an API, as mentioned in the article, which isn't even rate limited. And all of the data is hosted on Firebase, which is a YC company. It's fine.
- mikeevans 1y agoFirebase is owned and operated by Google (has been for a while).
- alt227 1y agoIf something is on the public web, it is already being scraped by thousands of bots.
- dangoodmanUT 1y agothere's literally an API they promote. Did you read that part before trying to cancel them?
- TZubiri 1y agoWell, it's called Hacker News, so hacking is fair game, at least in the good sense of the word.
- internetter 1y agoThere's literally a public database https://console.cloud.google.com/marketplace/product/y-combinator/hacker-news https://console.cloud.google.com/marketplace/product/y-combi...
- mattkevan 1y agoI did something similar a while back to the @fesshole Twitter/Bluesky account. Downloaded the entire archive and fine-tuned a model on it to create more unhinged confessions. Was feeling pretty pleased with myself until I realised that all I’d done was teach an innocent machine about wanking and divorce. Felt like that bit in a sci-fi movie where the alien/super-intelligent AI speed-watches humanity’s history and decides we’re not worth saving after all.
- falcor84 1y agoWhat's wrong with wanking and divorce? These are respectively a way for people to be happier and more self-reliant, and a way for people to get out of a situation that isn't working out for them. I think both are net positives, and I'm very grateful to live in a society that normalizes them.
- dcuthbertson 1y agoThe innocent machine can't do either. It's akin to having no mouth, but it must scream (apologies to Harlan Ellison)
- falcor84 1y agoThat is a fair point, but it would then apply to everything else we teach it about, like how we perceive the color of the sky or the taste of champagne. Should we remove these from the training set too? Is it not still good to be exposed to the experiences of others, even if one cannot experience these things themself?
- dcuthbertson 1y agoThanks for saying it's a fair point, but it's more of an offhand joke about "an innocent machine". In reality, a machine, even an LLM, has no innocence. It's just a machine.
- 1y ago
- hellostgeroge 1y ago[dead]
- deleted 1y ago[deleted]
- deadbabe 1y agoIs the 20GB JSON file available?
- a3w 1y agoCool project. Cool graphs. But any GDPR requests for info and deletion in your inbox, yet?
- arduanika 1y agoCome on, you wouldn't GDPR a whimsical toy project!
- shayway 1y agoHah, I've been scraping HN over the past couple weeks to do something similar! Only submissions though, not comments. It was after I went to /newest and was faced with roughly 9/10 posts being AI-related. I was curious what the actual percentage of posts on HN were about AI, and also how it compared to other things heavily hyped in the past like Web3 and crypto.
- alt227 1y agoHere, the entire history of HN with the ability to run queries on it directly in the browser :) https://play.clickhouse.com/play?user=play#U0VMRUNUICogRlJPTSBoYWNrZXJuZXdzX2hpc3RvcnkgV0hFUkUgbG93ZXIodGV4dCkgTElLRSAnJXB5dGhvbiUnIE9SREVSIEJZIHRpbWUgREVTQyBMSU1JVCAxMA== https://play.clickhouse.com/play?user=play#U0VMRUNUICogRlJPT...
- sebastianmestre 1y agoCan you remake the stacked graphs with the variable of interest at the bottom? Its hard to see the percentage of Rust when it's all the way at the top with a lot of noise on the lower layers Edit: or make a non-stacked version?
- jasonthorsness 1y agoLots of valid criticism here of these graphs and the queries; I'll write a follow-up article.
- xnx 1y agoI have this data and a bunch of interesting analysis to share. Any suggestions on the best method to share results? I like Tableau Public, because it allows for interactivity and exploration, but it can't handle this many rows of data. Is there a good tool for making charts directly from Clickhouse data?
- texodus 1y agoNo Clickhouse connector for free accounts yet, but if you can drop a Parquet file on S3 you can try https://prospective.co https://prospective.co
- xnx 1y agoThanks! I'll check that out. Thought it was a typo of "Perspective" for a moment: https://perspective.finos.org/ https://perspective.finos.org/
- texodus 1y agoYes! This is the pro version, we also develop open source https://github.com/finos/perspective https://github.com/finos/perspective (which Prospective is substantially built on, with some customizations such as a wasm64 runtime).
- Am4TIfIsER0ppos 1y agoI hope they snatched my flagged comments. I would be pleased to have helped make the AI into an asshole. Here's hoping for another Tay AI.
- wslh 1y agoIt would be great if it is available as a torrent. There also mutable torrents [1]. Not implemented everywhere but there are available ones [2]. [1] https://www.bittorrent.org/beps/bep_0046.html https://www.bittorrent.org/beps/bep_0046.html [2] https://www.npmjs.com/package/bittorrent-dht https://www.npmjs.com/package/bittorrent-dht
- th1nhng0 1y agoCan I ask how you draw the chart in the post?
- jasonthorsness 1y agolol it was Excel (save as picture / SVG format / edit colors to support dark/light mode)
- th1nhng0 1y agowow, I never expect that xD thanks for let me know
- byearthithatius 1y agoCan you scrape all of HN by just incrementing item?id (since its sequential) and using Python web requests with IP rotation (in case there is rate limiting)? NVM this approach of going item by item would take 460 days if the average request response time is 1 second (unless heavily parallelized, for instance 500 instances _could_ do it in a day but thats 40 million requests either way so would raise alarms).
- g8oz 1y agoI predict that in the coming years a lot of APIs will begin offer the option of just returning a duckdb file. If you're just going to load the json into a database anyway, why not just get a database in the response.
- vdm 1y agozstd parquets exported from my duckdb 1.2 files compress 2-3x more
- mike503 1y agoOther people have asked, probably for the same reason but I would love an offline version, packaged in zim format or something. For when the apocalypse happens it’ll be enjoyable to read relatively high quality interactions and some of them may include useful post-apoc tidbits!
- febeling 1y agoYou wonder what all the Rust talk was about before the programming language's release in Jan 2012.
- 0x008 1y agolike other's have said it likely includes partial matches as well (e.g. 'antitrust' etc)
- deleted 1y ago[deleted]
- dredmorbius 1y agoI've been tempted to look into API-based HN access having scraped the front-page archive about two years ago. One of the advantages of comments is that there's simply so much more text to work with. For the front page, there is up to 80 characters of context (often deliberately obtuse), as well as metadata (date, story position, votes, site, submitter). I'd initially embarked on the project to find out what cities were mentioned most often on HN (in front-page titles), though it turned out to be a much more interesting project than I'd anticipated. (I've somewhat neglected it for a while though I'll occasionally spin it up to check on questions or ideas.)