16 ms·

Show HN: Exploring HN by mapping and analyzing 40M posts and comments for fun

• thyrox 2y ago
  Very nice. Since Hn data spawns so many such fun projects, there should be a monthly or weekly updates zip file or torrent with this data, which hackers can just download instead of writing a scraper and starting from scratch all the time.
  • average_r_user 2y ago
    that's a nice idea
  • noman-land 2y ago
    I very much support this idea. Put them on ipfs and/or torrents. Put them on HuggingFace.
    • pfarrell 2y ago
      I’ve had this same thought but was unsure what the licensing for the data would be.
  • deleted 2y ago
    [deleted]
  • pfarrell 2y ago
    I have a daily updated dataset that has the HN data split out by months. I've published it on my web page, but it’s served from my home server so I don’t want to link to it directly. Each month is about 30mb of compressed csv. I’ve wanted to torrent it, but don’t know how to get enough seeders since each month will produce a new torrent file (unless I’m mistaken). If you’re interested, send me a message. My email is mrpatfarrell. Use gmail for the domain.
  • minimaxir 2y ago
    There is a public dataset of Hacker News posts on BigQuery, but it unfortunately has only been updated up to November 2022: https://news.ycombinator.com/item?id=19304326 https://news.ycombinator.com/item?id=19304326
  • zX41ZdbW 2y ago
    It is very easy to get this dataset directly from HN API. Let me just post it here: Table definition: CREATE TABLE hackernews_history ( update_time DateTime DEFAULT now(), id UInt32, deleted UInt8, type Enum('story' = 1, 'comment' = 2, 'poll' = 3, 'pollopt' = 4, 'job' = 5), by LowCardinality(String), time DateTime, text String, dead UInt8, parent UInt32, poll UInt32, kids Array(UInt32), url String, score Int32, title String, parts Array(UInt32), descendants Int32 ) ENGINE = MergeTree(update_time) ORDER BY id; A shell script: BATCH_SIZE=1000 TWEAKS="--optimize_trivial_insert_select 0 --http_skip_not_found_url_for_globs 1 --http_make_head_request 0 --engine_url_skip_empty_files 1 --http_max_tries 10 --max_download_threads 1 --max_threads $BATCH_SIZE" rm -f maxitem.json wget --no-verbose https://hacker-news.firebaseio.com/v0/maxitem.json clickhouse-local --query " SELECT arrayStringConcat(groupArray(number), ',') FROM numbers(1, $(cat maxitem.json)) GROUP BY number DIV ${BATCH_SIZE} ORDER BY any(number) DESC" | while read ITEMS do echo $ITEMS clickhouse-client $TWEAKS --query " INSERT INTO hackernews_history SELECT * FROM url('https://hacker-news.firebaseio.com/v0/item/{$ITEMS}.json')" done It takes a few hours to download the data and fill the table.
    • zX41ZdbW 2y ago
      Also, a proof that it is updated in real-time: https://play.clickhouse.com/play?user=play#U0VMRUNUICogRlJPTSBoYWNrZXJuZXdzX2hpc3RvcnkgV0hFUkUgdGV4dCBJTElLRSAnJUNsaWNrSG91c2UlJyBPUkRFUiBCWSB0aW1lIERFU0MgTElNSVQgMTAw https://play.clickhouse.com/play?user=play#U0VMRUNUICogRlJPT...
    • strooper 2y ago
      While trying the script, I am getting the following error - <Trace> ReadWriteBufferFromHTTP: Failed to make request to 'https://hacker-news.firebaseio.com/v0/item/40298680.json https://hacker-news.firebaseio.com/v0/item/40298680.json'. Error: Timeout: connect timed out: 216.239.32.107:443. Failed at try 3/10. Will retry with current backoff wait is 200/10000 ms. I googled with no luck. I was wondering if you have a solution for it.
  • remram 2y ago
    As a starting point, that project has Apache Arrow files. I don't know if they'll update them though. https://github.com/wilsonzlin/hackerverse/releases/tag/dataset-39996091 https://github.com/wilsonzlin/hackerverse/releases/tag/datas... The comments text table is 13 GB, to give you an idea. Can definitely be processed on a laptop.
• graiz 2y ago
  Would be cool to see member similarity. Finding like-minded commentors/posters may help discover content that would be of interest.
  • vsnf 2y ago
    Reminds me of a similar project a few months ago whose purpose was to unmask alt accounts. It wasn’t well received as I recall.
  • noman-land 2y ago
    Accidental dating app.
    • internetter 2y ago
      > Accidental dating app. Possibly the greatest indicator of social startup success.
  • naveen99 2y ago
    We implemented member similarity in our hacker read app: https://apps.apple.com/in/app/hacker-read/id6479697844 https://apps.apple.com/in/app/hacker-read/id6479697844 Once you register on ios, you can also login through webapp: https://hn.garglet.com https://hn.garglet.com probably not ready for a hacker news hug of death yet, but you can try.
• ed_db 2y ago
  This is amazing, the amount of skill and knowledge involved is very impressive.
  • wilsonzlin 2y ago
    Thank you for the kind words!
• seanlinehan 2y ago
  It was not obvious at first glance to me, but the actual app is here: https://hn.wilsonl.in/ https://hn.wilsonl.in/
  • uncertainrhymes 2y ago
    I'm curious if the link to the landing page was intentionally near the end. Only the people who actually read it would go to the site. (That's not a dig, I think it's a good idea.)
  • bravura 2y ago
    1) it doesn’t appear search links are shareable or have the query terms are in it 2) are you embedding the search phrases word by word? And using the same model as the documents used? Because I searched for „lead generation“ which any decent non-unigram embedding should understand, but I got results for lead poisoning.
  • oschvr 2y ago
    I found me and my post there ! Nice
• freediver 2y ago
  If you have a blog, add an RSS feed :)
  • breck 2y ago
    I tried to fetch his RSS too! :) Turns out, there's only 1 post so far on his blog. Hoping for more! This one is great.
• CuriouslyC 2y ago
  Good example of data engineering/MLops for people who aren't familiar. I'd suggest using HDBScan to generate hierarchical clusters for the points, then use a model to generate names for interior clusters. That'll make it easy to explore topics out to the leaves, as you can just pop up refinements based on the connectivity to the current node using the summary names. The groups need more distinct coloring, which I think having clusters could help with. The individual article text size should depend on how important or relevant the article is, either in general or based on the current search. If you had more interior cluster summaries that'd also help cut down on some of the text clutter, as you could replace multiple posts with a group summary until more zoomed in.
  • wilsonzlin 2y ago
    Thanks for the great pointers! I didn't get the time to look into hierarchical clustering unfortunately but it's on my TODO list. Your comment about making the map clearer is great and something I think there's a lot of low-hanging approaches for improving. Another thing for the TODO list :)
  • zetazzed 2y ago
    For folks with GPUs, note that HDBscan is very optimized in cuML (https://docs.rapids.ai/api/cuml/stable/api/#clustering https://docs.rapids.ai/api/cuml/stable/api/#clustering / https://developer.nvidia.com/blog/faster-hdbscan-soft-clustering-with-rapids-cuml/ https://developer.nvidia.com/blog/faster-hdbscan-soft-cluste...).
    • jszymborski 2y ago
      Ooo thanks for this
• NeroVanbierv 2y ago
  Really love the island map! But the automatic zooming on the map doesn't seem very relevant. E.g. try typing "openai" - I can't see anything related to that query in that part of the map
  • NeroVanbierv 2y ago
    Ok I just noticed there is a region "OpenAI" in the north-west, but for some reason it zooms in somewhere close to "Apple" (middle of the island) when I type the query
  • oersted 2y ago
    Indeed I've long been intreagued by the idea of rendering such clustering maps more like geographic maps for better readability. It would be cool to have analogous continents, countries, sub-regions, roads, different-sized settlements, and significant landmarks... This version looks great at the highest zoom level, but rapidly becomes hard to interpret as you zoom in, same as most similar large embedding or graph visualizations.
  • wilsonzlin 2y ago
    Thanks! Yeah sometimes there are one or two "far" away results which make the auto zoom seem strange. It's something I'd like to tune, perhaps zooming to where most but not all results are.
    • luke-stanley 2y ago
      Often embeddings are not so good for comparing similarity of text. A cross-encoder might be a good alternative, perhaps as a second-pass, since you already have the embeddings. https://www.sbert.net/docs/pretrained_cross-encoders.html https://www.sbert.net/docs/pretrained_cross-encoders.html Pairwise, this can be quite slow, but as a second pass, it might be much higher quality. Obviously this gets into LLM's territory, but the language models for this can be small and more reliable than cosine on embeddings.
• paddycap 2y ago
  Adding a subscribe feature to get an email with the most recent posts in a topic/community would be really cool. One of my favorite parts of HN is the weekly digest I get in my inbox; it would be awesome if that were tailored to me. What you've built is really impressive. I'm excited to see where this goes!
  • wilsonzlin 2y ago
    Thanks! Yeah if there's enough interested users I'd love to turn this into a live service. Would an email subscription to a set of communities you pick be something you'd be interested in?
• oersted 2y ago
  This is a surprisingly big endeavour for what looks like an exploratory hobby project. Not to minimize the achievement, very cool, I'm just surprised by how much was invested into it. They used 150 GPUs and developed two custom systems (db-rpc and queued) for inter-server communication, and this was just to compute the embeddings, there's a lot of other work and computation surrounding it. I'm curious about the context of the project, and how someone gets this kind of funding and time for such research. PS: Having done a lot of similar work professionally (mapping academic paper and patent landscapes), I'm not sure if 150 GPUs were really needed. If you end up just projecting to 2D and clustering, I think that traditional methods like bag-of-words and/or topic modelling would be much easier and cheaper, and the difference in quality would be unnoticeable. You can also use author and comment-thread graphs for similar results.
  • alchemist1e9 2y ago
    The author is definitely very skilled. I find it interesting they submit posts on HN but haven’t commented since 2018! And then embarked on this project. As far as funding/time, one possibility is they are between endeavors/employment and it’s self funded as they have had a successful career or business financially. They were very efficient at the GPU utilization so it probably didn’t cost that much.
    • wilsonzlin 2y ago
      Thanks! Haha yeah I'm trying to get into the habit of writing about and sharing the random projects I do more often. And yeah the cost was surprisingly low (in the hundreds of dollars), so it was pretty accessible as a hobby project.
  • deleted 2y ago
    [deleted]
  • wilsonzlin 2y ago
    Hey, thanks for the kind words. I wasn't able to mention the costs in the post (might follow up in the future) but it was in the hundreds of dollars, so was reasonably accessible as a hobby project. The GPUs were surprisingly cheap, and was only scaled up mostly because I was impatient :) --- the entire cluster only ran for a few hours. Do you have any links to your work? They sound interesting and I'd like to read more about them.
• ashu1461 2y ago
  This is pretty great. Feature request : Is it possible to show in the graph how famous the topic / sub topic / article is ? So that we can do an educated exploration in the graph around what was upvoted and what was not ?
  • wilsonzlin 2y ago
    Thanks! Do you mean within the sentiment/popularity analysis graph? Or the points and topics within the map?
    • ashu1461 2y ago
      Points and topics within the map.
• oersted 2y ago
  Here's a great tool that does almost exactly the same thing for any dataset: https://github.com/enjalot/latent-scope https://github.com/enjalot/latent-scope Obviously the scale of OP's project adds a lot of interesting complexity, this tool cannot handle that, but it's great for medium-sized datasets.
• password4321 2y ago
  Related a month ago: A Peek inside HN: Analyzing ~40M stories and comments https://news.ycombinator.com/item?id=39910600 https://news.ycombinator.com/item?id=39910600
  • deleted 2y ago
    [deleted]
  • deleted 2y ago
    [deleted]
• xnx 2y ago
  As a novice, is there a benefit to using custom Node as the downloader? When I did my download of the 40 million Hacker News api items I used "curl --parallel". What I would like to figure out is the easiest way to go from the API straight into a parquet file.
  • wilsonzlin 2y ago
    I think your curl approach would work just as fine if not better. My instinct was to reach for Node.js out of familiarity, but curl is fast and, given the IDs are sequential, something like `parallel curl ::: $(seq 0 $max_id)` would be pretty simple and fast. I did end up needing more logic though so Node.js did ultimately come in handy. As for the Arrow file, I'm not sure unfortunately. I imagine there are some difficulties because the format is columnar, so it probably wants a batch of rows (when writing) instead of one item at a time.
• chossenger 2y ago
  Awesome visualisation, and great write-up. On mobile (in portrait), a lot of longer titles get culled as their origin scrolls off, with half of it still off the other side of the screen - wonder if it'd be worth keeping on rendering them until the entire text field is off screen (especially since you've already got a bounding box for them). I stumbled upon [1] using it that reflects your comments on comment sentiment. This also reminded me of [2] (for which the site itself had rotted away, incidentally) - analysing HN users' similarity by writing style. [1] https://minimaxir.com/2014/10/hn-comments-about-comments/ https://minimaxir.com/2014/10/hn-comments-about-comments/ [2] https://news.ycombinator.com/item?id=33755016 https://news.ycombinator.com/item?id=33755016
  • wilsonzlin 2y ago
    Thanks for the kind words, and raising that problem --- I've added it as an issue to fix. Thanks for sharing that article, it was an interesting read. It was cool how deep the analysis went with a few simple statistical methods.
• abe94 2y ago
  This is impressive work, especially for a one man show! One thing that stood out to me was the graph of the sentiment analysis over time, I hadn't seen something like that before and it was interesting to see it for Rust. What were the most positive topics over time? And were there topics that saw very sudden drops? I also found this sentence interesting, as it rings true to me about social media "there seems to be a lot of negative sentiment on HN in general." It would be cool to see a comparison of sentiment across social media platforms and across time!
  • wilsonzlin 2y ago
    Thanks! Yeah I'd like to dive deeper into the sentiment aspect. As you say it'd be interesting to see some overview, instead of specific queries. The negative sentiment stood out to me mostly because I was expecting a more "clear-cut" sentiment graph: largely neutral-positive, with spikes in the positive direction around positive posts and negative around negative posts. However, for almost all my queries, the sentiment was almost always negative. Even positive posts apparently attracted a lot of negativity (according to the model and my approach, both of which could be wrong). It's something I'd like to dive deeper into, perhaps in a future blog post.
    • walterbell 2y ago
      Great work! Would you consider adding support for search-via-url, e.g. https://hn.wilsonl.in/?q=sentiment+analysis https://hn.wilsonl.in/?q=sentiment+analysis. It would enable sharing and bookmarks of stable queries.
      • wilsonzlin 2y ago
        Thanks for the suggestion, I've just added the feature: https://hn.wilsonl.in/s/sentiment%20analysis https://hn.wilsonl.in/s/sentiment%20analysis
    • luke-stanley 2y ago
      I did something related for my ChillTranslator project for translating spicy HN comments to calm variations which has a GGUF model that runs easily and quickly but it's early days. I did it with a much smaller set of data, using LLM's to make calm variations and an algo to pick the closest least spicy one to make the synthetic training data then used Phi 2. I used Detoxify then OpenAI's sentiment analysis is free, I use that to verify Detoxify has correctly identified spicy comments then generate a calm pair. I do worry that HN could implode / degrade if there is not able to be a good balance for the comments and posts that people come here for. Maybe I can use your sentiment data to mine faster and generate more pairs. I've only done an initial end-to-end test so far (which works!). The model, so far is not as high quality as I'd like but I've not used Phi 3 on it yet and I've only used a very small fine-tune dataset so far. File is here though: https://huggingface.co/lukestanley/ChillTranslator https://huggingface.co/lukestanley/ChillTranslator I've had no feedback from anyone on it though I did have a 404 in my Show HN post!
• gaauch 2y ago
  A long term side project of mine is to try to build a recommendation algorithm trained on HN data. I trained a model to predict if a given post will reach the front page, get flagged etc, I collected over a 1000 RSS feeds and rank the RSS entries with my ranking models. I submit the high ranking entries on HN to test out my models and I can reach the front page consistently sometimes having multiple entries on the front page at a given time. I also experiment with user->content recommendation, for that I use comment data for modeling interactions between users and entries, which seems to work fine. Only problem I have is that I get a lot of 'out of distribution' content in my RSS feeds which causes my ranking models to get 'confused' for this I trained models to predict if a given entry belongs HN or not. On top of that I have some tagging models trained on data I scraped from lobste.rs and hand annotated. I had been working on this on and off for the last 2 years or so, this account is not my main, and just one I created for testing. AMA
  • saganus 2y ago
    did you find if submitted entries are more likely to reach the frontpage depending on the title or the content? i.e. do HN users upvote more based on the title of the article or on actually reading them?
    • gaauch 2y ago
      I tried making an LLM generate different titles for a given article and compared their ranking scores. There seems to be a lot of variation in the ranking scores based on the way the title is worded. Titles that are more likely to generate 'outrage' seems to be getting ranked higher, but at the same time that increases is_hn_flagged score which tries to predict if a entry will get flagged.
  • deleted 2y ago
    [deleted]
  • Foreignborn 2y ago
    Could you explain more about what you mean by modeling interactions between comments and entities?
• swozey 2y ago
  I'm.. shocked there's been 40 million posts. Wow. Really neat work edit: Also had no idea HN went back to 2006. https://news.ycombinator.com/item?id=1 https://news.ycombinator.com/item?id=1 edit2: PG wrote this? https://news.ycombinator.com/item?id=487171 https://news.ycombinator.com/item?id=487171
  • c17r 2y ago
    An HN "item" is not just posts but everything: posts, comments, the parts of a poll, etc. Still an impressive number
• fancy_pantser 2y ago
  HN submissions and comments are very different on weekends (and US holidays). Your data could explore and quantify this in some very interesting ways!
• callalex 2y ago
  “Cloud Computing” “us-east-1 down” This gave me a belly laugh.
  • deleted 2y ago
    [deleted]
• replete 2y ago
  I think this is easily the coolest post I've seen on HN this year
• minimaxir 2y ago
  A modern recommendation for UMAP is Parametric UMAP (https://umap-learn.readthedocs.io/en/latest/parametric_umap.html https://umap-learn.readthedocs.io/en/latest/parametric_umap....), which instead trains a small Keras MLP to perform the dimensionality reduction down to 2D by minimizing the UMAP loss. The advantage is that this model is small and can be saved and reused to predict on unknown new data (a traditionally trained UMAP model is large), and training is theoetically much faster because GPUs are GPUs. The downside is that the implementation in the Python UMAP package isn't great and creates/pushes the whole expanded node/edge dataset to the GPU, which means you can only train it on about 100k embeddings before going OOM. The UMAP -> HDBSCAN -> AI cluster labeling pipeline that's all unsupervised is so useful that I'm tempted to figure out a more scalable implementation of Parametric UMAP.
  • bravura 2y ago
    From a quick glance, it appears that it's because the implementation pushes the entire graph (all edges) to the GPU. Sampling of edges during training could alleviate this.
    • minimaxir 2y ago
      Indeed, TensorFlow likes pushing everything to the GPU by default whereas many PyTorch DL implementations encourage feeding data from the CPU to the GPU as needed with a DataLoader. There have been attempts at a PyTorch port of Parametric UMAP (https://github.com/lmcinnes/umap/issues/580 https://github.com/lmcinnes/umap/issues/580) but nothing as good.
      • bravura 2y ago
        Looks like there is a little motion on this topic: https://github.com/lmcinnes/umap/pull/1103 https://github.com/lmcinnes/umap/pull/1103
  • Der_Einzige 2y ago
    It exists in cuML with a fast GPU implementation. Not sure why cuMl is so poorly known though…
• dfworks 2y ago
  If anybody found this interesting and would like some further reading, the paper below employed a similar strategy to analyse inauthentic content/disinformation on Twitter. https://files.casmconsulting.co.uk/message-based-community-detection-on-twitter.pdf https://files.casmconsulting.co.uk/message-based-community-d... If you would like to read about my largely unsuccessful recreation of the paper, you can do so here - https://dfworks.xyz/blog/partygate/ https://dfworks.xyz/blog/partygate/
• Lerc 2y ago
  A suggestion for analysis: Compare topics/sentiment etc. by number of users and by number of posts. Are some topics dominated by a few prolific posters? Positively or negatively. Also, How does one seperate negative/positive sentiment to criticism/advocacy? How hard is it to detect positive criticism, or enthusiastic endorsement of an acknowledged bad thing?
• nojvek 2y ago
  I'm impressed with the map component in canvas. It's very smooth, dynamic zoom and google-maps like. Gonna dig more into it. Exemplary Show HN! We need more of this.
• datguyfromAT 2y ago
  What a great read! Thats for taking the time and effort to provide the inside into your process
• gsuuon 2y ago
  This is super cool! Both the writeup and the app. It'd be great if the search results linked to the HN story so we can check out the comments.
• jxy 2y ago
  > We can see that in this case, where perhaps the X axis represents "more cat" and Y axis "more dog", using the euclidean distance (i.e. physical distance length), a pitbull is somehow more similar to a Siamese cat than a "dog", whereas intuitively we'd expect the opposite. The fact that a pitbull is "very dog" somehow makes it closer to a "very cat". Instead, if we take the angle distance between lines (i.e. cosine distance, or 1 minus angle), the world makes sense again. Typically the vectors are normalized, instead of what's shown in this demonstration. When using normalized vectors, the euclidean distance measures the distance between the two end points of the respective vectors. While the cosine distance measures the length of one vector projected onto the other.
  • GeneralMayhem 2y ago
    The issue with normalization is that you lose a degree of freedom - which when you're visualizing, effectively means losing a dimension. Normalized 2d vectors are really just 1d vectors; if you want to show a 2d relationship, now you have to use 3d vectors (so that you have 2 degrees of freedom again).
• Tollen 2y ago
  [dead]
• cyclecount 2y ago
  I can’t tell from the documentation on GitHub: does the API expose the flagged/dead posts? It would be interesting to see statistics on what’s been censored lately.
• coolspot 2y ago
  Absolutely wonderful project and even more so the writeup! Feedback: on my iOS phone, once you select a dot on the map, there is no way to unselect it. Preview card of some articles takes full screen, so I can’t even click to another dot. Maybe add a “cross” icon for the preview card or make that when you tap outside of a card, it hides whole card strip?
  • wilsonzlin 2y ago
    Thank you! And thanks for raising that issue. I've pushed a fix that should hopefully mitigate this for you: it's possible to unselect, card images are hidden on mobile, and the invisible results area around a card (caused by the tallest card stretching the results area) should no longer intercept map touches. Let me know if it helps!
• deleted 2y ago
  [deleted]
• Igor_Wiwi 2y ago
  how much you paid to generate those embeddings?
• sourcepluck 2y ago
  Where is lisp?! I thought it was a verifiable (urban) legend around these parts that this forum is obssessed with lisp..?
  • pinkmuffinere 2y ago
    Maybe lisp is so niche that even a rather small interest makes HN relatively lispy?
• gitgud 2y ago
  Very cool! I was hoping to be able to navigate to the HN post from the map though? Is that possible?
• dangoodmanUT 2y ago
  excellent work
• aeonik 2y ago
  I couldn't help but notice that Hy is on the map but Clojure isn't. Am I out of touch? https://hylang.org https://hylang.org
• chatman 2y ago
  Worth trying Cagra (Raft)/CuVS and Lucene-CuVS for the vector search. (https://github.com/SearchScale/lucene-cuvs https://github.com/SearchScale/lucene-cuvs)
• ComputerGuru 2y ago
  Amazing work, I'm impressed by the scope of your project! I must say though, is it jina or bge-3/flag - the embeddings (and tokenizer?) do not do a good job on tech topics. It's fine for natural words, but searching for tech concepts like "xaml", "simd", etc cause it fall back to tokenizing the inputs and tries to grab similar sounding words. Also, just some constructive feedback, if there were some way to stop it from showing the same "hn leaderboard" of results when there are no results because a topic is too niche would be nice. I get a lot of "Stephen Hawking has died" when searching for words the embeddings aren't familiar with. Edit: I'm not so sure how well the sentiment analysis is working. I had the feeling that there was too much negative sentiment that didn't match up to reality, so I tried looking up things HN would feel overwhelmingly positive about like "Mr Rogers", I mean, who could feel negatively about him? The results show some serious negative spikes. Look up "Carter" and there's a massive negative peak associated with the passing of Rosalynn Carter. It was an HN submission talking about all the wonderful things the Carters did. Also, I think the "popularity over time" needs to be scaled by the median number of votes a story got that month/year, because the trend lines just go up and up if you plot strictly the number of posts. Look at the popularity of "diesel" and you'll see what I mean - this is a term that peaked ten years ago! Or perhaps it should be some sort of keyword incidence rate or number of items with a cosine similarity index of less than x from the query rather than post score, maybe? Edit2: The dynamic "click a post to remove and recalculate similarity threshold" is awesome.
  • tarasglek 2y ago
    How does one tell programmatically that any given embedding model doesn't recognize a term or word?
• kriro 2y ago
  Very nice project and documented really well. I learned a lot reading the post. The examples of the improved HN search are pretty awesome. Any idea why password reuse is so far away from security? That was the only oddity of the map for me.
• deleted 2y ago
  [deleted]
• rantymcrant 2y ago
  I'd like to see an analysis of the rise of self promotion on HN. I define self promotion on HN as a "Show HN: I ..." post vs "Show HN: Something ..." Examples from the top 100 right now * "Show HN: Exploring HN by mapping and analyzing 40M posts and comments for fun" * "Show HN: Browser-based knitting (pattern) software" These are not self promotional titles. The subjects are the exploration and the software respectively. * "Show HN: I built a non-linear UI for ChatGPT" * "Show HN: I created 3,800+ Open Source React Icons" These are self promotional titles. The subject of each is "I" My own simple check just via algolia search results checking for titles that start with "Show HN: I" gave these results for years starting April 1st. Graphed divided by the total number of results for that year 2023 **************************************** 2022 *********************************** 2021 *************************** 2020 ************************************** 2019 ************************* 2018 ************* 2017 ******* 2016 ********** 2015 ******** 2014 ************ 2013 ********************* 2012 ***************** 2011 ********* 2010 *** I feel like maybe I grew up in a time when generally, self promotion was considered a bad character trait. Your actions are supposed to be what promotes you, calling attention to them is not but I feel that culture is changing. I wonder if the rise in self promotion (assuming there is a rise) has to do with social media etc... I perceive a similar rise on Youtube but I have no data, just a feeling from the number of youtube recommendations for videos of "I....."
  • Thorrez 2y ago
    Your definition of self promotion is a bit different from what I usually think. I usually consider self promotion to be someone promoting something that that same person did. Both of your non-self-promotion examples would be self promotion under my definition. So what you consider to be self promotion vs non-self-promotion, I consider to be self promotion with a title that very clearly indicates that vs self promotion with a title that less clearly indicates that. However, the "Show HN" phrase is only used for self promotion I think, so even without the "I", anyone familiar with the convention will know it's self promotion.
    • rantymcrant 2y ago
      > However, the "Show HN" phrase is only used for self promotion I think, so even without the "I", anyone familiar with the convention will know it's self promotion. I think that's an extremely cynical view though a common one. I've never thought of "Show HN" as self promotion if it doesn't include "I" unless I go through to the actual product/library/post and find it full of self promotion. I agree with you that a post that doesn't include "I" can be self promotion but I don't think it always is even if the person made/worked on it. "Show HN: XYZ and LLM library in rust" to me is informational. It's point is, more often than not, to inform people of something they might get use out of. I know that's true when I've posted something like that. It's meaning is "here's a useful resource that was just created". Sure I get pleasure from knowing I helped people with something but I'm not trying to promote myself, I'm trying to promote the library/post/info. "Show HN: I made an LLM Library in rust" to me is self promotional. It might be useful to others but it's intent was clearly self promotion given the subject is "I", not the library/post/product.
• ailephant 2y ago
  [dead]
• tomthe 2y ago
  I made something very similar a few weeks ago. I also included usernames with the average of their comments: https://tomthe.github.io/hackmap/ https://tomthe.github.io/hackmap/
• celltalk 2y ago
  It would be cool to see yearly changes of UMAP, by different years or the overall evolution in pseudotime on the embedding. Such a cool side project!
• redbell 2y ago
  Truly, amazing work! Not only because of the final results, but also because of the whole process it took the author to bring this to life. If I could upvote this by giving points from my karma, I wouldn't hesitate to easily give a hundred points. Without a doubt, I would classify this on par with "40k HN comments mentioning books, extracted using deep learning" (https://news.ycombinator.com/item?id=28595967 https://news.ycombinator.com/item?id=28595967), which is the highest-voted "Show HN" project related to hacker news so far with 1359 points. I'm not in the ML/AI arena yet, so I couldn't fully understand the second half of the article except for having a general idea about Embeddings and their potential, but the first part is what interests me as a software engineer. Following are some of the challenges the author came across, was able to overcome each of them, and published the full source code. Downloading HN database > There's also a maxitem.json API, which gives the largest ID. As of this writing, the max item ID is over 40 million. Even with a very nice and low 10 ms mean response time, this would take over 4 days to crawl, so we need some parallelism. > I've exported the HN crawler [1] (in TypeScript) to its own project, if you're ever in need to fetch HN items. Fetching and parsing linked URLs' HTML for metadata and text > For text posts and comments, the answer is simple. However, for the vast majority of link posts, this would mean crawling those pages being linked to. So I wrote up a quick Rust service [2] to fetch the URLs linked to and parse the HTML for metadata (title, picture, author, etc.) and text. This was CPU-intensive so an initial Node.js-based version was 10x slower and a Rust rewrite was worthwhile. Fortunately, other than that, it was mostly smooth and painless, likely because HN links are pretty good (responsive servers, non-pathological HTML, etc.). Recovering missing/dead links > A lot of content even on Hacker News suffers from the well-known link rot: around 200K resulted in a 404, DNS lookup failure, or connection timeout, which is a sizable "hole" in the dataset that would be nice to mend. Fortunately, the Internet Archive has an API that we can use to use to programmatically fetch archived copies of these pages. So, as a final push for a more "complete" dataset, I used the Wayback API to fetch the last few thousands of articles, some dating back years, which was very annoying because IA has very, very low rate limits (around 5 per minute). Finding a cost-effective cloud provider for GPUs > Fortunately, I discovered RunPod, a provider of machines with GPUs that you can deploy your containers onto, at a cost far cheaper than major cloud providers. They also have more cost-effective GPUs like RTX 4090, while still running in datacenters with fast Internet connections. This made scaling up a price-accessible option to mitigate the inference time required. This is the type of content that makes HN stands out from the crowd. _____________________________ 1. https://github.com/wilsonzlin/crawler-toolkit-hn/ https://github.com/wilsonzlin/crawler-toolkit-hn/ 2. https://github.com/wilsonzlin/hackerverse/tree/master/crawler https://github.com/wilsonzlin/hackerverse/tree/master/crawle...
• gardenhedge 2y ago
  AI is the most popular topic (by far) that I could find. Is there anything more popular?
• pudiklubi 2y ago
  This is wild. I've been creating my own dataset of trending articles and ironically this is how I came across your post. I'm doing a similar project for my uni thesis. I set out with similar hypotheses and goals like you (on a slightly different scale though, haha) but I've been completely stuck on the interactive map part. Definitely getting a lot of pointers from how you handled this! Maybe one key difference in approach is that I've put more emphasis on trying to extract key topics as keywords. For ex: article (title): "Useful Uses of cat" keywords: ['Software design', 'Contraction', 'Code changes', 'Modularity', 'Ease of extension'] My hypothesis is this will be a faster search solution than using the embeddings, but potentially not as accurate. Not that far yet to really prove this though. Would love to hear what you think! Any other cool ideas on what could be done with the keywords? I explain my process a bit more here if interested: https://hackernews-demo.streamlit.app/#data-aggregation-methodology https://hackernews-demo.streamlit.app/#data-aggregation-meth...
• racosa 2y ago
  Very cool project. Thanks for sharing it!
• Venkatesh10 2y ago
  This is the type of content I'm here for.
• carte_blanche 2y ago
  Getting "Argo tunnel error" on the page
  • wilsonzlin 2y ago
    Thanks for the heads up, just fixed this.
• stavros 2y ago
  This search engine is amazing. I was looking for an old story about curing acid reflux by some exercise, Google/DDG/Kagi/HN's Algolia were completely useless, this found it first hit. Well done, this is the HN search engine I've always wanted. Is it possible to keep it up to date?