7 ms·
28M Hacker News comments as vector embedding search dataset
- Gerardo1 11mo agoCan I submit a request somewhere to have my data removed?
- amarant 11mo agoDepends. Are you a European citizen?
- j4coh 11mo agoOh to have had a delete account/comments option.
- verdverm 11mo agothere are many replicas of the HN dataset out there, one should consider posts here as public content
- SilverElfin 11mo agoEven so, deletion would be nice. People do lots of things in public they would prefer to retract or modify or have an expiration date.
- sunaookami 11mo agoThe phrase "the internet does not forget" is popular for a reason.
- delichon 11mo agoThe words we type on this site diffuse rapidly onto innumerable independent devices where they are experimentation grist for herds of wild nerds around the globe. Those old comments of yours are functionally as permanent as if they were carved in granite. Mine of course will be treasured some day as ancient wisdom.
- bcjdjsndon 11mo ago> Those old comments of yours are functionally as permanent as if they were carved in granite. I've definitely heard that one before... Explain link rot to me then, or why the internet archive even exists?
- stephen_cagle 11mo agoI'd say link rot is more a reflection of the fragility of the system (the original source has been lost), however, the original source has probably been copied to innumerable other places. tldr: both of these things can be true.
- delichon 11mo agoFor one thing, this is part of the data set encoded in AI models, and those are rapidly heading toward being embedded in local devices. By the millions then billions. Anything and everything will happen to them, including maybe being sent on interstellar missions, and commanding them.
- lazide 11mo agoGranite decomposes, just not quickly or necessarily predictably.
- pessimizer 11mo ago> why the internet archive even exists As an archive that supplements my personal archive, and the archives of many others. Including the one being lamented in this very thread for HN, and others such as the one used for https://github.com/afiodorov/hn-search https://github.com/afiodorov/hn-search The way to eliminate your comments would be to take over world government, use your copy of the archives of the entire internet in order to track down the people who most likely have created their own copies, and to utilize worldwide swat teams with trained searchers, forensics experts and memory-sniffing dogs. When in doubt, just fire missiles at the entire area. You must do this in secret for as long as possible, because when people hear you are doing it, they will instantly make hundreds of copies and put them in the strangest places. You will have to shut down the internet. When you are sure you have everything, delete your copy. You still may have missed one.
- afiodorov 11mo agoI've been embedding all HN comments since 2023 from BigQuery and hosting at https://hn.fiodorov.es https://hn.fiodorov.es Source is at https://github.com/afiodorov/hn-search https://github.com/afiodorov/hn-search
- kylecazar 11mo agoI appreciate the architectural info and details in the GH repo. Cool project.
- Gerardo1 11mo agoCan users here submit an issue to have data associated with their account removed?
- vilocrptr 11mo agoGDPR still holds, so I don’t see why not if that’s what your request is under. However, it’s out there- and you have no idea where, so there’s not really a moral or feasible way to get rid of it everywhere. (Please don’t nuke the world just to clean your rep.)
- dangus 11mo agoThe law (at least, in the EU) grants a legal right to privacy, and the motivation behind it is really none of anyone’s business. Maybe commenters face threats to safety. Maybe commenters didn’t think AI companies profiting off of their non-commercial conversations would ever exist and wouldn’t have put data out there if that was disclosed ahead of time. Corporations have an unlimited right to bully and threaten to take down embarrassing content and hide their mistakes, they have greatly enhanced leverage over copyright enforcement compared to individuals, but then if individuals do a much less egregious thing to try and take down their content they don’t even get paid for it’s immoral. This community financially benefits YCombinator and its portfolio companies. Without our contributions, readership, and comments, their ability to hire and recruit founders is diminished. They don’t provide a delete button for profit-motivated reasons, and privacy laws like GDPR guard against that. (As you might guess, I am personally quite against HN’s policy forbidding most forms of content deletion. Their policy and solution involving manual modifications via the moderation team makes no sense - every other social media platform lets you delete your content)
- catapart 11mo agoAm I misunderstanding what a parquet file is, or are all of the HN posts along with the embedding metadata a total of 55GB?
- verdverm 11mo agobased on the table they show, that would be my inclination wanted to do this for my own upvotes so I can see the kind of things I like, or find them again easier or when relevant
- lazide 11mo agoCompressed, pretty believable.
- gkbrk 11mo agoI imagine that's mostly embeddings actually. My database has all the posts and comments from Hacker News, and the table takes up 17.68 GB uncompressed and 5.67 GB compressed.
- atonse 11mo agoThat’s crazy small. So is it fair to say that words are actually the best compression algorithm we have? You can explain complex ideas in just a few hundred words. Yes, a picture is worth a thousand words, but imagine how much information is in those 17GB of text.
- _zoltan_ 11mo agohow much?
- binary132 11mo agoI don’t think I would really consider it compression if it’s not very reversible. Whatever people “uncompress” from my words isn’t necessarily what I was imagining or thinking about when I encoded them. I guess it’s more like a symbolic shorthand for meaning which relies on the second party to build their own internal model out of their own (shared public interface, but internal implementation is relatively unique…) symbols.
- SchwKatze 11mo agoI know it's unrelated but does anyone knows a good paper comparing vector searches vs "normal" full text search? Sometimes I ask myself of the squeeze worth the juice
- verdverm 11mo agoNot aware of a specific paper. This account on Bluesky focuses on RAG and general information retrieval https://bsky.app/profile/reachsumit.com https://bsky.app/profile/reachsumit.com
- stephantul 11mo ago“Normal search” is generally called bm25 in retrieval papers. Many, if not all, retrieval papers about modeling will use or list bm25 as a baseline. Hope this helps!
- arboles 11mo agoCompared in what? Server load, user experience?
- ProofHouse 11mo agoScratches off one of my todos,
- delichon 11mo agoI think it would be useful to add a right-click menu option to HN content, like "similar sentences", which displays a list of links to them. I wonder if it would tell me that this suggestion has been made before.
- JacobThreeThree 11mo agoYou'd get sentences full of words like: tangential, orthogonal, externalities, anecdote, anecdata, cargo cult, enshittification, grok, Hanlon's razor, Occam's razor, any other razor, Godwin's law, Murphy's law, other laws.
- pessimizer 11mo agoClicking "Betteridge's" would bring down the site.
- iwontberude 11mo agoSomeone made a tool a few years ago that basically unmasked all HN secondary accounts with a high degree of certainty. It scared the shit out of me how easy it picked out my alts based on writing style.
- CraigJPerry 11mo agoI think that original post was taken down after a short while but antirez was similarly nerd sniped by it and posted this which i keep a link to for posterity: https://antirez.com/news/150 https://antirez.com/news/150
- dylan604 11mo ago"Well, the first problem I had, in order to do something like that, was to find an archive with Hacker News comments. Luckily there was one with apparently everything posted on HN from the start to 2023, for a huge 10GB of total data. You can find it here: https://huggingface.co/datasets/OpenPipe/hacker-news https://huggingface.co/datasets/OpenPipe/hacker-news and, honestly, I’m not really sure how this was obtained, if using scarping or if HN makes this data public in some way." This is funny to me in a number ways. I doubt anyone would be interested in post-2023 data dumps for fear it would be too contaminated with content produced from LLMs. It's also funny that the archive was hosted by huggingface which just removes any sliver of doubt they scarped (sic) the site.
- John-Tony 11mo ago[dead]
- baalimago 11mo agoFinetune LLM to post_score -> high quality slop generator
- GeoAtreides 11mo agoI don't remember licensing my HN comments for 3rd party processing.
- verdverm 11mo agohttps://www.ycombinator.com/legal/ https://www.ycombinator.com/legal/
- GeoAtreides 11mo agocorrect, my comments are licensed to HN and HN affiliated companies: >With respect to the content or other materials you upload through the Site or share with other users or recipients (collectively, “User Content”), you represent and warrant that you own all right, title and interest in and to such User Content, including, without limitation, all copyrights and rights of publicity contained therein. >By uploading any User Content you hereby grant and will grant Y Combinator and its affiliated companies a nonexclusive, worldwide, royalty free, fully paid up, transferable, sublicensable, perpetual, irrevocable license to copy, display, upload, perform, distribute, store, modify and otherwise use your User Content for any Y Combinator-related purpose
- cyberpunk 11mo agoAnd whoever created this database of our comments is affiliated with YCOM how?
- GeoAtreides 11mo agothat's exactly what I'm saying :)
- verdverm 11mo agoLooks like the relationship is not new https://clickhouse.com/deals/ycombinator https://clickhouse.com/deals/ycombinator
- 11mo ago
- isodev 11mo agoMaybe I’m reading this wrong, but commercial use of comments is prohibited by the HN Privacy and data Policy. So is creating derivative works (so technically a vector representation)
- hammock 11mo agoSomeone better go tell Open AI
- isodev 11mo agoI think a number of lawsuits are in progress of teaching them that particular lesson.
- lazide 11mo agoStill waiting for anything resembling a penalty, been a long time now. 5 years?
- verdverm 11mo agoMost of the time they are hardly penalties and look more like rounding errors to these companies
- sfn42 11mo agoI'm just wondering what gives HN, Reddit etc the right to our comments? If anyone owns this comment it's me IMO. So I don't see any reason why HN should be able to sue anyone for using this freely available information.
- handfuloflight 11mo agoWith Reddit, at least it's the legal agreement you enter into them by creating an account and using it.
- 11mo ago
- John-Tony12 11mo ago[dead]
- zkmon 11mo agoI don't know how to feel about this. Is the only purpose of the comments here is to train some commercial model? I have a feeling that, this might affect my involvement here going forward.
- wiseowise 11mo agoOkay, okay, party poopers.
- creata 11mo agoLLMs have drastically reduced my desire to post anything helpful on the internet. It used to be about helping strangers in some small way. Now it's helping people I don't like more than people I do like.
- ThrowawayR2 11mo agoNot me. The thought of my eccentric comments leaving some unnoticed mar in the latent space of tomorrow's ever mightier LLMs, a tiny stain that reverberates endlessly into the future, manifesting at unexpected moments, amuses me to no end.
- minimaxir 11mo agoDon't use all-MiniLM-L6-v2 for new vector embeddings datasets. Yes, it's the open-weights embedding model used in all the tutorials and it was the most pragmatic model to use in sentence-transformers when vector stores were in their infancy, but it's old and does not implement the newest advances in architectures and data training pipelines, and it has a low context length of 512 when embedding models can do 2k+ with even more efficient tokenizers. For open-weights, I would recommend EmbeddingGemma (https://huggingface.co/google/embeddinggemma-300m https://huggingface.co/google/embeddinggemma-300m) instead which has incredible benchmarks and a 2k context window: although it's larger/slower to encode, the payoff is worth it. For a compromise, bge-base-en-v1.5 (https://huggingface.co/BAAI/bge-base-en-v1.5 https://huggingface.co/BAAI/bge-base-en-v1.5) or nomic-embed-text-v1.5 (https://huggingface.co/nomic-ai/nomic-embed-text-v1.5 https://huggingface.co/nomic-ai/nomic-embed-text-v1.5) are also good.
- xfalcox 11mo agoI am partial to https://huggingface.co/Qwen/Qwen3-Embedding-0.6B https://huggingface.co/Qwen/Qwen3-Embedding-0.6B nowadays. Open weights, multilingual, 32k context.
- SteveJS 11mo agoAlso matryoshka and the ability to guide matches by using prefix instructions on the query. I have ~50 million sentences from english project gutenberg novels embedded with this.
- Joshua-Peter 11mo ago[dead]
- dangoodmanUT 11mo agoWhy all-MiniLM-L6-v2? This is so old and terribly behind the new models...
- SilverElfin 11mo agoIs there a dataset for the discussion links and the linked articles (archived without paywall)?
- slurrpurr 11mo agoThe most smug AI ever will be trained on this
- doctorslimm 11mo agolmao this is gold
- doctorslimm 11mo agowhy is this not on huggingface as a dataset yet? is anyone poutine this on hugginggface?
- dylan604 11mo agoMaybe you skimmed past this from TFA: "Well, the first problem I had, in order to do something like that, was to find an archive with Hacker News comments. Luckily there was one with apparently everything posted on HN from the start to 2023, for a huge 10GB of total data. You can find it here: https://huggingface.co/datasets/OpenPipe/hacker-news https://huggingface.co/datasets/OpenPipe/hacker-news and, honestly, I’m not really sure how this was obtained, if using scarping or if HN makes this data public in some way."
- notsahil 11mo agohttps://huggingface.co/datasets/labofsahil/hackernews-vector-search-dataset https://huggingface.co/datasets/labofsahil/hackernews-vector...
- dmezzetti 11mo agoFun project. I'm sure it will get a lot of interest here. For those into vector storage in general, one thing that has interested me lately is the idea of storing vectors as GGUF files and bring the familiar llama.cpp style quants to it (i.e. Q4_K, MXFP4 etc). An example of this is below. https://gist.github.com/davidmezzetti/ca31dff155d2450ea1b51634a814d37f https://gist.github.com/davidmezzetti/ca31dff155d2450ea1b516...
- rashkov 11mo agoIs there an affordable service for doing something like this?
- Kuraj 11mo agoI can't help but feel a bit violated by this.
- pizzafeelsright 11mo agoThe content you published was consumed yet you fell violated?
- Kuraj 11mo agoI dunno man. When I first joined it was unconcieveable that someone could just take everything and build a trivially queryable _conversational_ (that's a big part of it) model around everything I've posted _just like that_. Call me naiive but I would consider it some sort of a social contract that you would not do that. I feel the same way about LLMs being trained on Reddit. I suspect with a large enough dataset these models can infer things about you that you wouldn't know about yourself. To make another example, even though my reddit history is public (or was until recently because I didn't have a choice) I would still feel uneasy if I realized someone deliberately snooped through all of it. And I would be SUUUUPER uncomfortable if someone did that with my Discord history. It's not against the rules or anything, I just think it's rude.
- fragmede 11mo agohttps://news.ycombinator.com/threads?id=Kuraj https://news.ycombinator.com/threads?id=Kuraj It's two clicks to get to that page from this page. Say the wrong thing here and some troll will go through it and find something you said years ago that contradicts something you're saying today. If the mere thought of that bothers you, I don't know what to tell you other than to warn you of the possibility.
- Kuraj 11mo agoI don't know how to get my point across, I guess I'm just thinking emotionally more than logically right now lol. Either way it's not my comments being visible verbatim that irks me but rather the processing part. But I get your point and the "damage" is already done, so /shrug
- ForOldHack 11mo agoSo... asking for a friend, if I comment here, it becomes a vector?
- edwardzcn 11mo ago[dead]