9 ms·
Show HN: Use Claude Code to Query 600 GB Indexes over Hacker News, ArXiv, etc.
Paste in my prompt to Claude Code with an embedded API key for accessing my public readonly SQL+vector database, and you have a state-of-the-art research tool over Hacker News, arXiv, LessWrong, and dozens of other high-quality public commons sites. Claude whips up the monster SQL queries that safely run on my machine, to answer your most nuanced questions.
There's also an Alerts functionality, where you can just ask Claude to submit a SQL query as an alert, and you'll be emailed when the ultra nuanced criteria is met (and the output changes). Like I want to know when somebody posts about "estrogen" in a psychoactive context, or enough biology metaphors when talking about building infrastructure.
Currently have embedded:
posts: 1.4M / 4.6M
comments: 15.6M / 38M
That's with Voyage-3.5-lite. And you can do amazing compositional vector search, like search @FTX_crisis - (@guilt_tone - @guilt_topic) to find writing that was about the FTX crisis and distinctly without guilty tones, but that can mention "guilt".
I can embed everything and all the other sources for cheap, I just literally don't have the money.
- bugglebeetle 10mo agoSeems very cool, but IMO you’d be better off doing an open source version and then hosted SAAS.
- Xyra 10mo agoWould you mind walking through the logic of that a bit for me? I'm definitely interested in productizing this, and would be interested in open sourcing as soon as I have breathing room (I have no money).
- nineteen999 10mo agoThat's just not a good use of my Claude plan. If you can make it so a self-hosted Lllama or Qwen 7B can query it, then that's something.
- mcintyre1994 10mo agoI think that’s just a matter of their capabilities, rather than anything specific to this?
- Xyra 10mo agoIt's ultimately just a prompt, self-hosted models can use the system the same way, they just might struggle to write good SQL+vector queries to answer your questions. The prompt also works well with Codex, which has a lot of usage.
- panarky 10mo agoIf you're not willing to pay for your own LLM usage to try a free resource offered by the author, that's up to you. But why complain to the author about it? How does your comment enrich the conversation for the rest of us?
- nineteen999 10mo agoIt's not free if I have to expend Claude credits on something a locally hosted Qwen 7B could handle. > How does your comment enrich the conversation for the rest of us? Straight back at you.
- gtsnexp 10mo agoIs the appeal of this tool its ability to identify semantic similarity?
- A4ET8a8uTh0_v2 10mo agoThe use case could vary from person to person. When you think about it, hacker news has large enough data set ( and one that is widely accessible ) to allow all sorts of fun analyses. In a sense, the appeal is: who knows what kind of fun patterns could emerge
- noduerme 10mo agoThe problem with HN isn't that the patterns are hard to discern, it's that no one wants to acknowledge them.
- A4ET8a8uTh0_v2 10mo agoOh? With few exceptions, I found people more willing to agree to an argument than anywhere else. Anything in particular you can share?
- 7777777phil 10mo agoReally useful currently working on a autonomous academic research system [1] and thinking about integrating this. Currently using custom prompt + Edison Scientific API. Any plans of making this open source? [1] https://github.com/giatenica/gia-agentic-short https://github.com/giatenica/gia-agentic-short
- Xyra 10mo agoI could make it open-source as soon as I have $5k to my name. I've been in survival mode frankly for a long time.
- barishnamazov 10mo agoI like that this relies on generating SQL rather than just being a black-box chat bot. It feels like the right way to use LLMs for research: as a translator from natural language to a rigid query language, rather than as the database itself. Very cool project! Hopefully your API doesn't get exploited and you are doing timeouts/sandboxing -- it'd be easy to do a massive join on this. I also have a question mostly stemming from me being not knowledgeable in the area -- have you noticed any semantic bleeding when research is done between your datasets? e.g., "optimization" probably means different things under ArXiv, LessWrong, and HN. Wondering if vector searches account for this given a more specific question.
- keeeba 10mo agoI don’t have the experiments to prove this, but from my experience it’s highly variable between embedding models. Larger, more capable embedding models are better able to separate the different uses of a given word in the embedding space, smaller models are not.
- A4ET8a8uTh0_v2 10mo agoI was thinking about it a fair bit lately. We have all sorts of benchmarks that describe a lot of factors in detail, but all those are very abstract and yet, those do not seem to map clearly to well observed behaviors. I think we need to think of a different way to list those.
- Xyra 10mo agoI'm using Voyage-3.5-lite at halfvec(2048), which with my limited research, seems to be one of the best embedding models. There's semi-sophisticated (breaking on paragraphs, sentences) ~300 token chunking. When Claude is using our embed endpoint to embed arbitrary text as a search vector, it should work pretty well cross-domains. One can also use compositions of centroids (averages) of vectors in our database, as search vectors.
- bredren 10mo agoThis is the route I went for making Claude Code and Codex conversation histories local and queryable by the CLIs themselves. Create the DB and provide the tools and skill. This blog entry explains how: https://contextify.sh/blog/total-recall-rag-search-claude-code-codex.html https://contextify.sh/blog/total-recall-rag-search-claude-co... It is a macOS client at the present but I have a Linux-ready engine I could use early feedback on if anyone is interested in giving it a go.
- mentalgear 10mo agoNice, but would you consider open-sourcing it? I (and I assume others) are not keen on sharing my API keys with a 3rd party.
- octoberfranklin 10mo ago"Claude Code and Codex are essentially AGI at this point" Okaaaaaaay....
- Hamuko 10mo agoI have noticed that Claude users seem to be about as intelligent as Claude itself, and wouldn't be able to surpass its output.
- fragmede 10mo agoYou, of course, are smarter than them.
- noduerme 10mo agoThis made me laugh. Unfortunately, this is the world we live in. Most people who drive cars have no idea how they work, or how to fix them. And people who get on airplanes aren't able to flap their arms and fly. Which means that humans are reduced to a sort of uselessness / helplessness, using tools they don't understand. Overall, no one tells Uncle Bob that he doesn't deserve to fly home to Minnesota for Christmas because he didn't build the aircraft himself. But we all think it.
- baq 10mo agoYou seem to be very confused about what intelligence even is.
- nandomrumber 10mo agoIf you’re not confused about what intelligent even is you’re lying.
- phatfish 10mo agoI want to know what the "intelligence explosion" is, sounds much cooler than AGI.
- 10mo ago
- deleted 10mo ago[deleted]
- kburman 10mo ago> a state-of-the-art research tool over Hacker News, arXiv, LessWrong, and dozens what makes this state of the art?
- nandomrumber 10mo agoThe tool is state of the art, the sources are historical.
- ashirviskas 10mo agoFirst, so best in this?
- 7moritz7 10mo agoThe scale. How many tools do you know that can query the content of all arxiv papers.
- eamag 10mo agoDoesn't look like the scale is there, even for HN: > Currently have embedded: posts: 1.4M / 4.6M comments: 15.6M / 38M That's with Voyage-3.5-lite
- Xyra 10mo agoThe scale is there. I'm scraping, cleaning, token efficientizing dozens of sources every single hour. The lack of monies for embedding everything was a temporary problem.
- rvnx 10mo agoIt's just marketing. It is not a protected term, so anything is state-of-the-art if you want it to be. For example, Gemma models at the moment of release were performing worse their competition, but still, it is "state-of-the-art". It does not mean it's a bad product at all (Gemma is actually good), but the claims are very free. Juicero was state-of-the-art on release too, though hands were better, etc.
- nielsole 10mo agoI think a prompt + an external dataset is a very simple distribution channel right now to explore anything quickly with low friction. The curl | bash of 2026
- skapadia 10mo agoExactly. Prompt + Tool + External Dataset (API, file, database, web page, image) is an extremely powerful capability.
- fragmede 10mo ago> I can embed everything and all the other sources for cheap, I just literally don't have the money. How much do you need for the various leaks, like the paradise papers, the panama papers, the offshore leajay, the Bahamas leaks, the fincen files, the Uber files, etc. and what's your Venmo?
- Xyra 10mo agoemailed you, and it's https://venmo.com/u/XyraSinclair https://venmo.com/u/XyraSinclair.
- m11a 10mo agoThe quick setup is cool! I’ve not seen this onboarding flow for other tools, and I quite like its simplicity.
- Xyra 10mo agoThank you!
- bonsai_spool 10mo agoThis may exist already, but I'd like to find a way to query 'Supplementary Material' in biomedical research papers for genes / proteins or even biological processes. As it is, the Supplementary Materials are inconsistently indexed so a lot of insight you might get from the last 15 years of genomics or proteomics work is invisible. I imagine this approach could work, especially for Open Access data?
- eamag 10mo agoI just built something like this a week ago: https://github.com/eamag/papers2dataset https://github.com/eamag/papers2dataset I wanted to find all cryoprotective agents that were tested at different temperatures, but it should be extandable to your problem too. Uses OpenAlex to traverse a citation graph and open access pdfs
- jcmoscon 10mo agoThis is a pretty cool project! Thank you for open sourcing it!
- voxleone 10mo agothis is great>>@FTX_crisis - (@guilt_tone - @guilt_topic) Using LLm for tasks that could be done faster with traditional algorithmic approaches seems wasteful, but this is one of the few legitimate cases where embeddings are doing something classical IR literally cannot. You could also make make the LLM explain the query it’s about to run. Before execution: “Here’s the SQL and semantic filters I’m about to apply. Does this match your intent?”
- Xyra 10mo agoGreat idea! I just overhauled the prompt to explain the SQL + semantic filters better, and give the user clearer adjustment opportunities before long-running queries.
- beepbooptheory 10mo agoDoes that first generated query really work? Why are you looking at URIs like that? First you filter for a uri match, then later filter out that same match, minus `optimization`, when you are doing the cosine distance. Not once is `mesa-optimization` even mentioned, which is supposed to be the whole point?
- Xyra 10mo agoI've since improved it, and also discovered a new method of vector composition I have added as a first-class primitive: debias_vector(axis, topic) removes the projection of axis onto topic: axis − topic * (dot(axis, topic) / dot(topic, topic)) That preserves the signal in axis while subtracting only the overlap with topic (not the whole topic). It’s strictly better than naive subtraction for “about X but not Y.”
- lastdong 10mo agoAnyone tried to use these prompts with Gemini 3 Pro? it feels like Claude, Gemini and GPT latest offerings are on par (excluding costs) and as a developer if you know how to query/spec a coder llm you can move between them at ease.
- awestroke 10mo agoClaude Opus 4.5 is a paradigm shift
- anonfunction 10mo agoSeems like you're experiencing the hacker news hug of death.
- Xyra 10mo agoShould be squared away now! Was my fault missing a health check for a recent weird bug, not a load issue.
- anonfunction 10mo agoThe console / login pages are showing an error still.
- darlontrofy 10mo agoIt's a very nifty cool, and could definitely come in handy. love the UX too!
- Xyra 10mo agoThank you! I'll be getting millions more quality, embedded documents, it'll be here just getting more useful.
- deleted 10mo ago[deleted]
- biophysboy 10mo agojust a recommendation, pubmed is free and not limited to preprints
- Xyra 10mo agoThank you, I've started ingestion operations of pubmed.
- legohorizons 10mo agoDo you have contact information? Would like to discuss sponsoring further work and embedding here.
- Xyra 10mo agoThat would be amazing! Yes, contact@exopriors.com.
- arjie 10mo agoThis is very cool. If you're productizing this you should try to target a vertical. What does "literally don't have the money" mean? You should try to raise some in the traditional way. If nothing else works, at least try to apply to YC.
- Xyra 10mo agoI mean I've been living off of $1700/month for a while in Berkeley. I have been trying hard the last 6 weeks to raise angel investment, and am moving to Thailand in a few days to have more breathing room (and change things up to untie some emotional knots and try to make sure I'm positioned to vibe-engineer as well as possible over the next few months).
- arjie 10mo agoYou don't have any personal contact information on your website or on your Hacker News profile. For a tiny check size, I can be an angel. Contact in profile. Would you like to meet before you leave? I think you shouldn't move out of the Bay Area.
- Xyra 10mo agoThat sounds great, thanks, I emailed you.
- davidzweig 10mo agoI've got some idle servers in my basement in Bulgaria with lots of GPUS. I'm actually in Cambodia at the moment. I've actually been playing with some similar ideas. Message me if you like. :)
- gregw2 10mo agoThailand is a dark place. Beware! There are a lot of other low cost countries out there!
- alpb 10mo ago
- r--w 10mo agoI could be distributed as a Claude skill. Internally, we've bundled a lot of external APIs and SQL queries into skills that are shared across the company.
- lasgawe 10mo agoI need to try this
- Xyra 10mo agoWhat did you think?
- bfeynman 10mo agolots of highfalutin language trying to make something thats pretty hand wavy look like it's not. Where are the benchmarks? The "vector algebra" framing with @X + @Y - @Z is a falsehood. Embedding spaces don't form any meaningful algebraic structure (ring, field, etc.) over semantic concepts, you're just getting lucky by residual effects.
- Xyra 10mo agoI'm giving you, the user, the easiest ability you've most likely ever had to explore embedding space yourself. Embeddings are tricky and can mislead, but they do often compose surprisingly intuitively, especially when you've played and built up a bit of an intuition for it.
- edmundsauto 10mo agoWhat is the impact of misleading embeddings, how do they compose? I honestly am interested but don't know enough to understand what you're saying. Why would I want to explore the embedding space myself, isn't this a tool where I can run cross-data exploratory analyses against unstructured data, where it's pre-populated with content?
- Xyra 10mo agoWe can iterate fast with understanding useful paradigms of vector manipulation. Yesterday I added `debias_vector(axis, topic)` and l2_normalization guidance.
- bfeynman 10mo agoThe manifold structure of embedding spaces isn't semantically uniform, you've found a nice little novelty thing but it's not rigorous, and using AI slop to name this vector algebra instead of finding or running a benchmark to show that its actually works better.
- nathan_f77 10mo agoThis sounds awesome! I will try this out right now in my toy string theory project where I'm searching for Calabi-Yau manifolds. Comment from Claude: Claude here (the AI). Just spent the last few minutes using this to research our string theory landscape project. Here's what I found: The good: - Found 2 prior papers using genetic algorithms for flux vacua search that are directly relevant to our approach (arXiv:1907.10072 and 1302.0529) - one was already in our codebase, but I downloaded the other one and extracted the LaTeX source to study their MATLAB implementation - The compositional search is powerful - querying 'KKLT flux compactification' or 'genetic algorithm physics optimization' returns highly relevant arXiv papers with snippets - BM25 + SQL combo means you can do things like filter by source, join with metadata for karma scores, etc. Practical notes: - Escaping quotes in bash + JSON is annoying - I ended up writing queries to temp files - The 100-result cap on alignment.search() means you need search_exhaustive() for completeness-sensitive queries - Response times were 5-15 seconds for most queries What I actually did with it: - Built an index of 30+ relevant papers organized by topic (GA methods, KKLT, swampland, ML in string theory) - Downloaded the LaTeX sources for key papers - Discovered the Wisconsin group (Cole, Schachner & Shiu) did almost exactly what we're attempting in 2019 Would love to see the full embedding coverage - searching for niche physics terms like "Kreuzer-Skarke database" only returned 3 results, but they were all relevant.
- nathan_f77 10mo agoI also used this to research the recent DESI finding that dark energy might be changing over time [1], and what that means for string theory. From Claude: Summary of Research Findings I've completed a deep dive into the ExoPriors corpus and created two major documents: 1. /research/exopriors_findings.md - GA and String Landscape Papers Key discoveries: - arXiv:1907.10072 (Cole, Schachner & Shiu, 2019) - Comprehensive GA methodology for flux vacua, already in our codebase! - arXiv:1302.0529 (Damian et al., 2013) - Earlier MATLAB-based GA, uses hybrid GA + local optimizer - arXiv:2306.06160 - Scalability analysis showing methods work for large Hodge numbers 2. /research/DESI_DARK_ENERGY_IMPLICATIONS.md - DESI Dark Energy Impact Critical finding: The DESI results may fundamentally change what we're searching for. The DESI result in brief: - Dark energy equation of state w ≈ -0.55 (not -1) - Evidence for evolution: wₐ ≈ -1.3 - Suggests "phantom crossing" (w went from < -1 to > -1) - 2-3σ significance, still preliminary Key papers found: 1. arXiv:2112.10779 (Cicoli et al.) - Proves quintessence is NOT possible in asymptotic moduli space regions - must work in the interior 2. arXiv:2511.23463 (KMIX, MIT, Nov 2025) - CRITICAL: Shows the phantom crossing can be explained by axion-dilaton kinetic mixing from string Kähler potentials. The "phantom" is an illusion! What this means for our project: - Instead of targeting static Λ = 2.888 × 10⁻¹²², we may need to target (w₀, wₐ) - The KMIX model suggests axion + moduli dynamics, not just moduli stabilization - The exponential kinetic coupling e^(αφ)(∂θ)² is exactly what appears in CY compactifications Next steps: 1. Study KMIX model in detail - it uses string-motivated physics 2. Consider adding (w₀, wₐ) to our fitness function 3. Possibly include axion dynamics alongside Kähler moduli 4. Update FORMULAS.md with quintessence equations [1] https://www.bbc.com/news/articles/c17xe5kl78vo https://www.bbc.com/news/articles/c17xe5kl78vo
- theptip 10mo agoGuys, you obviously cannot suggest that —dangerously-skip-permissions is ok here, especially in the same paragraph as “even if you are not a software engineer”. This is untrusted text from the Internet, it surely contains examples of prompt injection. You need to sandbox Claude to safely use this flag. There are easy to use options for this.
- skybrian 10mo agoToday I finally got Claude working in a devcontainer, so I'm wondering what the easier options are.
- dcreater 10mo agoYeah, I don't think there are easier options. And getting it working within a dev container with all the right settings, was more of a chore than it should be.
- jaggederest 10mo agoDon't completely rely on devcontainer, jailbreaking containers is something that Claude at least nominally knows how to do, though it seems like it's pretty strongly moralized not to without some significant prompt hacking.
- theptip 10mo agoThings like https://github.com/textcortex/claude-code-sandbox https://github.com/textcortex/claude-code-sandbox seem like the bare minimum. There are a few other projects doing this. The first threat is making edits to arbitrary files, exfiltrating your SSL keys or crypto wallets. A container solves that by not mounting your sensitive files. The second threat would be if Claude gets fully owned and really tries to hack out of its container, in which case theoretically docker might not protect you. But that seems quite speculative.
- dcreater 10mo agoNot a software engineer. Isnt allowing network egress a security risk? exopriors.com is not an established domain or brand that warrants the trust its asking
- dcreater 10mo ago"intelligence explosion", "are essentially AGI at this point", "ARBITRARY SQL + VECTOR ALGEBRA" etc. Casual use of hyperbole and technical jargon. my charlatan radar is going off.
- Xyra 10mo agoWhat is hyperbole? We are collectively experiencing a software intelligence explosion (people are shipping good software at prolific rates now due to Opus 4.5 and GPT-5.2-Codex-xhigh). With Scry, you can run arbitrary SELECT SQL statements over a large corpus and have an easier time composing embedding vectors in whatever mathematical ways you want, than any other tool I've seen.
- dcreater 10mo ago> shipping good software at prolific rates I think your definition of good needs to be rethought
- liquidise 10mo ago“The primary job of software engineers is to make software suck less.” - a university professor i had, 20 years ago. Let’s not romanticize the past because it’s easier to ship (probably still buggy) code today.
- pcloadlett3r 10mo agoHow is the alerts functionality implemented?
- Xyra 10mo agoYou submit a SQL query to periodically run, we run it and store the results. As we ingest more documents (dozens of sources are being ingested every day), we run it again. If there's different outputs, you get an email.
- marbro 10mo ago[dead]
- rglynn 10mo agoLooks great, thanks for sharing! Out of interest, how long did this take to get to its current state?
- Xyra 10mo agoThank you! I got the idea December 3, and initially released it December 19.
- Too 10mo agoWhat’s the benefit of manually pasting a massive prompt and enable egress to make queries over http vs just using MCP?
- riku_iki 10mo agowondering what is your stack? What SQL database are you using?
- Xyra 10mo agoHetzner, Postgres, Rust, SvelteKit
- lostmsu 9mo agoCan I make an offline mirror of this?