4 ms·

Enabling Codex to Analyze Two Decades of Hacker News Data

87 points by ronfriedhaber 7mo ago

mike_hearn 7mo ago
I don't quite understand how Modolap differs from just asking AI to use any other OLAP engine? Both your website and the github readme just emphasise that it's idiosyncratic and your personal approach, without explaining what that is or why anyone should care.
- ronfriedhaber 7mo ago
  Appreciate the feedback. I shall certainly revamp the README; it is rather stale. > "how Modolap differs from just asking AI to use any other OLAP engine" There presently exist two components, the OLAP query engine and the remote infrastructure service. The service enables systems like Codex (or developers as well) to manage datasets, maintain version control over queries, and offload the computational burden to dedicated machines. This is especially beneficial given the current trend of running agents inside micro-VMs. In addition, it is designed with AI usage in mind. There is significant value in co-design. One could argue that models can use Polars or DuckDB just as well, and that there is no room for improvement, but I do not think this is true.
  - bastawhiz 7mo ago
    What room for improvement is there?
  - esafak 7mo ago
    I don't get the value proposition either; your landing page is underdeveloped. Tracking the query history is trivial. Offloading computation could be done with Polars Cloud or MotherDuck. Can you expand on the "manage datasets" part?
throwaway290 7mo ago
HN data is open? Under what conditions it's distributed?
- bastawhiz 7mo ago
  There's an API link at the bottom of every page.
  - throwaway290 7mo ago
    And? I'm talking about entire dataset. API is irrelevant unless you scrape it and because HN rate limits actual users I'm sure it doesn't like scraping.
- hsuduebc2 7mo ago
  There are even datasets created by users over time and shared publicly.
  - throwaway290 7mo ago
    Is this one of them? It sounded like it's some sort of official dataset.
- GeoAtreides 7mo ago
  no conditions! what, you didn't know your content would be free for LITERALLY EVERYONE when you made your account? and that you can't delete your comments? and it's a free for all? well, that's on you buddy, not HN, definitely not HN in any way
benreesman 7mo ago
[dead]
zeroxfe 7mo ago
I've done this kind of thing many times with codex and sqlite, and it works very well. It's one prompt that looks something like this: - inspect and understand the downloaded data in directory /path/..., then come up with an sqlite data model for doing detailed analytics and ingest everything into an sqlite db in data.sqlite, and document the model in model.md. Then you can query the database adhoc pretty easily with codex prompts (and also generate PDF graphs as needed.) I typically use the highest reasoning level for the initial prompt, and as I get deeper into the data, continuously improve on the model, indexes, etc., and just have codex handle any data migration.
- huflungdung 7mo ago
  [dead]
- rizzo94 7mo ago
  [dead]
avib99 7mo ago
[dead]
voidUpdate 7mo ago
When searching for references to Go, what does it actually look for? "Go" is a relatively common word, and I hardly see anyone referring to it as Golang
xnorswap 7mo ago
5% of all comments mention Claude code? Am I reading that right?
- epaga 7mo ago
  Well now it's 5.00001%.
- SyneRyder 7mo ago
  I don't think it's a percentage of all comments. I think it's either the percent of articles (topics) posted to the site that are about Claude Code, or maybe a percentage of topics where at least one person mentioned Claude Code in the comments. The latter seems easier to achieve. To borrow from another internet rule of thumb: "As an online discussion grows longer, the probability of someone mentioning Claude approaches one."
  - xnorswap 7mo ago
    I think that latter interpretation must be right, it seems possible, if high. Either raw comments or submissions doesn't seem credible.
huflungdung 7mo ago
[dead]
moralestapia 7mo ago
Do not estimate/plot DAUs/MAUs, it's not a pretty picture :'(.
- hakrgrl 7mo ago
  Why do you say that?
Brajeshwar 7mo ago
The “Hacker News - Complete Archive” on Hugging Face,[1] recently popped up here. “The data is stored as monthly Parquet files sorted by item ID, making it straightforward to query with DuckDB, load with the datasets library, or process with any tool that reads Parquet.” Out of curiosity, I tinkered with it using Claude to see trends and patterns (I did find a few embarrassing things about me!). 1. https://huggingface.co/datasets/open-index/hacker-news https://huggingface.co/datasets/open-index/hacker-news
hakrgrl 7mo ago
That last chart showing the average comment length shows a clear negative downtrend, especially in recent months. I wonder why that is.
- hakrgrl 7mo ago
  I noticed some topics and comments that were usually in violation of HN guidelines are no longer flagged, and discourse decays into reddit-like jabs and echo chambers. Only a small percent, but still, more than the previous 0% I was accustomed to. Would be interesting to see how many comments violate the guidelines over time. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
sd9 7mo ago
I'm kind of surprised that postgres was quite that dominated by mongodb back in the day. I remember the mongo fever, but I always thought postgres held reasonable market share. I guess it was other SQL dbs back then, I guess MySQL was still viable.
- hsuduebc2 7mo ago
  I worked on many projects that had used wrongly mongo instead of ordinary relation database and they needed rework in time. It was just hyped in it's days. Like micro service architecture in few years.
- tombert 7mo ago
  It could be that Postgres was so popular that people didn't really discuss it. Hyperbolic example; literally every human reading this consumes oxygen nearly every moment of the day, and as such no one talks about how great breathing is.
hsuduebc2 7mo ago
I really love codex. The price/value comparison to claude code is at least from my opinion much better.
- fragmede 7mo ago
  There's no doubt that $20 to OpenAI gets you more than $20 to Anthropic, the question is who's better? The bus to work could be free vs a car but there are cases where spending the extra money is simply just worth it.
  - hsuduebc2 7mo ago
    Well sure. No doubt abouth that. I just think that most people, especially non tech people vibe coding apps are simply just burning money with latest open claw saga. You simply do not need claude for simple tasks or simply scripts.
sam-bee 7mo ago
Nobody who actually codes in that language ever calls it 'Golang'
- Zambyte 7mo ago
  I write Go for my day job and I refer to it as Golang all the time, depending on context. What are you getting at with this comment?
cloudpeaklabs 7mo ago
[flagged]
RockyMcNuts 7mo ago
Could be interesting to chart quality of responses, toxicity/health of conversations, sentiment over time, impact of release of ChatGPT. (since AI can now answer many questions that might have been topics of conversation; people can use AI to participate; people may be reluctant to participate if AI can data mine everything and link it back to them, etc. similar to Stack Overflow)

87 points by ronfriedhaber 7mo ago

I don't quite understand how Modolap differs from just asking AI to use any other OLAP engine? Both your website and the github readme just emphasise that it's idiosyncratic and your personal approach, without explaining what that is or why anyone should care.

ronfriedhaber 7mo ago
Appreciate the feedback. I shall certainly revamp the README; it is rather stale. > "how Modolap differs from just asking AI to use any other OLAP engine" There presently exist two components, the OLAP query engine and the remote infrastructure service. The service enables systems like Codex (or developers as well) to manage datasets, maintain version control over queries, and offload the computational burden to dedicated machines. This is especially beneficial given the current trend of running agents inside micro-VMs. In addition, it is designed with AI usage in mind. There is significant value in co-design. One could argue that models can use Polars or DuckDB just as well, and that there is no room for improvement, but I do not think this is true.
- bastawhiz 7mo ago
  What room for improvement is there?
- esafak 7mo ago
  I don't get the value proposition either; your landing page is underdeveloped. Tracking the query history is trivial. Offloading computation could be done with Polars Cloud or MotherDuck. Can you expand on the "manage datasets" part?

throwaway290 7mo ago

HN data is open? Under what conditions it's distributed?

bastawhiz 7mo ago
There's an API link at the bottom of every page.
- throwaway290 7mo ago
  And? I'm talking about entire dataset. API is irrelevant unless you scrape it and because HN rate limits actual users I'm sure it doesn't like scraping.
hsuduebc2 7mo ago
There are even datasets created by users over time and shared publicly.
- throwaway290 7mo ago
  Is this one of them? It sounded like it's some sort of official dataset.
GeoAtreides 7mo ago
no conditions! what, you didn't know your content would be free for LITERALLY EVERYONE when you made your account? and that you can't delete your comments? and it's a free for all? well, that's on you buddy, not HN, definitely not HN in any way

benreesman 7mo ago

[dead]

zeroxfe 7mo ago

I've done this kind of thing many times with codex and sqlite, and it works very well. It's one prompt that looks something like this: - inspect and understand the downloaded data in directory /path/..., then come up with an sqlite data model for doing detailed analytics and ingest everything into an sqlite db in data.sqlite, and document the model in model.md. Then you can query the database adhoc pretty easily with codex prompts (and also generate PDF graphs as needed.) I typically use the highest reasoning level for the initial prompt, and as I get deeper into the data, continuously improve on the model, indexes, etc., and just have codex handle any data migration.

huflungdung 7mo ago
[dead]
rizzo94 7mo ago
[dead]

avib99 7mo ago

[dead]

voidUpdate 7mo ago

When searching for references to Go, what does it actually look for? "Go" is a relatively common word, and I hardly see anyone referring to it as Golang

xnorswap 7mo ago

5% of all comments mention Claude code? Am I reading that right?

epaga 7mo ago
Well now it's 5.00001%.
SyneRyder 7mo ago
I don't think it's a percentage of all comments. I think it's either the percent of articles (topics) posted to the site that are about Claude Code, or maybe a percentage of topics where at least one person mentioned Claude Code in the comments. The latter seems easier to achieve. To borrow from another internet rule of thumb: "As an online discussion grows longer, the probability of someone mentioning Claude approaches one."
- xnorswap 7mo ago
  I think that latter interpretation must be right, it seems possible, if high. Either raw comments or submissions doesn't seem credible.

huflungdung 7mo ago

[dead]

moralestapia 7mo ago

Do not estimate/plot DAUs/MAUs, it's not a pretty picture :'(.

hakrgrl 7mo ago
Why do you say that?

Brajeshwar 7mo ago

The “Hacker News - Complete Archive” on Hugging Face,[1] recently popped up here. “The data is stored as monthly Parquet files sorted by item ID, making it straightforward to query with DuckDB, load with the datasets library, or process with any tool that reads Parquet.” Out of curiosity, I tinkered with it using Claude to see trends and patterns (I did find a few embarrassing things about me!). 1. https://huggingface.co/datasets/open-index/hacker-news https://huggingface.co/datasets/open-index/hacker-news

hakrgrl 7mo ago

That last chart showing the average comment length shows a clear negative downtrend, especially in recent months. I wonder why that is.

hakrgrl 7mo ago
I noticed some topics and comments that were usually in violation of HN guidelines are no longer flagged, and discourse decays into reddit-like jabs and echo chambers. Only a small percent, but still, more than the previous 0% I was accustomed to. Would be interesting to see how many comments violate the guidelines over time. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html

sd9 7mo ago

I'm kind of surprised that postgres was quite that dominated by mongodb back in the day. I remember the mongo fever, but I always thought postgres held reasonable market share. I guess it was other SQL dbs back then, I guess MySQL was still viable.

hsuduebc2 7mo ago
I worked on many projects that had used wrongly mongo instead of ordinary relation database and they needed rework in time. It was just hyped in it's days. Like micro service architecture in few years.
tombert 7mo ago
It could be that Postgres was so popular that people didn't really discuss it. Hyperbolic example; literally every human reading this consumes oxygen nearly every moment of the day, and as such no one talks about how great breathing is.

hsuduebc2 7mo ago

I really love codex. The price/value comparison to claude code is at least from my opinion much better.

fragmede 7mo ago
There's no doubt that $20 to OpenAI gets you more than $20 to Anthropic, the question is who's better? The bus to work could be free vs a car but there are cases where spending the extra money is simply just worth it.
- hsuduebc2 7mo ago
  Well sure. No doubt abouth that. I just think that most people, especially non tech people vibe coding apps are simply just burning money with latest open claw saga. You simply do not need claude for simple tasks or simply scripts.

sam-bee 7mo ago

Nobody who actually codes in that language ever calls it 'Golang'

Zambyte 7mo ago
I write Go for my day job and I refer to it as Golang all the time, depending on context. What are you getting at with this comment?

cloudpeaklabs 7mo ago

[flagged]

RockyMcNuts 7mo ago

Could be interesting to chart quality of responses, toxicity/health of conversations, sentiment over time, impact of release of ChatGPT. (since AI can now answer many questions that might have been topics of conversation; people can use AI to participate; people may be reluctant to participate if AI can data mine everything and link it back to them, etc. similar to Stack Overflow)