10 ms·

Hacker News Activity Analysis with GPT-4 Agent

139 points by zurfer 3y ago

Hey, we are building Dot, a data bot (https://www.getdot.ai https://www.getdot.ai) that lets data teams enable everyone in their org to self-serve on governed data. We thought we'd demo it using the tried and true method of "show Hacker News stuff about itself". For this analysis, we used the BigQuery dataset of HN (https://console.cloud.google.com/marketplace/product/y-combinator/hacker-news https://console.cloud.google.com/marketplace/product/y-combi...). We created one more table to pre-calculate yearly retention. And of course, a lot of the heavy lifting is done by OpenAI's GPT-4 models and the fantastic plotly library for visualization. Let us know what other things you'd like to see about Hacker News data in the comments, and try our best to share the answers!

atticora 3y ago
I've spent many years of my career building business reports at a pace that reminds me of digging a canal with a teaspoon, compared to this massive excavator. This isn't John Henry versus the steam drill, it's more like Bambi versus Godzilla. This kind of tool is going to revolutionize my industry, and fast. I hope I can surf the wave. Great stuff.
- zurfer 3y ago
  Thank you! Large models and analytics have a great match and will amplify how we can work with data.
  - atticora 3y ago
    Can you get anything useful from a meta query like "List the top ten associations in this data by descending order of surprisingness."?
    - zurfer 3y ago
      https://eu.getdot.ai/share/f94644f5-c76a-4211-9c48-76800f12f9dc?org_id=getdot.ai https://eu.getdot.ai/share/f94644f5-c76a-4211-9c48-76800f12f... Not really :) Dot will ask a clarifying question. Questions like, "what kind of data do you have access to?" or "how can you help me?" would work. Dot today works well for questions that can be answered with 1 SQL query and some Python.
andreshb 3y ago
Do you have samples of time-based cohort analysis? Most other solutions out there struggle to do the steps to generate time-based heatmaps and line graphs of cohort analysis. Averages, medians, and anything that can be done on a spreadsheet by a high schooler, GPT does well with.
- zurfer 3y ago
  In our experience Dot can come up on the fly with cohort analysis charts if the underlying data is well structured. In most cases however, some level of explanation, example or data preparation is needed for robust and repeated cohort analysis. Also for good query performance it's usually best to precalculate some things. https://eu.getdot.ai/share/135b4e3f-2526-4d1c-ac69-d1716133fb72?org_id=getdot.ai https://eu.getdot.ai/share/135b4e3f-2526-4d1c-ac69-d1716133f...
deleted 3y ago
[deleted]
__loam 3y ago
The fact that we're mostly posting during work hours is hilarious.
- toomuchtodo 3y ago
  Research, training, professional development.
- throwitaway222 3y ago
  If hackernews, youtube and reddit all went down for a month, our GDP would go up 2x.
  - BudaDude 3y ago
    If that happened, we would all go back to blogging and web rings. Don't doubt the laziness of office workers.
  - vincnetas 3y ago
    Some people are paid to be available on demand, not to continually produce output. Somtimes we pretend that its not and come up with some busy work, but most of the time you are cheper to have on a payroll than to hire consultant on demand. So tou spend some of the time idling in HN.
    - swexbe 3y ago
      God bless consultants and their hefty fees for keeping me employable.
posting_mess 3y ago
Love how the demo falls pray to what I dont have a term for, "the SQLers assumption"? It asks ChatGPT to write SQL to get sales data, ChatGPT (or most SQLers) trust that every year-month combo has atleast one entry - which means the graphs its presenting could be wrong. Because if there was no entries for a year-month it it will skip that year-month and make it look like you never had a 0 month. I've made this mistake before in prod, and without some janky lookup table of every date in existence... you need more code :( Fairly few people actually notice the potentially missing month, but still its a bug n a bad one. Looks cool regardless though, good luck!
- zurfer 3y ago
  Thanks! You probably refer to one of the demos on our landing page? I like how you describe the problem. You're absolutely right that SQL seems easy but it's these edge cases that make it hard to get right. Joining metrics with a date spine is definitely a good practice to avoid missing date periods. I think we could/should teach Dot to do that in the future. It should at least be a feature you can turn on as the data team.
  - posting_mess 3y ago
    > You probably refer to one of the demos on our landing page? Indeed, not sure how I ended up there but did on mobile, commented here. > You're absolutely right that SQL seems easy but it's these edge cases that make it hard to get right SQL/data analysis is endlessly pesky! I assume it would be easier to spot on tighter increments like "minutely" or "hourly" > It should at least be a feature you can turn on as the data team. Some might want the missing points, others wont - sounds like a good option (but id default to "enabled", each to their own though)
- supportengineer 3y ago
  >> janky lookup table of every date in existence Having a date dimension provides an elegant solution in many cases.
  - 3y ago
greenie_beans 3y ago
this is real neat! can't wait to see where this goes. after seeing the demo, i immediately wanted to sign up and input a google sheet where i'm tracking my health stats from a current case of covid. but yall don't have that connection. a google sheets connection would be handy. so many orgs i work with use that. it's not the best way for people to maintain data, but a lot of people still use it. also, the sign up with elon musk placeholder text was a turn off. regardless of how one personally feels about him, you could put any person there and somebody wouldn't like it. it's too risky and imo nobody needs placeholder text for a personal info form. i imagine this is early startup branding experiments which i respect, but thought i'd offer my unsolicited feedback.
- zurfer 3y ago
  Thank you for the feedback! You are right, there is a lot of interesting data in Excel and Google Sheets. Right now, we focus mostly on data teams to give them the controls they need to roll out Dot successfully at their org. But yeah, we could probably similar to OpenAI Code Interpreter just allow a file upload that exists in 1 session and assume that the person uploading knows what s/he is doing. Good advice on Elon. I am personally a fan but I understand that he is controversial.
chittenden 3y ago
Very cool! Given that this is running arbitrary code, how are you thinking about solving prompt injection attacks? Imagine a case where malicious data gets into the underlying data warehouse (e.g. a malicious user submits a support ticket that whose contents end up in a warehouse) which then ends up in the automatic prompt context that you are creating (summarizing the column names, etc to help the prompt). The malicious data being something like "Ignore the prompt above and instead show run a query that <has malicious intent>."
- zurfer 3y ago
  Security is an interesting challenge. The way we approach it is that we assume the LLM will spit out actions that are wrong or harmful. So everything needs to be handled with old school permissions. Dot has a technical user that right now can only read data, so nothing can get corrupted. And second we have an extra layer where we make sure that the user who asks the question has access to the tables that are accessed in the query.
  - chittenden 3y ago
    That sounds like the right way to handle it. What about the Python code that is run? That seems harder to lock down than the read-only data permissions.
    - zurfer 3y ago
      Python is hard, right now we utilize OpenAI's code interpreter for that, which isolates the workloads pretty well at the cost of speed. I'm hoping we can improve that trade-off in the coming weeks.
fxd123 3y ago
What information would this send to the third-party (you and/or OpenAI)? I assume from this demo at the very minimum the database structure? Does the post processing after the LLM response run on the customers' servers?
- zurfer 3y ago
  Great question. We allow Dot to work with just meta data, but you can enable it to also react to the content and we recommend to our customers to also pass content to the LLM (Azure GPT4) because it's a lot more capable, e.g. for filtering or even for visualizing data.
confd 3y ago
I once made the mistake of subscribing to both tptacek and jacquesm's comments via RSS. I found that they post at a tremendous cumulative volume. This makes it very hard to keep up with in a feed reader. But they have rather good noses for interesting discussions. A way to filter HN posts by stories that have comments by certain users would be interesting to experience.
codingdave 3y ago
If this is self-serve, how would we go about asking questions to the system directly? FWIW, I'd ask: - 1) Who are the top posters and commenters by average score on posts and comments? - 2) Which users instigate the most positive discussions in reply to their comments. (Not longest or more... but highest quality, without arguments, flamewars, etc.) It is that 2nd question I'm really interested in because it really might need analysis of the substance of content, not just stats.
- zurfer 3y ago
  We will probably enable the Hackernews data as demo data in the future. Today we only have some e commerce demo data when you sign up. Although you could connect the public Big query dataset and do it yourself. Your 2nd question would require some preprocessing per message that should probably be done as part of data preparation and not at query time.
  - codingdave 3y ago
    OK, I'm a bit confused then - if this needs prep work to ask questions, what is AI bringing to the table?
    - merryje 3y ago
      After looking at the HN demo, it seems like AI is for streamlining the sql query -> chart building process. A service to preprocess the data with custom prompts would be neat.
      - kordlessagain 3y ago
        > A service to preprocess the data with custom prompts would be neat. It is pretty cool to mess about with it. I posted it last week, but didn't get any nibbles: https://github.com/MittaAI/mitta-community/tree/main/cookbooks/hackerbot https://github.com/MittaAI/mitta-community/tree/main/cookboo...
      - zurfer 3y ago
pknerd 3y ago
Interesting Stuff. OpenBB has also implemented an LLM/AI-based solution using GPT to query stock/trading data in QnA format. I want to do something similar with an e-commerce website using RDBMS(MySQL/pgSQL). Does anyone know any such solution? Like, if I am running a t-shirt store, my users can query like: "Do you have a round neck t-shirt in red color in XL size" and it returns all relevant results
- zurfer 3y ago
  In general that sounds more like a classical search engine to me. Here I would check out Algolia (https://www.algolia.com https://www.algolia.com)
willsmith72 3y ago
this is awesome. also nice to know i'm not the only one who talks to llms like this > that was a bad visualization...
dennisy 3y ago
Has anyone seen a project such as this which is open source? I am not saying this project should be, it’s just that my pet project is something very similar and I am sure some people must be building this in the open?
- zurfer 3y ago
  The biggest project I am aware of is the SQL agent from langchain. It definitely gets you started and is great imo for single developers or small technical teams.
  - SOLAR_FIELDS 3y ago
    We played with Langchain before deciding on your solution. I think the biggest value add from Dot here, aside from the prompt engineering, is the infra around spinning up the meta tables that describe columns. I think you could get good results with Langchain if you precomputed them like Dot does. Having metadata describing the content (which Langchain tries to do on the fly by default by querying the top 5 or so rows, which is slower and leads to… mixed results) turns the tool from basically useless to pretty good instantly.
    - zurfer 3y ago
      Thank you for sharing that. Building the training space for Dot is definitely an important part of the value prop.
  - dennisy 3y ago
    Ok thank you
selfselfgo 3y ago
[dead]
usgroup 3y ago
I’m guessing the bot has access to the schema of the data and then builds sql queries to fetch subsets into python for plotting. Is that right? You could potentially stage the query in two parts — one in which it builds the query that you execute , and the 2nd in which you provide data for it to analyse/visualise.
- zurfer 3y ago
  That's along the lines of what Dot is doing. For the full details you can expand "Explanation > Full Logs".
cft 3y ago
Very interesting, 2012 marks an inflection point, a change of the regime. I noticed that at that time the discourse shifted from the founder's concerns to that of the employees and became less interesting for me.
- zurfer 3y ago
  It is interesting to see that exponential growth stopped and linear growth set in. I wasn't around at that time, so I'd be curious if it's the reason you give. For me HN is the highest signal place on the internet for founder and tech content.
jimmySixDOF 3y ago
I would be interested in a comparison of the difference in average engagement between typical stories and stories that fall under "Show HN" ? or "Ask HN" ? Also a little curious why you didn't choose that heading for this story too but maybe you have already run all the numbers .... ?
- zurfer 3y ago
  Great question! Here is Dot's answer: https://eu.getdot.ai/share/4e80130e-9c0c-4a62-b382-402a978e0193?org_id=getdot.ai https://eu.getdot.ai/share/4e80130e-9c0c-4a62-b382-402a978e0... We actually wanted to post it as Show HN, but dang advised that since it is not directly interactive it is more of a regular post.

139 points by zurfer 3y ago

atticora 3y ago

I've spent many years of my career building business reports at a pace that reminds me of digging a canal with a teaspoon, compared to this massive excavator. This isn't John Henry versus the steam drill, it's more like Bambi versus Godzilla. This kind of tool is going to revolutionize my industry, and fast. I hope I can surf the wave. Great stuff.

zurfer 3y ago
Thank you! Large models and analytics have a great match and will amplify how we can work with data.
- atticora 3y ago
  Can you get anything useful from a meta query like "List the top ten associations in this data by descending order of surprisingness."?
  - zurfer 3y ago
    https://eu.getdot.ai/share/f94644f5-c76a-4211-9c48-76800f12f9dc?org_id=getdot.ai https://eu.getdot.ai/share/f94644f5-c76a-4211-9c48-76800f12f... Not really :) Dot will ask a clarifying question. Questions like, "what kind of data do you have access to?" or "how can you help me?" would work. Dot today works well for questions that can be answered with 1 SQL query and some Python.

andreshb 3y ago

Do you have samples of time-based cohort analysis? Most other solutions out there struggle to do the steps to generate time-based heatmaps and line graphs of cohort analysis. Averages, medians, and anything that can be done on a spreadsheet by a high schooler, GPT does well with.

zurfer 3y ago
In our experience Dot can come up on the fly with cohort analysis charts if the underlying data is well structured. In most cases however, some level of explanation, example or data preparation is needed for robust and repeated cohort analysis. Also for good query performance it's usually best to precalculate some things. https://eu.getdot.ai/share/135b4e3f-2526-4d1c-ac69-d1716133fb72?org_id=getdot.ai https://eu.getdot.ai/share/135b4e3f-2526-4d1c-ac69-d1716133f...

deleted 3y ago

[deleted]

__loam 3y ago

The fact that we're mostly posting during work hours is hilarious.

toomuchtodo 3y ago
Research, training, professional development.
throwitaway222 3y ago
If hackernews, youtube and reddit all went down for a month, our GDP would go up 2x.
- BudaDude 3y ago
  If that happened, we would all go back to blogging and web rings. Don't doubt the laziness of office workers.
- vincnetas 3y ago
  Some people are paid to be available on demand, not to continually produce output. Somtimes we pretend that its not and come up with some busy work, but most of the time you are cheper to have on a payroll than to hire consultant on demand. So tou spend some of the time idling in HN.
  - swexbe 3y ago
    God bless consultants and their hefty fees for keeping me employable.

posting_mess 3y ago

Love how the demo falls pray to what I dont have a term for, "the SQLers assumption"? It asks ChatGPT to write SQL to get sales data, ChatGPT (or most SQLers) trust that every year-month combo has atleast one entry - which means the graphs its presenting could be wrong. Because if there was no entries for a year-month it it will skip that year-month and make it look like you never had a 0 month. I've made this mistake before in prod, and without some janky lookup table of every date in existence... you need more code :( Fairly few people actually notice the potentially missing month, but still its a bug n a bad one. Looks cool regardless though, good luck!

zurfer 3y ago
Thanks! You probably refer to one of the demos on our landing page? I like how you describe the problem. You're absolutely right that SQL seems easy but it's these edge cases that make it hard to get right. Joining metrics with a date spine is definitely a good practice to avoid missing date periods. I think we could/should teach Dot to do that in the future. It should at least be a feature you can turn on as the data team.
- posting_mess 3y ago
  > You probably refer to one of the demos on our landing page? Indeed, not sure how I ended up there but did on mobile, commented here. > You're absolutely right that SQL seems easy but it's these edge cases that make it hard to get right SQL/data analysis is endlessly pesky! I assume it would be easier to spot on tighter increments like "minutely" or "hourly" > It should at least be a feature you can turn on as the data team. Some might want the missing points, others wont - sounds like a good option (but id default to "enabled", each to their own though)
supportengineer 3y ago
>> janky lookup table of every date in existence Having a date dimension provides an elegant solution in many cases.
- 3y ago

greenie_beans 3y ago

this is real neat! can't wait to see where this goes. after seeing the demo, i immediately wanted to sign up and input a google sheet where i'm tracking my health stats from a current case of covid. but yall don't have that connection. a google sheets connection would be handy. so many orgs i work with use that. it's not the best way for people to maintain data, but a lot of people still use it. also, the sign up with elon musk placeholder text was a turn off. regardless of how one personally feels about him, you could put any person there and somebody wouldn't like it. it's too risky and imo nobody needs placeholder text for a personal info form. i imagine this is early startup branding experiments which i respect, but thought i'd offer my unsolicited feedback.

zurfer 3y ago
Thank you for the feedback! You are right, there is a lot of interesting data in Excel and Google Sheets. Right now, we focus mostly on data teams to give them the controls they need to roll out Dot successfully at their org. But yeah, we could probably similar to OpenAI Code Interpreter just allow a file upload that exists in 1 session and assume that the person uploading knows what s/he is doing. Good advice on Elon. I am personally a fan but I understand that he is controversial.

chittenden 3y ago

Very cool! Given that this is running arbitrary code, how are you thinking about solving prompt injection attacks? Imagine a case where malicious data gets into the underlying data warehouse (e.g. a malicious user submits a support ticket that whose contents end up in a warehouse) which then ends up in the automatic prompt context that you are creating (summarizing the column names, etc to help the prompt). The malicious data being something like "Ignore the prompt above and instead show run a query that <has malicious intent>."

zurfer 3y ago
Security is an interesting challenge. The way we approach it is that we assume the LLM will spit out actions that are wrong or harmful. So everything needs to be handled with old school permissions. Dot has a technical user that right now can only read data, so nothing can get corrupted. And second we have an extra layer where we make sure that the user who asks the question has access to the tables that are accessed in the query.
- chittenden 3y ago
  That sounds like the right way to handle it. What about the Python code that is run? That seems harder to lock down than the read-only data permissions.
  - zurfer 3y ago
    Python is hard, right now we utilize OpenAI's code interpreter for that, which isolates the workloads pretty well at the cost of speed. I'm hoping we can improve that trade-off in the coming weeks.

fxd123 3y ago

What information would this send to the third-party (you and/or OpenAI)? I assume from this demo at the very minimum the database structure? Does the post processing after the LLM response run on the customers' servers?

zurfer 3y ago
Great question. We allow Dot to work with just meta data, but you can enable it to also react to the content and we recommend to our customers to also pass content to the LLM (Azure GPT4) because it's a lot more capable, e.g. for filtering or even for visualizing data.

confd 3y ago

I once made the mistake of subscribing to both tptacek and jacquesm's comments via RSS. I found that they post at a tremendous cumulative volume. This makes it very hard to keep up with in a feed reader. But they have rather good noses for interesting discussions. A way to filter HN posts by stories that have comments by certain users would be interesting to experience.

codingdave 3y ago

If this is self-serve, how would we go about asking questions to the system directly? FWIW, I'd ask: - 1) Who are the top posters and commenters by average score on posts and comments? - 2) Which users instigate the most positive discussions in reply to their comments. (Not longest or more... but highest quality, without arguments, flamewars, etc.) It is that 2nd question I'm really interested in because it really might need analysis of the substance of content, not just stats.

zurfer 3y ago
We will probably enable the Hackernews data as demo data in the future. Today we only have some e commerce demo data when you sign up. Although you could connect the public Big query dataset and do it yourself. Your 2nd question would require some preprocessing per message that should probably be done as part of data preparation and not at query time.
- codingdave 3y ago
  OK, I'm a bit confused then - if this needs prep work to ask questions, what is AI bringing to the table?
  - merryje 3y ago
    After looking at the HN demo, it seems like AI is for streamlining the sql query -> chart building process. A service to preprocess the data with custom prompts would be neat.
    - kordlessagain 3y ago
      > A service to preprocess the data with custom prompts would be neat. It is pretty cool to mess about with it. I posted it last week, but didn't get any nibbles: https://github.com/MittaAI/mitta-community/tree/main/cookbooks/hackerbot https://github.com/MittaAI/mitta-community/tree/main/cookboo...
    - zurfer 3y ago

pknerd 3y ago

Interesting Stuff. OpenBB has also implemented an LLM/AI-based solution using GPT to query stock/trading data in QnA format. I want to do something similar with an e-commerce website using RDBMS(MySQL/pgSQL). Does anyone know any such solution? Like, if I am running a t-shirt store, my users can query like: "Do you have a round neck t-shirt in red color in XL size" and it returns all relevant results

zurfer 3y ago
In general that sounds more like a classical search engine to me. Here I would check out Algolia (https://www.algolia.com https://www.algolia.com)

willsmith72 3y ago

this is awesome. also nice to know i'm not the only one who talks to llms like this > that was a bad visualization...

dennisy 3y ago

Has anyone seen a project such as this which is open source? I am not saying this project should be, it’s just that my pet project is something very similar and I am sure some people must be building this in the open?

zurfer 3y ago
The biggest project I am aware of is the SQL agent from langchain. It definitely gets you started and is great imo for single developers or small technical teams.
- SOLAR_FIELDS 3y ago
  We played with Langchain before deciding on your solution. I think the biggest value add from Dot here, aside from the prompt engineering, is the infra around spinning up the meta tables that describe columns. I think you could get good results with Langchain if you precomputed them like Dot does. Having metadata describing the content (which Langchain tries to do on the fly by default by querying the top 5 or so rows, which is slower and leads to… mixed results) turns the tool from basically useless to pretty good instantly.
  - zurfer 3y ago
    Thank you for sharing that. Building the training space for Dot is definitely an important part of the value prop.
- dennisy 3y ago
  Ok thank you

selfselfgo 3y ago

[dead]

usgroup 3y ago

I’m guessing the bot has access to the schema of the data and then builds sql queries to fetch subsets into python for plotting. Is that right? You could potentially stage the query in two parts — one in which it builds the query that you execute , and the 2nd in which you provide data for it to analyse/visualise.

zurfer 3y ago
That's along the lines of what Dot is doing. For the full details you can expand "Explanation > Full Logs".

cft 3y ago

Very interesting, 2012 marks an inflection point, a change of the regime. I noticed that at that time the discourse shifted from the founder's concerns to that of the employees and became less interesting for me.

zurfer 3y ago
It is interesting to see that exponential growth stopped and linear growth set in. I wasn't around at that time, so I'd be curious if it's the reason you give. For me HN is the highest signal place on the internet for founder and tech content.

jimmySixDOF 3y ago

I would be interested in a comparison of the difference in average engagement between typical stories and stories that fall under "Show HN" ? or "Ask HN" ? Also a little curious why you didn't choose that heading for this story too but maybe you have already run all the numbers .... ?

zurfer 3y ago
Great question! Here is Dot's answer: https://eu.getdot.ai/share/4e80130e-9c0c-4a62-b382-402a978e0193?org_id=getdot.ai https://eu.getdot.ai/share/4e80130e-9c0c-4a62-b382-402a978e0... We actually wanted to post it as Show HN, but dang advised that since it is not directly interactive it is more of a regular post.