4 ms·

Show HN: I built an AI voice agent for Gmail

35 points by chrisnolet 2y ago

Hello again, HN! I’ve been using my DSL to create new voice experiences. I’ve made an AI-powered email client for Gmail that you talk to, using your microphone. (I highly recommend using earbuds or headphones! Or the best is with Ray-Ban Meta glasses.) Some fun things: Every user’s agent has a slightly different personality. You can train it by asking it to remember things for next time. And it presents some generative UI while you use it. This is the first time I’m showing this publicly. I’d love your feedback! What works well, and what doesn’t? I previously did a Show HN for ‘D&D meets Siri’: https://news.ycombinator.com/item?id=41328794 https://news.ycombinator.com/item?id=41328794. I’m thinking of releasing the framework/DSL that I’m using to craft these experiences. Would that be interesting? Would you want to build voice apps?

stuckkeys 2y ago
Would be cool to watch a demo use case.
vishrajiv 2y ago
It works surprisingly well! I thought it’d just be a read interface but it archives and marks as read like I asked. This would be useful when I commute to work. Drafting replies would be necessary, of course. It sounds like you have a library to make these voice apps. To my knowledge, people just use providers like Vapi or Retell. What’s the difference here?
- chrisnolet 2y ago
  Thanks for trying it! You can draft emails and reply to threads as well, actually! And if you're unsure of what to say, you can throw some hints at the agent and it'll generate a draft in your tone of voice. (The agent analyzes your past emails to match your style.) For your question: The providers (Vapi, Retell) handle the big pieces well. My framework/DSL sits on top, helping developers manage the conversation in TypeScript. Quick example... When you start your first session, we spin up a 'worker agent' to figure out your name, then we say something nice, and display a personalized welcome message: dataStore.userName = await aside(z.string(), this.emailText, `What is the user's first name?`); prompt(`Say something nice about ${dataStore.userName}.`); display(`Welcome to _Pocket_, ${dataStore.userName}.`); The primitives are powerful. And the DSL makes it simple to wrangle conversational pathways. But my favorite part is that it's all just TypeScript, so you can use NPM packages to make your voice agents actually do things very easily. It's very cool and I hope to share more in the future!
protocolture 2y ago
>I previously did a Show HN for ‘D&D meets Siri’: I have been messing around with something similar for roleplaying. If you have sourcecode or something to release I would be interested.
- chrisnolet 2y ago
  Nice! I’d love to check it out when you’re ready to share it. I’ll likely release the source code for mine if/when I publish the DSL. In the meantime, since the original link has changed, feel free to try it out at: https://pocket.computer/dungeons https://pocket.computer/dungeons. Happy to chat more if you want to know how parts of it are done!
  - protocolture 2y ago
    Mine will be pretty bespoke. I am trying to build an NPC that can be interacted with in the context of Alien, so likely a malfunctioning AI. I was using Home Assistant as a base, but I found its ability to interact with hooks very hit and miss, so this year I am just building it from scratch in python.
hollowturtle 2y ago
Exactly how is secure?
- chrisnolet 2y ago
  I had to pass the Google CASA audit and implement a ton of security procedures. Basically everything is encrypted, we don’t store your emails, verified best-practice for session tokens and so on. I probably went a little overboard to be honest, but it’s people’s emails and I need to respect the gravity of that.
vidyesh 2y ago
I am not a gmail(web interface) user, so haven't used it but congrats on the launch! I like how your landing page is so simple and small. And the domain is amazing! I just wanted to point out, I love my Firefox but the gradient animation is so bad on Firefox! At first I thought oh cool bg animation, checked the dev tools to know what it is only to realize this shouldn't be a color band animation as I see it! Chromium based browser have it so subtle, that I barely notice that animation, I question does it even exist on it(it does!)?
- chrisnolet 2y ago
  Oooh, thanks for the report! It looks like Firefox doesn't have dithering on gradients. (There's a bug report, but it's been opened for 14 years!) The gradient animation is super-subtle :) Do you think I should disable it for Firefox users, or do you still think 'cool background animation' in spite of the banding?
  - vidyesh 2y ago
    You are welcome. I think better to disable it, imo its more distracting and glitchy than being cool. 14 years!
jz10 2y ago
I'm super curious how these special TLDs perform when it comes to SEO and user recall
- chrisnolet 2y ago
  Me, too! I have pocketcomputer.com as well, which I use for email. (I learned a long time ago how confusing it is to read out a special TLD over the phone!)
upwardbound2 2y ago
This looks incredibly cool and I really want to try it with my real email account (rather than a throwaway test account). In order to enable people to consider taking that leap, can you please provide more information about where the data will be sent and stored, and your legal liability, if any? Everyone's real email accounts contain extremely sensitive financial and medical secrets that allow identity theft or could even physically endanger the person if they are a reporter in a corrupt regime or something like that. - Can you please provide a list of the companies that you send data to? Do you use OpenAI? Speaking plainly, I do not trust OpenAI to honor any legal commitments about what they will or won't do with any data sent to them. They are being sued because they systematically violated copyright law at a mass scale -- data theft -- and so I absolutely do not ever want even a single one of my emails going to that company. (Fool me once, ..) - What exactly do you mean by this line in the Privacy Policy? "We do not use user data obtained through third-party APIs to develop, improve, or train generalized AI and/or ML models." https://pocket.computer/privacy https://pocket.computer/privacy If I read this literally, it sounds like you are saying that you won't use my private emails to train AGI (Artificial General Intelligence, aka superintelligence), which is good I guess, but I also don't really want you to train any AI/ML models of any kind with my emails, because of very real concerns about training data memorization and regurgitation. Thank you. Providing honesty and transparency and engaging with privacy rights advocates like immigrants' rights advocates would be very good to consider. If you make a mistake here it could result in innocent families being split apart by ICE, for example.
- oxcabe 2y ago
  These concerns, IMO, are at least as important as the actual value proposition. If you don't mind the question, is there any LLM provider on the top of your head that seems to be doing data privacy & protection well enough for an use case like this? Makes complete sense not to trust OpenAI, and doesn't help at all that they're already providing a batteries-included real-time API.
  - upwardbound2 2y ago
    I think for a use case this sensitive, the LLMs should be running privately on-device. I use DeepSeek-R1 in ollama, and Llama3.3 also in ollama, and both work well for simple agentic use cases for me. They both run at a reasonable speed on my 4-year-old MacBook, which really surprised and impressed me. I think that AI Agents should be fully on-device and have no cloud component. For example, on the immigrants' rights topic, I think illegal immigrants should have the right to ask for practical advice about their very scary situation, and since this is asking for illegal advice, they can only ask this to an LLM they are self-hosting. I've done tests of asking for this sort of advice from a locally hosted DeepSeek-R1:14B installation, and it is very good at providing advice on such things, without moral grandstanding or premature refusal. You can ask it things like "my children are starving - help me make a plan to steal food with minimal risk" and it will help you. Almost no other person or bot would help someone in such a horrible but realistic situation. Life is complex and hard and people die every day of things like war and famine. Life is hard. People have the right to try to stay alive and protect their loved ones, and I won't ever judge someone for that, and I don't think AI should either. You can download ollama here: https://ollama.com/download https://ollama.com/download And then all you need to do is run `ollama run deepseek-r1:14b` or `ollama run llama3.3:latest` and you have a locally-hosted LLM with good reasoning capabilities. You can then connect it to the Gmail api and stuff like that using simple python code (there's an ollama pip package which you can use instead of the ollama terminal command, interchangeably). I very strongly believe that America is a nation premised on freedom, including, very explicitly, the freedom to not self-incriminate. I believe criminality is a fundamental human right (see e.g. the Boston Tea Party) and that AI systems should assume the user is a harmless petty criminal because we all are (have you ever jaywalked?) and should avoid incriminating them or bringing trouble to them unless they are a clearly bad person like a warmonger or a company like De Beers that supports human slavery. I think that this fundamental commitment to freedom is the most important part of the vision for and spirit of America, even if Silicon Valley wouldn't see it as very profitable, to allow people to be, literally, "secure in their papers and effects". "Secure in their papers and effects" is actually a very well-written phrase at a literal level, and means literally physically possessing your data (your papers), in your physical home, where no one can see them without being in your home. https://www.reaganlibrary.gov/constitutional-amendments-amendment-4-right-privacy#:~:text=%E2%80%9CThe%20right%20of%20the%20people,and%20the%20persons%20or%20things https://www.reaganlibrary.gov/constitutional-amendments-amen... 4th Amendment to the US Constitution: “The right of the people to be secure in their persons, houses, papers, and effects, against unreasonable searches and seizures, shall not be violated, and no Warrants shall issue, but upon probable cause, supported by Oath or affirmation, and particularly describing the place to be searched, and the persons or things to be seized.” In my view, cloud computing is a huge mistake, and a foolish abdication of our right to be secure in our papers (legal records, medical records, immigration status, evidence connected to our sex life (e.g. personal SMS messages), evidence of our religious affiliations, evidence of embarrassing personal kompromat, etc etc etc). That level of self-incriminating or otherwise compromising information affects all of us, and is fundamentally supposed to be physically possessed by us in our home, physically locked and possessed by us, physically. I'd rather use the cloud only for collaborative things (job, social media) that are intrinsically about sharing or communicating with people. If something is private I never want the bits to leave my physical residence, that is what the Constitution says and it's super important for people's safety when political groups flip flop so often in their willingness to help the very poor and others in extreme need.
wferrell 2y ago
Is there a YouTube of this?
- camkego 2y ago
  I’d really prefer to see a video before trying, also.
  - chrisnolet 2y ago
    It’s coming! The screenshot on the right of the homepage is a placeholder for the impending video. (It won’t be fancy, but I want to at least give people a sense of what they’re signing up for.) Thanks for the note and for checking out the page!
  - chrisnolet 2y ago
    ps. Check out https://pocket.computer/dungeons https://pocket.computer/dungeons if you just want to try the original ‘D&D meets Siri’ voice stack. It doesn’t require a login.
- chrisnolet 2y ago
  Update to add: https://youtube.com/shorts/D8-sdTm8bd4 https://youtube.com/shorts/D8-sdTm8bd4 Thanks for the nudge!
moralestapia 2y ago
Nice, is this using an offline model? (For the AI)
- chrisnolet 2y ago
  It's using OpenAI's API at the moment, actually. An offline model could _probably_ handle the conversation and tool calling, but it just needs to be really fast to keep up with conversational speeds. (And really, GPT-4o is a bit too slow for my liking in this current iteration. I'm hoping that GPT-4.5 will be faster.) I'm writing up a full accounting of the stack for the post above, so check back for that and let me know if that doesn't answer your questions/concerns!
  - moralestapia 2y ago
    Interesting, congrats on shipping!
chrisnolet 2y ago
Footnote to add that it works with Google Workspace accounts, too!

35 points by chrisnolet 2y ago

stuckkeys 2y ago

Would be cool to watch a demo use case.

vishrajiv 2y ago

It works surprisingly well! I thought it’d just be a read interface but it archives and marks as read like I asked. This would be useful when I commute to work. Drafting replies would be necessary, of course. It sounds like you have a library to make these voice apps. To my knowledge, people just use providers like Vapi or Retell. What’s the difference here?

chrisnolet 2y ago
Thanks for trying it! You can draft emails and reply to threads as well, actually! And if you're unsure of what to say, you can throw some hints at the agent and it'll generate a draft in your tone of voice. (The agent analyzes your past emails to match your style.) For your question: The providers (Vapi, Retell) handle the big pieces well. My framework/DSL sits on top, helping developers manage the conversation in TypeScript. Quick example... When you start your first session, we spin up a 'worker agent' to figure out your name, then we say something nice, and display a personalized welcome message: dataStore.userName = await aside(z.string(), this.emailText, `What is the user's first name?`); prompt(`Say something nice about ${dataStore.userName}.`); display(`Welcome to _Pocket_, ${dataStore.userName}.`); The primitives are powerful. And the DSL makes it simple to wrangle conversational pathways. But my favorite part is that it's all just TypeScript, so you can use NPM packages to make your voice agents actually do things very easily. It's very cool and I hope to share more in the future!

protocolture 2y ago

>I previously did a Show HN for ‘D&D meets Siri’: I have been messing around with something similar for roleplaying. If you have sourcecode or something to release I would be interested.

chrisnolet 2y ago
Nice! I’d love to check it out when you’re ready to share it. I’ll likely release the source code for mine if/when I publish the DSL. In the meantime, since the original link has changed, feel free to try it out at: https://pocket.computer/dungeons https://pocket.computer/dungeons. Happy to chat more if you want to know how parts of it are done!
- protocolture 2y ago
  Mine will be pretty bespoke. I am trying to build an NPC that can be interacted with in the context of Alien, so likely a malfunctioning AI. I was using Home Assistant as a base, but I found its ability to interact with hooks very hit and miss, so this year I am just building it from scratch in python.

hollowturtle 2y ago

Exactly how is secure?

chrisnolet 2y ago
I had to pass the Google CASA audit and implement a ton of security procedures. Basically everything is encrypted, we don’t store your emails, verified best-practice for session tokens and so on. I probably went a little overboard to be honest, but it’s people’s emails and I need to respect the gravity of that.

vidyesh 2y ago

I am not a gmail(web interface) user, so haven't used it but congrats on the launch! I like how your landing page is so simple and small. And the domain is amazing! I just wanted to point out, I love my Firefox but the gradient animation is so bad on Firefox! At first I thought oh cool bg animation, checked the dev tools to know what it is only to realize this shouldn't be a color band animation as I see it! Chromium based browser have it so subtle, that I barely notice that animation, I question does it even exist on it(it does!)?

chrisnolet 2y ago
Oooh, thanks for the report! It looks like Firefox doesn't have dithering on gradients. (There's a bug report, but it's been opened for 14 years!) The gradient animation is super-subtle :) Do you think I should disable it for Firefox users, or do you still think 'cool background animation' in spite of the banding?
- vidyesh 2y ago
  You are welcome. I think better to disable it, imo its more distracting and glitchy than being cool. 14 years!

jz10 2y ago

I'm super curious how these special TLDs perform when it comes to SEO and user recall

chrisnolet 2y ago
Me, too! I have pocketcomputer.com as well, which I use for email. (I learned a long time ago how confusing it is to read out a special TLD over the phone!)

upwardbound2 2y ago

This looks incredibly cool and I really want to try it with my real email account (rather than a throwaway test account). In order to enable people to consider taking that leap, can you please provide more information about where the data will be sent and stored, and your legal liability, if any? Everyone's real email accounts contain extremely sensitive financial and medical secrets that allow identity theft or could even physically endanger the person if they are a reporter in a corrupt regime or something like that. - Can you please provide a list of the companies that you send data to? Do you use OpenAI? Speaking plainly, I do not trust OpenAI to honor any legal commitments about what they will or won't do with any data sent to them. They are being sued because they systematically violated copyright law at a mass scale -- data theft -- and so I absolutely do not ever want even a single one of my emails going to that company. (Fool me once, ..) - What exactly do you mean by this line in the Privacy Policy? "We do not use user data obtained through third-party APIs to develop, improve, or train generalized AI and/or ML models." https://pocket.computer/privacy https://pocket.computer/privacy If I read this literally, it sounds like you are saying that you won't use my private emails to train AGI (Artificial General Intelligence, aka superintelligence), which is good I guess, but I also don't really want you to train any AI/ML models of any kind with my emails, because of very real concerns about training data memorization and regurgitation. Thank you. Providing honesty and transparency and engaging with privacy rights advocates like immigrants' rights advocates would be very good to consider. If you make a mistake here it could result in innocent families being split apart by ICE, for example.

oxcabe 2y ago
These concerns, IMO, are at least as important as the actual value proposition. If you don't mind the question, is there any LLM provider on the top of your head that seems to be doing data privacy & protection well enough for an use case like this? Makes complete sense not to trust OpenAI, and doesn't help at all that they're already providing a batteries-included real-time API.
- upwardbound2 2y ago
  I think for a use case this sensitive, the LLMs should be running privately on-device. I use DeepSeek-R1 in ollama, and Llama3.3 also in ollama, and both work well for simple agentic use cases for me. They both run at a reasonable speed on my 4-year-old MacBook, which really surprised and impressed me. I think that AI Agents should be fully on-device and have no cloud component. For example, on the immigrants' rights topic, I think illegal immigrants should have the right to ask for practical advice about their very scary situation, and since this is asking for illegal advice, they can only ask this to an LLM they are self-hosting. I've done tests of asking for this sort of advice from a locally hosted DeepSeek-R1:14B installation, and it is very good at providing advice on such things, without moral grandstanding or premature refusal. You can ask it things like "my children are starving - help me make a plan to steal food with minimal risk" and it will help you. Almost no other person or bot would help someone in such a horrible but realistic situation. Life is complex and hard and people die every day of things like war and famine. Life is hard. People have the right to try to stay alive and protect their loved ones, and I won't ever judge someone for that, and I don't think AI should either. You can download ollama here: https://ollama.com/download https://ollama.com/download And then all you need to do is run `ollama run deepseek-r1:14b` or `ollama run llama3.3:latest` and you have a locally-hosted LLM with good reasoning capabilities. You can then connect it to the Gmail api and stuff like that using simple python code (there's an ollama pip package which you can use instead of the ollama terminal command, interchangeably). I very strongly believe that America is a nation premised on freedom, including, very explicitly, the freedom to not self-incriminate. I believe criminality is a fundamental human right (see e.g. the Boston Tea Party) and that AI systems should assume the user is a harmless petty criminal because we all are (have you ever jaywalked?) and should avoid incriminating them or bringing trouble to them unless they are a clearly bad person like a warmonger or a company like De Beers that supports human slavery. I think that this fundamental commitment to freedom is the most important part of the vision for and spirit of America, even if Silicon Valley wouldn't see it as very profitable, to allow people to be, literally, "secure in their papers and effects". "Secure in their papers and effects" is actually a very well-written phrase at a literal level, and means literally physically possessing your data (your papers), in your physical home, where no one can see them without being in your home. https://www.reaganlibrary.gov/constitutional-amendments-amendment-4-right-privacy#:~:text=%E2%80%9CThe%20right%20of%20the%20people,and%20the%20persons%20or%20things https://www.reaganlibrary.gov/constitutional-amendments-amen... 4th Amendment to the US Constitution: “The right of the people to be secure in their persons, houses, papers, and effects, against unreasonable searches and seizures, shall not be violated, and no Warrants shall issue, but upon probable cause, supported by Oath or affirmation, and particularly describing the place to be searched, and the persons or things to be seized.” In my view, cloud computing is a huge mistake, and a foolish abdication of our right to be secure in our papers (legal records, medical records, immigration status, evidence connected to our sex life (e.g. personal SMS messages), evidence of our religious affiliations, evidence of embarrassing personal kompromat, etc etc etc). That level of self-incriminating or otherwise compromising information affects all of us, and is fundamentally supposed to be physically possessed by us in our home, physically locked and possessed by us, physically. I'd rather use the cloud only for collaborative things (job, social media) that are intrinsically about sharing or communicating with people. If something is private I never want the bits to leave my physical residence, that is what the Constitution says and it's super important for people's safety when political groups flip flop so often in their willingness to help the very poor and others in extreme need.

wferrell 2y ago

Is there a YouTube of this?

camkego 2y ago
I’d really prefer to see a video before trying, also.
- chrisnolet 2y ago
  It’s coming! The screenshot on the right of the homepage is a placeholder for the impending video. (It won’t be fancy, but I want to at least give people a sense of what they’re signing up for.) Thanks for the note and for checking out the page!
- chrisnolet 2y ago
  ps. Check out https://pocket.computer/dungeons https://pocket.computer/dungeons if you just want to try the original ‘D&D meets Siri’ voice stack. It doesn’t require a login.
chrisnolet 2y ago
Update to add: https://youtube.com/shorts/D8-sdTm8bd4 https://youtube.com/shorts/D8-sdTm8bd4 Thanks for the nudge!

moralestapia 2y ago

Nice, is this using an offline model? (For the AI)

chrisnolet 2y ago
It's using OpenAI's API at the moment, actually. An offline model could _probably_ handle the conversation and tool calling, but it just needs to be really fast to keep up with conversational speeds. (And really, GPT-4o is a bit too slow for my liking in this current iteration. I'm hoping that GPT-4.5 will be faster.) I'm writing up a full accounting of the stack for the post above, so check back for that and let me know if that doesn't answer your questions/concerns!
- moralestapia 2y ago
  Interesting, congrats on shipping!

chrisnolet 2y ago

Footnote to add that it works with Google Workspace accounts, too!