10 ms·

Show HN: Search your Twitter history with Algolia

171 points by transitivebs 7y ago

transitivebs 7y ago
The idea for this product came last night from Guillermo Rauch tweeting that he wished there was a service to search your personal Twitter history (tweets, likes, retweets, filters, etc) using an Algolia-type search UI (super fast search as you type). A whole bunch of people echoed that they'd pay for this type of product. So I decided to build it overnight and open source it. The majority of the functionality is handled by Algolia and Saasify which is the only reason this was possible to build so quickly. Would love to hear people's thoughts - thanks!
- mraza007 7y ago
  Hey your UI is Pretty cool what front end framework did you use to build it
  - transitivebs 7y ago
    The base webapp template uses https://saasify.sh https://saasify.sh (which uses CRA and antd under the hood). The core functionality when logged in is written in React using Chakra UI.
- kreetx 7y ago
  What does Saasify do? I browsed their website (https://www.saasify.net https://www.saasify.net) and don't quite get it - how did they help you?
  - pdimitar 7y ago
    They are using https://saasify.sh/ https://saasify.sh/.
nzeeshan 7y ago
This is a neat feature! Kudos How about allowing to add other's twitter handle and searching their TL?
- transitivebs 7y ago
  Yeah, I really like this idea. Given that the initial version was built in 8 hours, I'm sure there are a lot of features that I'll be adding over the next few weeks :) I'm thinking this would make sense as a paid feature?
- Timac 7y ago
  If you want to monitor keywords on Twitter and not just in your timeline, you might be interested to check the iOS app Clatters : https://apps.apple.com/us/app/clatters/id1480930237 https://apps.apple.com/us/app/clatters/id1480930237 Full disclosure : I'm the developer That being said, I am really impressed by this new service. Creating such a solution in just a couple of hours/days is astonishing.
  - transitivebs 7y ago
    Thanks, Timac. You can read more about how I built Twitter Instant Search here: https://www.indiehackers.com/post/launched-a-solid-saas-in-8-hours-574cb5282b https://www.indiehackers.com/post/launched-a-solid-saas-in-8...
jchook 7y ago
Twitter only stores your last 3200 likes. What a bummer. I like a lot of very funny or informative stuff on twitter and the tweets often pop into my head even years later. Sometimes I wanna find that tweet (eg to share) and I can’t. I started using pinboard.in, which archives all my likes, but the search suuuucks. So, I started building my own archive tool, but the search part is still hard. I have looked into annoy, perkeep, tf-idf, Doc2Vec, and various other tools, but nothing really works as well as I had hoped. Your project definitely gave me some ideas! Thanks.
- pgt 7y ago
  Wow, I did not know that. This is such a no brainer paid feature for a premium Twitter plan: storage & search. I suspect they probably keep all likes, but do not expose them. Another premium service that searches your Twitter like is https://www.favourites.io/ https://www.favourites.io/, but it's too costly for me. I might pay $10 for a yearly subscription, but not 29 GBP/year, which is ~10% of my monthly rent.
  - Kinrany 7y ago
    If they do keep all the data, you should be able to get it via GDPR
    - transitivebs 7y ago
      Yeah, you can access all the data of your own account, but third-party apps accessing your data via OAuth is another issue. In the case of Twitter's API, third-party apps can access all of your history if you upgrade to a paid partnership plan.
- dewey 7y ago
  > I started using pinboard.in, which archives all my likes, but the search suuuucks. What's also annoying is that I don't think there's a way to exclude from_twitter bookmarks in Pinboard when you search. So now that I enabled this feature a while ago everything is cluttered with tweets and Pinboard become useless for me. Something like a -from_twitter search would be helpful but there's only a "+" search right now.
harlanji 7y ago
Nice job. Had a similar idea after seeing the same Tweet, but based around importing the downloaded data archive, as I’m working with that myself now. Open source, so I might see if I can weave it in when I get a chance, if nobody else does it first.
- transitivebs 7y ago
  Yeah, I'm definitely down to collaborate -- feel free to reach out via email and I'm happy to share any profits if you add features to the open source repo.
h43z 7y ago
I use the standard twitter search a lot to find some of my own tweets. Mostly a "from:@h43z term" does the job for me. Though it's not instant twitter provides many useful filters you may want to check out (https://twitter.com/h43z/status/1234614509400215554 https://twitter.com/h43z/status/1234614509400215554)
- transitivebs 7y ago
  Will definitely look into adding support for some of these common filters. Thanks for the heads up!
stevenpetryk 7y ago
Seems this was posted primarily to market Saasify, so I'll take the bait: pretty cool that this was set up so fast. You've built a neat thing, Travis :)
- transitivebs 7y ago
  Hahaha you got me :) Thanks, Steven!
  - imdsm 7y ago
    But hey, Saasify looks neat! I'm interested for sure
iamaelephant 7y ago
You mean that insanely shitty tool that powers HN search? No thanks.
sneak 7y ago
Strange to think that building an actual web spider to make an independent index of public tweets is probably illegal these days. This is basically a frontend to Twitter’s own datastore via the API.
- transitivebs 7y ago
  Yeah; it's just strange that this sort of thing is necessary given the resources that Twitter has available.
101008 7y ago
Does anyone know a tool to download all tweets from an account in JSON format? No need to include likes or RTs. In fct, better if I can avoid those.
- dorkinspace 7y ago
  You could make a tool pretty easily using Twitter's api: Get started here: https://developer.twitter.com/en/docs/tweets/curate-a-collection/api-reference/get-collections-list https://developer.twitter.com/en/docs/tweets/curate-a-collec...
  - transitivebs 7y ago
    Unfortunately, Twitter limits the number of historical tweets they allow access to from their API (3200 iirc). You have to pay for a partner account in order to have full API access.
- simonw 7y ago
  If it's your own account the API will let you fetch everything. If it's someone else's account the API limits you to around 3,200 tweets. I built https://github.com/dogsheep/twitter-to-sqlite https://github.com/dogsheep/twitter-to-sqlite tool for archiving my own tweets. I've used it to pull tweets from other interesting accounts too.
anw 7y ago
I'm very happy to see a tool like this that can be self-hosted. Back around 2010 there was a company called Topsy[0] who was supposed to be the "Google of Social Media". It would index Twitter tweets and Google+ posts. The idea was you could read discussions over the years and find trends that could be used for research, analytics or marketing. Sadly, it was bought up by Apple who shut it down a year or so later. There is still a giant hole left around search for Social Media, with each of them being their own walled garden. [0] https://en.wikipedia.org/wiki/Topsy_Labs https://en.wikipedia.org/wiki/Topsy_Labs
- molsongolden 7y ago
  Gnip was also working in a similar space before being acquired by Twitter in 2014. https://en.wikipedia.org/wiki/Gnip https://en.wikipedia.org/wiki/Gnip
bastijn 7y ago
You should put a license in that repo. Witbout a license, the default copyright laws apply, meaning that you retain all rights to your source code and no one may reproduce, distribute, or create derivative works from your work. If you're creating an open source project, it is strongly encouraged to include an open source license.
- transitivebs 7y ago
  The readme clearly states the MIT license at the bottom. Are you referring to adding a separate license.md file?
- transitivebs 7y ago
  Just added a license file.
  - bastijn 7y ago
    Thanks!
gge 7y ago
Can I use this to search other peoples tweet history?
- transitivebs 7y ago
  This is a feature that we'll be adding shortly.
zelly 7y ago
from:my_username my query Twitter dot com slash search

171 points by transitivebs 7y ago

The idea for this product came last night from Guillermo Rauch tweeting that he wished there was a service to search your personal Twitter history (tweets, likes, retweets, filters, etc) using an Algolia-type search UI (super fast search as you type). A whole bunch of people echoed that they'd pay for this type of product. So I decided to build it overnight and open source it. The majority of the functionality is handled by Algolia and Saasify which is the only reason this was possible to build so quickly. Would love to hear people's thoughts - thanks!

mraza007 7y ago
Hey your UI is Pretty cool what front end framework did you use to build it
- transitivebs 7y ago
  The base webapp template uses https://saasify.sh https://saasify.sh (which uses CRA and antd under the hood). The core functionality when logged in is written in React using Chakra UI.
kreetx 7y ago
What does Saasify do? I browsed their website (https://www.saasify.net https://www.saasify.net) and don't quite get it - how did they help you?
- pdimitar 7y ago
  They are using https://saasify.sh/ https://saasify.sh/.

nzeeshan 7y ago

This is a neat feature! Kudos How about allowing to add other's twitter handle and searching their TL?

transitivebs 7y ago
Yeah, I really like this idea. Given that the initial version was built in 8 hours, I'm sure there are a lot of features that I'll be adding over the next few weeks :) I'm thinking this would make sense as a paid feature?
Timac 7y ago
If you want to monitor keywords on Twitter and not just in your timeline, you might be interested to check the iOS app Clatters : https://apps.apple.com/us/app/clatters/id1480930237 https://apps.apple.com/us/app/clatters/id1480930237 Full disclosure : I'm the developer That being said, I am really impressed by this new service. Creating such a solution in just a couple of hours/days is astonishing.
- transitivebs 7y ago
  Thanks, Timac. You can read more about how I built Twitter Instant Search here: https://www.indiehackers.com/post/launched-a-solid-saas-in-8-hours-574cb5282b https://www.indiehackers.com/post/launched-a-solid-saas-in-8...

jchook 7y ago

Twitter only stores your last 3200 likes. What a bummer. I like a lot of very funny or informative stuff on twitter and the tweets often pop into my head even years later. Sometimes I wanna find that tweet (eg to share) and I can’t. I started using pinboard.in, which archives all my likes, but the search suuuucks. So, I started building my own archive tool, but the search part is still hard. I have looked into annoy, perkeep, tf-idf, Doc2Vec, and various other tools, but nothing really works as well as I had hoped. Your project definitely gave me some ideas! Thanks.

pgt 7y ago
Wow, I did not know that. This is such a no brainer paid feature for a premium Twitter plan: storage & search. I suspect they probably keep all likes, but do not expose them. Another premium service that searches your Twitter like is https://www.favourites.io/ https://www.favourites.io/, but it's too costly for me. I might pay $10 for a yearly subscription, but not 29 GBP/year, which is ~10% of my monthly rent.
- Kinrany 7y ago
  If they do keep all the data, you should be able to get it via GDPR
  - transitivebs 7y ago
    Yeah, you can access all the data of your own account, but third-party apps accessing your data via OAuth is another issue. In the case of Twitter's API, third-party apps can access all of your history if you upgrade to a paid partnership plan.
dewey 7y ago
> I started using pinboard.in, which archives all my likes, but the search suuuucks. What's also annoying is that I don't think there's a way to exclude from_twitter bookmarks in Pinboard when you search. So now that I enabled this feature a while ago everything is cluttered with tweets and Pinboard become useless for me. Something like a -from_twitter search would be helpful but there's only a "+" search right now.

harlanji 7y ago

Nice job. Had a similar idea after seeing the same Tweet, but based around importing the downloaded data archive, as I’m working with that myself now. Open source, so I might see if I can weave it in when I get a chance, if nobody else does it first.

transitivebs 7y ago
Yeah, I'm definitely down to collaborate -- feel free to reach out via email and I'm happy to share any profits if you add features to the open source repo.

h43z 7y ago

I use the standard twitter search a lot to find some of my own tweets. Mostly a "from:@h43z term" does the job for me. Though it's not instant twitter provides many useful filters you may want to check out (https://twitter.com/h43z/status/1234614509400215554 https://twitter.com/h43z/status/1234614509400215554)

transitivebs 7y ago
Will definitely look into adding support for some of these common filters. Thanks for the heads up!

stevenpetryk 7y ago

Seems this was posted primarily to market Saasify, so I'll take the bait: pretty cool that this was set up so fast. You've built a neat thing, Travis :)

transitivebs 7y ago
Hahaha you got me :) Thanks, Steven!
- imdsm 7y ago
  But hey, Saasify looks neat! I'm interested for sure

iamaelephant 7y ago

You mean that insanely shitty tool that powers HN search? No thanks.

sneak 7y ago

Strange to think that building an actual web spider to make an independent index of public tweets is probably illegal these days. This is basically a frontend to Twitter’s own datastore via the API.

transitivebs 7y ago
Yeah; it's just strange that this sort of thing is necessary given the resources that Twitter has available.

101008 7y ago

Does anyone know a tool to download all tweets from an account in JSON format? No need to include likes or RTs. In fct, better if I can avoid those.

dorkinspace 7y ago
You could make a tool pretty easily using Twitter's api: Get started here: https://developer.twitter.com/en/docs/tweets/curate-a-collection/api-reference/get-collections-list https://developer.twitter.com/en/docs/tweets/curate-a-collec...
- transitivebs 7y ago
  Unfortunately, Twitter limits the number of historical tweets they allow access to from their API (3200 iirc). You have to pay for a partner account in order to have full API access.
simonw 7y ago
If it's your own account the API will let you fetch everything. If it's someone else's account the API limits you to around 3,200 tweets. I built https://github.com/dogsheep/twitter-to-sqlite https://github.com/dogsheep/twitter-to-sqlite tool for archiving my own tweets. I've used it to pull tweets from other interesting accounts too.

anw 7y ago

I'm very happy to see a tool like this that can be self-hosted. Back around 2010 there was a company called Topsy[0] who was supposed to be the "Google of Social Media". It would index Twitter tweets and Google+ posts. The idea was you could read discussions over the years and find trends that could be used for research, analytics or marketing. Sadly, it was bought up by Apple who shut it down a year or so later. There is still a giant hole left around search for Social Media, with each of them being their own walled garden. [0] https://en.wikipedia.org/wiki/Topsy_Labs https://en.wikipedia.org/wiki/Topsy_Labs

molsongolden 7y ago
Gnip was also working in a similar space before being acquired by Twitter in 2014. https://en.wikipedia.org/wiki/Gnip https://en.wikipedia.org/wiki/Gnip

bastijn 7y ago

You should put a license in that repo. Witbout a license, the default copyright laws apply, meaning that you retain all rights to your source code and no one may reproduce, distribute, or create derivative works from your work. If you're creating an open source project, it is strongly encouraged to include an open source license.

transitivebs 7y ago
The readme clearly states the MIT license at the bottom. Are you referring to adding a separate license.md file?
transitivebs 7y ago
Just added a license file.
- bastijn 7y ago
  Thanks!

gge 7y ago

Can I use this to search other peoples tweet history?

transitivebs 7y ago
This is a feature that we'll be adding shortly.

zelly 7y ago

from:my_username my query Twitter dot com slash search