6 ms·

Hey everyone, Josh here the creator. As a developer, I’m always looking out for an emerging market trend on which to bootstrap my own SaaS product. This is sinc

by jhow15 7y ago

Hey everyone, Josh here the creator. As a developer, I’m always looking out for an emerging market trend on which to bootstrap my own SaaS product. This is since market awareness and timing are often so critical, on top of execution skill of course! So I built Trennd. Under the hood it continually monitors the web for interesting keywords & topics, classifies them using Google Trends data and packages everything as a neat web app where other people can contribute too. The app itself is built with the Next.js React framework along with Express, Bootstrap and MongoDB. Next.js was new to me, but made sense since it comes with so much out of the box, including server side rendering. Let me know if you have any questions, and any feedback is much appreciated!

deadcoder0904 7y ago
You said you are monitoring interesting keywords & topics but how do you know which keywords & topics to monitor? Also, if you are choosing the keywords then wouldn't the data be inaccurate for predicting trends as trends occur randomly?
- jhow15 7y ago
  I have scripts that are crawling the web in places like twitter, reddit and facebook to generate keyword ideas. So the trends that it detects is biased towards the corners of the web that I am monitoring. Which skew heavily tech!
  - mooreds 7y ago
    This is a great service. It would be great for you to take suggestions on other locations to crawl.
    - jhow15 7y ago
      That's a brilliant idea, will add somewhere for people to make suggestions. Thank you!
maxencecornet 7y ago
What could be a really neat addition is having trends sorted by countries
- jhow15 7y ago
  Great idea, thank you!
omarhaneef 7y ago
Nice work Josh. It can probably be used for lots of things: startups, as you mentioned, but also journalism ideas, investing ideas, job hunting and many more.
- jhow15 7y ago
  Thank you! Yeah totally agree, I hadn't thought of job hunting though!!
nurettin 7y ago
Hi, it would be nice if the keywords had short explanations. I had to find out what an affinity designer and an enneagram was. Bubble tea was also quite new.
yuy910616 7y ago
Hi Josh, Very interesting stuff. Can I ask what is the math behind defining the trend? How did you calculate the most trendy words, etc? I'm working on something similar for a internal tool, so I'm curious to see what your solution is. Thanks!
- jhow15 7y ago
  Of course - I have scripts constantly monitoring the web for potentially interesting keywords. I then check google trends data for that topic/keyword and analyze the gradient of the line of best fit over different timeframes. If the gradient is higher than a certain (arbitrary) amount, then I class it as a trend!
  - peterhil 7y ago
    Is the gradient just a linear least squares fit?
    - yuy910616 7y ago
      Not the author. Seems like it, it doesn't look like Google Trend gives other variables, so just simple 2 var regression I'd guess
charlesism 7y ago
I don’t have much feedback, except to say I think this is a really good, useful concept.
- jhow15 7y ago
  Thanks so much!
simonebrunozzi 7y ago
Hey Josh, very interesting concept. Thanks for building it! I tried to use the "technology" category, but the results are not what I expected. Perhaps it would be more useful if I could narrow it down to more specific things, e.g. programming languages, or databases. Keep up the good work!
gondo 7y ago
would be interesting to also to see the opposite, retreating trends
- vernie 7y ago
  It seems to already have that (the "gradient" arrow)
ashelmire 7y ago
These results aren't filtered in some way? I find it suspicious that almost all of the top ~100 results are tech related.
- concert-gilled 7y ago
  > I have scripts that are crawling the web in places like twitter, reddit and facebook to generate keyword ideas. So the trends that it detects is biased towards the corners of the web that I am monitoring. Which skew heavily tech!
markdown 7y ago
OT but... > built with the Next.js React framework Wait wait wait... do you mean to say there's now a framework for a framework? Javascriptland continues to confound.
- hashbig 7y ago
  I wouldn't call React a framework, it's more like a library.
tasssko 7y ago
Awesome I enjoyed making some trend charts, and the results are fun (i.e I added DevOps, Erlang, Elixir). I think there is a risk it could conceal too many errors and biases for it to be useful for trend analysis. Using services like Twitter and Reddit might introduce biases into your data skewing trends. The challenge is using data and not understanding the intent (is it positive, negative or neutral/sarcastic potentially negative/positive). Google Adwords is good at this. I applaud your effort; I hope you keep going but its a tough nut to crack. I use markets like Google Adwords Tools to do this at the moment, have you looked at it? I think more info around time, intent and volume.
kevinguh 7y ago
Pretty neat, I see the most granular time scale currently supported is a 3-month window -- are there any plans to increase granularity towards a (near) real-time system? I get your crawler might be rate-limited at that scale and you would likely have to rethink the entire system architecture, but I think real-time data would open up a whole new dimension of possibilities for detecting/predicting trends perhaps even before public awareness catches up. I'd imagine there would be parties interested in paying for that kind of capability if that's the direction you're planning on taking this idea
- jhow15 7y ago
  I'd love to go as granular as possible, at least 30 days. Real-time would be ideal, but as you note - Google Trends rate limiting soon becomes a hurdle especially when you potentially multiply by different geographies.
taneem 7y ago
Hi Josh - it would be incredible to have this filtered by location, and I think you can get that from Google Trends.
- jhow15 7y ago
  Absolutely agree! How specific/granular would be the most valuable to you - at the city level perhaps?
  - taneem 7y ago
    Yes - City would be great for people doing stuff around activities, events and local places. But Country would be more powerful for e-commerce I suspect (my usecase).

by jhow15 7y ago

deadcoder0904 7y ago

You said you are monitoring interesting keywords & topics but how do you know which keywords & topics to monitor? Also, if you are choosing the keywords then wouldn't the data be inaccurate for predicting trends as trends occur randomly?

jhow15 7y ago
I have scripts that are crawling the web in places like twitter, reddit and facebook to generate keyword ideas. So the trends that it detects is biased towards the corners of the web that I am monitoring. Which skew heavily tech!
- mooreds 7y ago
  This is a great service. It would be great for you to take suggestions on other locations to crawl.
  - jhow15 7y ago
    That's a brilliant idea, will add somewhere for people to make suggestions. Thank you!

maxencecornet 7y ago

What could be a really neat addition is having trends sorted by countries

jhow15 7y ago
Great idea, thank you!

omarhaneef 7y ago

Nice work Josh. It can probably be used for lots of things: startups, as you mentioned, but also journalism ideas, investing ideas, job hunting and many more.

jhow15 7y ago
Thank you! Yeah totally agree, I hadn't thought of job hunting though!!

nurettin 7y ago

Hi, it would be nice if the keywords had short explanations. I had to find out what an affinity designer and an enneagram was. Bubble tea was also quite new.

yuy910616 7y ago

Hi Josh, Very interesting stuff. Can I ask what is the math behind defining the trend? How did you calculate the most trendy words, etc? I'm working on something similar for a internal tool, so I'm curious to see what your solution is. Thanks!

jhow15 7y ago
Of course - I have scripts constantly monitoring the web for potentially interesting keywords. I then check google trends data for that topic/keyword and analyze the gradient of the line of best fit over different timeframes. If the gradient is higher than a certain (arbitrary) amount, then I class it as a trend!
- peterhil 7y ago
  Is the gradient just a linear least squares fit?
  - yuy910616 7y ago
    Not the author. Seems like it, it doesn't look like Google Trend gives other variables, so just simple 2 var regression I'd guess

charlesism 7y ago

I don’t have much feedback, except to say I think this is a really good, useful concept.

jhow15 7y ago
Thanks so much!

simonebrunozzi 7y ago

Hey Josh, very interesting concept. Thanks for building it! I tried to use the "technology" category, but the results are not what I expected. Perhaps it would be more useful if I could narrow it down to more specific things, e.g. programming languages, or databases. Keep up the good work!

gondo 7y ago

would be interesting to also to see the opposite, retreating trends

vernie 7y ago
It seems to already have that (the "gradient" arrow)

ashelmire 7y ago

These results aren't filtered in some way? I find it suspicious that almost all of the top ~100 results are tech related.

concert-gilled 7y ago
> I have scripts that are crawling the web in places like twitter, reddit and facebook to generate keyword ideas. So the trends that it detects is biased towards the corners of the web that I am monitoring. Which skew heavily tech!

markdown 7y ago

OT but... > built with the Next.js React framework Wait wait wait... do you mean to say there's now a framework for a framework? Javascriptland continues to confound.

hashbig 7y ago
I wouldn't call React a framework, it's more like a library.

tasssko 7y ago

Awesome I enjoyed making some trend charts, and the results are fun (i.e I added DevOps, Erlang, Elixir). I think there is a risk it could conceal too many errors and biases for it to be useful for trend analysis. Using services like Twitter and Reddit might introduce biases into your data skewing trends. The challenge is using data and not understanding the intent (is it positive, negative or neutral/sarcastic potentially negative/positive). Google Adwords is good at this. I applaud your effort; I hope you keep going but its a tough nut to crack. I use markets like Google Adwords Tools to do this at the moment, have you looked at it? I think more info around time, intent and volume.

kevinguh 7y ago

Pretty neat, I see the most granular time scale currently supported is a 3-month window -- are there any plans to increase granularity towards a (near) real-time system? I get your crawler might be rate-limited at that scale and you would likely have to rethink the entire system architecture, but I think real-time data would open up a whole new dimension of possibilities for detecting/predicting trends perhaps even before public awareness catches up. I'd imagine there would be parties interested in paying for that kind of capability if that's the direction you're planning on taking this idea

jhow15 7y ago
I'd love to go as granular as possible, at least 30 days. Real-time would be ideal, but as you note - Google Trends rate limiting soon becomes a hurdle especially when you potentially multiply by different geographies.

taneem 7y ago

Hi Josh - it would be incredible to have this filtered by location, and I think you can get that from Google Trends.

jhow15 7y ago
Absolutely agree! How specific/granular would be the most valuable to you - at the city level perhaps?
- taneem 7y ago
  Yes - City would be great for people doing stuff around activities, events and local places. But Country would be more powerful for e-commerce I suspect (my usecase).