2 ms·

Hey Josh, this is a really neat idea! I can definitely see the value in real-time forum and Twitter monitoring, especially for brand reputation management and l

Hey Josh, this is a really neat idea! I can definitely see the value in real-time forum and Twitter monitoring, especially for brand reputation management and lead generation. A few thoughts and questions: Scalability: With so many forums and the real-time aspect, how do you handle data processing and ensure timely notifications without delays? Filtering: False positives could be a pain. How robust is the keyword matching, and are there options for filtering by sentiment, forum quality, or specific user profiles? Integrations: Down the line, integrations with popular marketing/CRM tools would be super valuable (e.g., pushing leads into a CRM). Any plans for this? I'm curious about the technical side too. Combining various forum APIs and handling the data firehose must be quite the challenge. Did you build any custom scraping/parsing solutions?

joshwaller7 2y ago
Thank you very much :) Scalability: This is more of a cost/resource issue than a technical one. Right now, ForumScout checks every hour for new posts. This isn't ideal, but it allows me to offer the product for much cheaper than if we were to check, say, every 5-10 minutes. Scalability is much more manageable with an hourly check, but the goal eventually is to get as close to real time as possible. I plan on offering a higher pricing tier for real-time checks to account for the extra resources. Filtering: By default, the searches are performed via exact match. From testing, there are no false positives; there's some code that double-checks that each post contains an exact match of the keyword(s). Users can manually turn off exact match if they want broader searches. Sentiment searching is a VERY good idea. I will add it to my list! Integrations: This is a great point, and something I had not yet considered. Right now, the feed can be exported via CSV, but direct integrations would be much better for UX. Technical: Honestly, it wasn't too difficult - every hour, the automation will make a call to all the different APIs and search using the parameters provided by the user at setup. Then we format and combine all the results, remove duplicates, perform some more filtering (depending on what settings the user has chosen), and add the results to the user's feed. If emails are turned on, we send an email with the new results. As mentioned, making this real-time will be more difficult, but it would really just mean doing this every few minutes, requiring more resources. I did build a custom scraping function for Reddit comments, because the native Reddit API doesn't allow direct searching of comments like you can on their website/app, but everything else is third-party.
- msnkarthik 2y ago
  Awesome

joshwaller7 2y ago

Thank you very much :) Scalability: This is more of a cost/resource issue than a technical one. Right now, ForumScout checks every hour for new posts. This isn't ideal, but it allows me to offer the product for much cheaper than if we were to check, say, every 5-10 minutes. Scalability is much more manageable with an hourly check, but the goal eventually is to get as close to real time as possible. I plan on offering a higher pricing tier for real-time checks to account for the extra resources. Filtering: By default, the searches are performed via exact match. From testing, there are no false positives; there's some code that double-checks that each post contains an exact match of the keyword(s). Users can manually turn off exact match if they want broader searches. Sentiment searching is a VERY good idea. I will add it to my list! Integrations: This is a great point, and something I had not yet considered. Right now, the feed can be exported via CSV, but direct integrations would be much better for UX. Technical: Honestly, it wasn't too difficult - every hour, the automation will make a call to all the different APIs and search using the parameters provided by the user at setup. Then we format and combine all the results, remove duplicates, perform some more filtering (depending on what settings the user has chosen), and add the results to the user's feed. If emails are turned on, we send an email with the new results. As mentioned, making this real-time will be more difficult, but it would really just mean doing this every few minutes, requiring more resources. I did build a custom scraping function for Reddit comments, because the native Reddit API doesn't allow direct searching of comments like you can on their website/app, but everything else is third-party.

msnkarthik 2y ago
Awesome