4 ms·

Sure! Right now a community admin that adds Linen to your Slack workspace. I have webhooks listening for any message events which gets mirror and saved to the A

Sure! Right now a community admin that adds Linen to your Slack workspace. I have webhooks listening for any message events which gets mirror and saved to the Aurora postgres DB. On install I also have a cronjob that hits the Slack api and pulls the old Slack conversations. For scaling wise the I host the webapp on Vercel where it is backed by lambda functions. I also am using AWS aurora for postgres which in theory should help a ton with scaling the DB. I've done Heroku db hosting before but found that AWS tends to be much cheaper.

maxmcd 4y ago
Since your data is append-only have you considered something like just sticking all of the data in s3? Might be awkward to deal with things like updating threads, but just editing the s3 file containing the json response for a given thread might work, and then you can skip having a database entirely.
- cheeseblubber 4y ago
  Oh interesting. I think like you said updates might be a little bit tricky as well as search. I'm using full text search from Postgres so that would be tough on S3. I'm doing some aggressive caching with Vercel and Nextjs which essentially turns in to putting the data in S3.
  - maxmcd 4y ago
    Ah didn't think about search, makes sense. Yeah agree the benefits caching are roughly equivalent to storing in s3.
  - rmbyrro 4y ago
    You can do text search and other SQL queries in S3 data by plugging AWS Athena to it. It isn't very scalable, though. Last time I checked it would handle few dozens of concurrent queries, at most. But for fulltext search, what really makes sense is something like OpenSearch.
treis 4y ago
Is there a how to or any documentation you followed to do this? I wanted to do something where I consumed all the messages but didn't really see how in my brief documentation reading.
- charcircuit 4y ago
  The meat of it would presumably use: https://api.slack.com/methods/conversations.list https://api.slack.com/methods/conversations.list https://api.slack.com/methods/conversations.history https://api.slack.com/methods/conversations.history https://api.slack.com/events/message https://api.slack.com/events/message
- cheeseblubber 4y ago
  Like @charcircuit said those were the meat of it. Main part is you have to iterate through conversation.history api. The syncing portion definitely is the most complicated part of the code base.

maxmcd 4y ago

Since your data is append-only have you considered something like just sticking all of the data in s3? Might be awkward to deal with things like updating threads, but just editing the s3 file containing the json response for a given thread might work, and then you can skip having a database entirely.

cheeseblubber 4y ago
Oh interesting. I think like you said updates might be a little bit tricky as well as search. I'm using full text search from Postgres so that would be tough on S3. I'm doing some aggressive caching with Vercel and Nextjs which essentially turns in to putting the data in S3.
- maxmcd 4y ago
  Ah didn't think about search, makes sense. Yeah agree the benefits caching are roughly equivalent to storing in s3.
- rmbyrro 4y ago
  You can do text search and other SQL queries in S3 data by plugging AWS Athena to it. It isn't very scalable, though. Last time I checked it would handle few dozens of concurrent queries, at most. But for fulltext search, what really makes sense is something like OpenSearch.

treis 4y ago

Is there a how to or any documentation you followed to do this? I wanted to do something where I consumed all the messages but didn't really see how in my brief documentation reading.

charcircuit 4y ago
The meat of it would presumably use: https://api.slack.com/methods/conversations.list https://api.slack.com/methods/conversations.list https://api.slack.com/methods/conversations.history https://api.slack.com/methods/conversations.history https://api.slack.com/events/message https://api.slack.com/events/message
cheeseblubber 4y ago
Like @charcircuit said those were the meat of it. Main part is you have to iterate through conversation.history api. The syncing portion definitely is the most complicated part of the code base.