4 ms·

Exactly. We use Supabase too but are at a scale where it just made sense to use a second, dedicated vector db (Pinecone) than to bloat our Postgres db that has

Exactly. We use Supabase too but are at a scale where it just made sense to use a second, dedicated vector db (Pinecone) than to bloat our Postgres db that has a completely different workload

crazy_marksman 3y ago
How do you deal with security and access control across postgres and pinecone?
- weird-eye-issue 3y ago
  We use Cloudflare Workers for our API and just handle auth calls by checking the JWTs with Supabase and caching it. So we already had the necessary auth setup to do this. For basic CRUD we use the Supabase endpoints directly but none of that involves querying a vector db :P
BoorishBears 3y ago
Bloat your DB... or pull in an entirely new vendor and bloat your entire operational outlay. I'd really love to know what kind of insane scale justifies that tradeoff...
- hobs 3y ago
  Don't forget your latency!
  - weird-eye-issue 3y ago
    They are separate systems. We don't touch Postgres for the same code that needs to access Pinecone.
- weird-eye-issue 3y ago
  That's a very exaggerated way to look at things lol. Nothing got bloated at all in this process, we are just using the right tools for the job. I'm a solo founder and the only backend developer. I can assure you this decision only made my life easier by choosing the correct tech from the get go.
  - BoorishBears 3y ago
    Nothing exaggerated, your comment implied scale was your justification: in which case there'd better have been some crazy high load that just brought the tool you already had to its knees to justify paying an additional closed source platform and manually having to pipe data to it in addition to your main data store. Of course if I sounded incredulous it's because I didn't think you had that scale, and it sounds like I was correct?
    - weird-eye-issue 3y ago
      No, you're not correct. We have over $2M ARR and with the amount of data we are storing it would be downright stupid to use Supabase. We don't also "pipe" our data to Supabase, we use a couple different data stores depending on the best use case. For example we also use R2 and Durable Objects. Just because you have a hammer doesn't mean everything is a nail.
      - BoorishBears 3y ago
        Well maybe we have different definitions of scale: I think my team spends about $2M a month on compute, so we don't pride ourselves on randomly pulling in new vendors.
        weird-eye-issue 3y ago
        You are incredibly dense
        BoorishBears 3y ago
        When you're playing checkers people playing by the rules of chess might seem dense.
        weird-eye-issue 3y ago
        It's pretty incredible how you know more about our needs, usage, and infrastructure than we do :D Also don't you think it's funny calling out somebody else's tech choices when you have zero insight into it and when it's worked out perfectly for us? By the way, how many TBs of vector data are you storing in Postgres and needing to retrieve with minimal latency?

Exactly. We use Supabase too but are at a scale where it just made sense to use a second, dedicated vector db (Pinecone) than to bloat our Postgres db that has a completely different workload

crazy_marksman 3y ago

How do you deal with security and access control across postgres and pinecone?

weird-eye-issue 3y ago
We use Cloudflare Workers for our API and just handle auth calls by checking the JWTs with Supabase and caching it. So we already had the necessary auth setup to do this. For basic CRUD we use the Supabase endpoints directly but none of that involves querying a vector db :P

BoorishBears 3y ago

Bloat your DB... or pull in an entirely new vendor and bloat your entire operational outlay. I'd really love to know what kind of insane scale justifies that tradeoff...

hobs 3y ago
Don't forget your latency!
- weird-eye-issue 3y ago
  They are separate systems. We don't touch Postgres for the same code that needs to access Pinecone.
weird-eye-issue 3y ago
That's a very exaggerated way to look at things lol. Nothing got bloated at all in this process, we are just using the right tools for the job. I'm a solo founder and the only backend developer. I can assure you this decision only made my life easier by choosing the correct tech from the get go.
- BoorishBears 3y ago
  Nothing exaggerated, your comment implied scale was your justification: in which case there'd better have been some crazy high load that just brought the tool you already had to its knees to justify paying an additional closed source platform and manually having to pipe data to it in addition to your main data store. Of course if I sounded incredulous it's because I didn't think you had that scale, and it sounds like I was correct?
  - weird-eye-issue 3y ago
    No, you're not correct. We have over $2M ARR and with the amount of data we are storing it would be downright stupid to use Supabase. We don't also "pipe" our data to Supabase, we use a couple different data stores depending on the best use case. For example we also use R2 and Durable Objects. Just because you have a hammer doesn't mean everything is a nail.
    - BoorishBears 3y ago
      Well maybe we have different definitions of scale: I think my team spends about $2M a month on compute, so we don't pride ourselves on randomly pulling in new vendors.
      - weird-eye-issue 3y ago
        You are incredibly dense
        BoorishBears 3y ago
        When you're playing checkers people playing by the rules of chess might seem dense.
        weird-eye-issue 3y ago
        It's pretty incredible how you know more about our needs, usage, and infrastructure than we do :D Also don't you think it's funny calling out somebody else's tech choices when you have zero insight into it and when it's worked out perfectly for us? By the way, how many TBs of vector data are you storing in Postgres and needing to retrieve with minimal latency?