jgraettinger1

8 ms·

31.
by jgraettinger1 3y ago
If one-way sync will do, we (Estuary) have been investing in a NetSuite connector that we've _just_ started testing with a few customers. Both SuiteAnalytics and SuiteQL with full support for custom tables & schema. [1]: https:&#x
32.
by jgraettinger1 3y ago
They're not literally passing around the hash. Holders of hash(email) <=> browser cookie associations are heavily incentivized for both regulatory and also competitive reasons to not blast that information around the internet --
33.
by jgraettinger1 3y ago
Not having to evolve or understand or staff 4,000 micro services. An ability to easily change the boundaries of your conceptual components, because they WILL be wrong now or in the future.
34.
by jgraettinger1 3y ago
Hi fellow Dark Tower friend. Yes, marketing and docs are not our strongest suits and we need to do better. To be fair, though, we're also trying not to scare off less technical users who see a bullet list like above and think "wel
35.
by jgraettinger1 3y ago
No. But this is neat, and at a glance it looks straight forward to add. Happy to discuss further!
36.
by jgraettinger1 3y ago
No. We implemented our own [1] for a few reasons: * Scaling well to multi-TB DBs without pinning the write-ahead log (potentially filling your DB's disk) while the backfill is happening. Instead, our connector constantly reads the WAL
37.
by jgraettinger1 3y ago
Estuary ( https://estuary.dev ; I'm CTO) gives you a real time data lake'd change log of all the changes happening in your database in your cloud storage -- complete with log sequence number, database time, and even bef
38.
by jgraettinger1 3y ago
> they told me the new CDO was already convinced that said "RT-based" datalake was the way to go forward Is the desire for "RT-based datalake" itself misplaced? Or just that the implementation isn't up to the job
39.
by jgraettinger1 3y ago
It does involve pinning the WAL while the backfill is happening, though, which is quite different from the incremental backfill achieved by DBLog. The former can be faster, because you’re able to walk the table in physical storage order, bu
40.
by jgraettinger1 3y ago
A uuid key would crush performance vs rowid, though, especially for ordered scans, and you’d still have conflicts if parallel r/w accesses are temporally correlated (you wrote X and Y at the ~same time, and tend to read them at the ~sa
41.
by jgraettinger1 3y ago
Right, unless they’re marking non-leaf pages as “read” — which would imply marking the root — it would have to be snapshot, right? Otherwise you can’t properly abort the txn because a “select x where y” that was previously empty, no longer
42.
by jgraettinger1 3y ago
> Postgres has a different issue in that replication slots can grow really fast, esp on AWS! [2]. We ended up writing our own custom snapshotter for Postgres that is Debezium compatible to onboard customers that have a massive dataset an
43.
by jgraettinger1 3y ago
estuary.dev may be a fit for you (am CTO). (Competitive product and I feel weird about replying in Artie's thread, but tang8330 has said they're not serving this segment)
44.
by jgraettinger1 3y ago
Hi, I'm Estuary's CTO ( https://estuary.dev ). Mind speaking a bit more about what didn't work? We put quite a bit of effort into our CDC connectors, as it's a core competency. We have numerous customers using
45.
by jgraettinger1 3y ago
Estuary Flow derivations [1], I suspect (I work here)? You write a program which has messages pushed to it, which maintains and updates an internal state, and which publishes output messages. You can build cyclic data-flows [2] - not just D
46.
by jgraettinger1 3y ago
futures::join!() and friends do what you’re after admirably. Is the complaint that something is still missing, or that it’s not in the standard library?
47.
by jgraettinger1 3y ago
We use a macaddr8 that embeds a wall-clock timestamp (so they're ascending order, achieving data locality) with some additional shard and sequence-number bits. It's worked really well for us: https://github.com/est
48.
by jgraettinger1 3y ago
Building contexts for structured data used in AI / GPT tasks is something I've seen little written about, but is obviously quite important. Confluent calls it a "customer 360" problem [1] and I don't disagree. We (E
49.
by jgraettinger1 3y ago
Totally. And 3.5-turbo actually does a great job on this (and lots of other) tasks, in my experience, and is a fraction of the price. The entire endeavor cost us a few bucks of OpenAI tokens.
50.
Continuous Slack=>ChatGPT=>Google Sheets Pipeline(estuary.dev)
9 points by jgraettinger1 3y ago|2 comments
51.
by jgraettinger1 3y ago
Also 39, also three young kids. It’s hard. One thing I’ve noticed, though, is that my difficultly in getting into a flow state is more often than not because of a subconscious hang-up about the approach that I haven’t articulated yet. I’ve
52.
by jgraettinger1 3y ago
Do you have an index on `updated_at` ?
53.
by jgraettinger1 3y ago
I'm having a hard time relating to this comment given our own experience. We use RLS extensively with PostgREST implementing much of our API. It _absolutely_ uses WHERE clauses and those are evaluated / indexes consulted before
54.
by jgraettinger1 3y ago
> Row level security is their "foundational piece", but there is a reason why we moved away from database functions and application logic in database over a decade ago: that stuff in unmaintainable. Funny. In my experience, app
55.
Ask HN: AI Pipelines have a “Customer 360” problem?
2 points by jgraettinger1 3y ago|0 comments
56.
by jgraettinger1 3y ago
Don’t you think billionaires would also rather sip cocktails on a beautiful beach vs their concrete bunker? There’s only so much you can do to spruce up a bunker. They’re not rooting for global warming, as a class.
57.
by jgraettinger1 3y ago
Tell a person “have as many as you want” and they’ll have none. Tell them they can’t have one and they’ll raise a rebellion.
58.
by jgraettinger1 3y ago
If you're a SQLite fan who does stream processing, we (Estuary) recently introduced a capability to write transformations as event-driven SQLite [0]. Basically you get a provisioned SQLite DB to which you apply whatever migrations you
59.
by jgraettinger1 3y ago
This is what cost-of-living salary adjustments are intended to address.
60.
by jgraettinger1 3y ago
A question to answer is, what does orchestration look like if data flows are continuous, updating as quickly as possible? That's where we're headed. Suppose N sources of data, M places where you want that data to be, and T streami

More ›