ctc24

14 ms·

61.
by ctc24 4y ago
They're applied to the caller, so "in total" in this instance. They're pretty high, however -- 300 write requests per minute per GCP project.
62.
Show HN: Syncing data to your customer’s Google Sheets
103 points by ctc24 4y ago|33 comments
63.
by ctc24 4y ago
Very cool to see a walkthrough with actual benchmarks. Not entirely surprised that Parquet shines here. Another big advantage of Parquet over CSV is that you don't have to worry about data integrity. Perhaps less relevant for GIS data,
64.
by ctc24 4y ago
Appreciate that and will do!
65.
by ctc24 4y ago
Agreed! It's kind of fascinating that a better standard hasn't emerged yet, but not surprising given that each one is developed in isolation by a different organization. Definitely a lot of room for improvement. Curious if anyone
66.
by ctc24 4y ago
Understand the sentiment. On the flip side, and as noted in the post, we don't want it to serve as a "name and shame" situation. We have a lot of respect for the folks maintaining the drivers we use, they do a pretty hard job
67.
Database drivers: Naughty or nice?(prequel.co)
94 points by ctc24 4y ago|44 comments
68.
by ctc24 4y ago
Thanks! In the case of traditional ETL tools, a lot of that transformation is usually schema cleaning, which is necessary because the data is extracted from an API and so needs a little love before it’s in analysis-ready shape (renaming col
69.
by ctc24 4y ago
1. I think in practice, people are already using a mix of sources for ingestion today. It’s rare that a data team would rely on a single tool to ingest all their data – instead, they might get some data via one or more ETL tools, some data
70.
by ctc24 4y ago
As you can imagine, the underlying tech is pretty much the same – we’re still moving bits from one db/dwh to another – so we can use our existing transfer infra. The only difference is the UX and API we surface on top of it. We got thi
71.
by ctc24 4y ago
There’s definitely some similarities (both are generic-ish ways to get data out of an API). As mentioned in another comment, we’re still deciding whether to adopt an existing protocol or roll our own for those API connections. We’ve done th
72.
by ctc24 4y ago
That’s something that we spend a lot of time working on. Our general approach is to do both the simple and predictable thing (ie what format would I want to get the data in if I were the data team receiving it). For the specific example you
73.
by ctc24 4y ago
That’s actually pretty close to how we connect to APIs today, but I love the idea of streamlining it as a protocol. We’re still working on improving the dev experience, especially around APIs, and would love to chat more if you have more fe
74.
by ctc24 4y ago
Very much hear you about the amount of logic contained within API layers. That said, it actually hasn’t come up a lot so far. What we’re finding is that a lot of teams that have complex API-layer logic also end up landing their data in a da
75.
Launch HN: Prequel (YC W21) – Sync data to your customer’s data warehouse
118 points by ctc24 4y ago|40 comments
76.
by ctc24 4y ago
Hey team, Charles here, co-founder of prequel.co (YC W21). We also help companies share data with their customers. As someone who's been playing in the data sharing space, it's really exciting to see it get more attention! All the