4 ms·
Very much hear you about the amount of logic contained within API layers. That said, it actually hasn’t come up a lot so far. What we’re finding is that a lot o
by ctc24 4y ago
Very much hear you about the amount of logic contained within API layers. That said, it actually hasn’t come up a lot so far. What we’re finding is that a lot of teams that have complex API-layer logic also end up landing their data in a data warehouse and replicating the API’s data model there. In those cases, we can use that as the source. For those that don’t have a dwh, we do actually support connecting to an API as a source, it just requires a bit more config and it’s less efficient and so we don’t promote it quite as much :).
As far as the tweet goes, there’s a couple points we might disagree with. One of the assumptions made is that the problem is solved by an outsourced third-party. That’s actually exactly what we want to change! We think the problem should be solved by a first-party (the software vendor), and we want to give them the tools to accomplish that. Another assumption in there is that companies who charge for dashboards wouldn’t do this because they’d lose money. If they charge for dashboards, they can also choose to charge for data exports.
- cayleyh 4y agoThe API-based part sounds a lot link https://www.singer.io https://www.singer.io, the API-based data ETL tooling Stitch Data developed -- is that accurate?
- ctc24 4y agoThere’s definitely some similarities (both are generic-ish ways to get data out of an API). As mentioned in another comment, we’re still deciding whether to adopt an existing protocol or roll our own for those API connections. We’ve done the latter so far but it’s a work in progress.
- yevpats 4y agoI very much hope it will work for you and it will succeed. This can be def an amazing turning point in data integration and ELT world. I do still have hard time understanding how this is technically feasible because let's say a company have their data stored in PostgreSQL but the data model their is not exactly as in their API because they are doing some minimal transformation and then expose it to the user. How the user will know what to expect the new data model is not documented at all and they usually want what is documented and exposed via the API. so seems you still have to go via the API and if you are there already then your are in the ELT space. Per your comment that it falls into the user and it should fall on the vendor - I agree but if it's a matter in the ELT space then they should just maintain a plugin to CloudQuery (https://github.com/cloudquery/cloudquery https://github.com/cloudquery/cloudquery) or AirByte or something similar. Similarly how vendors maintain terraform or pulumi plugins.