3 ms·

Hmm not at all. But isn't it just another persistence storage? I mean yes, it's a column instead of row-oriented one. But essentially it saves data. The core p

Hmm not at all. But isn't it just another persistence storage? I mean yes, it's a column instead of row-oriented one. But essentially it saves data. The core problem I want to solve is basically getting data from various SaaS tools. Alone getting the data is a big challenge especially when it's usually not available in the right form or the API is not accessible. Another one is then consolidating data of several tools with complete different data models.

noduerme 4y ago
FWIW it probably wouldn't solve your problem with any particular SaaS, but the CLI app is quite clever, doesn't have anything to do with their commercial offerings and just lets you query CSV files as if they were SQL, and join them to SQL queries... https://clickhouse.com/docs/en/interfaces/cli/ https://clickhouse.com/docs/en/interfaces/cli/ So to some extent, this helps with writing scripts for merging different data models, and different formats. To be fair I've never used it for anything serious. I've written so many web scrapers and little one-time tools to match up data, I lose track. It's hard to think of a way to fix this as a global problem when every service wants to lock you in somehow... except to demand they make your data open for you. I suppose you could imagine a standard and try to enforce it with a badge or something. There's an incentive for services not to comply. If I read your post right, you're talking about tackling this as a Big Problem that could be solved technically. In my experience - not to be jaded - you'd be chasing the endless dragon; and there's no money in it. You only live once, and spending hours reverse engineering non-existent APIs for fickle SaaS companies is probably not the ideal use of one's resources. Just speaking as an indie dev. Partly because any dev plugging into such a service would have to trust the service to keep up with whatever non-API they wanted to mangle, and in any single use case a dev is better off rolling their own. It'll always be fractured. /sorry, I'm drunk and thinking out loud. But your idea isn't bad, just too glorious for reality ;)
- mathiasn 4y ago
  That's great advice actually. Thanks a lot for your open words, appreciate it!

noduerme 4y ago

FWIW it probably wouldn't solve your problem with any particular SaaS, but the CLI app is quite clever, doesn't have anything to do with their commercial offerings and just lets you query CSV files as if they were SQL, and join them to SQL queries... https://clickhouse.com/docs/en/interfaces/cli/ https://clickhouse.com/docs/en/interfaces/cli/ So to some extent, this helps with writing scripts for merging different data models, and different formats. To be fair I've never used it for anything serious. I've written so many web scrapers and little one-time tools to match up data, I lose track. It's hard to think of a way to fix this as a global problem when every service wants to lock you in somehow... except to demand they make your data open for you. I suppose you could imagine a standard and try to enforce it with a badge or something. There's an incentive for services not to comply. If I read your post right, you're talking about tackling this as a Big Problem that could be solved technically. In my experience - not to be jaded - you'd be chasing the endless dragon; and there's no money in it. You only live once, and spending hours reverse engineering non-existent APIs for fickle SaaS companies is probably not the ideal use of one's resources. Just speaking as an indie dev. Partly because any dev plugging into such a service would have to trust the service to keep up with whatever non-API they wanted to mangle, and in any single use case a dev is better off rolling their own. It'll always be fractured. /sorry, I'm drunk and thinking out loud. But your idea isn't bad, just too glorious for reality ;)

mathiasn 4y ago
That's great advice actually. Thanks a lot for your open words, appreciate it!