3 ms·

How are you solving the problem of scale? Wouldn't everyone else be using this approach if the need to e.g. maintain indexes was overruled by your table join te

How are you solving the problem of scale? Wouldn't everyone else be using this approach if the need to e.g. maintain indexes was overruled by your table join technology? Or is the idea that you specify the type of dataset you want, and then wait a few hours for the table to be generated from the activity stream? Good luck with your launch, this is a cool and novel take on data analytics.

cedricd 6y ago
Thanks! We've found that this scales fairly well. Tables generated from the activity stream are generally done in seconds or minutes at the worst case. One reason for this is that most warehouses are column-oriented. This means our table, which is 11 columns wide but pretty deep, is really fast to query. We had a customer on a 3 Node (tiny) DS large Redshift warehouse dealing with billions of rows. Their Looker queries literally took 8 hours. We moved them to the Narrator structure and they saw the same queries assembled using the activity stream down to minutes (<5 minutes) Edit: I should also point out that all queries in Narrator are compiled to SQL that runs directly on the warehouse. We don't use any external data stores or anything like that.
- mariushn 6y ago
  If I understand correctly: 1. you define activities, each with an associated SQL query eg customer opened support ticket: SELECT * FROM tickets WHERE customer_id=$customer_id 2. users use Narrator UI to build a report, similar to Looker 3. Narrator creates a table for that report in the same db and populates it with data, based on 1. 4. Narrator maintains all the reports tables (updates, deletes when report is deleted) ?
  - cedricd 6y ago
    Yes, that's basically it. 2. The report (we call it Dataset) you build with the Narrator UI is a table that you can aggregate different ways, plot, and export (including writing back to the warehouse as a materialized view) 3. done optionally as part of 2 4. Yes, we keep anything written back to the warehouse up to date. You can control the cadence. Because of 4. we work well with BI tools like Looker. Once you have the data you want just point Looker to the right table in the warehouse.
    - mariushn 6y ago
      Thanks. Interesting approach!

cedricd 6y ago

Thanks! We've found that this scales fairly well. Tables generated from the activity stream are generally done in seconds or minutes at the worst case. One reason for this is that most warehouses are column-oriented. This means our table, which is 11 columns wide but pretty deep, is really fast to query. We had a customer on a 3 Node (tiny) DS large Redshift warehouse dealing with billions of rows. Their Looker queries literally took 8 hours. We moved them to the Narrator structure and they saw the same queries assembled using the activity stream down to minutes (<5 minutes) Edit: I should also point out that all queries in Narrator are compiled to SQL that runs directly on the warehouse. We don't use any external data stores or anything like that.

mariushn 6y ago
If I understand correctly: 1. you define activities, each with an associated SQL query eg customer opened support ticket: SELECT * FROM tickets WHERE customer_id=$customer_id 2. users use Narrator UI to build a report, similar to Looker 3. Narrator creates a table for that report in the same db and populates it with data, based on 1. 4. Narrator maintains all the reports tables (updates, deletes when report is deleted) ?
- cedricd 6y ago
  Yes, that's basically it. 2. The report (we call it Dataset) you build with the Narrator UI is a table that you can aggregate different ways, plot, and export (including writing back to the warehouse as a materialized view) 3. done optionally as part of 2 4. Yes, we keep anything written back to the warehouse up to date. You can control the cadence. Because of 4. we work well with BI tools like Looker. Once you have the data you want just point Looker to the right table in the warehouse.
  - mariushn 6y ago
    Thanks. Interesting approach!