3 ms·

Would it be easy to pump Cloud Datastore into Bigquery and then read from that? (genuine question)

• bduerst 10y ago
  You can load *.info Datastore backups into BigQuery tables natively. You could probably write up a quick apps script to automate data refreshes it if you wanted. Still not the same as reporting directly from a live datastore though :)
  • obulpathi 10y ago
    That's exactly my issue. I have a crawler that collects data and stores into Datastore. I need to periodically scoop up the snapshots and export them into BigQuery, then run the visualizations. I want my visualization to be live and also don't want the overhead of scooping the data.
    • skybrian 10y ago
      Using App Engine, I schedule task queue tasks to insert rows into a BigQuery table on the fly [1]. It only works for inserts, though. [1] https://cloud.google.com/bigquery/streaming-data-into-bigquery https://cloud.google.com/bigquery/streaming-data-into-bigque...
      • obulpathi 10y ago
        Thanks! And yes, as you mentioned, the problem is that it works only for inserts. I use Datastore as my primary database. So requesting for supporting Datastore from Datastudio.
• vdm 10y ago
  I would try a Cloud Dataflow streaming job. Both Datastore and Bigquery have "IO" connectors in Dataflow.