3 ms·

I also do this, but be weary of situations where you’re using ephemeral containers and you have to download, scrape, then reupload a bare repo.. Sure, you can u

I also do this, but be weary of situations where you’re using ephemeral containers and you have to download, scrape, then reupload a bare repo.. Sure, you can use GitHub or Gitlab, but I opted for stashing and archiving into GCS / S3 directly and had to write some pretty intricate code to essentially perform a mitosis each month so the repo only got to about 100 mb. Fortunately, got it working before we hit 2gb flying around every 15m.. Also running garbage collection is tricky and didn’t appear to be possible with libgit2, so a lot of nuance here too don’t let the simplicity of the idea fool you. I learned a lot of things about Git from this project.. so yeah, just be careful is all in certain architectural contexts. However, I agree, it’s an extremely useful technique to approach scraping this way. Edit: PS we use Apache arrow and save as parquet files, which has lots of benefits over SQLite because you can democratize the dataset by using Athena / BigQuery.. and I diplomatically agreed to use Go to consume the data and serve the API.. so that’s been time consuming.. but fun!

ZeroGravitas 5y ago
Could you expand, or point me in the direction of more information about what you mean by "benefits over SQLite because you can democratize the dataset by using Athena / BigQuery" as it sounds very interesting.
- awill88 5y ago
  So we have a semi-large organization where our product people are very savvy, but not to a point where they are going to jump with joy having to download a huge tool chain with docker just to grab a SQLite database and fiddle with some analytical data. However, they absolute will learn commands and SQL or some python if they just login to a portal and start chomping away.. kind of like the whole SAP / data warehousing industry. Just don’t make people download a bunch of crap (figuratively speaking). Also, this particular data isn’t sensitive, it’s technically derived from public sources so it’s safe to “democratize” in that sense. I’m not sure how the approach would change with user data. So how can I as an engineer do the most good for the widest possible audience? Well, I was in a meeting and I was taking about using SQLite and my peer was like “I dunno it sounds a little complicated” and I took that to heart and responded with, well ya know from where I sit it seems like all the big players are just using flat files in cloud storage as data lakes.. didn’t sleep well that night and then just perusing other options I came across Apache Drill and then it’s spiritual successor, Arrow and it hit me.. well if Athena / BigQuery understands parquet files and Apache arrow writes parquet files without any sort of infra like spark.. what if our data pipeline didn’t save it in SQLite but Apache Parquet and by doing so (with still plenty of complexity ironically), we all of a sudden have a dataset that can technically be read by API developers and the cloud data lake products. All one really needs at that point is access to a cloud bucket.. It’s still been a lot of work, and I haven’t realized all my promises as we’re still trying to ship the product, but I think in the long run, cloud storage + Apache arrow will be the de facto approach to time series data. And in some ways, it also can give event stores in the context of eventsourcing a facelift too.. but that’s not as developed of a thought for now.

ZeroGravitas 5y ago

Could you expand, or point me in the direction of more information about what you mean by "benefits over SQLite because you can democratize the dataset by using Athena / BigQuery" as it sounds very interesting.

awill88 5y ago
So we have a semi-large organization where our product people are very savvy, but not to a point where they are going to jump with joy having to download a huge tool chain with docker just to grab a SQLite database and fiddle with some analytical data. However, they absolute will learn commands and SQL or some python if they just login to a portal and start chomping away.. kind of like the whole SAP / data warehousing industry. Just don’t make people download a bunch of crap (figuratively speaking). Also, this particular data isn’t sensitive, it’s technically derived from public sources so it’s safe to “democratize” in that sense. I’m not sure how the approach would change with user data. So how can I as an engineer do the most good for the widest possible audience? Well, I was in a meeting and I was taking about using SQLite and my peer was like “I dunno it sounds a little complicated” and I took that to heart and responded with, well ya know from where I sit it seems like all the big players are just using flat files in cloud storage as data lakes.. didn’t sleep well that night and then just perusing other options I came across Apache Drill and then it’s spiritual successor, Arrow and it hit me.. well if Athena / BigQuery understands parquet files and Apache arrow writes parquet files without any sort of infra like spark.. what if our data pipeline didn’t save it in SQLite but Apache Parquet and by doing so (with still plenty of complexity ironically), we all of a sudden have a dataset that can technically be read by API developers and the cloud data lake products. All one really needs at that point is access to a cloud bucket.. It’s still been a lot of work, and I haven’t realized all my promises as we’re still trying to ship the product, but I think in the long run, cloud storage + Apache arrow will be the de facto approach to time series data. And in some ways, it also can give event stores in the context of eventsourcing a facelift too.. but that’s not as developed of a thought for now.