3 ms·

I'm in a camp of "do the minimum". The cost of adding some call to a logger to dump data into file/syslog/etc is microscopic. The cost of adding that data into

I'm in a camp of "do the minimum". The cost of adding some call to a logger to dump data into file/syslog/etc is microscopic. The cost of adding that data into BigTable and building tooling and web interface and reporting around it is probably higher than the cost of the entire project. In my experience companies are experiencing data overload and data paralysis because they have tons of data that is absolutely useless now and will be even more useless tomorrow. And they keep adding data about the data because that's the current koolaid everyone is drinking.

hobs 7y ago
I think based on your experience that's a fair line to take; I've spent a lot of time on the ops side, and seen much of "well we did nothing and we're all out of ideas AND budget" The best answer is somewhere between these two extremes, and I like your approach of starting simple and clear - a file based approach also easily lends itself to centralizing or bootstrapping to another medium - the bigtable approach calls for a migration and potentially even a change in the guarantees made by the underlying system because someone started to repurpose it as not just an audit trail because "of course we can just query it." (ugghgghghgh)
dustinmoris 7y ago
Spreadsheets work with BigTable out of the box in the G-Suite. I love dumping stuff in BigTable for analytics so other non engineering people can easily interact with that data and run all sorts of complex and non complex queries against it with tooling which they are already know, Out-Of-The-Box! This is as cheap as it gets when it comes to building a "web interface and reporting" and the cost is zero.
- notyourday 7y ago
  Does your app handle all the errors that insert into a bigtable could trigger? Does it do retries? Does it queue messages for those retries via a separate queue manager? Does it synchronize the state of the transaction and the state of the transaction inside of the big table? If one succeeded and one failed, what do you do? That's the extra complexity. If dumps into data lake are that important, then it becomes its own special beast that needs to be fed. If the dumps into the data lake aren't important, then the question to ask "Do I need the data lake at all or am I doing it because everyone else is doing it?" If the answer is "yes" then absolutely build that functionality. If the answer is "Eh" the question becomes much murkier.

hobs 7y ago

I think based on your experience that's a fair line to take; I've spent a lot of time on the ops side, and seen much of "well we did nothing and we're all out of ideas AND budget" The best answer is somewhere between these two extremes, and I like your approach of starting simple and clear - a file based approach also easily lends itself to centralizing or bootstrapping to another medium - the bigtable approach calls for a migration and potentially even a change in the guarantees made by the underlying system because someone started to repurpose it as not just an audit trail because "of course we can just query it." (ugghgghghgh)

dustinmoris 7y ago

Spreadsheets work with BigTable out of the box in the G-Suite. I love dumping stuff in BigTable for analytics so other non engineering people can easily interact with that data and run all sorts of complex and non complex queries against it with tooling which they are already know, Out-Of-The-Box! This is as cheap as it gets when it comes to building a "web interface and reporting" and the cost is zero.

notyourday 7y ago
Does your app handle all the errors that insert into a bigtable could trigger? Does it do retries? Does it queue messages for those retries via a separate queue manager? Does it synchronize the state of the transaction and the state of the transaction inside of the big table? If one succeeded and one failed, what do you do? That's the extra complexity. If dumps into data lake are that important, then it becomes its own special beast that needs to be fed. If the dumps into the data lake aren't important, then the question to ask "Do I need the data lake at all or am I doing it because everyone else is doing it?" If the answer is "yes" then absolutely build that functionality. If the answer is "Eh" the question becomes much murkier.