3 ms·
Does your app handle all the errors that insert into a bigtable could trigger? Does it do retries? Does it queue messages for those retries via a separate queue
by notyourday 7y ago
Does your app handle all the errors that insert into a bigtable could trigger? Does it do retries? Does it queue messages for those retries via a separate queue manager? Does it synchronize the state of the transaction and the state of the transaction inside of the big table? If one succeeded and one failed, what do you do?
That's the extra complexity. If dumps into data lake are that important, then it becomes its own special beast that needs to be fed. If the dumps into the data lake aren't important, then the question to ask "Do I need the data lake at all or am I doing it because everyone else is doing it?" If the answer is "yes" then absolutely build that functionality. If the answer is "Eh" the question becomes much murkier.