4 ms·

Nightly to seconds is definitely great. Seconds to milliseconds is what is questionable I think.

• throwaway_pdp09 6y ago
  > Nightly to seconds is definitely great Why? Can a business mobilise in anything less than days? If a report is minutes out of data, is that any loss? Given that some largish proportion of reports are never used, perhaps better management is key. not disagreeing but efficiency is not just a matter of quickness.
  • atomicity 6y ago
    Nightly to seconds = developers can see the data generated within seconds. This generally has implications on data quality, because you aren't fixing data quality issues once they occur in prod. This also makes it far easier to develop ETL pipelines. You don't need complex tooling to see whether your ETL pipeline works. You can technically fix data quality and dev velocity issues without low data freshness, but a quick glance at the data engineering landscape tells you they aren't being solved enough.
  • olavgg 6y ago
    I work in oil and gas with process optimization, and we are working on autonomous tuning of the production process in a way that is not possible with daily data updates and manual human intervention. Quite simply, the end goal is increased oil production and reduced power consumption. Realtime data flows are a big deal when reading from thousands of sensors for just one process system and there are hundreds of tunable parameters. All this has to be safe also.