3 ms·

How does snowplow solve this problem? By writing to disk?

• alexatkeplar 10y ago
  Snowplow is implemented as a unidirectional data pipeline: tracking -> collection -> enrichment -> storage Between each step there is typically some kind of persistent queue (mostly S3/Kinesis), and data won't be lost if a downstream component is not operational. Examples: * If your event collector is unavailable, raw events will be cached in the tracker in localStorage, SQLite or similar * If your Redshift database is read-only for maintenance, enriched events will be held back until Redshift is writeable again
  • shermozle 10y ago
    There is of course the opportunity for a collector to be down, but the aim is to keep those components super simple and rely on really durable storage (Kinesis, S3) managed by someone else.