4 ms·
Hey HN — Segment PM on the project here! Happy to answer questions. Under the hood we're using NSQ as a queuing layer, S3 for storage and batched uploads, Amaz
by n2parko 11y ago
Hey HN — Segment PM on the project here! Happy to answer questions.
Under the hood we're using NSQ as a queuing layer, S3 for storage and batched uploads, Amazon Aurora (for S3 indexing), DynamoDB for billing and metadata storage, and several distinct Go services that handle batching, transformation, schema updating, deduplication and internal consistency checking.
It's been in beta for several months and we're loading about 10,000 events per second into customers' databases today.
- troels 11y agoCan you say a bit more about how you use Aurora for indexing S3? What are these indices used for? (Or what do you mean by indexing)
- schmatz 11y agoSure! We batch together data to load into warehouses by time and a few other properties. Usually, to figure out what objects are in S3, you have to issue an S3 list objects command. That operation tends to be relatively slow, especially if there are many objects. Instead, when we put a new object, we update a table in Aurora which tracks all of the relevant objects. That way, we can query information like "what objects were uploaded in a certain time range" very quickly.
- balls187 11y agoHow do you ensure that Aurora and S3 stay in sync?
- schmatz 11y agoWe have a worker consuming S3 events and updating the index. On a related note, we have experimented with Lambda to do something similar; AWS has done a fantastic job integrating their products :)
- balls187 11y agoCool, that part makes sense, but how are you ensuring they stay in sync? That is, what happens if a db write fails? How are you handling concurrent updates? Is there a reconcile process that runs periodically? (those details would be an awesome engineering blog post :) ) Edited to add: Any reason that AWS Lambda didn't work out? Was it due to the public endpoint requirements? Double Edited to add: I totally geek out on composing AWS Services, and this is fascinating to me.
- schmatz 11y agoThis is a good idea for a blog post! The way we have set up the system ensures that we requeue upon failure and concurrent updates are not an issue. We are eagerly waiting for Lambda VPC support! I'm happy you're interested in this sort of stuff; shoot me an email, let's chat :) michael@segment.com
- chris_wot 11y agoHesitate to ask this, but will anyway: can you access the data from with Tableau or Qliview?
- n2parko 11y agoYep! You can use those BI tools on top of your Redshift or Postgres instance :)
- vyrotek 11y agoHow real-time is this stuff? If an order record is inserted can I instantly see on a dashboard the +1 based on a custom report query?
- pkrein 11y agoScrolling down the posted page you can see the details on loading latency: it ranges from daily loading on the free plans to 30 minutes on the business tier.
- mshron 11y agoWhat about companies that have additional data sources? For example, a CRM. It seems like it's hardly a data warehouse if it's _only_ event data. Or am I missing something?