vtuulos

16 ms·

91.
by vtuulos 10y ago
First time I hear about Elasticluster. Regarding cluster provisioning, auto-scaling is critical for cost and convenience. We managed to do this with a simple job queue that outputs a CloudWatch metric that in turn controls an auto-scaling g
92.
by vtuulos 10y ago
Thanks! Pachyderm's version control over datasets is a killer feature.
93.
by vtuulos 10y ago
We have been using this architecture for containerized batch processing for more than a year: http://tech.adroll.com/blog/data/2015/09/22/data-pipelines-d... It is very convenient to be able to defi
94.
Querying Data in Amazon S3 Directly with User-Space Page Fault Handling(tech.adroll.com)
12 points by vtuulos 10y ago|0 comments
95.
by vtuulos 10y ago
In 2006-2007, just before iPhone was released, we tried to pitch Nokia to adopt Python as the main language for mobile application development. We even wrote a book about it: https://www.amazon.com/Mobile-Python-prototyping-
96.
by vtuulos 10y ago
Fun question, even if it wasn't a serious one. Since I happen to work at a company whose business is related to ad placement prediction, I can try to answer. In our case, we have 1000-2000 instances (the number is elastic, because AWS)
97.
by vtuulos 10y ago
AdRoll - San Francisco If you like developing open-source code, languages such as Python, JS, C, D, Lua, Erlang, AWS, petabytes of data, this may be your dream job. AdRoll is one of the most widely used adtech companies, already producing $
98.
by vtuulos 10y ago
AdRoll - San Francisco If you like languages such as Python, JS, C, D, Lua, Erlang, AWS, and/or petabytes of data, this is your dream job. AdRoll is one of the most widely used adtech companies, already producing $100M+ in revenue with
99.
by vtuulos 10y ago
TrailDB is designed for discrete events (think JSON objects) rather than continuous metrics (like CPU utilization). Most time-series databases manage to store a large number of data points by aggregating data over time. Discrete events can&
100.
by vtuulos 10y ago
Good question. Judy arrays tend to be really efficient for low-level mappings like the 128-bit -> 64-bit map ( https://github.com/traildb/traildb/blob/master/src/judy_128_... ) that we use to keep
101.
by vtuulos 10y ago
You can find the R bindings now at https://github.com/traildb/traildb-r
102.
by vtuulos 10y ago
We were just talking about it earlier today. It should be easy to do with https://github.com/node-ffi/node-ffi Contributions are welcome :)
103.
by vtuulos 10y ago
TrailDB is optimized for a very specific data model ( http://traildb.io/docs/technical_overview/#data-model ) which allows it to do compression that you couldn't do with other data layouts. TrailDB is a C libra
104.
by vtuulos 10y ago
It is coming soon, sorry for the delay!
105.
by vtuulos 10y ago
Thanks for you kind words!
106.
by vtuulos 10y ago
Pipelinedb.com is a great product and very complementary to TrailDB. PipelineDB is all about querying streaming, real-time data in SQL whereas TrailDB is great for computationally intensive analysis of historical data using any programming
107.
by vtuulos 10y ago
Our infrastructure is in AWS, so S3 was a natural choice for us. You can easily use any other object store or filesystem with TrailDB. We use Kinesis amongst other things to stream raw data to S3.
108.
by vtuulos 10y ago
Thanks! I hope we can open-source the rest of the stack now that the core libraries are out. Btw, I am happy to answer any questions about the project here.
109.
TrailDB – An Efficient Library for Storing and Processing Event Data(tech.adroll.com)
212 points by vtuulos 10y ago|54 comments
110.
by vtuulos 10y ago
AdRoll - San Francisco If you like languages such as Python, JS, C, D, Lua, Erlang, AWS, and/or petabytes of data, this is your dream job. AdRoll is one of the fastest growing adtech companies, already producing $100M+ in revenue with
111.
by vtuulos 11y ago
AdRoll - San Francisco If you like languages such as Python, JS, C, D, Lua, Erlang, AWS, and/or petabytes of data, this is your dream job. AdRoll is one of the fastest growing adtech companies, already producing $100M+ in revenue with
112.
by vtuulos 11y ago
No, we haven't used Luigi with Hadoop. For batch processing we use containerized jobs with a simple job queue, like described in the blog article.
113.
by vtuulos 11y ago
We have been using Luigi in production for a year now at AdRoll, to manage a graph of tens of data processing tasks. We have been really happy with it. You can read more about our setup in these two blog posts: http://tech.adroll
114.
by vtuulos 11y ago
AdRoll - San Francisco If you like languages such as Python, JS, C, D, Lua, Erlang, AWS, and/or petabytes of data, this is your dream job. AdRoll is one of the fastest growing adtech company, already producing $100M+ in revenue with a
115.
by vtuulos 11y ago
We are routinely pulling a dataset of hundreds of GBs to 100+ instances (1600+ cores) in parallel. We have never noticed throughput going down with the number of nodes. S3 delivers the maximum throughput of 2-4Gbps / instance very cons
116.
by vtuulos 11y ago
AdRoll - San Francisco If you like languages such as Python, JS, C, D, Lua, Erlang, AWS, and/or petabytes of data, this is your dream job. AdRoll is one of the fastest growing adtech company, already producing $100M+ in revenue with a
117.
by vtuulos 11y ago
AdRoll - San Francisco If you like languages such as Python, JS, C, D, Lua, Erlang, AWS, and/or petabytes of data, this is your dream job. AdRoll is one of the fastest growing adtech company, already producing $100M+ in revenue with a
118.
Managing Containerized Data Pipeline Dependencies with Luigi(tech.adroll.com)
8 points by vtuulos 11y ago|0 comments
119.
by vtuulos 11y ago
We have a few cases of containers pulling (meta)data from databases. We use Postgres on RDS, so setting up a read-replica is very easy. Containers can safely access these replicas. Raw data is collected by our bidders and adservers that pus
120.
Petabyte-Scale Data Pipelines with Docker, Luigi and Elastic Spot Instances(tech.adroll.com)
51 points by vtuulos 11y ago|5 comments

More ›