tlipcon

9 ms·

31.
by tlipcon 11y ago
Yep, I've been taking part in those design discussions. We hope to have Kudu tablet servers support generating this in-memory format in shared memory as the result of scans, so the Impala server (client from Kudu's perspective) ca
32.
by tlipcon 11y ago
Yep, that's correct. HDFS+Parquet is more accurate but doesn't fit quite as well on slides and short descriptions. The idea is to get the analytic scan performance of Parquet while still allowing for in-place updates and row-by-ro
33.
by tlipcon 11y ago
9 worker nodes plus one master node. Each with 48GB of RAM, 3 data disks, and dual 4-core Xeon L5630 processors at 2.13GHz. So, pretty old stuff. But note that the 100G dataset fit comfortably in RAM. If you don't fit in RAM, then the
34.
by tlipcon 11y ago
Quick example of how TSAN makes it easy to understand and fix races -- here's a commit message which shows its output: https://github.com/cloudera/kudu/commit/e402d5ed79a9c98283b6... Apparently Google ha
35.
by tlipcon 11y ago
I spent a lot of time in 2011 or so struggling with GC on the JVM: http://blog.cloudera.com/blog/2011/02/avoiding-full-gcs-in-h... has some of the gory details. Even hacked a bit on G1: http://mail
36.
by tlipcon 11y ago
We're working on running some more thorough YCSB benchmarks, but here are some of the percentiles on the uniform "workload A" running on a 9 node cluster for 1 hour: Throughput: 28280 ops/sec Read: 2821us avg, 467us min,
37.
by tlipcon 11y ago
We haven't scoped out the security features. Cell level security can be difficult to implement efficiently, but if we see enough demand for it, I could imagine it happening. My guess is that the first pass will be table and column leve
38.
by tlipcon 11y ago
Should add that those looking for a technical deep dive might enjoy our draft paper: http://getkudu.io/kudu.pdf and/or browsing our source: http://github.com/cloudera/kudu
39.
by tlipcon 11y ago
Todd from the Kudu team here. If anyone has any questions, feel free to ask them here, will try to check back throughout the day.
40.
by tlipcon 12y ago
True it's a pain to embed. We embedded it in my current project, and it only took a few hours to add a CMakeLists.txt and make it reasonably embeddable. I guess if your project doesn't want to include C++ (and you don't want
41.
by tlipcon 12y ago
Existing library which does the same thing, and faster, with arbitrary polynomials: http://code.google.com/p/crcutil/
42.
by tlipcon 12y ago
If this were more like Spotify for sheet music, I'd pay for it -- instant access to music for any published song is valuable. Otherwise, I'm not sure how it's much better than dropbox and PDFs.
43.
by tlipcon 12y ago
If only that were true -- the shuffle is typically seek-bound when the intermediate data doesn't fit into cache (plenty of papers show this pretty conclusively).
44.
Announcing Support for Impala with Amazon Elastic MapReduce(aws.amazon.com)
27 points by tlipcon 13y ago|4 comments
45.
Inside Cloudera Impala: Runtime Code Generation with LLVM(blog.cloudera.com)
6 points by tlipcon 14y ago|0 comments
46.
by tlipcon 14y ago
Cloudera isn't primarily consulting. In fact we have about 8x as many engineers in product development compared to consulting, last I counted.
47.
Quorum-based Journaling for the Hadoop HDFS NameNode(blog.cloudera.com)
1 points by tlipcon 14y ago|0 comments
48.
Analyzing Twitter Data with Hadoop(cloudera.com)
4 points by tlipcon 14y ago|0 comments
49.
One Year, Six Products: What I’ve Built and Learned(alexlod.com)
78 points by tlipcon 14y ago|34 comments
50.
by tlipcon 15y ago
Early in your career, trading salary for experience is a smart move. In my case, I turned down a Google offer to work at a startup after college (taking a 15%-30% pay cut) and my equity ended up being worth less than I paid to exercise it.
51.
Caching in HBase: SlabCache(cloudera.com)
15 points by tlipcon 15y ago|0 comments
52.
by tlipcon 15y ago
Both APIs are available and will continue to be available for the foreseeable future. -Todd
53.
by tlipcon 15y ago
This isn't quite true - data is streamed from the client through a pipeline made up of all of the replicas, as it's written. It's true you'll lose data if you crash in the middle of a block, _unless_ you call the sync() function which makes
54.
by tlipcon 15y ago
The 1.0.0 release is actually formerly known as the 0.20.205.1 release -- ie just bugfixes since 0.20.205. Hadoop's been "production ready" for years - there are hundreds of companies running it in business critical applications. But some p
55.
by tlipcon 15y ago
Several comments here indicate that this practice is outdated. I disagree strongly. Even in 2011 autoneg is flaky on many switches - in a recent shipment of ~100 machines, ~5 of them autonegotiated to 100mbit instead of gigabit. And these a
56.
by tlipcon 15y ago
I was an intern in Leo's group at Google 6 years ago and learned a lot from him. So, if you're looking for LA jobs or internships, I would highly recommend checking out his team! Of course, if you're based in the bay area, also check out my
57.
by tlipcon 15y ago
INTERN or FULLTIME (REMOTE for some positions) Offices in downtown San Francisco, Palo Alto, and Research Triangle Park Cloudera offers enterprises a powerful new data platform built on the popular Apache Hadoop open-source software package
58.
by tlipcon 15y ago
I turned down a job at Google (not 300k but hey, it's Google!) to join a startup. The startup started to sink about 2 years later. I learned a ton and didn't regret it for an instant. Moved on to a new startup 2 years ago when it became cle
59.
Video: HBase Dos and Don'ts(readwriteweb.com)
2 points by tlipcon 15y ago|0 comments
60.
by tlipcon 16y ago
Yes, they currently depend on kernel patches to provide bulk map/unmap APIs as well as some other vm_area trickery to allow parts of physical memory to be mapped to multiple places in virtual memory with different protection flags. (assumin

More ›