frankmcsherry

51 ms·

181.
by frankmcsherry 11y ago
By way of comparison, the single-threaded numbers for PageRank on the `twitter_rv` graph on my laptop are 1.5B edges in 5s, which is about 300M edges per second per core. PageRank isn't doing much other than a load, a few += operations
182.
by frankmcsherry 11y ago
> "Rust can leak memory" is an overgeneralization of the issue and misrepresents it. I thought it was the exactly correct response to a comment that says Rust's main feature is that it doesn't let you leak memory. =&#
183.
by frankmcsherry 11y ago
It's a good question. Part two goes in to this a bit more (the post was too long). But, the performance seems to be due to a few things: 1. Rust/C#/etc. vs Java. Stuff just goes faster with less effort. 2. The model is more &
184.
by frankmcsherry 11y ago
I'm sorry, this hasn't been my experience, or the experience of anyone I know writing stream processing systems. Having written one each over a GC and not (Naiad, and the timely dataflow in Rust linked above), dev time on the form
185.
by frankmcsherry 11y ago
In my experience, concurrent access to shared data is relatively rare in a well-designed data-parallel platform. Data are exchanged rather than shared, at which point ownership of memory transfers from one worker to another. Rust is great
186.
by frankmcsherry 11y ago
You can contribute to: https://github.com/frankmcsherry/timely-dataflow https://github.com/frankmcsherry/differential-dataflow Or, just tell your friends. :) Better yet, write some python / p
187.
by frankmcsherry 11y ago
Shameless self-promotion for big data in Rust: https://github.com/frankmcsherry/timely-dataflow https://github.com/frankmcsherry/differential-dataflow Nothing much useful to contribute, except tha
188.
by frankmcsherry 11y ago
SIGMOD 2015 proceedings are open access, meaning the conference has arranged with ACM to provide the content to pretty much everyone. That being said, the submission is to someone's (Adrian's) interpretation of the work, and is ar
189.
by frankmcsherry 11y ago
The paper contains the motivation: "For our needs at Twitter, we needed a stream processing platform that was open-source, high-performance, scalable, and was compatible with the current Storm API." I guess 3/4 isn't bad
190.
by frankmcsherry 11y ago
Oh, it's coming around, alright. https://www.usenix.org/conference/hotos15/workshop-program/p...
191.
by frankmcsherry 11y ago
> What do we need to do to prevent people like you from writing an article like this again? I thought it was a great read, and totally didn't take the "Haskell programmers are liars" literally ... ... until I saw #4 on you
192.
by frankmcsherry 11y ago
yeah, totally. there's now a method `embalm(&mut self)` run just before the binary data gets written. That and `exhume` would be great places to go to and from little-endian.
193.
by frankmcsherry 11y ago
Would you find it equally worrying if stack-based languages claimed they improved over x86 because you needn't worry about clobbering values in registers? I guess it depends on how well you know the code and its context. Might still be
194.
by frankmcsherry 12y ago
Well, I'm looking at the code and it looks like it doesn't report false positives. I understand that you expect it to hash the content down and store only the hash, but it doesn't do that. EDIT: I suppose my point was just th
195.
by frankmcsherry 12y ago
Yes, you can also write safe code in an unsafe language, but until you are literally willing to code review every program anyone writes in your unsafe language of choice, there is value in having safe languages where we don't have to r
196.
by frankmcsherry 12y ago
This probably isn't crystal clear from this post alone, but for PageRank the 42GB working set has a good mix of read and write. Specifically, about 14GB of target rank vector gets +='d all over the place. If they got hammered rand
197.
by frankmcsherry 12y ago
That's a good point, thanks. I've added the timings from the original post to the tables in this post, which also makes clearer the intended point: the existing systems don't yet have measurements for data at this scale (exce
198.
by frankmcsherry 12y ago
I was formerly in the same lab as Leslie. As a side project, we (Tom Rodeheffer, mainly) used TLA to verify the correctness of Naiad's coordination protocol. The protocol is very simple (distributed reference counting), and my initial
199.
by frankmcsherry 12y ago
The thrust of the post wasn’t meant to be that it is surprising that you can even do these things on one core; sorry if it came across that way. Rather, it’s that when doing the work on one core (with a way simple implementation) you go fas
200.
by frankmcsherry 12y ago
With all due respect, your code seems to be a copy/paste of my code, where you've removed the Pair and Option implementation (allowing the alternate default I discussed in my second blog post) and replaced my string `ColumnarVec`
201.
by frankmcsherry 12y ago
I totally left the #[inline(always)] out from the code, in the interest of explaining. That's what prevents the copying! :D And also what Steve said about Rust being rly rly smart, or at least move semantics opening the door for that.
202.
by frankmcsherry 12y ago
Hi, author here. Unless I misunderstand your post, I think you may have missed important parts! More is going on than "just wrapping the data of the original vector as a Vec<u8>". That is what happens when we are handed a Ve