raja_sekar

5 ms·

1.
by raja_sekar 7y ago
It is in a very initial POC stage and distributed mode is pretty basic, but it is moving faster than I expected. Python integration is definitely one of the primary objectives as I suspect that no one is going to learn Rust for this, althou
2.
by raja_sekar 7y ago
In my experience too I observed that distributed code introduces a lot of redundancy and it requires a lot of data to beat the performance of a single-threaded/single machine implementation. Check out McSherrys' Timely Dataflow, i
3.
by raja_sekar 7y ago
It is indeed my opinion too. In non-standard workflows, handcrafted code/application will most likely beat generic frameworks(not true for some cases). I have conflicting thoughts about this. Nowadays industries are very fast-moving, t
4.
by raja_sekar 7y ago
I didn't mean it in that way. Instead of sitting idly on my laptop, it might at least be useful for someone, and if it really proves to be beneficial, then people might contribute to it. Yeah, not denying your point, it does require hu
5.
by raja_sekar 7y ago
Author of the article here. Sorry about Medium restriction. I will just use Github to host my content hereafter. For now, you can use this friend link. https://medium.com/@rajasekar3eg/fastspark-a-new-fast-native...
6.
by raja_sekar 7y ago
The author of the repo here. It is definitely not orders of magnitude faster. I didn't mention it anywhere also I guess. But yeah, JVM is sometimes a problem for in-memory computing for big data processing. Spark itself tried to addres
7.
by raja_sekar 7y ago
The author of the repo here. It is still very much in the POC stage. It will definitely have python APIs in the future. One of the primary reasons to choose a native language is to have better python integration. I intend to have to APIs al
8.
by raja_sekar 7y ago
I haven't actually monetized it. I think without medium distribution, it will be limited to my followers. That is the only reason I switched on distribution. I have decided to just use Github for my future blog.
9.
by raja_sekar 7y ago
Sorry about that. I will just use Github to host my content hereafter. For now, you can use this friend link. https://medium.com/@rajasekar3eg/fastspark-a-new-fast-native...