faizshah

63 ms·

781.
by faizshah 11y ago
Another major difference is that Flink's scheduler doesn't have a notion of data locality like Spark's. If you want to use data local to the node you have to query whatever you're storing your stuff in (HDFS) and filter
782.
by faizshah 11y ago
I guess the only thing left is to design a game where you can change the rules of the game as a turn.
783.
by faizshah 11y ago
The more recent results are even more impressive: https://news.berkeley.edu/2015/05/21/deep-learning-robot-mas...
784.
by faizshah 11y ago
I think it's the same as what happened when the perceptron was the big thing in the 50s/60s. Many made far overreaching conclusions about how the problem of AI had been solved.
785.
In future datacenter technology, what will the layers of responsibility be?(quora.com)
1 points by faizshah 11y ago|0 comments
786.
by faizshah 11y ago
I agree that a good data model is key, in fact the way you structure your data is probably the most important thing. But the reason that the hadoop ecosystem is so good is because it's a growing ecosystem with a lot of momentum and a f
787.
by faizshah 11y ago
I agree, disk is cheap it's compute that's expensive. One thing to note though is that lz4 will use a lot more RAM relative to gzip but compared to processing tasks it's not much.
788.
by faizshah 11y ago
It's due to the decompression algorithm, gzip gets better file sizes and is a better general solution. lz4 is usually 10% worse but will get vastly better decompression speeds. The I/O of an SSD is usually 300+MBps, lz4 will usual
789.
by faizshah 11y ago
I agree but if their problem is that they want a scalable solution for processing a growing amount of data cheaply and robustly and potentially doing more complex work in the future (it's a growing business). This is basically the tagl
790.
by faizshah 11y ago
In my experience the difference is ~10% (.3 ratio vs ~.23 ratio) but for the increase in processing speed it's well worth it. If the tool supports Hadoop's FileInputFormat or a stream of whatever underlying data (you can use the 4
791.
by faizshah 11y ago
Yes but to add to this, for generating reports it sounds like they will probably not be working with really complex SQL queries in which case bigquery might be better. But the full solution to their problems is to write the solution using g
792.
by faizshah 11y ago
I think shopping around and recommending/evaluating these tools is pretty fun that's why these threads are always a blast.
793.
by faizshah 11y ago
I agree with stay away from AWS, but centurylink does have lower prices for compute servers comparatively and google cloud has a better pricing model for compute. I'm about to test this out empirically but based on my initial research
794.
by faizshah 11y ago
This is good advice but lz4 and it's chunked couterpart 4mc are much better suited for big data imo. This is the difference between 1 hour and 4+ hours for an archive of 300GB uncompressed on a single SSD based node.
795.
by faizshah 11y ago
Note: This is based on solutions I have been researching for a current project and I haven't used these in production. Short answer: I think you're looking in the wrong direction, this problem isn't solved by a database but a
796.
by faizshah 11y ago
After searching through these kinds of products for the past couple weeks, this product looks great and is priced reasonably (unlimited projects!). Theres a few features it's missing though. It needs a calendar view, daily agenda for s
797.
by faizshah 11y ago
Anybody find a download for the dataset? Would prefer to take a look at it locally.
798.
by faizshah 11y ago
After wrestling with creating a nice gulpfile for the past two days (essentially wasting two days), I think this is the right route. My biggest gripe with gulp is that creating reusable pipelines even with lazypipe is unnecessarily difficul
799.
by faizshah 11y ago
Thanks, I appreciate the advice!
800.
by faizshah 11y ago
Ok, so in general you would say that when releasing a product and trying to get traction you should only worry about creating a basic TOS&PP before launch? Then if you start getting revenue >10k you should form an LLC/incorporat
801.
by faizshah 11y ago
Yeah, I was just wondering if I start selling without all that stuff will it come back to bite me? Or is it not a big deal and I can just do it later?
802.
Ask HN: Do I need a TOS&PP, Accountant, Incorporation, Trademark before launch?
3 points by faizshah 11y ago|7 comments
803.
by faizshah 11y ago
That's a pretty good app I tried it out right now. It's missing a kanban board but that isn't a big deal since the task lists can be used for states. I think the most important thing that's missing is scheduling time for
804.
by faizshah 11y ago
Thanks for the suggestion. I tried it out and it's a pretty great alternative to other task boards but it's missing a daily/weekly/monthly planner which is a killer feature for me. I will keep an eye on it though as it&#
805.
by faizshah 11y ago
I'm looking for something more visual than Org, thanks for the suggestion though.
806.
Ask HN: App that supports task boards, time tracking and a daily/weekly planner?
8 points by faizshah 11y ago|12 comments
807.
by faizshah 11y ago
I like it! Can you add a comparison section to mithriljs and/or infernojs (the current leader in vdom performance)? Also an example for routing would be nice. Keep up the great work!
808.
by faizshah 11y ago
No you're correct, gzip -1 outperformed lz4 in my test in compression ratio. I don't know why I typed "30% compression" instead of "compression ratio of 30%." Sorry about that.
809.
by faizshah 11y ago
I just tested this on my laptop, I used the first 5 million JSONLines of /u/stuck_in_the_matrix reddit dataset (~4.6GB). For compression lz4 took ~22 seconds (~210 MB/s) and I got ~30% compression, gzip -1 took ~56 seconds (~
810.
by faizshah 11y ago
It's not a problem for me since I mostly use Java. However, you can probably just pipe in your data from the lz4 CLI then use that InputStream for whatever python parser you're using and you should be fine. The biggest problem is

More ›