jdoliner

65 ms·

1. 691.▲
   Pachyderm (YC W15) version 0.4: s3 and Git integration(medium.com)

   3 points by jdoliner 12y ago|0 comments
2. 692.▲
   by jdoliner 12y ago

   It's been talked about a few times in this thread already so I'll try not to rehash too much. We think Spark is way cool but disagree that it's a complete replacement for MapReduce. There were people else where in this thread
3. 693.▲
   by jdoliner 12y ago

   Hi, JD @ Pachyderm here. We think Manta is pretty cool but it wasn't actually an inspiration for us. I'm not sure I fully understand your criticism of the filesystem. Pfs was actually initially conceived as almost exactly what you
4. 694.▲
   by jdoliner 12y ago

   Hi, JD @ Pachyderm here. Our feeling isn't that Java is bad it's that it shouldn't be the only option for big data processing. There are a lot of people that want to do this type of computing and a lot of them hate Java. We w
5. 695.▲
   by jdoliner 12y ago

   Thanks for the tips :D. These all seem like very likely places to look for low hanging fruit. We were actually early employees at RethinkDB so we've been around the block with low level optimizations like this. I'm really looking
6. 696.▲
   by jdoliner 12y ago

   Hi, JD @ Pachyderm here. Our feeling is that there's a big difference between having projects like this in the ecosystem and having a canonical implementation that comes preloaded in the software. "Batteries included but removable
7. 697.▲
   by jdoliner 12y ago

   Hi, jd @ Pachyderm here. Long term we don't want to mandate either of these technologies. We'd like to offer users a variety of job formats (docker and rocket come to mind more domain specific things like SQL are interesting as we
8. 698.▲
   by jdoliner 12y ago

   JD @ Pachyderm here. We try to talk to as many people as possible about their data infrastructure so I have a decent sample size on this. My experience is that there a specific use cases for which streaming solutions are gaining ground but
9. 699.▲
   by jdoliner 12y ago

   MapReduce is actually an incredibly flexible paradigm. If you have a clean implementation of MapReduce with well defined interfaces you can make behave in a number of different ways by putting it on top of different storage engines. We&#x27
10. 700.▲
    by jdoliner 12y ago

    Hi! We unfortunately don't right now. It's surprisingly hard to get benchmarks that give you a true apples to apples comparison between us and Hadoop. However it seems from the comments in this thread that there's a lot of in
11. 701.▲
    by jdoliner 12y ago

    Hi. Co-founder and lead Pachyderm dev here. This is a really interesting comment and raises some questions that I haven't fully considered. Building the infrastructure in this decoupled way has let us move quickly early on and made it
12. 702.▲
    by jdoliner 12y ago

    Hi, co-founder This is something we probably should have spoken directly to in the article because of how popular spark is. The main reason we didn't is that we like Spark and don't feel it needs to be replaced. It doesn't se
13. 703.▲
    by jdoliner 12y ago

    Hi cofounder / lead Pachyderm dev here. Pachyderm actually does a lot more with MapReduce than batch processing. We need to work on making this clearly in our marketing. The way this works under the hood is pretty cool though. Pfs is a
14. 704.▲
    by jdoliner 12y ago

    Hi, Founder here. We'll hanging out in this thread all day to answer any questions.
15. 705.▲
    Hadoop Vs Pachyderm: A Technical Comparison(pachyderm.io)

    4 points by jdoliner 12y ago|1 comments
16. 706.▲
    by jdoliner 12y ago

    Hi!! Just to clarify, is CoreOS in general a no go for you guys and is that due to its newness or something else? I'm unfortunately pretty ignorant on what the approval process looks like for security folks. The good news is that pfs i
17. 707.▲
    by jdoliner 12y ago

    Hi, Just read up on Arvados. I'm totally new to it but it seems really cool. I think pfs and Arvados are very philosophically aligned. Arvados seems to believe as we do that the most important thing missing from data science right now
18. 708.▲
    by jdoliner 12y ago

    Hi! Yes, pfs is designed with the "batteries included but removable" ethos in mind and is only loosely tied to CoreOS. Getting it to run on Kubernetes and Mesos is something we're definitely going to do at some point in the n
19. 709.▲
    by jdoliner 12y ago

    I haven't actually used Manta but I think in terms of computation model this comparison is fair. According to the docs: "Users express computation in terms of shell scripts, which can make use of any programs installed in the defa
20. 710.▲
    by jdoliner 12y ago

    Hadoop streaming is definitely a great way to use Hadoop and actually provided a lot of the early inspiration for Pachyderm. The thing that I always found difficult when using it was dependency management. If I wanted to use libraries it wa
21. 711.▲
    by jdoliner 12y ago

    Hi! We actually did that demo as part of a post. We just released it right here: https://medium.com/@jdoliner/when-grandmasters-blunder-a8198... . Hope you enjoy it.
22. 712.▲
    by jdoliner 12y ago

    Hi guys, founder here. We'll be hanging out in this thread all day to chat with you guys about what we're doing. Ask us questions!
23. 713.▲
    PopSlate, an E ink case for your iPhone(popslate.com)

    67 points by jdoliner 12y ago|57 comments
24. 714.▲
    by jdoliner 12y ago

    Hi Everyone, 1/2 of founding team here. We'll be following this thread all day to chat about the release. Excited to hear from all of you.
25. 715.▲
    Do MapReduce with Docker containers in pfs v0.3(pachyderm.io)

    8 points by jdoliner 12y ago|1 comments
26. 716.▲
    by jdoliner 12y ago

    According to http://techreport.com/news/27619/intel-compute-stick-is-a-14... it's micro usb right now but they're going to make it HDMI only eventually.
27. 717.▲
    by jdoliner 12y ago

    What sorts of stuff do you guys have planned for indexing methods? It looks like right now there's full text search which is already really useful. I'd be really interested to see some more structured search based on the AST of th
28. 718.▲
    by jdoliner 12y ago

    We very much don't want to take over the orchestration plane. We'd much rather interoperate nicely with existing orchestration systems. We just want to add the ability to store and access large datasets within these existing syste
29. 719.▲
    by jdoliner 12y ago

    > if you have a clean and simple API like REST or even plain pipes, you open yourself up to a whole new world of composability Totally agree with this and that's one of the core tenants of our API design. We should probably be more
30. 720.▲
    by jdoliner 12y ago

    Hi, I wasn't previously familiar with Irmin so I can't say this with certainty but it looks like the data models of pfs and Irmin are very similar. They are targeting slightly different use cases though so I expect some performanc