5 ms·

BTW, I'm one of the co-authors of streamparse, one of the DARPA-supported projects that is being developed by my company, Parse.ly. It lets you integrate Apache

BTW, I'm one of the co-authors of streamparse, one of the DARPA-supported projects that is being developed by my company, Parse.ly. It lets you integrate Apache Storm cleanly with Python. I just gave a talk about streamparse at PyCon US (https://www.youtube.com/watch?v=ja4Qj9-l6WQ https://www.youtube.com/watch?v=ja4Qj9-l6WQ) a few days ago, it was entitled "streamparse: defeat the Python GIL with Apache Storm". I'm glad to answer any questions about it.

look_lookatme 11y ago
How does one get their commercial project supported by DARPA?
- pixelmonkey 11y ago
  Hmm, not sure I could answer that question, as in this case, DARPA is supporting our open source projects, not our commercial projects. Or is that what you are asking? That said, FastCompany covered the story of how we got involved with DARPA here: http://www.fastcompany.com/3040363/the-future-of-search-brought-to-you-by-the-pentagon http://www.fastcompany.com/3040363/the-future-of-search-brou...
  - look_lookatme 11y ago
    I looked at the MEMEX page and say a bunch of companies represented and was genuinely curious... thanks for sharing the link.
- phy6 11y ago
  Send your proposal in response to the Broad Agency Announcements (BAA) that the agency puts out.
strgrd 11y ago
With only a brief skim of your talk, I wonder what you think of the moral implications of this project being DARPA supported. > ...DARPA said Memex wasn’t about destroying the privacy protections offered by Tor, even though it wanted to help uncover criminals’ identities. “None of them [Tor, the Navy, Memex partners] want child exploitation and child pornography to be accessible, especially on Tor. We’re funding those groups for testing...” Doesn't this sound like the same "protect the kids" line embedded in every press release for not-so-subtle government spy programs? $1 million is a lot of money, and I'm sure being able to name drop DARPA in any conversation about your company carries its own cache -- surely you feel pressured to design your optimizations to fit DARPA's needs. Does it feel weird to write code that's being used to track people? Or is that off base?
- def_illiterate 11y ago
  >With only a brief skim of your talk, I wonder what you think of the moral implications of this project being DARPA supported. You're not adding to the conversation by pointing this out. We can all clearly see this for what it is.
- mpyne 11y ago
  > Does it feel weird to write code that's being used to track people? Does it feel weird to design mechanical implements designed for the sole purpose of destroying human life? I'm not speaking of drones and missiles, mind you; I'm speaking of small arms, the very same tools so staunchly defended by libertarian lovers of the Second Amendment everywhere. There are plenty of valid reasons to want to track someone over a network like Tor, just as there are insidious reasons. E.g. all the reasons that make legal, warrant-protected wiretaps a legitimate function of governments worldwide. But even if there weren't valid reasons, other countries will develop (or already have) similar capabilities, so making DARPA your line in the sand for this is missing the point anyways.
- pixelmonkey 11y ago
  To be clear, our projects (at Parse.ly) don't have anything to do with Tor. In fact, I didn't know much about Tor until researching DARPA and the other participants involved in the program. But, I'll address your general question, which is, do I have a moral/ethical problem with DARPA funding some of our open source work, such as streamparse and pykafka? The answer is a resounding "no". There are very few funding sources for open source work. Part of DARPA's funding supports fundamental tech advancements (famously, the Internet itself and GPS) and recently, important open source projects (such as, Apache Spark and the Julia language). Now, there is no doubt in my mind that open source software is used for intelligence purposes, regardless of its funding source. To restrict ones contribution to F/OSS based on the fear that some government or entity may use it toward an end you disagree with seems a battle you can only win through willful ignorance. The nature of open source software is that people can use it however they please (within legal limits, of course). This is a trade-off I accept with eyes wide open, and in my mind, the benefit to the community for F/OSS always wins out.
anon3_ 11y ago
What's the GitHub URL?
- pixelmonkey 11y ago
  https://github.com/Parsely/streamparse https://github.com/Parsely/streamparse
  - anon3_ 11y ago
    Solid. Apache licensed. You're inside tmux too. This is legit. Docs: http://streamparse.readthedocs.org/en/latest/ http://streamparse.readthedocs.org/en/latest/ How did you make that screenshot / animated preview?
    - pixelmonkey 11y ago
      I used a Linux program called byzanz. The bash alias I use to record gif screencasts is here: https://github.com/amontalenti/home/blob/master/.bash_aliases#L36 https://github.com/amontalenti/home/blob/master/.bash_aliase...

look_lookatme 11y ago

How does one get their commercial project supported by DARPA?

pixelmonkey 11y ago
Hmm, not sure I could answer that question, as in this case, DARPA is supporting our open source projects, not our commercial projects. Or is that what you are asking? That said, FastCompany covered the story of how we got involved with DARPA here: http://www.fastcompany.com/3040363/the-future-of-search-brought-to-you-by-the-pentagon http://www.fastcompany.com/3040363/the-future-of-search-brou...
- look_lookatme 11y ago
  I looked at the MEMEX page and say a bunch of companies represented and was genuinely curious... thanks for sharing the link.
phy6 11y ago
Send your proposal in response to the Broad Agency Announcements (BAA) that the agency puts out.

strgrd 11y ago

With only a brief skim of your talk, I wonder what you think of the moral implications of this project being DARPA supported. > ...DARPA said Memex wasn’t about destroying the privacy protections offered by Tor, even though it wanted to help uncover criminals’ identities. “None of them [Tor, the Navy, Memex partners] want child exploitation and child pornography to be accessible, especially on Tor. We’re funding those groups for testing...” Doesn't this sound like the same "protect the kids" line embedded in every press release for not-so-subtle government spy programs? $1 million is a lot of money, and I'm sure being able to name drop DARPA in any conversation about your company carries its own cache -- surely you feel pressured to design your optimizations to fit DARPA's needs. Does it feel weird to write code that's being used to track people? Or is that off base?

def_illiterate 11y ago
>With only a brief skim of your talk, I wonder what you think of the moral implications of this project being DARPA supported. You're not adding to the conversation by pointing this out. We can all clearly see this for what it is.
mpyne 11y ago
> Does it feel weird to write code that's being used to track people? Does it feel weird to design mechanical implements designed for the sole purpose of destroying human life? I'm not speaking of drones and missiles, mind you; I'm speaking of small arms, the very same tools so staunchly defended by libertarian lovers of the Second Amendment everywhere. There are plenty of valid reasons to want to track someone over a network like Tor, just as there are insidious reasons. E.g. all the reasons that make legal, warrant-protected wiretaps a legitimate function of governments worldwide. But even if there weren't valid reasons, other countries will develop (or already have) similar capabilities, so making DARPA your line in the sand for this is missing the point anyways.
pixelmonkey 11y ago
To be clear, our projects (at Parse.ly) don't have anything to do with Tor. In fact, I didn't know much about Tor until researching DARPA and the other participants involved in the program. But, I'll address your general question, which is, do I have a moral/ethical problem with DARPA funding some of our open source work, such as streamparse and pykafka? The answer is a resounding "no". There are very few funding sources for open source work. Part of DARPA's funding supports fundamental tech advancements (famously, the Internet itself and GPS) and recently, important open source projects (such as, Apache Spark and the Julia language). Now, there is no doubt in my mind that open source software is used for intelligence purposes, regardless of its funding source. To restrict ones contribution to F/OSS based on the fear that some government or entity may use it toward an end you disagree with seems a battle you can only win through willful ignorance. The nature of open source software is that people can use it however they please (within legal limits, of course). This is a trade-off I accept with eyes wide open, and in my mind, the benefit to the community for F/OSS always wins out.

anon3_ 11y ago

What's the GitHub URL?

pixelmonkey 11y ago
https://github.com/Parsely/streamparse https://github.com/Parsely/streamparse
- anon3_ 11y ago
  Solid. Apache licensed. You're inside tmux too. This is legit. Docs: http://streamparse.readthedocs.org/en/latest/ http://streamparse.readthedocs.org/en/latest/ How did you make that screenshot / animated preview?
  - pixelmonkey 11y ago
    I used a Linux program called byzanz. The bash alias I use to record gif screencasts is here: https://github.com/amontalenti/home/blob/master/.bash_aliases#L36 https://github.com/amontalenti/home/blob/master/.bash_aliase...