23 ms·
AOL Moloch: open-source, large scale, packet-capturing, indexing database system
- sparrish 7y agoWhat a horrible name for a project. Why would you want it associated with child sacrifice? It's disgusting and distracts from any merits of the tech.
- mumphster 7y agocounterpoint: I like it and owls are pretty rad
- dudeinjapan 7y agoAnd also Moloch isn't a good name either.
- ignoramous 7y agoNot the greatest choice of name: Moloch[a] (also Molech, Mollok, Milcom, or Malcam) is the biblical name of a Canaanite god associated with child sacrifice, through fire or war. From: https://en.wikipedia.org/wiki/Moloch https://en.wikipedia.org/wiki/Moloch
- totetsu 7y agoSheesh, I hope no-one looks up my user name on wikipedia..
- pnathan 7y agoThat was my immediate thought as well. Profoundly and deeply unsavory.
- simlevesque 7y agoYou are offended by the name of an open source project ? Your life must be easy.
- pgsbathhouse2 7y agoAre you going to be that daft?
- simlevesque 7y agoI'm really sorry you don't approve the name.
- nicehat 7y agoNo, that guy disapproved of you being rather daft. See, names matter.
- pgsbathhouse2 7y agoI don't have a stake in this argument. But nice signaling attempt.
- pnathan 7y agoNames have symbolic value beyond the intended referent. Typically, wisdom dictates choosing names that aren't immediately offensive out of the gate. I won't speculate on whatever led you to such fatuous conclusions, because ad hominem is a bad reasoning.
- striking 7y agoPerhaps they were going for an association with Allen Ginsburg's Howl (https://www.poetryfoundation.org/poems/49303/howl https://www.poetryfoundation.org/poems/49303/howl) instead?
- lsb 7y agoReminds me of when someone saw that the NSA's public key (http://www.cypherspace.org/adam/hacks/lotus-nsa-key.html http://www.cypherspace.org/adam/hacks/lotus-nsa-key.html) to be included in Lotus Notes had an organizational name of "MiniTruth", and a common name of "Big Brother". Moloch isn't the good guy in Howl. Moloch! Solitude! Filth! Ugliness! Ashcans and unobtainable dollars! Children screaming under the stairways! Boys sobbing in armies! Old men weeping in the parks! Moloch! Moloch! Nightmare of Moloch! Moloch the loveless! Mental Moloch! Moloch the heavy judger of men! Moloch the incomprehensible prison! Moloch the crossbone soulless jailhouse and Congress of sorrows! Moloch whose buildings are judgment! Moloch the vast stone of war! Moloch the stunned governments!
- peterwwillis 7y agoMoloch the dude in the cubicle across from mine that always eats his lunch at his desk and chews with his mouth open! Moloch the meddling micromanager from Maine! Moloch the Junior Architect! Moloch the 3-space indenter!
- hprotagonist 7y agowhich is the same referent, and begs the same question: "What sphinx of cement and aluminum bashed open their skulls and ate up their brains and imagination?"
- theprotocol 7y agoWasn't Allen Ginsburg also in favor of child abuse?... He was a member of NAMBLA... At least Moloch isn't real.
- pas 7y agohttps://slatestarcodex.com/2014/07/30/meditations-on-moloch/ https://slatestarcodex.com/2014/07/30/meditations-on-moloch/
- _jal 7y agoKind of fitting for AOL though. I can name three startups that went to die there off the top of my head.
- chews 7y agoIt's an inside joke to the Deep State that actively needs captures of network data, AOL... is a Verizon company after all. In these discordian times, even Eris would blush.
- mensetmanusman 7y agoFunny you mentioned the deep state. I remember being perturbed that the email leak from the 2016 race had references to child-sacrifice moloch. Strange times
- scandox 7y agoI think this is a bit distracting. It's an ancient name. Probably a better paragraph to fixate on would be: > Moloch has been used figuratively in English literature from John Milton's Paradise Lost (1667) to Allen Ginsberg's "Howl" (1955), to refer to a person or thing demanding or requiring a very costly sacrifice. I mean a project called Zeus would hardly founder on it's namesakes fetish for turning into a bull and assaulting human women.
- HideousKojima 7y agoZeus is known a bit better and for more things than Moloch
- sildur 7y agoBut today everyone knows about Zeus huge sex drive.
- derptron 7y agoAm I the only one who thinks of Moloch aka Edgar Jacobi, the old villain in Watchmen dying of cancer?
- smoe 7y agoInteresting, newer knew the word is of biblical origin. In German the word Moloch is in somewhat common use to refer to an abstract merciless, all-consuming or all devouring power. E.g big cities might get referred to as molochs the way they eat into nature around it or suck people into its anonymity
- woodpanel 7y agoI was thinking the same. In German Moloch is mostly used for groteskly big/growing cities. But thinking about it, the "eating into nature"-theme aligns well with the biblical "eating into kids"-theme. For me at least, because of the city-context, Moloch matches: The moloch could stand for the amount of activity in a system that is humanly insurmountable to get a better-than-superficial understanding of.
- smoe 7y agoHa, so I learned two things about Moloch today. I didn't have that strong of an association of Moloch with big cities as it felt like when I looked it up for the above answer, so I googled a bit of its usage on news sites and it looks like in Germany it is almost exclusively used for big cities, whereas in Switzerland, cities is the most common usage but it is applied way more liberally. Within a minute or so I saw it used to refer to: The EU, a government bicycle program, a 2km long tunnel in Zürich, iTunes, the Tour de France, Goldman Sachs and fashion.
- finnthehuman 7y agoWhy not? That's a pretty metal name. Don't tell me anyone still has appetite for yet another saccharine cutesy-wutesy "yay software squee" name.
- Hackbraten 7y agoThere’s always room for more cute in software.
- kfrzcode 7y agoHard disagree. Software should be cold, calculated and cannibalistic. Leave cuteness to the experts: puppies and such.
- runawaybottle 7y agoHow about AOL PackagieWackigie?
- cryptoquick 7y agoMore like UWU Mowoch x3
- Thaxll 7y agoAnyone who works on Linux probably knows that sometime you need to sacrifice a child...
- gyehuda 7y agoNaming open source projects is a challenging task. The team hears this all the time, but hey, open source is about code you can use, a community you can join to make the code better, and the pride that people around the world want to use your code. If you are on a blue-team, you'll want to look at this project. If you are making lists of names that distract attention this goes on it.
- ignoramous 7y agoIf someone names their CLI JDahmer which then requires me to use it jdahmer <command> <options>, it is going to cringe me out. That said, I totally respect the project's decision to name it so. I'm sorry if it comes off as an outrage or an unnecessary rant. I understand it takes away the energy which is otherwise better spent on a constructive discussion of this excellent project, which is also surprisingly well-maintained. But, I hope you realise why the name is controversial, especially since everyone that has to use the project has to get used to seeing/reading/using/typing Moloch everywhere from the docs, to the CLI, to the FAQ pages, and what-not. May be the project can explain their rationale in a separate webpage (if they want to) and that might help?
- xenihn 7y agoHonestly the first thing that came to my mind was Watchmen.
- travbrack 7y agoAt least it's not trafficly
- phs318u 7y agohttps://slatestarcodex.com/2014/07/30/meditations-on-moloch/ https://slatestarcodex.com/2014/07/30/meditations-on-moloch/ Makes more sense.
- yingw787 7y agoThis is cool, thanks for sharing! I'll be honest, I didn't think AOL did anything...interesting...but this is interesting to me! What are some open-source alternatives / analogues to this product?
- gyehuda 7y agoSorta like https://www.wireshark.org/ https://www.wireshark.org/. But Moloch is a very active project, used by many, and used internally at Verizon Media. Aol is part of Verizon Media (which brought AOL and Yahoo together). Open source is very active here. ;-) Compare also to https://enterprise.verizon.com/products/security/advanced-threat-analytics-and-detection/network-detection-response/ https://enterprise.verizon.com/products/security/advanced-th... (formerly known as Protectwise), which is not open source.
- solarkraft 7y agoIgnorant question for you: What does AOL do these days (I can't view the website for the same reason I don't read TechCrunch articles)? Is it fully about online publications now?
- gyehuda 7y agoYou can see https://www.verizonmedia.com/our-brands https://www.verizonmedia.com/our-brands to see the collection of online brands in the family. You might use lots of these brands today without really noticing. There's a lot of internet content you get via https://www.verizondigitalmedia.com/ https://www.verizondigitalmedia.com/ which is also part of the same company. Aol is still a thing, people do use it. Many people use lots of these brands as part of their internet experience. If you are here, you probably care about internet security. Verizon Media runs a pretty significant Bug Bounty program https://hackerone.com/verizonmedia https://hackerone.com/verizonmedia which you can read about here https://www.protocol.com/hackerone-bug-bounty-virtual-verizon https://www.protocol.com/hackerone-bug-bounty-virtual-verizo... and their blog here https://www.verizonmedia.com/technology/security#/bug-bounty https://www.verizonmedia.com/technology/security#/bug-bounty You might even be looking for a job as an information security professional. You can join "The Paranoids" team (now that's a good name, don't you think!) by checking out some of their jobs. https://www.verizonmedia.com/careers/search.html?q=paranoids https://www.verizonmedia.com/careers/search.html?q=paranoids
- seemslegit 7y agoThis should work nicely with Google Behemoth and Facebook Beelzebub
- burpsnard 7y agoSee also : SATAN https://en.m.wikipedia.org/wiki/Security_Administrator_Tool_for_Analyzing_Networks https://en.m.wikipedia.org/wiki/Security_Administrator_Tool_...
- awsanswers 7y agoAOL github account's location is Dulles, VA. Thinking of what this does, the name's meaning and its location in 'the corridor' is giving me the creeps
- gyehuda 7y agoI fixed that for you https://github.com/aol https://github.com/aol
- awsanswers 7y agoYeah much less creepy
- nerdbaggy 7y agoIt’s been there since before the Mapquest days.
- cfeduke 7y agoAOL's location has been there for a very long time.
- deleted 7y ago[deleted]
- dguido 7y agoFun fact, Moloch was initially created with funding from DARPA's famous Cyber Fast Track program! It's great to see that Moloch is still going strong since ~2013. There is some previous discussion of Moloch when it was released in this older thread: https://news.ycombinator.com/item?id=20586005 https://news.ycombinator.com/item?id=20586005
- awick 7y agoNot sure if you are joking, but Moloch was never part of DARPA's short lived Cyber Fast Track program. :) We do welcome contributions from everyone, and lots of different folks use Moloch. If interested join us over at https://molo.ch https://molo.ch
- dguido 7y agoThat's odd. Are you one of the original authors? The CFT project list had Moloch on it. I'll try and dig it up, it's probably floating around my Google Drive. You may want to speak with Eoin Miller, as I believe he was the point of contact for the project in the document I'm thinking of.
- dguido 7y agoFuckkkkk I think I found the source of my confusion. I am wrong, you are right. I DID find documents about Moloch floating around my Google Drive from ~2013-ish. I believe I invited your co-author Eion to present at a conference I was running, THREADS, in 2014 and that he was not able to make it. The focus the _year prior_ was exclusively on DARPA CFT. I combined those two events in my head and thought your project got some seed funding from DARPA too. I'm sorry! Here is the conference: THREADS 2014 when you were invited: https://github.com/trailofbits/threads/tree/master/2014 https://github.com/trailofbits/threads/tree/master/2014 THREADS 2013 was a retrospective on DARPA CFT: https://github.com/trailofbits/threads/tree/master/2013 https://github.com/trailofbits/threads/tree/master/2013
- whyhead 7y agoThanks, John. I tried to do something a while back with wireshark but I couldn't get the thing to scale. Gonna give this one a hard look.
- dang 7y agoA thread from last year: https://news.ycombinator.com/item?id=20586005 https://news.ycombinator.com/item?id=20586005
- dang 7y agoUrl changed from https://github.com/aol/moloch https://github.com/aol/moloch, which points to this.
- jstewartmobile 7y agoAs the n95 masks are going on, other masks are coming off.
- hprotagonist 7y agoThis gets posted once in a while and I always, without fail, think of Ginsburg. Moloch whose mind is pure machinery! Moloch whose blood is running money! Moloch whose fingers are ten armies! Moloch whose breast is a cannibal dynamo! Moloch whose ear is a smoking tomb!
- caconym_ 7y agoYeah, I literally heard that line in my head immediately after reading the title of this post. What else are we supposed to think of?
- burpsnard 7y agoThe game
- bigiain 7y agoAnd whenever anything here reminds me of Ginsberg, I always, without fail, remember: “The best minds of my generation are thinking about how to make people click ads.” –Jeff Hammerbacher
- mdonahoe 7y agoFor the uninitiated (like myself) Allen Ginsberg's poem "Howl" starts with the line: "I saw the best minds of my generation destroyed by madness"
- sergiotapia 7y agoSeriously "Moloch", was Lucifer or Satan taken already? Edit: And the logo is an owl lmao hard pass - what can they gain from this?
- simlevesque 7y agowhat is your problem with owls ?
- sergiotapia 7y agoHN ain't the place.
- simlevesque 7y agofor owls ?
- kfrzcode 7y agoFor discourse, it seems.
- cryptoquick 7y agoHaha, tell me about it. The downvoters are out in droves. Cowards.
- sibane 7y agoI believe it refers to the symbolic use of the owl of Athena/Minerva in Enlightenment-era secret societies and elite clubs like the Bohemian Grove. Alex Jones tends to call the owl Moloch, an old Semitic deity linked to child sacrifice (and actually more commonly depicted as a horned bovine kind of creature), because he believes those societies are at their core satanic. That's pretty heavy symbolism for an open source project.
- rsynnott 7y agoWell, there's already a database called Voldemort...
- gyehuda 7y agoAccording to my sources, DARPA wasn't involved in any of the child sacrifices, but they might use Moloch to help secure their networks.
- dguido 7y agoYep! As far as I'm aware, a number of military services (I think the USAF) have been public about using Moloch on their networks. It was money well spent by DARPA.
- gyehuda 7y agoMoloch is free. :-)
- cryptoquick 7y agoWell, Moloch is free now. DARPA had to make a lot of sacrifices to make that happen, though.
- gyehuda 7y agoThey do an amazing job keeping it secret too. Can't even find them on the contributor list https://github.com/aol/moloch/graphs/contributors https://github.com/aol/moloch/graphs/contributors
- fireattack 7y agoOff-topic question: does this graph mean that top two people write 99% of the code? Or they're just sharing the account(s)?
- cryptoquick 7y agoThose are those who contributed code. Unfortunately, defense contract money doesn't have a tab on GitHub. Trust me, I've used palantir/tslint
- omarforgotpwd 7y agoI’m sorry. AOL?
- kfrzcode 7y agoThe amount of professional-grade hand-wringing virtue-signaling in this comment section makes me feel ill. Why get bent out of shape over the name of a software project? It's virtually a meaningless factor in day-to-day life. What about hearing phrases "Sacrificing a Chicken to Moloch," the "spirit cooking" culture and related symbolism rampant in elite political circles? Shouldn't we be more interested in that? Lots of low-hanging fruit to pick, I guess. Someday I might unlock the "downvote" ability on this platform. Until then my opinions don't carry weight here. Also, uhh who decides the "threshold" for downvoting? Hint: nobody knows. [0] This platform has become more of an echo chamber than a host of rational discussion based on merit. I suppose that's a problem with growth. There's a lot of conversation around "Moloch" as a name and the subsequent emotional responses... but not a lot of discussion about the tech at-hand. And it's a repost. Where's the value? [0]: https://news.ycombinator.com/newsfaq.html https://news.ycombinator.com/newsfaq.html Edit: while this is high-ranking comment, I'd point out that if I had the ability I would have just downvoted the comments I didn't like. Take that for whatever it's worth.
- gyehuda 7y agoPreach Brother! Moloch is a serious open source project, run by serious people who care about network security. They published their code under an open source license, showing off just how confident they are that it is solid. You can use this project to inspect packets on your network, you can learn how they built it and become a valuable network security engineer with a job somewhere finding people trying to hack in to your site. You can propose modifications to make this even better (and if your code is good enough, it will get accepted and used by security teams around the world). Or you could focus on the name they chose and the name of the company at the time this was published. I'd seize the opportunity to talk tech and focus on network security. Infosec jobs pay better than brand marketing jobs.
- kfrzcode 7y agoThe fact we have access to tools like this (and many, many others) is so damn motivating. I am working through some structured course material in preparation for net-sec baseline `$vendor-$certification` exams, and there are so many complexities and rabbit holes that it's easy to be overwhelmed. Some nights are harder than others to see forest for the trees when it feels I'm trying to understand how to pick up a single pine cone to study. Then I see projects like this, or Jaeger/OpenZipkin, or Chaos Monkey, and I'm simply inspired. Much like listening to a killer record. I don't spend time feeling sorry for myself because an ancient, somewhat esoteric proper noun was used to describe the project.
- frequentnapper 7y agointeresting. didn't realize till now that .ch was a TLD for switzerland!
- simlevesque 7y agoIt means Confœderatio Helvetica
- voz_ 7y agoPardon my ignorance - Is this the same AOL as America Online?
- kfrzcode 7y agoIt certainly is!
- gyehuda 7y agoWhen this project was first published, the name of the company that published it was AOL. Before that, the company was called America Online. Subsequently it was named Oath, and now it's Verizon Media. Names change. The common theme: names are not easy, sometimes they are beloved brands, sometimes they fall out of favor. Sometimes they were just bad ideas from the start, but happened anyway. This gives rise to an interesting challenge for open source projects when you have an open source project in a github org, and the name of your company changes (or your company gets acquired), should you move the project? The problem is real since you don't want to lose your community, but you don't want to be stuck in the past.
- deleted 7y ago[deleted]
- narrator 7y agoYou'd think with a name like that that this would be a daemon that would take as input the cores of killed child processes.
- pgsbathhouse2 7y agoNaming things is hard in software. But some people won't even bother trying.
- X6S1x6Okd1st 7y agoIs there anyone that uses something like this in a home/personal setting?
- jlgaddis 7y agoThe Filesystem Hierarchy Standard [0] has been around for ~25 years but it still took quite a long time before most Linux distros decided to adhere to it (for the most part; some still do "non-standard" things at times). Now that we're to that point, please stop screwing it up and coming up wih your own locations for application binaries, data, etc. To be clear, the "/data" directory -- under which Moloch's pre-built packages apparently install to, according to the README [1] -- is not part of the FHS. --- [0]: https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard [1]: https://raw.githubusercontent.com/aol/moloch/master/release/README.txt https://raw.githubusercontent.com/aol/moloch/master/release/...
- traden210 7y agoThis is probably intended to be run in a container, but as that's not specified, it's certainly extremely weird.
- deathanatos 7y agoEven inside a container, you want stuff in the standard places, for minimal surprises.
- peterwwillis 7y agoIf you look at the source code of most open source software, the devs typically install by default to arbitrary directories. They either don't know about, or care about, operating system standards. When Linux distributions package that software, they choose to change how that software works to align with standards. Usually their own standards, but those can often align with distro-independent standards too. But as a packager, you expect that you'll be modifying paths and making patches. So really it doesn't matter where a random dev decides to install their software to by default, since either A) it's proprietary and it's non-standard anyway, or B) it's up to us to package it the way we want it.
- brigandish 7y agoNot everyone is using a package manager for every install, and even then it's not a magic bullet. I'm a Mac user and I've no idea why devs insist on installing to places like /usr/local or putting configuration in a dot file in my home directory when there's a standard place to put those (~/Library/Application Support), among other well thought out directory structuring[1]. Further to this, the moment you want to install somewhere else or have a dependency elsewhere (e.g. /opt/anything) builds break because of hard coded paths and poor assumptions. To top it all, I've no idea why a package manager like Homebrew then contributes to all of this by acting like no of this matters because a Mac is just like Linux, apparently. If they don't like Apple's way then why not XDG[2]? Reinventing the wheel badly is more than just hubris, it breaks things and it's annoying. [1] https://developer.apple.com/library/archive/documentation/MacOSX/Conceptual/BPFrameworks/Concepts/FrameworkAnatomy.html https://developer.apple.com/library/archive/documentation/Ma... [2] https://specifications.freedesktop.org/basedir-spec/latest/ https://specifications.freedesktop.org/basedir-spec/latest/
- jzer0cool 7y agoI read the instructions but was not clear. Is this just a pcap viewer or something more? The way I read it, it appears to be a running daemon which listens on all port which saves the pcap file, which then exposes API's for accessing such data. If you had an application http server running, is traffic sent to Moloch first, and forwarded to the http server like a proxy?
- jlgaddis 7y agoWhen running something like this on a large scale to capture all traffic going across a network, you'd typically use a "network packet broker" (cf. Google) that sends a copy of all traffic to the machine(s) running this software. Your hypothetical application server would not even be aware that this was taking place.
- yters 7y agoThere should be a competition for most useful software with most offensive name. Then we can make all the software stacks of top companies likely to result in jail time when discussed in public. That would be funny.
- deleted 7y ago[deleted]
- nodaemon 7y agoSeriously, AOL named this after the god of child sacrifice, you might want to put a new marketeer on this project, I won't be touching anything with a name like that.