7 ms·
Show HN: Midas, a Streaming Anomaly Detector. Now Implemented in Go
- siddhartb_ 6y agoCode is quite neat! What are the changes it will need for including fit and predict API?
- steve0hh 6y agoI've just added the fit/predict api for Midas. Will be doing MidasR next when I've time. :) The change needed is basically to create a MirasR struct and re-purpose the MidasR function into fit and predict.
- pmfgpmfg 6y agoit's not clear to me how this API supports streaming edges. Looks like it takes lists of sources and destinations and loads them into memory.
- steve0hh 6y agoThanks for your comments. I'll be updating the readme to better reflect it.. I've just added the midas struct, in which you can do the following: ``` m := midas.NewMidasModel(2, 769, 9460) m.FitPredict(2,3,1) // inputs will be your src,dst and time (from your stream) m.FitPredict(2,3,1) // same here ``` The MidasR model is next. :)
- floren 6y agoCould you go into the selection criteria for those 3 numbers? I see 2 and 769 on the original MIDAS repo as default number of hash functions and buckets, but without explanation of why they're chosen. The third argument, "m", I don't see explained anywhere.
- steve0hh 6y agoI adapted the code from the original paper from https://github.com/bhatiasiddharth/MIDAS/ https://github.com/bhatiasiddharth/MIDAS/ . And from one of the issues(https://github.com/bhatiasiddharth/MIDAS/issues/7#issuecomment-597185695 https://github.com/bhatiasiddharth/MIDAS/issues/7#issuecomme...), M can be any reasonable value because it's just used to compute the edge hash (https://github.com/steve0hh/midas/blob/master/edgehash.go#L42 https://github.com/steve0hh/midas/blob/master/edgehash.go#L4...). Hope it clarifies.
- siddhartb_ 6y agoHi, I'm the author of the MIDAS algorithm. We choose the number of hash functions and bucket according to the maximum error we can tolerate and the false positive probability theoretical guarantee we want. Please refer to the AAAI paper here: https://www.comp.nus.edu.sg/~sbhatia/assets/pdf/midas.pdf https://www.comp.nus.edu.sg/~sbhatia/assets/pdf/midas.pdf Let me know if you need more details.
- dabei 6y agoWhat are some interesting applications of edge anomaly detection?
- siddhartb_ 6y agoDetecting Intrusions, Denial of Service (DoS) attacks, Distributed Denial of Service (DDoS) attacks. It can also be used to detect fake profiles in Social Networks like Twitter, Facebook, Amazon reviews, and Financial Frauds. Basically any suspicious similar group of edges in time-evolving/dynamic graphs.
- MauranKilom 6y agoNeither title, comments here nor Github repo indicated to an outsider like me what this is or what field it's even from. The abstract of the linked paper helps though: "Given a stream of graph edges from a dynamic graph, how can we assign anomaly scores to edges in an online manner, for the purpose of detecting unusual behavior, using constant time and memory? Existing approaches aim to detect individually surprising edges. In this work, we propose MIDAS, which focuses on detecting microcluster anomalies, or suddenly arriving groups of suspiciously similar edges, such as lockstep behavior, including denial of service attacks in network traffic data. MIDAS has the following properties: (a) it detects microcluster anomalies while providing theoretical guarantees about its false positive probability; (b) it is online, thus processing each edge in constant time and constant memory, and also processes the data 162−644 times faster than state-of-the-art approaches; (c) it provides 42%-48% higher accuracy (in terms of AUC) than state-of-the-art approaches."
- specialist 6y agoThanks. I'd love some ELI5s on this. My last gig had some newly minted CS graduates who proposed complimenting our monitoring with some anomaly detection. Having done a bit of both data mining and optimization, a lifetime ago, I could kinda follow the big data and machine learning stuff the kids were doing. Whereas anomaly detection and fault prediction seem magical to me.
- siddhartb_ 6y agoThere is a recorded presentation of the paper at https://youtu.be/Bd4PyLCHrto https://youtu.be/Bd4PyLCHrto The first 5-10 minutes or so should be quite explanatory. Please feel free to let me know if you have any specific doubts. Thanks.
- steve0hh 6y agoThanks for your comments. I know the documentation and examples is still a bit raw, will be working on it. When doing the implementation, I was trying to mimic the original API(https://github.com/bhatiasiddharth/MIDAS https://github.com/bhatiasiddharth/MIDAS) that was done in the C++ first. This was in case someone wants to use the exact same API(for familiarity) in Go, they could. Posted this here to show that there is now a Go implementation of it. :) After this, I'll be implementing a fit/predict api that mimics SKLearn's API and showing more examples on how to use it in a streaming fashion.
- ugh123 6y agoman... again with this project? this is a many-re-submit with astroturfed discussion created by the author
- jchw 6y ago(I have removed the text from this comment since it is past the deletion deadlines, does not adhere to HN guidelines, and was based primarily on speculation that may not be true. Sorry.)
- arun_kumar2 6y agoThe current post is a different Go implementation of the algorithm by https://github.com/steve0hh/ https://github.com/steve0hh/ unlike the previous posts.
- jchw 6y agoI see, that makes more sense. It still doesn't explain what was going on before, but at least this thread doesn't seem to have the same issues previous threads seemed to have.
- tptacek 6y agoIf you're concerned, this belongs in an email to hn@ycombinator.com, not on the thread itself; the guidelines ask specifically for us not to have discussions like this in threads.
- jchw 6y agoThat’s a good point, but unfortunately there is no action to be taken here specifically, since it turns out this thread is not really related. So I have nothing to add. I’m already past the deadline for deleting the comment. I’ll leave it to moderators discretion since I can’t do anything. edit: I am within edit deadline, so I removed the body of the text.