5 ms·

If anyone has looked at web frameworks in Rust, they'll have noticed one known as Rocket (http://rocket.rs http://rocket.rs). The Rocket project is authored by

by Dowwie 9y ago

If anyone has looked at web frameworks in Rust, they'll have noticed one known as Rocket (http://rocket.rs http://rocket.rs). The Rocket project is authored by none other than Sergio Benitez, who is teaching Stanford CS140e.

mcguire 9y ago
How is Rocket, compared to the others? (I don't know that the one I used before is still going.)
- robto 9y ago
  There's a really good talk from StrangeLoop about it[0], I highly recommend checking it out. I don't write rust, but it made me want to. [0]https://www.youtube.com/watch?v=QS8mrbAPLJc https://www.youtube.com/watch?v=QS8mrbAPLJc
akhilcacharya 9y ago
I really should not have read that resume. Really doesn't help the overall inadequacy problem.
- shawn 9y ago
  On the other hand: At Google I worked with the Gmail performance team to decrease server downtime by detecting anomalous behavior before server failure. I designed and implemented an original anomaly detection algorithm based on local outlier factors. Does anyone have any info about anything related to this? I want to write a program to detect anomalous behavior in my systems too. (Mostly so I can completely ignore the useful information, but still.)
  - adtac 9y ago
    I can talk about my experience writing such a tool during my internship at a bank that rhymes with Oldman Bachs. Bear in mind, this wasn't my original project; since I had finished my first project, I was allowed to work on something else I find interesting. So it will probably not be very robust, but it worked. The tool would run in trading engines and monitor different metrics of performance. CPU usage, while allowed to burst, shouldn't be sustained at 100% for a long time. Typically, these machines are monsterous and have hundreds of gigs of memory; therefore, they shouldn't come anywhere near 100% memory usage either. There's a bunch of other proprietary internal metrics that I can't really explain (stuff like latency of a particular kind of response). Anyway, since I was most comfortable with writing the tool in Python (also, time was a constraint as I started working on the project towards the end of my internship), I wrote a modular tool with pluggable metrics. The reason why it wasn't a monolithic block: if the company decides to add a new metric they want to measure and follow, they should just have to write the logic necessary to retrieve that metric. The rest should be handled by my framework: anomaly detection, alerting relevant parties, and so on. So banged out the framework and had just enough time to write two modules: CPU usage and memory usage (probably the easiest ones). With a simple moving window average and standard deviation, you can identify outliers: I marked everything outside 2*sigma as an outlier as it's better to have false positives than false negatives (but not excessively many false positives as that would be counter-productive and make humans ignore reports). The alerting part was fairly straightforward as there was a built-in tool to do that. I ran a few tests with historical data and it was pretty good at detecting the anomalies that I would mark myself as a human. It could be better if someone smarter than could come up with a machine learning algo to do the same thing more accurately. (I didn't productionize it as I didn't have the time.)
  - iandanforth 9y ago
    You might also be interested in this older project from etsy https://github.com/etsy/skyline https://github.com/etsy/skyline Has a fairly good mixture of models approach to anomaly detection even if it's not maintained.
  - jradd 9y ago
    First thing that comes to mind is tracing system calls with strace (linux), truss (bsd), or Dtrace (recommended). A common probability theory such as Markov Chains could be used as a reference: Markov Chains http://cucis.ece.northwestern.edu/projects/DMS/publications/AnomalyDetection.pdf http://cucis.ece.northwestern.edu/projects/DMS/publications/... There is no limit of items this model could be applied to, but a few that come to mind; given the context might be; Load Generation or emulation at the filesystem, network or application level. iowait, vmstats, memory leaks (Why am I unable to account for N percent memory), Cache hit vs miss, Network stats, heuristics, buffer overflows, denial of service (listen/accept queue overflow), TCP RTT, NTP Jitter, Response Time, etc…
  - massaman_yams 9y ago
    Netflix has some good posts on outlier / anomaly detection. https://medium.com/netflix-techblog/tracking-down-the-villains-outlier-detection-at-netflix-40360b31732 https://medium.com/netflix-techblog/tracking-down-the-villai... https://medium.com/netflix-techblog/rad-outlier-detection-on-big-data-d6b0494371cc https://medium.com/netflix-techblog/rad-outlier-detection-on...
mark_l_watson 9y ago
Thanks for that call-out for Rocket. I skimmed a Rust book over the holidays so I could experiment with the Exonum blockchain platform. I had tentatively decided to not pursue an interest in Rust but looking through the docs for Rocket has changed my mind. Looking through the CS140e class description made me a little envious of students who are both able to take the class live and also have 20 hours a week or so to really get the most out of the class.

by Dowwie 9y ago

mcguire 9y ago

How is Rocket, compared to the others? (I don't know that the one I used before is still going.)

robto 9y ago
There's a really good talk from StrangeLoop about it[0], I highly recommend checking it out. I don't write rust, but it made me want to. [0]https://www.youtube.com/watch?v=QS8mrbAPLJc https://www.youtube.com/watch?v=QS8mrbAPLJc

akhilcacharya 9y ago

I really should not have read that resume. Really doesn't help the overall inadequacy problem.

shawn 9y ago
On the other hand: At Google I worked with the Gmail performance team to decrease server downtime by detecting anomalous behavior before server failure. I designed and implemented an original anomaly detection algorithm based on local outlier factors. Does anyone have any info about anything related to this? I want to write a program to detect anomalous behavior in my systems too. (Mostly so I can completely ignore the useful information, but still.)
- adtac 9y ago
  I can talk about my experience writing such a tool during my internship at a bank that rhymes with Oldman Bachs. Bear in mind, this wasn't my original project; since I had finished my first project, I was allowed to work on something else I find interesting. So it will probably not be very robust, but it worked. The tool would run in trading engines and monitor different metrics of performance. CPU usage, while allowed to burst, shouldn't be sustained at 100% for a long time. Typically, these machines are monsterous and have hundreds of gigs of memory; therefore, they shouldn't come anywhere near 100% memory usage either. There's a bunch of other proprietary internal metrics that I can't really explain (stuff like latency of a particular kind of response). Anyway, since I was most comfortable with writing the tool in Python (also, time was a constraint as I started working on the project towards the end of my internship), I wrote a modular tool with pluggable metrics. The reason why it wasn't a monolithic block: if the company decides to add a new metric they want to measure and follow, they should just have to write the logic necessary to retrieve that metric. The rest should be handled by my framework: anomaly detection, alerting relevant parties, and so on. So banged out the framework and had just enough time to write two modules: CPU usage and memory usage (probably the easiest ones). With a simple moving window average and standard deviation, you can identify outliers: I marked everything outside 2*sigma as an outlier as it's better to have false positives than false negatives (but not excessively many false positives as that would be counter-productive and make humans ignore reports). The alerting part was fairly straightforward as there was a built-in tool to do that. I ran a few tests with historical data and it was pretty good at detecting the anomalies that I would mark myself as a human. It could be better if someone smarter than could come up with a machine learning algo to do the same thing more accurately. (I didn't productionize it as I didn't have the time.)
- iandanforth 9y ago
  You might also be interested in this older project from etsy https://github.com/etsy/skyline https://github.com/etsy/skyline Has a fairly good mixture of models approach to anomaly detection even if it's not maintained.
- jradd 9y ago
  First thing that comes to mind is tracing system calls with strace (linux), truss (bsd), or Dtrace (recommended). A common probability theory such as Markov Chains could be used as a reference: Markov Chains http://cucis.ece.northwestern.edu/projects/DMS/publications/AnomalyDetection.pdf http://cucis.ece.northwestern.edu/projects/DMS/publications/... There is no limit of items this model could be applied to, but a few that come to mind; given the context might be; Load Generation or emulation at the filesystem, network or application level. iowait, vmstats, memory leaks (Why am I unable to account for N percent memory), Cache hit vs miss, Network stats, heuristics, buffer overflows, denial of service (listen/accept queue overflow), TCP RTT, NTP Jitter, Response Time, etc…
- massaman_yams 9y ago
  Netflix has some good posts on outlier / anomaly detection. https://medium.com/netflix-techblog/tracking-down-the-villains-outlier-detection-at-netflix-40360b31732 https://medium.com/netflix-techblog/tracking-down-the-villai... https://medium.com/netflix-techblog/rad-outlier-detection-on-big-data-d6b0494371cc https://medium.com/netflix-techblog/rad-outlier-detection-on...

mark_l_watson 9y ago

Thanks for that call-out for Rocket. I skimmed a Rust book over the holidays so I could experiment with the Exonum blockchain platform. I had tentatively decided to not pursue an interest in Rust but looking through the docs for Rocket has changed my mind. Looking through the CS140e class description made me a little envious of students who are both able to take the class live and also have 20 hours a week or so to really get the most out of the class.