8 ms·
Problems with AI-based monitoring startups (2018)
- matsemann 6y agoLots of monitoring setups are either too simple or too advanced. We had a system accepting a certain kind of applications. We would normally get about ~1500 a day, following a pattern with a steady amount during working hours, many in the evening, and almost nothing during the night. Any human could take a glance at the graph and quickly tell if something was wrong or everything ok. We wanted to catch if something went wrong and people couldn't submit applications. We set up an alarm that triggered "if less than X applications last hour", but that was hard to tweak so it didn't go off during the night and was unusable. The tool had no way to set that rule to only apply for certain time periods or any way of making more advanced rules. But even if it had, I think it would have been a game of whack-a-mole with false positives and obvious errors slipping through. Instead we could set up some kind of AI "anomaly detection". That was almost even worse. Firstly, because no one could tell us how it really worked, how does it know what we consider an anomaly? I mean, if there is a holiday in the middle of the week and we get fewer applications than normal that's an anomaly, but nothing to sound an alarm for.
- social_quotient 6y agoI have similar thoughts here. It’s almost like we don’t need to AI we just need the I. What you have described is a fairly straightforward pattern that seems like it could be expressed by looking at the same period (ago) to see if it’s acting roughly right. It reminds me of a chat I had with a customer in their slack. They wanted build and deploy notifications but only if a build didn’t happen on a day at an expected time. They don’t want 700 notifications a day, it creates a deafness to the signal. The request was comically difficult.
- sarusso 6y agoActually, good anomaly detection can take into account what "normal" is for your use-case. For example, Facebook Prophet (a forecasting procedure, which can in turn be used for anomaly detection) has extensive support for seasonalities and holidays, and would probably work well for the use-case you mentioned.
- a_imho 6y agoFirstly, because no one could tell us how it really worked, how does it know what we consider an anomaly This hits the nail on the head for me. Unless the AI magic handwaving is 100% infallible a human in the loop needs to verify whether there is in fact any problem. Traceability is at least as important as accuracy and statistical methods have a very good track record there.
- craigching 6y agoI don't think of anomaly detection as being the end, it's triage. It shouldn't do more than "I found something that doesn't follow the normal patterns, let me page someone to take a look." And you need to use it strategically, it's not for use everywhere. Like anything, it's a tool and it really depends on how you use it.
- amelius 6y agoI'd like to see anomaly detection as a standard part of (image) classification libraries, as it's incredibly useful to know if a given image is "too far" outside of your trained dataset to be accurately classified.
- jacques_chester 6y ago> Firstly, because no one could tell us how it really worked, how does it know what we consider an anomaly? Some old-fashioned techniques (eg. just triggering on high deviations with adjustment for seasonality) are at least explicable, given a little time. But they don't get much buzz. My own view is that the untreated problem for most attempts to apply time series analysis / anomaly detection / concept drift / process control is that the time series observed are highly variable to begin with. A highly variable series means (1) you need more data to build the predictions, (2) those predictions are less certain and (3) the false positives will be higher. Put another way: the need for such tools is a symptom, and a treatment, but not a cure. Making software more predictable is more value-adding than building ever more elaborate predictors.
- linuxftw 6y agoThe pattern is simple. Collect metrics, filter, apply rules, alert. The problem is, people try to put all of those pieces into a single product. This sometimes works, other times not. "Filter" and "apply rules" is the real work, and it's rather intractable. Some products embed a DSL to attempt to empower users to create these bits themselves. Presumably the "AI" could write the filters and rules by "training" on some data. The AI can't automatically deduce the dimensions of your data like 'consider the time and day of this request' without instruction from the human. It can't conceptualize 'is a holiday/is not a holiday' without input from a human. At the end of the day, just collect the metrics and write a small application with whatever alerting rules you want. This might not scale, but it might not need to.
- eric_khun 6y agoDatadog works really well for us. Their "watchdog" feature , that monitor every metric you send to them, based on anomaly detection, helped us to uncover many issues, with a low % of false positives. Kudos to their AI team (but yeah, Datadog is hella expensive)
- deleted 6y ago[deleted]
- that_guy_iain 6y agoI'm actually working on an AI-based monitoring product[1] that I plan on launching next month. Mines is based on anomaly detection for ecommerce systems with the emphasis on anomalies with the orders. My take is, normally IT just look to see if they're still processing payments and still taking orders. They'll often have a dashboard within their office where they can see the orders and go check that dashboard manually throughout the day. Any process that has manual actions are prone to failure. They don't look for 6 hours or maybe forget for a few days. But even then, if you're looking to ensure that you're still accepting orders, you may be accepting orders, but a random feature may be reducing the sales. An example that I've been told happens a lot, the dropdown to select which variant you want may break. There are also other things other than technical errors that could affect sales, such as changing the layout, it may all work but conversion rates could drop. Noticing a 25% drop in sales is a hard thing to notice by eye and often something that will only get picked up weeks later and cause a whole bunch of painful meetings. Overall, I think anomaly detection is a major thing we need in monitoring. I generally want to have it with my logging which is where I got the idea for my project, and I will be extending it to work with my logging system even if it's just for me. There are so many issues in legacy production systems where if it happens once that's ok, but if it happens 100 times, it's not ok. Writing rules for all of these seems near impossible if you have a large startup system you need to make reliable. [1] https://www.ootliers.com https://www.ootliers.com
- teekert 6y agoWell, now you know why it will fail ;) Kidding of course. In fact I tend to skip titles like this because actually they are not true and that is often even admitted in the text. It annoys me to no end. Good luck with you AI-based monitoring startup!
- that_guy_iain 6y ago> Well, now you know why it will fail ;) I'm preparing my CV already, got to think of the best sounding job title to impress for my next gig. :)
- qayxc 6y ago
- phenkdo 6y agoI think the author is painting with too broad a brush here, companies like Pagerduty, Datadog et al have done very well. Yeah there are many me-too businesses in this space - just like any other, and they are probably doomed. I feel the author is being too harsh here.
- KaiserPro 6y ago> Pagerduty, Datadog et al have done very well. They aren't an AI monitoring company. Pagerduty is a rules engine based alerting system. Datadog is nagios 2.0 on the web, with really aggressive sales people, and really expensive. None of them are "feed me your raw data and I'll make sane alerts and root cause analysis" Which is the core argument of the post, if you can't do alert routing, or root cause pinpointing then AI isn't going to help you. Its like saying that AI is going to make your UX, or backend app.
- phenkdo 6y agotbf ""feed me your raw data and I will..." is any space is a bunch of hooey. Nothing special about the monitoring space.
- phenkdo 6y ago> Pagerduty is a rules engine based alerting system erm that's AI too.
- kristiandupont 6y agoDoes a user have to write the rules?
- tauwauwau 6y agohttps://en.wikipedia.org/wiki/Expert_system https://en.wikipedia.org/wiki/Expert_system Manual rule writing and AI are not mutually exclusive.
- 6y ago
- melomal 6y agoIt's not far off to be fair: https://www.ft.com/content/21b19010-3e9f-11e9-b896-fe36ec32aece https://www.ft.com/content/21b19010-3e9f-11e9-b896-fe36ec32a... Again, buzzwords and whatever get's money in the bank is what leads tech development. Which is probably why we have stagnated on new, radical ideas or at least they are not nearly enough getting awareness.
- aaron695 6y ago> Which is probably why we have stagnated on new, radical ideas or at least they are not nearly enough getting awareness. We don't need new radical ideas. Almost nothing in IT has been new for the past 30 year (Exception - Blockchain) We do prioritise badly. We don't acknowledged real problems. We do consistently pour dev money into scam technologies. We do run around in circles pretending we haven't been here before (like the article mentions) It's the same as any industry. Like quack medicine, and real medicine which is mostly wrong. But IT should have the ability fix itself at a faster rate.
- sgt101 6y ago>Almost nothing in IT has been new for the past 30 year eermm ! - iPhones came out in 2007 and it took maybe 5 years for mass adoption of mobile devices to do work tasks (see iPads in bank branches as an example). This is new. - This year the workforce in much of the world has gone from 90% in the office to 99% at home. This is new. - Data bases have gone from 10's MB to 1000's of GB, and from scores of tables to tens of thousands of tables. - Cloud computing; when was it that your company outsourced it's entire IT infrastructure? I bet it wasn't 30 years ago! - Outsourcing itself, 30 years ago very few people had lumps of their IT done by SI's, this is a huge change that everyone has just got used to. - Security and tfa and firewalls and all that jazz: 30 years ago this was all terribly naive and often implemented with airgaps. I remember one project doing billing automation at a power utility where we installed networking, workstations (in branches) and built a completely independent network out from the mainframes. It did not touch the internet, sign on was via user name and password only with no changing policy. I don't even remember seeing a security policy! - End user consumption; this is big. 30 years ago the c-suite never saw emails. Their secretaries printed them out. Execs didn't type, or look at screens. There was a weekly report and it was given out on friday, reviewed at the weekend and discussed / actioned monday morning. Now every exec has a notebook, and a phone (see above) they want dashboards, they want to explore the data themselves - and they do do that (often badly). Their level of interaction and requirement from IT is orders of magnitude different from 30 years ago. - Lack of cash. Every company outside China with the exception of FAANGS has had the cash squeezed out of it by investors in the last 30 years. 30 years ago utilities and investment companies sat on what (in hindsight) were rivers of cash. Funding was so heavy that there were special offices set up to oversee it's dispersal (who else remembers central program offices?) Now projects are managed in line and are part of 0 build budgets (start every year with everyone on 0 budget and make them make the case to add $1 at a time). It is incredible to think of how we used to act in projects and how empowering and facilitating of progress and quality it was - but also how expensive. People would think nothing of spending 10 or 20 times their yearly salary on machines and software - there was no motivation to imagine what else could be done with the cash! Of course all that spending floated great chunks of the economy along with the spend going into other peoples pockets. It was another world.
- bassdigit 6y agoLook, three paragraphs beginning with "Look," give the article a rather condescending tone.
- exporectomy 6y agoThat grates on me too. I translate it as "Shut up and listen." which causes me to try to disengage from the speaker. Though, here it's aimed at a 3rd party, so it's not quite as offensive.
- helsinkiandrew 6y ago> “We can predict failures. You don’t have to write alert rules any more.” If AI monitoring tools can predict potential failures before they happen based on existing rules and events - then they have value but writing the alert rules defines what you're interested in - what is and is not an acceptable error.
- BorisTheBrave 6y ago> If I have one NOC for all of my services and they’re chasing their tail trying to figure out root causes and who is responsible for each service… you have a management problem. Is it inconcievable to the author that buying software may be easier than fixing a management problem, at least in the short term.
- kristiandupont 6y agoCreating solutions looking for problems is a common issue for would-be founders and it certainly has been for me for years.. It's a bit weird to hear the dismissal of an idea that I could totally have seen myself run with (and agreeing with said dismissal!)
- laichzeit0 6y agoWeird. Moogsoft [1] has been around for years and they're an AI-based monitoring startup. A lot of other monitoring tools (AppDynamics, DynaTrace) incorporate some ML to help with root-cause and alarm deduplication. It's not the core of the product though, it's more like a helpful feature. I personally never found any of that shit to work in practice. [1] https://www.moogsoft.com/ https://www.moogsoft.com/
- phonebucket 6y agoSomewhat of a tangent, but a pet peeve of mine is people misusing ‘by definition’ where there is no clear definition to be leveraged. > By definition, an AI researcher does not have operational experience. Why? What definition?
- thinkloop 6y agoAn AI researcher, by definition, is one who spends their time researching AI, so they would have no time remaining to get operational experience.
- TomasEkeli 6y agothat's just silly. someone with operational experience can switch to ai research later in their career. probably not super common, but nothing definitionally wrong with it
- deleted 6y ago[deleted]
- sakarisson 6y agoA stamp collector, by definition, is someone who spends all of their time collecting stamps, so they would have no time remaining to get operational experience.
- speedgoose 6y agoThere is applied science.
- lwhi 6y agoThey probably have experience of using e-commerce though.
- dijksterhuis 6y agoI'm a researcher in ML security. I also have five years experience in music copyright and royalties. I still know stuff about weird esoteric distribution policies PRS for Music has, like the Educational Recording Agreement or how the pubs and clubs scheme analogous apportionment of £40 million worked. Then there's the digital signals synthesis techniques I've had to learn in the last two years as well. Literally building bare bones software based synthesisers. And then there's all the ML stuff and the security stuff... So I wholeheartedly disagree with your definition. Knowledge is often much fuzzier than person is working on X, therefore they must only know about X.
- sarusso 6y agoCurious. I had to read until mid-article to figure out what kind of monitoring we were talking about. Because for a lot of other monitoring use-case (i.e. infrastructural, environmental, biological etc.) it's another story. Feels like the article has a strong sysadmin/devops bias. Which is fine, but maybe adding "IT" in the title would have been better :)
- trabant00 6y agoThe article should be taken in the context of system administration. And in that context it's on point. When he writes monitoring he means alerts that mean something important is surely broken and that somebody should wake up right now and log on to fight the fire. I myself call that "state monitoring". And I find the Nagios model still the best for this. The other kind I call "trend monitoring". And here you can feed data to some program that might detect anomalies, find correlations between different data deltas and so on. This can be very valuable and obviously a computer can crunch a lot more data a lot faster than humans. For state monitoring I find black box tests to work very good. You don't need to understand how the system works, you don't need any inside data, just try to use it. Have a bot try to buy the product from the web store. It's a lot more reliable than trying to deduce the state of the system from how internal components act and interact. Again, the scope here is to alert somebody that action is surely needed ASAP. Ofc you also need trend monitoring, you don't want to wait until you're out of space when you can project from the current growth trend. But in keeping with that example a data import will always trick the AI into thinking we're heading for disaster so it's not very reliable as a fire alarm.
- omneity 6y ago> some program that might detect anomalies Do you happen to know standard approaches for anomaly detection in this context?
- kalal 6y agoDoes the tone bothers anybody? After reading first couple of sentences I got the impression that the author knows it all and everybody else is just stupid... This may be cultural difference, or too much sensitivity on my side. I don't know.
- tachyonbeam 6y agoIt's not just you. The article starts out with a very inflammatory title. If the author had any regard for effective communication, they would begin by explaining what exactly is a "monitoring startup", but instead they assume that there can really be only one kind of "monitoring". About 3/4 through the article, it becomes clear they probably mean monitoring servers, and not, say, monitoring a physical location with cameras, or monitoring an assembly line or baby monitors. Then, the meat of the article is basically saying "everyone else failed, so you will too. Also, Google can do it better than you." It just sounds snarky and arrogant. IMO much of the point of startups is to be allowed to take risks. If everyone has a conservative attitude that risky ideas shouldn't even be tried, progress never happens. What do you think this guy would have said when Elon Musk announced his plans to launch SpaceX? Good thing some people don't listen to cynical asshats. Also, two-sentence paragraphs. They convey a fundamental misunderstanding of how paragraphs are used.
- xwdv 6y agoIf it was a comment on HN, it’d be downvoted to hell. Instead it’s written as an article and soared to the front page. Maybe I should write my comments as blogs instead, since I have a large amount of downvotes.
- htrp 6y agoYou aren't wrong, the author definitely conveys the impression of the stereotypical toxic engineer.
- coding123 6y agoI think one AI that would be nice to have is something to sell to AWS: A tool that examines data that is publicly on S3 and tries to analyze if it's sensitive data or just media assets, etc... S3 seems to be our largest source of data breaches sadly. There's a whole branch of "security" AI that needs more exploring: shell analysis (hacker or not), IP connection analysis, change analysis. lots of crap.
- michaelbuckbee 6y agoThis is part of what Polyrize (now Varonis) does - https://www.polyrize.com/ https://www.polyrize.com/
- panpanna 6y agoYou don't need ai for that. Can't find the link, but there are a few projects doing just that.
- btbuildem 6y agoA pretty narrow conception of "AI monitoring" -- I'm guessing the author is talking about monitoring in context of IT? Bit annoying tone tbh, like a tadpole in a puddle bragging it's got the ocean all figured out. Failure prediction, process modelling and other applications of AI monitoring in an industrial context are more and more mainstream these days. It's not just startups trying to pitch, it's established players deploying real-world solutions, and infrastructure giants providing the building blocks.
- dicroce 6y agoI think the problem is that these AI solutions should integrate at a lower level. Instead of a giant brain that does everything (and screws it up), just give people ML based tools... and let people hook them up and decide what they need.
- ram_rar 6y ago> Each team should set up their own monitoring and alerting rules. I agree with the general sentiment of it. But in practice this leads to a lot more chaos. There are a lot of benefits of templatizing standards metrics for any service that is launched with right SLI/SLOs. Associating it with high and slow burn rates bifurcates issues that needs immediate attention to be fixed vs something thats slowing eating into your service, but doesn't need to be acted on right away.