7 ms·
Introducing Thanos: Prometheus at Scale
- nickthemagicman 8y agoThe name Thanos is not the greatest branding right now.
- hhsnopek 8y agoBecause of the Avengers or?
- nickthemagicman 8y agoI mean it's the 5th highest grossing film of all time and still moving up.
- nickthemagicman 8y agoSorry film is a strong word for it. Movie I should say.
- chrisseaton 8y agoHow is 'film' a strong word? What's the difference between a 'film' and a 'movie'? I'd normally call them films myself. Does it mean anything else to you?
- ryanwaggoner 8y agoThe distinction is a snobby put-down of a popular film :)
- bkul_ 8y agoMy guess is they're using "film" as a term for a movie with greater redeeming value, sort of like the "literature" distinction with books. Pedantically, Infinity War was likely not recorded on actual film.
- curun1r 8y agoFilm is inappropriate for most modern movies because they're no longer filmed, they're digitally recorded and distributed. When there's no actual film stock used in the production of a movie, you shouldn't call it a film. In the case of a Marvel movie, it's almost more accurate to call it an animation, but movie covers all production methods.
- arcticfox 8y agoHuh, funny thing, I can film something using my cellphone camera no problem. The power of language.
- curun1r 8y agoNo, you can't. That's a malapropism that's become common as a result of non-technical people appropriating a technical term. Just because a lot of people say it, doesn't make it right.
- chrisseaton 8y agoIt’s not a malapropism - that means accidentally using a word that sounds similar, not just using any incorrect word.
- arbie 8y agoThe irony is blinding.
- ryanwaggoner 8y agoJust because a lot of people say it, doesn't make it right. Actually, that’s exactly how the English language works. There’s no governing body that matters who determines what counts as correct English. The closest thing is the dictionaries but those change all the time, in response to....people using words differently.
- curun1r 8y ago
- yathern 8y agoIf you consider yourself having more sophisticated, artistic taste in movies, you may reserve the term 'film' for those. For the movie snob, Casablanca is a film, Spiderman 4 is a movie. Nothing too wrong with making that separation in my mind. I agree with the sentiment that Avengers may not be the most sophisticated movie out there. Most people know this and would agree. But randomly calling attention to it to make sure people notice that you have a sophisticated taste is a little pretentious.
- chrisseaton 8y ago> If you consider yourself having more sophisticated, artistic taste in movies, you may reserve the term 'film' for those. For the movie snob, Casablanca is a film, Spiderman 4 is a movie. I had no idea this was a distinction. I just thought 'film' was what the British called them and 'movie' was the Americanism. We say 'I'm going to the see a film' rather than 'going to see a movie'.
- abiox 8y agodoes this account for inflation?
- ashleyn 8y agoAlmost immediately, a joke about half my data disappearing came to mind.
- ayekat 8y agoFrom Wikipedia: > In the Western classical tradition, Prometheus became a figure who represented human striving, particularly the quest for scientific knowledge, and the risk of overreaching or unintended consequences. In particular, he was regarded in the Romantic era as embodying the lone genius whose efforts to improve human existence could also result in tragedy [...] ... effectively, Thanos is this at large scale.
- nickthemagicman 8y agoYour data is using up too many resources!
- zitterbewegung 8y agoThat and simple confusion. Also, it will get harder and harder to Google as time goes on especially with the next movie coming out.
- zython 8y ago... Balanced. As all things should be. or alternatively: Mr. database engineer. I dont feel so good.
- ryandrake 8y agoI don’t know what a Thanos is, but upon first reading my brain said “Theranos”. Probably not a great brand typo-association either.
- bobwaycott 8y agoI did exactly this when I read the title, as well. I was very confused for a couple seconds.
- Diederich 8y agoI would love to hear any experiences folks here have with this. We are seriously looking at it right now.
- witcher 8y ago(disclaimer: Blog post co-author) You are welcome to join our growing community to know more. (: Follow slack join button here: https://github.com/improbable-eng/thanos https://github.com/improbable-eng/thanos
- tostaki 8y agoI suggest the second part of this talk from the last Kubecon https://www.youtube.com/watch?v=IpGfmmJ2hcw https://www.youtube.com/watch?v=IpGfmmJ2hcw
- Diederich 8y agoFantastic, thanks!
- MetalMatze 8y agoWow. Nice to see my talk here. If you have any questions, feel free to ask.
- Dowwie 8y agoWhat app did you use to make those beautiful diagrams?
- witcher 8y agoMy hands + Google Drawing (: (blog post co-author here)
- 52-6F-62 8y agoNot directly related, but I just watched the demonstration video for SURVIVAL— I am pretty blown away by the apparent scale of it. I haven't heard of Improbable before now and it looks like so much work has been done already. Has anybody worked with SpatialOS here yet?
- domgreenhn 8y agohttps://youtu.be/lGWON5TtS04 https://youtu.be/lGWON5TtS04 so cool
- mwitkow 8y ago(disclaimer: I work at Improbable) The reason why we built Thanos was to enable monitoring of large scale simulation systems, which are inherently stateful, such as the Survival demo (link https://youtu.be/lGWON5TtS04 https://youtu.be/lGWON5TtS04).
- 52-6F-62 8y agoOh yes I already caught the demo. So Thanos was your solution for building the energy economy and the larger ecosystem? The energy simulation is what really struck me as a fundamentally challenging element in a game. Maybe largely because in the first world we don't really have to face that problem at its root anymore. Frankly, I also want to know if there are plans to release something based on that demo
- mwitkow 8y agoWe use Thanos to provide the observability features (monitoring in particular) to Workers (i.e. user processes we run on our Cloud) that perform the simulation. You can have multiple Workers collaborating on a simulation of the economics or ecology that export monitoring variables that you want to track. Since the simulation is inherently dynamic, and the number of Workers can change, Thanos helps us with achieving the necessary scale and retention for a hosted platform that is SpatialOS.
- 8y ago
- sytse 8y agoReally great project. Instead of a fragile centralized approach Thanos embraced the federated nature of Prometheus. We see us using this for GitLab.com. I just discussed Thanos with Ben who works on the Prometheus project itself and who leads monitoring for GitLab: https://youtu.be/JzlwwGZ3yQ4 https://youtu.be/JzlwwGZ3yQ4
- INTPenis 8y agoMy interpretation is that Thanos is for massive scale, perhaps suitable for gitlab but most organisations would manage with just prometheus to start with and might scale up to thanos later. Is this a correct observation?
- sytse 8y agoYes, a single Prometheus server can probably handle the scale of most SaaS applications.
- witcher 8y agoEither that or just use basic Thanos setup (as described in a blog post) that gives you better Prometheus HA support and global view. You can always add setup for long-term metric retention later on.
- irrational 8y agoRead the title - so disappointed this was not an Avengers - Aliens crossover.
- stephengillie 8y agoIf only it were hosted on Pandora Nexus. (Unoriginal names are a pet peeve of mine. This is worse - it's "drafting" like a racecar behind an unrelated recent marketing campaign.)
- bboreham 8y agoGreat project - just wanted to mention Cortex which is a different take on long-term large-scale Prometheus. https://github.com/weaveworks/cortex https://github.com/weaveworks/cortex Best to read the design document [1] to learn more about the different choices that Cortex makes. Also no connotations of large-scale destruction :-) (Note I work on Cortex) 1: http://goo.gl/prdUYV http://goo.gl/prdUYV
- audip 8y agoIs Weaveworks Cortex completely open source? Does it require Weave Cloud subscription to post messages to Slack?
- bboreham 8y agoYes it is Open Source: the whole project is on GitHub, Apache 2 licenced. Posting to Slack (and OpsGenie, PagerDuty, etc) is a feature of the Prometheus AlertManager, which Cortex builds upon, no subscription required. The commercial Weave Cloud product gives you a hosted instance of the same code, storage - we ingest all your metrics and store them for a year - and a nice GUI with user logins, team permissions, etc. Plus the Deploy and Explore features which are hosted versions of two more Open Source projects.
- AaronM 8y agoCan grafana query Thanos directly? or will a datasource plugin be required?
- tedreed 8y agoThe Thanos query nodes have the same interface as Prometheus itself, including the web UI (with a few small changes), so you can just use the same Prometheus plugin pointed at Thanos.
- latchkey 8y agoStoring the Prometheus data in long term storage raises one question for me... what is the process for upgrading the TSDB data format when it changes over time?
- fabr 8y agoThe format already went through one format change since Thanos was started. The format encodes a version itself and Thanos simply supports reading multiple ones.
- kintoandar 8y agoThanos seems like the closest thing to a silver bullet for Prometheus missing features (as by design). Quick question: In a multi Prometheus setup, if all the Thanos nodes are behind a load balancer (without sticky sessions), do a particular query from a dashboard interface like Grafana to that Load Balancer result in the same dataset, if run multiple times?
- witcher 8y agoIf by "Thanos nodes" you mean Thanos querier instances, then yes -> Does not matter to which one you actually ask. All have the same view and access to the old metrics (Store Gateway) and fresh ones (Prometheus+Sidecar - Scraper)
- kintoandar 8y agoThanks, that's exactly what I was looking for