7 ms·

OPM – Open PostgreSQL Monitoring

102 points by rayshan 12y ago

tvon 12y ago
The "fork" link goes nowhere, FYI.
- daamien 12y ago
  fixed.
- rjuju 12y ago
  This has been fixed. Thanks for the report !
rpedela 12y ago
I am unsure what some of the graphs measure and labels mean when viewing the demo. Tooltips, descriptions, etc would be helpful in my opinion. Overall it seems like a great start though.
- rjuju 12y ago
  For all PGSQL graphs, the signification of each graph and label is (only :/) documented in the check_pgactivity "--help" option, as it's the direct source of these graphs. Sadly, we didn't have time yet to write a proper documentation on the wiki page, or on the UI. We'll add a better documentation as soon as possible. For the SYSTM graphs, they correspond to standard nagios system checks.
mw44118 12y ago
There's no install instructions in the .zip file, but there is a github wiki. But that has obsolete instructions. But there's an open issue, and the comments for that issue include fixes to that. I stopped messing with it once I saw how they wanted me to install something into root's crontab. My hypothesis: postgresql has none of these jazzy monitoring tools because it never crashes. Sure, as traffic grows, slow queries need to be optimized, crappy N+1 ORMs need to be replaced, and boxes need more RAM and faster disks, but when was the last time you heard of anyone complain about postgresql or blame it for any downtime?
- majorsc2noob 12y ago
  I want to be able to scale out by adding more machines. I want to be able to failover automatically to another data center when the first one goes down. I have yet to see a straight forward way to accomplish this with PG. Their wiki lists a bunch of tools related to this but they are either abandoned or does not cover this as far as I can tell. I can't understand all the positive I read about PG..
  - sitharus 12y ago
    Postgres has had hot standby replication since 9.1, along with streaming replication. https://wiki.postgresql.org/wiki/Hot_Standby https://wiki.postgresql.org/wiki/Hot_Standby As for scaling horizontally at the db level that can be achieved with foreign data wrappers calling out, but isn't built in. While Postgres doesn't rival oracle and ms SQL server in feature checkboxes it's a very solid DBMS with many advanced SQL features, and it's free and open source. You can do a lot without hitting the scaling problems.
    - majorsc2noob 12y ago
      To be honest, I'm pretty tired of the "You can do a lot without hitting the scaling problems.". I hear it all the time. Of course, if you don't have a scaling problem you don't. But when you have, you need a solution to it.
      - AlisdairO 12y ago
        Well, in the world of free software you have a choice - an easily-scaled database system that offers you few guarantees and is very hard to code safely against, or one that is harder to scale but easier to code safely against. That doesn't make PG bad - it's just a choice you make. If you want to pay, you can get a little bit closer to having both. Of course you hear lines like you quoted all the time - that's because it's true. The overwhelming majority of applications don't have needs beyond a master-slave pair of DB systems - and trends are moving in favour of that direction every day as RAM gets cheaper. Stack Overflow runs on a single not-that-beefy master-slave DB system. Typical web applications are insanely amenable to read caches, and that's where you do most of your scaling. If you're at the point where your scaling needs truly exceed a large single DB system, I'd hope you have some money to throw at the problem.
Cieplak 12y ago
htop like application for PostgreSQL server activity monitoring: https://github.com/julmon/pg_activity/ https://github.com/julmon/pg_activity/
STRiDEX 12y ago
I'm not sure how these two projects are different, but they seem to be worked on by the same people/person. http://dalibo.github.io/powa/ http://dalibo.github.io/powa/
- moondowner 12y ago
  Just noticed that as well. I wonder why two different projects for the same thing? Are there any differences between OPM and PoWA? Is OPM built on top of PoWA?
  - daamien 12y ago
    Hi ! I'm involved in both projects. Here' s some insight : OPM and POWA are not related. They were born at different times and developed by two different R&D teams at DALIBO, a French PostgreSQL company. The fact that they've been released publicly almost at the same time is merely a coincidence OPM is designed to help a DBA to manage a very large number of servers. It provides answers to the questions : "Is there a problem somewhere on my PostgreSQL servers ? Which instances need to improvements ?" POWA is more focused showing the "real-time" activity of one specific instance. It provides answers to questions like : "Which queries are slowing down my server right now ?" If you're familiar with Oracle products, the difference between OPM and POWA is more-or-less the difference between Oracle AWR and Oracle Grid Control. In a nutshell, if you only have one or two PostgreSQL servers in production, then POWA is easier to install and probably enough for you. If you want more stats and an overview of your servers, then OPM is a better choice. Both projects are autonomous and moving fast. We're just pushing code on github without direct economic goals or marketing plans :-) In the long run we'll see which one finds its user base and how they evolve. That being said, we also have other guys working on other similar PostgreSQL tools, in particular : pgBadger ( http://dalibo.github.io/pgbadger/ http://dalibo.github.io/pgbadger/ ) is a log analyzer which provides rich reports of the database activity. Basically it answers to the question "Which queries should be optimized ?" pgCluu ( http://pgcluu.darold.net/ http://pgcluu.darold.net/ ) is a audit/reporting tools, more focused on system and resource utilization. It answers to questions such as "What's the load on my CPUs ? Is my memory swapping ?" pg_activity ( https://github.com/julmon/pg_activity/ https://github.com/julmon/pg_activity/ ): is like linux "top" command but for PostgreSQL queries. pg_stat_kcache ( https://github.com/dalibo/pg_stat_kcache https://github.com/dalibo/pg_stat_kcache ) will gather statistics about reads done by the filesystem layer So basically regarding PostgreSQL monitoring, we're just moving in every possible directions at the same time... Some say it's a "waste of time", we prefer the term "Darwinian Approach" :P
sarnowski 12y ago
Since we (Zalando) are heavy PostgreSQL users, we also had the need for monitoring and part of the result we came up with is http://zalando.github.io/PGObserver/ http://zalando.github.io/PGObserver/ and https://github.com/zalando/pg_view https://github.com/zalando/pg_view
davidw 12y ago
Minor grammar nitpick: "Open PostgreSQL Monitoring is a free software" is incorrect English, as software is an uncountable: we don't say "a software", but a program, a system, or something like that. You could also say "is free software".
- daamien 12y ago
  fixed.
therealmarv 12y ago
Is there anything similar for Newrelic out there? Nice project btw.

102 points by rayshan 12y ago

tvon 12y ago

The "fork" link goes nowhere, FYI.

daamien 12y ago
fixed.
rjuju 12y ago
This has been fixed. Thanks for the report !

rpedela 12y ago

I am unsure what some of the graphs measure and labels mean when viewing the demo. Tooltips, descriptions, etc would be helpful in my opinion. Overall it seems like a great start though.

rjuju 12y ago
For all PGSQL graphs, the signification of each graph and label is (only :/) documented in the check_pgactivity "--help" option, as it's the direct source of these graphs. Sadly, we didn't have time yet to write a proper documentation on the wiki page, or on the UI. We'll add a better documentation as soon as possible. For the SYSTM graphs, they correspond to standard nagios system checks.

mw44118 12y ago

There's no install instructions in the .zip file, but there is a github wiki. But that has obsolete instructions. But there's an open issue, and the comments for that issue include fixes to that. I stopped messing with it once I saw how they wanted me to install something into root's crontab. My hypothesis: postgresql has none of these jazzy monitoring tools because it never crashes. Sure, as traffic grows, slow queries need to be optimized, crappy N+1 ORMs need to be replaced, and boxes need more RAM and faster disks, but when was the last time you heard of anyone complain about postgresql or blame it for any downtime?

majorsc2noob 12y ago
I want to be able to scale out by adding more machines. I want to be able to failover automatically to another data center when the first one goes down. I have yet to see a straight forward way to accomplish this with PG. Their wiki lists a bunch of tools related to this but they are either abandoned or does not cover this as far as I can tell. I can't understand all the positive I read about PG..
- sitharus 12y ago
  Postgres has had hot standby replication since 9.1, along with streaming replication. https://wiki.postgresql.org/wiki/Hot_Standby https://wiki.postgresql.org/wiki/Hot_Standby As for scaling horizontally at the db level that can be achieved with foreign data wrappers calling out, but isn't built in. While Postgres doesn't rival oracle and ms SQL server in feature checkboxes it's a very solid DBMS with many advanced SQL features, and it's free and open source. You can do a lot without hitting the scaling problems.
  - majorsc2noob 12y ago
    To be honest, I'm pretty tired of the "You can do a lot without hitting the scaling problems.". I hear it all the time. Of course, if you don't have a scaling problem you don't. But when you have, you need a solution to it.
    - AlisdairO 12y ago
      Well, in the world of free software you have a choice - an easily-scaled database system that offers you few guarantees and is very hard to code safely against, or one that is harder to scale but easier to code safely against. That doesn't make PG bad - it's just a choice you make. If you want to pay, you can get a little bit closer to having both. Of course you hear lines like you quoted all the time - that's because it's true. The overwhelming majority of applications don't have needs beyond a master-slave pair of DB systems - and trends are moving in favour of that direction every day as RAM gets cheaper. Stack Overflow runs on a single not-that-beefy master-slave DB system. Typical web applications are insanely amenable to read caches, and that's where you do most of your scaling. If you're at the point where your scaling needs truly exceed a large single DB system, I'd hope you have some money to throw at the problem.

Cieplak 12y ago

htop like application for PostgreSQL server activity monitoring: https://github.com/julmon/pg_activity/ https://github.com/julmon/pg_activity/

STRiDEX 12y ago

I'm not sure how these two projects are different, but they seem to be worked on by the same people/person. http://dalibo.github.io/powa/ http://dalibo.github.io/powa/

moondowner 12y ago
Just noticed that as well. I wonder why two different projects for the same thing? Are there any differences between OPM and PoWA? Is OPM built on top of PoWA?
- daamien 12y ago
  Hi ! I'm involved in both projects. Here' s some insight : OPM and POWA are not related. They were born at different times and developed by two different R&D teams at DALIBO, a French PostgreSQL company. The fact that they've been released publicly almost at the same time is merely a coincidence OPM is designed to help a DBA to manage a very large number of servers. It provides answers to the questions : "Is there a problem somewhere on my PostgreSQL servers ? Which instances need to improvements ?" POWA is more focused showing the "real-time" activity of one specific instance. It provides answers to questions like : "Which queries are slowing down my server right now ?" If you're familiar with Oracle products, the difference between OPM and POWA is more-or-less the difference between Oracle AWR and Oracle Grid Control. In a nutshell, if you only have one or two PostgreSQL servers in production, then POWA is easier to install and probably enough for you. If you want more stats and an overview of your servers, then OPM is a better choice. Both projects are autonomous and moving fast. We're just pushing code on github without direct economic goals or marketing plans :-) In the long run we'll see which one finds its user base and how they evolve. That being said, we also have other guys working on other similar PostgreSQL tools, in particular : pgBadger ( http://dalibo.github.io/pgbadger/ http://dalibo.github.io/pgbadger/ ) is a log analyzer which provides rich reports of the database activity. Basically it answers to the question "Which queries should be optimized ?" pgCluu ( http://pgcluu.darold.net/ http://pgcluu.darold.net/ ) is a audit/reporting tools, more focused on system and resource utilization. It answers to questions such as "What's the load on my CPUs ? Is my memory swapping ?" pg_activity ( https://github.com/julmon/pg_activity/ https://github.com/julmon/pg_activity/ ): is like linux "top" command but for PostgreSQL queries. pg_stat_kcache ( https://github.com/dalibo/pg_stat_kcache https://github.com/dalibo/pg_stat_kcache ) will gather statistics about reads done by the filesystem layer So basically regarding PostgreSQL monitoring, we're just moving in every possible directions at the same time... Some say it's a "waste of time", we prefer the term "Darwinian Approach" :P

sarnowski 12y ago

Since we (Zalando) are heavy PostgreSQL users, we also had the need for monitoring and part of the result we came up with is http://zalando.github.io/PGObserver/ http://zalando.github.io/PGObserver/ and https://github.com/zalando/pg_view https://github.com/zalando/pg_view

davidw 12y ago

Minor grammar nitpick: "Open PostgreSQL Monitoring is a free software" is incorrect English, as software is an uncountable: we don't say "a software", but a program, a system, or something like that. You could also say "is free software".

daamien 12y ago
fixed.

therealmarv 12y ago

Is there anything similar for Newrelic out there? Nice project btw.