6 ms·

Osquery

224 points by duck 9y ago

kbd 9y ago
Posted because it came up in the discussion of FSQL: https://news.ycombinator.com/item?id=14343967 https://news.ycombinator.com/item?id=14343967 Previous discussions: Release: https://news.ycombinator.com/item?id=8528460 https://news.ycombinator.com/item?id=8528460 Windows release: https://news.ycombinator.com/item?id=12600790 https://news.ycombinator.com/item?id=12600790 (Links courtesy of https://news.ycombinator.com/item?id=14344549 https://news.ycombinator.com/item?id=14344549) Personally upvoted because it's news to me that it supports Windows now, and because it still appears to be very actively developed despite existing for years now.
- nthcolumn 9y ago
  I only heard about this yesterday on that thread. And by facebook? I really should pay more attention to what they are doing over there... FSQL is pretty cool too and surprisingly fast.
dduarte 9y ago
Can it be used to find Windows machines without MS17-010 installed? :)
- jeremymcanally 9y ago
  Definitely. And you could write queries for the various indicators/files/etc., too (or use the built in YARA support and grab the rules from US-CERT). I just started working at Kolide (http://kolide.com http://kolide.com) a couple weeks ago where we're building a whole product on top of osquery. I'm constantly surprised by "can it do (x)" and the answer is almost always yes. It's pretty solid!
lmeyerov 9y ago
If you're in a security team adding this to your stack, we're building a visual playbook & pivoting system, and have been wanting to experiment with scenarios that include an osquery connector. Can share more @ info@graphistry.com , would love to explore!
- dfc 9y ago
  What is a "visual playbook and pivoting system"?
  - allanmacgregor 9y ago
    We don't know but sales said those terms and trending hard, ohh and we also have AI .... not sure in which part of the system but we do!!!!!!
    - lmeyerov 9y ago
      Edit: I guess the above post was trolling, but I think it's fair to ask what pivoting & playbooks are. They're coming from the investigative & ops sides of the enterprise security world. Not terms I used when I was on the R&D side, but useful when they describe big chunks of your job: -- "Visual pivoting" is where a visual tool -- typically in node-link diagrams like Maltego, Palantir, and ours (Graphistry) -- lets you click on an entity like a user etc, and get the result of running queries. For example, "according to vpn/dhcp logs, what hostname was this yesterday? Did the IDS logs say anything funny about that hostname?", or, "Looks like this patient had a bad experience with that doctor, what are the interactions other patients had with that doctor?" AFAICT, "pivoting" became the shorthand for "a left-join, including across different APIs instead of within one database". -- We're using "Visual playbooks" to describe a way of helping turn a team's text documents describing their manual operating procedures for incident investigation & response into interactive software. They often can't just automatically trigger chef/puppet scripts because someone needs to check incident data. For example, we find a lot of teams want help triaging an endpoint incident alerts: resolving the host/user involved, what other alerts were happening for them, were other users/hosts getting similar alerts, and if external IPs are involved, what threat intel feeds say + is anyone else internally talking to those. A visual playbook helps you kick off an executable sequence of context-gathering pivots + maybe API action calls, and instead of returning a text file / dashboard, drops you into a visual analytics session that shows you how the pivot results chain together. So basically visual etl + visual analytics + visual querying. It's new but fun. Hopefully you can see why osquery is a nice tool for some of the steps within an investigation... or visual playbook ;-)
burdzwastaken 9y ago
cannot upvote this enough! seriously great open source project. I have been able to use it in large scale cross-platform deployments with great success.
onida 9y ago
Outside of facebook, what's the largest install base of osquery?
- mavam 9y ago
  We have an ongoing project and existing prototype that combines OSquery with the Bro network security monitor. This enables correlation of network and host state in the same framework. More details here: https://www.bro.org//bro4pros2017/Haas_OSquery_Bro4Pros2017.pdf https://www.bro.org//bro4pros2017/Haas_OSquery_Bro4Pros2017....
josteink 9y ago
I realize WMI[1] is a Windows-thing only, but if you're already on the Windows-platform, WMI comes shipped out of the box and has been for probably a decade or more, and is overall very well supported. It has its quirks, but there's practically no part of the system you shouldn't be able to get information about using WQL[2]. [1] https://msdn.microsoft.com/en-us/library/ms186146(vs.80).aspx https://msdn.microsoft.com/en-us/library/ms186146(vs.80).asp... [2] https://en.m.wikipedia.org/wiki/WQL https://en.m.wikipedia.org/wiki/WQL
- Already__Taken 9y ago
  The nice thing osquery can have is disconnecting what's currently turned on and connected this second from what records you're getting back. OSquery can pump results into a DB that you look through, WMI I've got to check for myself what's turned on, did it all report etc. A driver for wmi in osquery would be a great addition.
kossmoboleat 9y ago
I've worked on something similar except that we used a stream database. Today you might use RethinkDB or something similar. The osquery events seem to solve these kind of monitoring/alerting problems. But I didn't find easily find their documentation.
usrme 9y ago
Can anyone shed some light on how exactly Osquery can be used to leverage checking, for example, of vulnerabilities? I wasn't able to gather information on how it exactly communicates with the rest of the infrastructure. Am I correct to understand that I can set up a single host that has Osquery installed and it can go out and automatically query a heterogeneous set of other hosts based on pre-existing configuration files, given that (I assume) some set of ports are open?
- lmeyerov 9y ago
  Indeed -- Osquery, and the commercial precursor Tanium and the native Microsoft equivalents, can help when you've identified one incident and want to find active repeats. Ex: "I just cleaned evil.exe from this box, are any other hosts running the same process? What other processes do they have in common?". I've seen hunting use cases as well. Your example is why we're building our visual playbook system. In this case, query your network scanner for open ports, and then feed the identified hosts into more targeted osquery calls.
  - usrme 9y ago
    Thank you for the reply; however, I am still having a hard time understanding how exactly Osquery works under the hood. How does it communicate with other hosts? With what does it query for information once it reaches a remote node? What kind of overhead does it have in terms of network transfer when querying? Maybe I am completely off base and am mistaking how this works... Is Osquery set up on every individual host and I query for that information remotely using whatever tools I have at my disposal?
    - fiorix 9y ago
      You can configure osquery to execute periodic queries (scheduled queries) of all kinds: computing md5 of your binaries and other files, taking a snapshot of sockets/connections per process, and so on. By default, osquery uses glog, which means it'll output the results to a local file that you can ship anywhere you want. There's also logging plugins to help you push the results of scheduled queries to other systems. Once you have that data flowing through your pipelines you can start doing security/anomaly detection on things.
xcopy 9y ago
like
jvgiuiigvuuc 9y ago
Useless out of the box: "If there are no USB devices added or removed to the laptop, this query would never log a result again. The query would still run every 60 seconds but the results would match the previous run and thus no state change would be detected. If a USB memory stick was inserted and left in the laptop for 60 seconds the daemon would log" So if you unplug your evil USB earlier, this won't tell you. I wonder if they've heard of udev.
- JosephRedfern 9y ago
  Oh please. It may be useless out of the box at detecting evil USB devices, but to accuse it of being outright useless is ridiculous. I'm sure it was sarcastic snark, but of course they have heard of udev: https://github.com/facebook/osquery/search?q=udev https://github.com/facebook/osquery/search?q=udev.
DominoTree 9y ago
Not directly related to Osquery, but we're using Tanium to run queries against ~180,000 endpoints and we get good results back in about a minute.
- rbolla 9y ago
  Tanium says, the max # of endpoints that the "core" can support is 33K. How did you scaled this to 180 K.
tlrobinson 9y ago
How do you issue queries programmatically from another program? Just execute "osqueryi"? I see mention of a Thrift API but no details.
drumttocs8 9y ago
Can this be used to troubleshoot DPC-related audio latency issues?

224 points by duck 9y ago

kbd 9y ago

Posted because it came up in the discussion of FSQL: https://news.ycombinator.com/item?id=14343967 https://news.ycombinator.com/item?id=14343967 Previous discussions: Release: https://news.ycombinator.com/item?id=8528460 https://news.ycombinator.com/item?id=8528460 Windows release: https://news.ycombinator.com/item?id=12600790 https://news.ycombinator.com/item?id=12600790 (Links courtesy of https://news.ycombinator.com/item?id=14344549 https://news.ycombinator.com/item?id=14344549) Personally upvoted because it's news to me that it supports Windows now, and because it still appears to be very actively developed despite existing for years now.

nthcolumn 9y ago
I only heard about this yesterday on that thread. And by facebook? I really should pay more attention to what they are doing over there... FSQL is pretty cool too and surprisingly fast.

dduarte 9y ago

Can it be used to find Windows machines without MS17-010 installed? :)

jeremymcanally 9y ago
Definitely. And you could write queries for the various indicators/files/etc., too (or use the built in YARA support and grab the rules from US-CERT). I just started working at Kolide (http://kolide.com http://kolide.com) a couple weeks ago where we're building a whole product on top of osquery. I'm constantly surprised by "can it do (x)" and the answer is almost always yes. It's pretty solid!

lmeyerov 9y ago

If you're in a security team adding this to your stack, we're building a visual playbook & pivoting system, and have been wanting to experiment with scenarios that include an osquery connector. Can share more @ info@graphistry.com , would love to explore!

dfc 9y ago
What is a "visual playbook and pivoting system"?
- allanmacgregor 9y ago
  We don't know but sales said those terms and trending hard, ohh and we also have AI .... not sure in which part of the system but we do!!!!!!
  - lmeyerov 9y ago
    Edit: I guess the above post was trolling, but I think it's fair to ask what pivoting & playbooks are. They're coming from the investigative & ops sides of the enterprise security world. Not terms I used when I was on the R&D side, but useful when they describe big chunks of your job: -- "Visual pivoting" is where a visual tool -- typically in node-link diagrams like Maltego, Palantir, and ours (Graphistry) -- lets you click on an entity like a user etc, and get the result of running queries. For example, "according to vpn/dhcp logs, what hostname was this yesterday? Did the IDS logs say anything funny about that hostname?", or, "Looks like this patient had a bad experience with that doctor, what are the interactions other patients had with that doctor?" AFAICT, "pivoting" became the shorthand for "a left-join, including across different APIs instead of within one database". -- We're using "Visual playbooks" to describe a way of helping turn a team's text documents describing their manual operating procedures for incident investigation & response into interactive software. They often can't just automatically trigger chef/puppet scripts because someone needs to check incident data. For example, we find a lot of teams want help triaging an endpoint incident alerts: resolving the host/user involved, what other alerts were happening for them, were other users/hosts getting similar alerts, and if external IPs are involved, what threat intel feeds say + is anyone else internally talking to those. A visual playbook helps you kick off an executable sequence of context-gathering pivots + maybe API action calls, and instead of returning a text file / dashboard, drops you into a visual analytics session that shows you how the pivot results chain together. So basically visual etl + visual analytics + visual querying. It's new but fun. Hopefully you can see why osquery is a nice tool for some of the steps within an investigation... or visual playbook ;-)

burdzwastaken 9y ago

cannot upvote this enough! seriously great open source project. I have been able to use it in large scale cross-platform deployments with great success.

onida 9y ago

Outside of facebook, what's the largest install base of osquery?

mavam 9y ago
We have an ongoing project and existing prototype that combines OSquery with the Bro network security monitor. This enables correlation of network and host state in the same framework. More details here: https://www.bro.org//bro4pros2017/Haas_OSquery_Bro4Pros2017.pdf https://www.bro.org//bro4pros2017/Haas_OSquery_Bro4Pros2017....

josteink 9y ago

I realize WMI[1] is a Windows-thing only, but if you're already on the Windows-platform, WMI comes shipped out of the box and has been for probably a decade or more, and is overall very well supported. It has its quirks, but there's practically no part of the system you shouldn't be able to get information about using WQL[2]. [1] https://msdn.microsoft.com/en-us/library/ms186146(vs.80).aspx https://msdn.microsoft.com/en-us/library/ms186146(vs.80).asp... [2] https://en.m.wikipedia.org/wiki/WQL https://en.m.wikipedia.org/wiki/WQL

Already__Taken 9y ago
The nice thing osquery can have is disconnecting what's currently turned on and connected this second from what records you're getting back. OSquery can pump results into a DB that you look through, WMI I've got to check for myself what's turned on, did it all report etc. A driver for wmi in osquery would be a great addition.

kossmoboleat 9y ago

I've worked on something similar except that we used a stream database. Today you might use RethinkDB or something similar. The osquery events seem to solve these kind of monitoring/alerting problems. But I didn't find easily find their documentation.

usrme 9y ago

Can anyone shed some light on how exactly Osquery can be used to leverage checking, for example, of vulnerabilities? I wasn't able to gather information on how it exactly communicates with the rest of the infrastructure. Am I correct to understand that I can set up a single host that has Osquery installed and it can go out and automatically query a heterogeneous set of other hosts based on pre-existing configuration files, given that (I assume) some set of ports are open?

lmeyerov 9y ago
Indeed -- Osquery, and the commercial precursor Tanium and the native Microsoft equivalents, can help when you've identified one incident and want to find active repeats. Ex: "I just cleaned evil.exe from this box, are any other hosts running the same process? What other processes do they have in common?". I've seen hunting use cases as well. Your example is why we're building our visual playbook system. In this case, query your network scanner for open ports, and then feed the identified hosts into more targeted osquery calls.
- usrme 9y ago
  Thank you for the reply; however, I am still having a hard time understanding how exactly Osquery works under the hood. How does it communicate with other hosts? With what does it query for information once it reaches a remote node? What kind of overhead does it have in terms of network transfer when querying? Maybe I am completely off base and am mistaking how this works... Is Osquery set up on every individual host and I query for that information remotely using whatever tools I have at my disposal?
  - fiorix 9y ago
    You can configure osquery to execute periodic queries (scheduled queries) of all kinds: computing md5 of your binaries and other files, taking a snapshot of sockets/connections per process, and so on. By default, osquery uses glog, which means it'll output the results to a local file that you can ship anywhere you want. There's also logging plugins to help you push the results of scheduled queries to other systems. Once you have that data flowing through your pipelines you can start doing security/anomaly detection on things.

xcopy 9y ago

jvgiuiigvuuc 9y ago

Useless out of the box: "If there are no USB devices added or removed to the laptop, this query would never log a result again. The query would still run every 60 seconds but the results would match the previous run and thus no state change would be detected. If a USB memory stick was inserted and left in the laptop for 60 seconds the daemon would log" So if you unplug your evil USB earlier, this won't tell you. I wonder if they've heard of udev.

JosephRedfern 9y ago
Oh please. It may be useless out of the box at detecting evil USB devices, but to accuse it of being outright useless is ridiculous. I'm sure it was sarcastic snark, but of course they have heard of udev: https://github.com/facebook/osquery/search?q=udev https://github.com/facebook/osquery/search?q=udev.

DominoTree 9y ago

Not directly related to Osquery, but we're using Tanium to run queries against ~180,000 endpoints and we get good results back in about a minute.

rbolla 9y ago
Tanium says, the max # of endpoints that the "core" can support is 33K. How did you scaled this to 180 K.

tlrobinson 9y ago

How do you issue queries programmatically from another program? Just execute "osqueryi"? I see mention of a Thrift API but no details.

drumttocs8 9y ago

Can this be used to troubleshoot DPC-related audio latency issues?