DFIR @scale - osquery

# Responding @ Scale
<img src="https://osquery.io/static/site/img/logo.png">
## osquery for Mass Incident Detection & Response

---

# Heroku
## _"A company that runs apps for you"_
## Tens of thousands of Linux servers in production

---

## DFIR Engineer @ GitHub
## @sroberts

# GitHub
## _"Making it easier to work together than alone"_
## Hundreds of Macs

---

# A Story in Three Parts
## osquery
--

## osquery use
--

## osquery use at scale

---

---

# What is osquery?
## Expose system information via "SQL tables"
### users, groups
### installed packages, kernel modules
### network communication, file system events
## Runs scheduled queries of tables

???

- They aren't actually SQL tables, they're "virtual tables". The DB behind osquery is actually RocksDB.

---

class: middle
# The People
## Facebook
## @marpaia & @theopolis
## 2991 commits with 95 contributors and a community of 241 as of this writing...

???
- Built off of Facebook's BigMac & Etsy's TripYarn tools (Combined as MIDAS)
- 2420 Total Commits, 1635 Pull Requests

---

# Out of the Box Tools*
.left-column[
### osqueryi]
.right-column[
## osquery's Run, Evaluate, Print, Loop
## Useful for testing & one off checks
]

---

# Out of the Box Tools
.left-column[
### osqueryi
### osqueryd]
.right-column[
## osquery's Daemon Tool
## Useful for continuous detection
## Schedules and runs pre-set queries and writes them to a logger
## Takes some setup (we'll get to that)
]

---

# Out of the Box Tools
.left-column[
### osqueryi
### osqueryd
### osqueryctl]
.right-column[
## osquery's System Control Tool
## Turns osqueryd on, gets status, turns it off, etc
## Takes care of system specific stuff (like LaunchAgents, which aren't fun)
]

---

---

---

# What it means?
### List all the information about all the sources where apt can download & install software

???

This is a ubuntu specific query.

---

class: middle
# Advanced Query
```sql
SELECT name, path FROM kernel_extensions WHERE name NOT LIKE 'com.apple%';
```

# What it means?
### Get the name & path about any loaded OSX Kernel Extensions that were not created by Apple

---

---

```sql
SELECT uid, name FROM listening_ports l, processes p WHERE l.pid=p.pid;
```

# What it means?
### Get the uid and name from any process with an open listening socket

---

# Real World Query Examples

## Detects LoginWindow Persistence Mechanism
```sql
select key, subkey, value from preferences where
path = '/Library/Preferences/com.apple.loginwindow.plist';
```

## Detect RAT used by Hacking Team
```sql
select * from apps where bundle_identifier = 'com.ht.RCSMac'
or bundle_package_type like 'OSAX';
```

## Detect the Careto Malware LaunchDaemon
```sql
select * from launchd where path like '%com.apple.launchport.plist';
```

---

class: middle, center
# Query Writing Philosophy*

## Haystack Approach
### _Write generalized queries that grab lots of potentially interesting information and sort it out later_

## Needle Approach
### _Write very specific queries looking only for verified indications of compromise and act immediately_

???

## Haystack
- All of our servers are uniform. So one of the things that we look for is something
that is different from the norm. Are there any kernel mods that are not present on
all the other servers? Are there any hashes not present across the board?

## Needle
- Useful because GitHub is really concerned about Hubber privacy
- Highly actionable
- Not useful for anomalies

---

class: middle
# Special Capabilities
## Ad Hoc Queries
## File Integrity Monitoring
## Yara

---

# osqueryd Configuration
.left-column[
### osquery.conf
]
.right-column[
## JSON configuration file
## holds the scheduled queries
## file paths to monitor
## read from local filesystem or https
]

---

# osqueryd Configuration
.left-column[
### osquery.conf
### osquery.flags
]
.right-column[
## configures osquery administration
## where do logs go
## where is the pidfile
## where to find JSON config file
]

---

class: middle
# Managing Intelligence with Packs
## _osquery query packs are groups of queries to be added to the osquery schedule_
## Lets you group queries for easier management and distribution
## Overall easier management without an unwieldy `osquery.conf`

---

---

class: middle
# How Heroku is Using osquery
## Only on the servers
## Our environment is very uniform (by type)
## Look for things that are not uniform
## Lookup IOCs just to get a heads up

---

class: middle
# How GitHub is Using osquery
## Only on OSX laptops
## Hunting specific known IOCs & verifiying configurations
## A work in progress with new features often

---

class: middle
# Managing Many Endpoints
## Earlier mentioned that `osquery.conf` tells `osqueryd` where to find config file
## That config file can come from an https server
## The same server can receive logs as well

???

- But that HTTPS server didn't exist... until now!

---

class: center, middle, dark-purple
# Windmill
## Developed with by Heroku & GitHub

???
- Except somehow a few people have found this project and already started PRs (which is awesome)

---

class: middle
# Windmill
## Open source Ruby TLS Configuration Endpoint
### Get it now on at [heroku/windmill](https://www.github.com/heroku/windmill)
## Organizes endpoints into __Configuration Groups__
## Enables intelligent endpoint management
## Recieves and forwards raw logs

---

class: middle
# Windmill
## Focused on safety
### versioned config files
### canary deployments

---

---

class: middle
# So... What now?
## __Transport:__ syslog, logstash forwarder, fluentd, etc
## __Analysis:__ Splunk, ELK, or a SIEM
## You have logs now... _GO FIND BAD STUFF!!!_

???
-

---

class: middle
# Resources
## [osquery.io](https://osquery.io/), [facebook/osquery](https://github.com/facebook/osquery), & [osquery-python](https://github.com/osquery/osquery-python)
## [heroku/windmill](https://github.com/heroku/windmill)
## [blackfist/osq_simulator](https://github.com/blackfist/osq_simulator)
## Also
### [mwielgoszewski/doorman](https://github.com/mwielgoszewski/doorman), [mephux/kolide](https://github.com/mephux/kolide)
???

# Worth adding?
## [mephux/envdb](https://github.com/mephux/envdb)

---

---

???
- __osquery:__ what it is, who made it, where it came from
- __osquery use:__ basic installation & use
- __osquery use at scale:__ how to deploy widely, how to manage configs, how to look at logs

---

---