@sroberts advanced persistent incident responder

How I Atom

During the time I’ve been at GitHub one of the coolest projects to come out has been Atom, GitHub’s own text editor. I’ve been using it since the day it got released internally at GitHub and I can say Atom is one of my 3 top used applications and an essential part of my work flow.

What makes Atom shine, even next to a great editor like Sublime Text (My old favorite) or a classic like VIM, is it’s extensibility. Getting started with such highly extensible tool is tough, so here’s a couple of ideas based on how I use Atom that might get you started.

Screenshots FTW

How I Use Atom

% Language Reason
45% Markdown Markdown is the standard document format for most things at GitHub and given how comfortable Markdown is I’ve shifted most of my personal documents and writing (including this blog) to Markdown.
35% Python GitHub is a Ruby shop it’s true, but Python is the language of security (yes yes, I know about Metasploit). I typically write smaller Python utilities, but occasionally dig into bigger Django projects like GRR.
10% CoffeeScript Not a terribly common language for a security engineer CoffeeScript is the language of Hubot. DevOps-ing security tools is so awesome I’ve already talked about itmultiple times.
5% HTML/CSS/JS Mostly for this blog.
5% Randomness A little Go these days. Some system automation like Shell and Ansible.

Packages

The easy place to start with Atom is packages. Here are my favorites:

Zen

You can pay a fair bit of money to buy a minimalist text editor. Or you can just use Zen. This package (by GitHub’s CEO @defunkt no less) turns Atom into a 100% focused text only editor experience.

Zen Editor

atom-beautify

This package takes structured file types and normalizes them, presenting them in a cleaner, more humanly grep-able way. I most commonly use it to take JSON from APIs, which is often presented as one line, and adding the new lines and indenting that makes it human readable.

dash

I think of Atom as more than a text editor and less than an IDE, in the best ways possible. Two big IDE features that I’ve added to my Atom setup and the first one is getting documentation from Dash using the dash & dash-on-cursor Atom packages. Just hitting Ctrl+h instantly does a context specific search for the term you’re on and pulls it up in Dash. Whats even better is this is offline documentation, so even in low/no bandwidth situations you can keep working.

DASH OF DASH!

You don’t have Dash? If you’re a developer using a Mac then you’re doing it wrong, I can’t recommend it enough.

date

Date lets you print a variety of date and timestamps. Highly useful if you’re writing up documentation on something like an incident. Which I do…. a lot.

file-icons

This package gives a prettier, more expressive icon set for the sidebar and tabs of Atom. These visual identifiers are often handy given modern projects base on web frameworks have so many different types of files involved.

go-plus

If you’re using Go you’ll take a look at this package, marvel in the awesome, and instantly install it. If you aren’t it’s basically useless, and pops a lot of annoying errors.

linter, linter-write-good, linter-pylint, & jsonlint

The other major IDE like function I get out of Atom is linters for realtime syntax checking. Linters, which let you know when you’re not complying with the language specification, are a great example of one of the IDE like benefits you can have in Atom. Think of it like spell check for programming languages, linters let you know when you’re writing code incorrectly.

The linter package is the start, then you can hunt down the linters most relevant to your own development.

merge-conflicts

Ahhh the bane of every Git users existence, the merge conflict. The merge-conflicts package helps make these easier to deal with, giving you visual queues and instantly snapping to each conflict.

Bad Hotlinked Merge Conflict Gif

I don’t hit conflicts that often, but when I do this plug makes it easy to sort through them.

regex-railroad-diagram

Get on the Train!

Regular Expressions are highly useful but complicated things and it’s often easy for one character to change the meaning or function of a regex entirely. The regex-railroad-diagram gives you a visual aid for understanding any regex. I rely on it heavily.

sort-lines

Super simple, but super useful. While you wouldn’t want to sort code this comes in useful when working with lists.

Themes

I won’t go too nuts talking about themes. It’s 100% personal preference. Atom Themes come in two pieces, UI & syntax. I find the best option is to play with them. Some favorites are:

Personally I swap mine in and out every couple weeks. Right now I’m using the built in Atom Light option for both.

  • vim-mode: Coming from everyone’s favorite terminal editor and miss your key commands, you can hold on to them!
  • minimap: Hold over from Sublime Text? minimap will make you feel at home.

Tricks

  • Meta-Shift-p: Everything. Much of Atom’s power hides behind the text menu. You’ll miss it otherwise.
  • Take a look at each package & theme for their own settings. These tweaks can often make a big difference. For instance I always check the “Force Show” option on the file-icons package.
  • Browsing the Packages & Themes sections of the Atom sites will probably have piles of other useful packages based on your specific needs.

How I Don’t Use Atom

Logs. It’s that simple. Atom doesn’t handle files 2mb or bigger, so it’s a no go for log analysis except for the smallest examples. During my switch to Atom log analysis was the main situation I’d go back to Sublime Text, but at this point I do almost everything on the commandline. Sed/Awk/Grep and ./jq are amazing tools and at this point I think I’m faster working through logs on the commandline than I ever was in a text editor.

pbcopy and pbpaste

pbpaste & pbcopy give you direct access to the OSX clipboard from a shell and makes it easy to tie together data from GUI based apps with command line apps.

pbpaste

So say you use ⌘+c to copy something from a browser that you want to then feed through a command line tool like ./jq:

$ pbpaste | jq ‘.’

pbpaste feeds the text from the clipboard to jq through standard in, which then allows jq to manipulate it as you see fit. The result gets sent to standard out (or wherever the second tool sends it).

pbcopy

The reverse also works:

$ cat file.txt | pbcopy

pbcopy takes data from standard in and saves it to the OSX clipboard. You can paste that into any other app using ⌘+v. This is great for grabbing the contents of files without having to open them in another application.

Put it Together

One of the most common uses is to manipulate data using sed:

$ pbpaste | sed -e ‘s/foo/bar/g’ | pbcopy

These two commands, like most Unix style commands, seem super basic. They’re deceptively simple and combined with other CLI tools pbpaste & pbcopy can create powerful combinations that simplify all kinds of tasks.

Imposter Syndrome in DFIR

Impostor syndrome can be defined as a collection of feelings of inadequacy that persist even in face of information that indicates that the opposite is true. It is experienced internally as chronic self-doubt, and feelings of intellectual fraudulence.

Imposter Syndrome ~ The CalTech Counseling Center

There isn’t an easy way to start a post like this and there doesn’t need to be. Imposter Syndrome is something most people don’t know a lot about (I’d never heard the idea until I started working at ) but it’s something everyone is intimately familiar with.

Which one is the boogie man?

Imposter Syndrome is like the boogie man: open the closet, turn on the lights, look around, and you see that nothing is there. You feel better. Exposing it, taking a good hard look, and considering the thing is what nullifies it. But it never goes away entirely. It’s just temporarily gone, and you’ll have to expose it again.

While everyone experiences imposter syndrome, it manifests itself in unique and individualized ways. So with that in mind, and knowing that openness and honesty are the these keys (both to help myself and others), this is my take on the symptoms of imposter syndrome I have experienced as a Digital Forensics & Incident Response professional:

The “Blinky Lights” Inferiority

“What if they make a box that does what I do? Like the Terminator but for SOC work.”

Early in my career I read an article the then-upcoming Windows XP SP2 release and its focus on security (built-in firewall, DEP/NX, Security Center, etc). This was a formative time in my career and this was a major advancement that I misunderstood entirely.

A Blink Light Box

My first thought was simple: “I may as well give up on security, they’re just going to make hacking impossible with this release.”

I was naive and mistaken but this fear has never gone away completely. I thought the same thing years later when I first saw a FireEye appliance. What if they fix all the bugs? What if someone builds a box that does what I do? What if machine learning actually becomes a real thing and it’s better than I am?

Reality is much different. I sometimes feel like an imposter because someday someone will build a box that does what I do better than I can. Someday won’t be today and tomorrow isn’t looking promising either. This is the easiest inferiority to look at and dismiss as silly, but that doesn’t make it less potent from time to time.

The Offensive Inferiority

“Red team bros just constantly say how easy it is to get around what I spent months building.”

There’s a joke I love that goes like this (with my own small modification):

How do you tell if someone is a vegan, a crossfitter, or red team bro? Give them two minutes and they’ll tell you.

Hang out with offensive types (pentesters, exploit developers, etc) and you’ll hear stories that can basically get boiled down to “Defense is garbage, defenders are useless, we always win, I’m super awesome.” It’s never said that way (otherwise get new friends) but as a DFIR, this feels like being called an idiot all night.

Hacker Bragging Origins

To be clear: This is my hang up. This is not my offensive friends’ fault. We all tell stories focused on our successes, not our failures. It’s tough when you never hear a pentester talk about the time they failed for a week straight to compromise a target. Instead you hear about the epic success when they got their shell. You never hear a vulnerability researcher describe the weeks in IDA (or Vivisect) getting frustrated. But you do hear about when they launched their exploit and saw calc.exe pop up.

A defensive team can succeed 999 out of 1000 times and the thing everyone cares about is the 1 in 1000 where they failed. In the same way, offensive teams never discuss the 999 times they failed, only the final time when they succeed. This leads to the feeling among DFIR folks that thinking they can stop APT:EquationShark (or even the local red team bros) is in vain — that they’re imposters. I would argue defense succeeds often; we just rarely acknowledge it.

The Jack of All Trades Inferiority

“I’m a worse RE than our RE and a worse 4n6 than our 4n6icator and a worse developer then our developer and a worse ops than our ops.”

Shifting to an individual perspective, incident handlers are not T-Shaped People. While most people have one skill they’re a 8 of 10 on, and lots of skills they’re a 3 of 10 in, most DFIR types are a 5 out of 10 in four or five skills and 3s on the rest.

Because of this we Fake the Funk, assuming we should know more and trying to avoid admitting we don’t. This lack of self honesty to preserve face has caused me to wastes talented coworkers time & skill, wastes valuable learning experiences, and it has kept me from focusing on the tasks I do better than anyone.

It’s easy to feel like an imposter when the rest of your team seems deeply skilled, even if it’s their area of expertise. I’m not as good at working with malware as a RE, but I’m much better at reversing than most forensics engineers, and I know more about intelligence too. It’s a unique set of skills to see the big picture that has a unique and important place on a team.

The “80/20” Inferiority

“About 80% of the time I just sit around being a mediocre software developer or systems engineer.”

I structured this in the order of my imposter syndrome issues. I’m less concerned that RSA is going to release a @SRoberts Replacement Box than the Pentester Bros frustrate me by bragging about their conquests (and ignoring their defeats). The toughest? DFIR types live for emergencies in the way ER doctors live for injuries. What do you do when everything is healthy?

80/20 Rule

No organization is without incidents, but the fact is that (even though there’s always something to do) there are also non-emergency times where as an incident handler it’s easy to feel like a 3rd wheel, unnecessary, an imposter. I spend a lot of that time building new systems, either as an ops engineer or a developer. This ties into the Jack of All Trades issue, but I know I’m not nearly as good a developer as our real developers. Nor am I as good an ops engineer as our real ops engineers. It’s easy to feel somewhat ancillary until the next emergency.

The “Human Being” Inferiority

Don’t think everyone feels like an imposter sometimes? I remember reading a note by a noteworthy colleague that no one would ever expect to feel like an imposter, writing a brutally honest account of feeling like others were going to find out she was a fraud. The text was a moment of pure humility on the author’s part. It didn’t say woe is me, but it was instead expressing a sense that she was handling this but wanted others to know it was normal and they could handle it too.

So What’s the Solution?

  • Acknowledge Imposter Syndrome is real. It’s not unique, you aren’t alone.
  • Stop faking it until you make it. Reach out to experts and either rely on them or learn from them.
  • Use it as motivation. Feel like you’re weak in RE? Take some time to read Practical Malware Analysis. Imposter Syndrome isn’t an excuse.
  • Share your experience handling your own Imposter Syndrome with others.

In The End

I often don’t feel like I’m prepared to be a part of this profession I’ve chosen. I feel like I should know more, read more, write more, and be a whole lot better at RE, forensics, coding, etc. I’m pretty sure all that is normal.

No one gets away from Imposter Syndrome entirely. It’s a human condition, but we can seek to understand it, to control it, and not let it control us. I think a willingness to confront this, with ourselves and our teams, can make everyone stronger.

Incident Response Hunting Tools

Great, you’ve decided to move beyond reactive incident response and start hunting. While hunting is primarily a way of thinking about incident response it does rely on your technical capabilities, so what tools should you use? The focus for me is always on open source tools with tools with wide ranging applications. Here are my favorites:

Endpoint Alerting Tools: Facebook osquery

Facebook osquery Screenshot

osquery is a tool from Facebook that describes itself as:

osquery allows you to easily ask questions about your Linux and OSX infrastructure. Whether your goal is intrusion detection, infrastructure reliability, or compliance, osquery gives you the ability to empower and inform a broad set of organizations within your company.

If I’m honest I wasn’t crazy about osquery at first (Sorry Mike). The SQL syntax, while familiar, felt forced and awkward to me. That said I’ve looked high and low and have yet to find a better endpoint alerting tool than osquery. It’s fast, flexible, and fully featured. Tools like Mozilla MIG are interesting and maturing quickly, but osquery is capable and ready to deploy, so I’ll eat my works and osquery is my first pick. New tools like Mephux’s EnvDB are just making it even better.

Endpoint Interrogation Tools: Google GRR

Google GRR Screenshot

Google Rapid Response (GRR) was built by Google in the wake of their Aurora Incident to do incident response at scale across a wide variety of operating systems and geographies. Billed as:

GRR Rapid Response is an incident response framework focused on remote live forensics.

GRR makes it possible to directly interrogate individual systems or hunt across a wide swath of systems. Able to pull memory, search for file system artifacts, audit user activity, search network usage, GRR provides a comprehensive experience for investigating malicious activity. The server can be stood up with one script and then just start deploying the packaged agents to start hunting.

Comprehensive Logging: Elasticsearch, Logstash, & Kibana

Elastic ELK Screenshot

Three tools in one ELK combines a datastore (Elasticsearch), log ingestion pipeline (Logstash), and an analysis interface (Kibana) aka ELK. Combined they form a powerful log management and analysis platform, scalable to tremendous amounts of ingestion and retention.

For a hunt team this provides a singular location to store logs from all relevant systems, whether they be application logs such as your VPN or email server logs, network & system logs like Bro (Read @jshlbrd’s great post on using Bro for Hunting) or auditd, and provide a unified place to analyze them. This enables the kind of telemetry that makes a singular indicator valuable across a myriad of systems.

Wide Ranging Network Visibility: AOL Moloch

AOL Moloch Screenshot

As far as network visibility goes the ideal world is full content monitoring. Before Moloch there were limited options.

  • A homebuilt system based on tcpdump, where all you’re paying for is storage, but with a clunky interface.
  • An expensive commercial system, great interfaces, but wildly limited storage coming with a crazy price tag.

Full stop.

Then came Moloch:

Moloch is an open source, large scale IPv4 packet capturing (PCAP), indexing and database system. A simple web interface is provided for PCAP browsing, searching, and exporting. […] Moloch is not meant to replace IDS engines but instead work along side them to store and index all the network traffic in standard PCAP format, providing fast access.

Moloch is one of the best designed, truly enterprise worthy open source projects out there, and an amazing solution for doing full content monitoring. It’s a challenging tool to deploy, and can be expensive depending on retention, but it’s the most flexible and featureful. Alternatives like flow monitoring or intrusion detection systems are highly effective as well, but less flexible.

Tracking: Societe Generale CERT’s FIR

Societe Generale FIR Screenshot

It’s easy to get caught up in systems that improve telemetry, but what you do with the information collected is just as important. Tracking a hunt is a complicated thing, with lots of data, lots of people, and often an adaptive adversary. It’s important to have a structured way to record varied types of data, not just facts but also observations, hunches, etc.

I haven’t personally spent a lot of time with it (something I plan to remedy soon), but as someone who’s used numerous commercial systems, endless wikis, even just plain notebooks and whiteboards, I think FIR looks like one of the best systems at any price, and the best open source option by far. For a less incident centric, more intelligence centric tool give CRITs a look as well.

How it Fits Together

At it’s core these tools enable the F3EAD Cycle that is the core of the hunt. Telemetry tools and their use makes up the operations phase (Find, Fix, & Finish) while the Tracking provide a datastore and structure for the intelligence phase (Exploit, Analyze, & Disseminate). These tools make it possible to establish the F3EAD cycle. As you improve the efficency of this interplay you shorten your OODA cycle

Conclusion

“It’s the man, not the machine”
~ Chuck Yeager

As important as tools are General Yeager is certainly correct in this case. Tools enable a team to hunt, but ultimately hunting is a team perspective, an attitude, more than a series of tools. Focus on building people, processes, and then focus on technology.

Incident Response is Dead...
Long Live Incident Response

Talk to anyone in the DFIR Illuminati and one of the topics that always comes up is Hunting. Much like threat intelligence & string theory, people talk a lot about this, but nearly no one knows what it actually means.

Get it? Hunting?

Proactive vs. Reactive

At its core, Hunting is about taking a proactive vs a reactive approach to identifying incidents.

  • In Reactive organizations, an incident starts when notification comes in; whether that’s a vendor IDS or AV alert, or worse a phone call from the FBI or getting Krebs-ed. At that point an IR begins, reacting to the incoming event.
  • In Hunting organizations, the IR team actively goes looking for incidents based on known patterns of activity, intelligence, or even just hunches. Once the hunt team finds a new incident they begin an IR as usual.

Building a Hunting Capability

What does it take to build a hunting team? A change in technology, mindset, and procedures.

Incident Response

The Hunting vs Non-Hunting process is singularly relevant to the Preparation & Detection phases (See the SANS Incident Response Process) of an incident. Good hunting is useless unless your team can execute on the mitigation, remediation, and recovery phases effectively. Without that, all you have is a larger incident load you’ll be unable to respond to effectively. Before focusing on hunting, make sure you have the processes in place to respond efficiently. Once your team has effective responses prepared, you’ll be ready to start hunting.

Wide Ranging Telemetry

A good hunt team can use a basic indicator, like an IPv4 address, in diverse ways while hunting. The more options a team has for getting telemetry (data), the better their chances of identifying and corroborating malicious actions. Successful hunt teams have myriad technologies available to identify different types of activity in different ways.

An IPv4Address is useful for searching:

  • Application Logs: Like your VPN, Email Server, or custom internal apps that attackers may target or co-opt.
  • Network Traffic: Traffic analysis is ideal for long running historical investigation. Full PCAP gives the full story.
  • Endpoint Logs: Getting data directly to clients makes correlating internal activity possible, such as lateral movement. It’s also valuable when systems like laptops may exist outside your network perimeter.

Having diverse vantage points provides redundancy as well as the option to use the best tool for the job.

Scalability & Automation

The tough thing about hunting is that you’re trying to prove a true negative. Put more plainly: you’re trying to prove that there isn’t something bad. What makes this tough is the fact that it’s impossible to prove a true negative. You can never ultimately be sure you aren’t digging deep enough.

The way to counter this is by hunting efficiently and widely. That requires a combination of scalability and automation. The ability to deploy, run, and collect all forms of telemetry should be centrally and programmatically managed. While hunting, you never know whether you’ll be scanning a small team worth of systems, or an entire division. Look at the advances the devops movement has made for the operations world: a great hunt team needs similar function at scale.

A Target Centric Mindset

“Know thyself…“
~ Ancient Greek Proverb

The first question people ask when building a hunting capability is where to start (If you read about F3EAD this is the Find section), otherwise known as targeting. The first approach to take is “target centric.”

The hunt team starts by evaluating internal projects and resources to determine which are most valuable and begins hunting assuming attackers want to compromise those valuable resources. If the hunt team knows Project Unicorn is valuable and possibly targeted, start hunting around people & systems involved with Project Unicorn. The challenge of target-centric hunting is understanding the organization as a whole well enough to recognize important resources. Collaborating with higher level leadership can often make this process easier.

An Actor Centric Mindset

“The way I wrestle five-year-olds makes me think if I were ever attacked by a pack of midgets, I’d be OK.
”
~Jarod Kintz

In the age of named (and numbered) attack groups and big nation state attack exposés, much of the focus is on actor-centric hunting. Rather than focusing on the potential targets, actor-centric hunting requires the hunt team to develop intelligence about potential attackers, either from their own research or 3rd party sources, and to identify specific actors. This is a powerful approach but first requires an understanding of the attacks you face. While threat intelligence firms may track dozens of different actors, at best a fraction will be relevant to your organization. By building dossiers based on target-driven hunts, you develop the necessary understanding to mature into actor driven hunting.

Management Buy In

Ok… #REALTALK: You would think that your leadership would want to know about all these newly discovered incidents and have the IR team remediate them. Turns out they often aren’t, and it’s worse if you work in a regulated industry (like finance, medicine, or the DIB). Often leadership would rather keep their head in the sand than deal with the reality (like regulators & reporting requirements), and the hunt team becomes a victim of it’s own success. I know of one organization where management told a mature hunting team to watch movies and play video games rather than track down more malicious activity. #truestory

Operational Tempo

The incident response team being a victim of their own success aren’t limited to leadership. Once a hunt team begins to mature their ability to constantly find new malicious activity, they will continue to improve until the point where they can basically find new malicious activity at will. As a result, the IR team is basically in a constant state of emergency, always responding to something.

At this point the hunt team lead needs to step in and develop an operational tempo to keep the team sane. While in reactive organizations treating every incident as an all hands on deck emergency makes sense, this can’t be the reality of a hunt team as the team might always be in a state of emergency. You may soon discover that throwing out an entrenched attacker on Friday may lead to a new phishing campaign on Monday, and every day can’t be an emergency or the fatigue will overwhelm and undermine your entire program. Even during an incident, and there will always be one, vacations are important and day to day work needs to get done. Exceptions happen, but they’re rare.

So what’s the big deal?

A Borne Hunter!

In some organizations, hunting seems like a mythical concept, in others it seems like business as usual. Hunting is still just a mindset, an idea of focusing on proactive action, rather than reactive response. Done well this mindset can change your organizations entire posture. It’s not comfortable, it’s not easy, but it’s the difference between being passive and taking action. In my next post I’ll talk about some of the better open source tools for equipping an up-and-coming hunt team.