@sroberts advanced persistent incident responder

Python for CND

One thing I constantly harp on while talking to people beginning in the security community is the importance of learning to code. I think it is awful that we have so many security professionals cannot write a line of code. It’s useful for automating common tasks, gathering & manipulating data, almost anything you can imagine. I think everyone should learn some coding and Python is the best place to start.

What is Python

From Python.org/about:

Python is powerful… and fast; plays well with others; runs everywhere; is friendly & easy to learn; is Open.Python is powerful… and fast; plays well with others; runs everywhere; is friendly & easy to learn; is Open.

Python is an interpreted language, meaning the python application takes the same .py file you write code in and runs that for you, without a compilation step in between (if you want a compiled language I recommend looking at Golang). Python runs about anywhere and if you have a MacOS or Linux computer you probably already have Python installed.

Python is beautifully simple. Here is Hello World:

It rarely gets much simpler than that. Lets demonstrate something a bit more complicated:

Here we’re using a library called Requests to grab the SANS Infocon status from their web service. Then we check to make sure the response had an HTTP status code 200 and return the value to print, otherwise if we can’t get the Infocon successfully we’ll return an error. Contrived? Yes. Simple? Of course. Useful as a jumping off point? Absolutely.

Why Use Python for Security?

Because all the cool kids are doing it? Python is the default programming language for most network security types, having largely taken the mantle from Perl. It’s easy to write, whether its a basic script for rearranging data or elaborate large scale distributed security tools, has lots of great libraries for common security tasks (we’ll look at some of them later), and quick to learn.

Resources for Learning Python

Learning any language is never just about learning to write code, though that’s the first place to start. Learning a language means learning it’s syntax, but also conventions, libraries & tooling, as well as how to test. It also often helps to look at projects others have done to get the lay of the land. Here’s some resources to get started:

Core Language Syntax

Start with CodeAcademy: Python to learn the basic syntax. It’s free, there’s no setup, and you’ll learn enough to be dangerous in an afternoon. Ready to go deeper? You want Learn Python the Hard Way for a deep dive into Python, using your own machine. If you need a few syntax reminders check out the Learn X in Y minutes: Python3 cheatsheet.

Aside: Python 2.7 vs Python 3 If you want to get Python folks at each other just bring up Python3. Released a few years Python3 was meant to be a significant upgrade, fixing lots of old structural issues that couldn’t be fixed earlier because they would break old code. Many people have been resistant, and even though it’s been out for years Python3 still isn’t the default. I focus on writing Python3, but your mileage may vary and your organization may do things very differently.

One of the nicest things about learning an interpreted language like Python (or Ruby, Perl, etc) is being able to learn using a Read Evaluate Print Loop (Usually referred to as a REPL). You can start a REPL for Python by just typing python at a command prompt without any arguments. This lets you write Python line by line, getting the output immediately. It’s a great way to do things quickly or just experiment.

Python Security Books

Python is so common in security that there are a number of great security centric Python books. These are great ways to learn Python in a security specific context.


One of the best things about Python is the multitude of libraries that already exist before you write your first line of code. These range from basic data manipulation libraries all the way up to complex web frameworks and data mining toolsets.

In Python all packages are managed using pip (yeah yeah easy_install & eggs and a bunch of other ways, but lets keep it basic). Searching for a new tool is as easy as pip search virustotal, which will return all the Python packages related to everyone’s favorite malware service. Decide you want to use one? A quick pip install virustotal-api and you’re off to the races (Once you get a bit further down the road you’ll want to learn about using pip & virtualenv together.)

What libraries you want to use depends a lot on the work you’re doing, but here are a few core ones most security minded Python developers will want to use:

Library Description
Flask The go to for small webapps, Flask is especially useful for REST API based tools.
Requests Requests is the only Non-GMO HTTP library for Python, safe for human consumption.
yolothreat/utilitybelt A Python library for being a CND Batman….

There are literally thousands more, but these are three I reach to often.

Awesome Python Projects

There are plenty of examples of great security tools using Python, many of which I’ve already talked about. Here’s a new set I’d recommend looking into:

These are all under active development and great tools to consider sending pull requests to.

Ready For More?

Programming is often far more than syntax and libraries. Here are some things you’ll want to start investigating next:

Companies Using Python for Security

Basically everyone. Honestly it seems counterproductive to list them. Aside from Metasploit Python is the go to scripting language for most IR firms and the language of choice for the open source security world.

In Conclusion

While not everyone in security will write code as a job almost everyone can benefit from writing some code to assist in their job. Even basic data manipulation is a key skill that becomes more useful the more you do it. Given Python’s wide use, great libraries, and ease to learn it’s an ideal place to get started. While there are other languages that do specific things better almost none can claim to do everything as well as Python.

Intelligence Collection Priorities

Meaningless globe picture of stuff and things..

One of the hardest things when starting a threat intelligence program is deciding where to start collection. This begins with an initial set of requirements and evolves from there. Everyone will give you a different opinion and insist on a different approach probably biased by their favorite collection sources. As for me I think the best approach is to start with the small things, the easy things, and build up from there. What’s easy depends a lot on your organization but this is my general list of useful collection sources for most organizations.

Internal Incident Data

Listen to your enemy, for God is talking. ~ Jewish Proverb

I love this quote and it couldn’t fit this topic any better. If you want to know who your enemies are look at the attacks you’re already experiencing.

The hardest part of threat intelligence isn’t collecting data. That’s the easy part. The hard part is deciding which data is worth collecting. There’s no better way to focus intelligence collection than by focusing on past attacks. This comes from mining internal sources, preferably your incident management system (certsocietegenerale/fir is a good start). Nothing is worse than an attacker using the same resource twice, so when you know a domain is used as C2 then keep watching that domain (and preferably the IP addresses that have pointed to it in the past as well).

Honeypots & The Like (Emphasis on The Like)

I’ll be honest… I hate honeypots. They’re trendy right now (it’s cyclical, like bell bottoms). The idea of letting a bad guy onto my network, even with the intention of gathering more information, scares the heck out of me. My favorite honeypot I ever used in operation was our Active Directory server and while we found it a great collection source it certainly wasn’t our choice. There are some cool tools out there for it, from the classic Honeynet Project to Secureworks new DCEPT project but I’m not sure I’d have the guts to run them.

What is interesting to me is the idea of using passive sources of data to enhance your detection. Want to know your most targeted users? Look at who has the most failed logins. Gathering firewall drops helps you understand what systems attackers may try to access. These passive sources of data provide internal insight that’s difficult to come by otherwise.

Vendor Reports

While opinions differ I like long form vendor reports. If you can match threat actors up with your organization’s threat model you can learn a lot (don’t assume everything) of what FireEye/Symantec/PaloAlto/etc knows about a given threats. The best reports also include IOCs.

Keep in mind attackers respond differently when outed. Some plow ahead like nothing happened, some change TTPs and even exploits, others shutdown entirely. The Use By date on these is often short, but that doesn’t mean they’re useless. In the end I think they’re free, high signal reports that give insight you might not have access to otherwise.

Sharing Communities

The cool kid cliques of the network defense world sharing communities are supposedly the end all be all answer to all our security problems. They are not. They’re far from perfect. Perfect though is the enemy of good and these communities do provide a trusted place for sharing threat information. At their most basic these communities can be as simple as a Slack channel. At their most complex they’re built on sharing platforms like ThreatExchange or ThreatConnect.

Getting in is the trick. These groups whether they’re chat or a purpose built threat-threat platform are a mechanism for sharing trust. In most cases this trust is a proxy for real life trust, so take the time to cultivate your relationships.

Free IOC Feeds

Why are free feeds my fourth choice? They’re free and all that and everyone talks about them. Well not to be blunt but they’re low signal to noise and generally unfocused. Don’t trust me? Ask smarter people like Alex Pinto & Kyle Maxwell’s Defcon 22 Talk Measuring the IQ of your Threat Intelligence Feeds (Slides & Video). I bet you’ll agree with them.

Paid IOC Feeds

Ice Cream Sundae

I know I made it last on my list but I’m not saying paid feeds are bad, they’re often great actually. I tend to think of it like a sundae. The ice cream is your first source, your base. The whip cream? That’s when you started mining APTNotes. Rainbow sprinkles… sure I guess that’s your new honeypot.

I view paid feeds as the cherry on top of the threat intelligence sundae. They take a good system and make it great. Take your time, press your vendor, and make sure you can make the most out of whatever feed you get.

The Beginning is the End is the Beginning

Now that you’ve combed through your incidents what’s next? Well we’re quick to forget it but collection isn’t the end of the intelligence cycle, it’s just the beginning. The goal isn’t a list of IOCs. The goal is an analyzed product that meets the needs set out in the requirements phase (you did that too right?). Which hopefully came from output from your feedback phase…

Golang for DFIR


One of my goals for this year was getting comfortable with a new programming language. I’ve been a Python devotee for a long time and it’s almost always gets the job done, but I wanted a little bit more. There are times Python works against you:

  • Dependency Nightmares: While virtualenv and a requirements.txt file work ok for developers it can often make use by non-developers or some deployment stories quite complicated.
  • Speed & Scale: This is a funny one. 99% of the time Python is more than fast enough and scales widely enough. Except when it doesn’t. Chewing through a few hundred megabytes of logs? No big deal. Gigabytes or terabytes? Not so much. Hundreds of network connections? Sure. Hundreds of thousands or millions? You might need something more.
  • Discipline: Python is easy. Almost too easy sometimes. It lets you cut corners, it takes care of a lot in the background. It’s a great language for getting things done but sometimes I want to learn/control/understand more. Beyond that sometimes I want software that’s more than functional, I want it to be engineered. More structure, more safety, etc.

Working for company made up of largely developers there’s always talk about new development tools and languages. Rust and Golang have been hotly discussed in these circles (along with Erlang/Elixer/Node). I was considering both until I had a discussion at ArchCon with Liam Randall who effused about how great Golang had been for his organization. Good enough for Liam is good enough for me, so I dove in.

What is Golang?

According to the Golang Lanugage site:

Golang is an open source programming language that makes it easy to build simple, reliable, and efficient software.

Go is a strongly typed, not quite object oriented, complied language with a syntax similar to C & Python (with a few little quirks). It’s a complied language (meaning you create an actual binary instead of just running the script like in Ruby or Python). It’s garbage collected, meaning you don’t have to deal with memory management (mostly), and highly concurrent, letting you run multiple threads easily. Golang was built to be a Google Scale language, meaning it’s meant for highly distributed large scale applications.

Ready for code? Here is Hello World in Go:

Simple and easy.

Golang also has a highly developed set of tools that include things like managing dependencies (these get statically complied into your application so end users don’t have to worry about them, just you). There are tons of libraries out there, from a well developed standard library and tons of 3rd party libraries for everything from building websites to using Yara rules.

The result? You get platform specific binaries (running native, without any dependencies) that have speed, scalablity, & safety along with lots useful libraries before you even start. All without having to write lower level, higher learning curve languages like C. Pretty much just what I was looking for.

It’s not all upside. Golang is strict. It has right ways and wrong ways to do things and it enforces those ideas. It is slower to develop compared to Python but in many cases the complications are wroth it.

Why use Golang for security?

I continue to think Python is the best language to start with for Security, but there are situations where you need a little bit more. For me though the first serious benefit was the compiled nature.

I was building a tool that pulled in data from various sources to use on the commandline and with Maltego. Initially these started as Python scripts specifically built for Maltego but they had a couple problems:

  • While they worked well enough for me they were buggy, no tests, inconsistent, and often needed me to jump in and fix mid-use.
  • The configuration was highly specific to my machine. That’s ok for me, but getting the same transforms setup on someone else’s machine would have been hard. (Yes, we could get a TDS, but for only a handful of people that seemed like overkill.)
  • Lastly as useful as it was for Maltego there were times I wanted the output on the commandline instead, to use with other scripts or tools similar to ivanlei/threatbutt. Quite a change to my codebase.

The solution? Well if you haven’t guessed by now I can’t help you, but yes, I began rewriting the scripts in Golang. The result? A single binary that anyone can run whether on the commandline or with Maltego, feels far more consistent and runs on the commandline. It also has the upside of being a bit faster and easier to understand.

Resources for Learning

One fascinating thing about people is how they learn in such diverse ways. Some people like reading while others prefer video. Some prefer diving right in. I’m a fan of a mixed approach, diving into real code, then backing out to learn some theory, then having references while I build. So here’s a little bit of everything so you can build the best Golang curriculum for you!

Hands on Resources

Go is a pretty straight forward language, especially if you’ve done a language like C, Java, or Python before. As a result one of the best ways to learn it is just diving into code itself, play with it, and see how it works. Here are the easiest ways to do that:

  • Your first stop should definitely be the A Tour of Go right on the Golang website. This walks through basic core structures with actual runable code, all within a web browser, letting you avoid Golang’s somewhat tedious setup before you know it’s for you.
  • After you have Golang installed the next great hands on tool is Exercism.io. This project shares example problems that run live on your own machine using built in Golang tools such as go test to learn not just syntax, but the tool chain as well. Exercism goes a step beyond by having users collaborate on improving each other’s solutions. Not to mention the Exercism commandline tool is itself written in Golang.

Sites & Blogs

The web is full of resources to help you get started with Golang (almost as much as to help you catch Pokemon). Here are a few favorites:

  • Golang.org: The official site of Golang run by the project maintainers this site is a tremendous resource including software, documentation, blogs, and all kinds of other resources.
  • Learn X in Y Minutes: Golang I’m a big fan of a good cheatsheet. This is the best one I’ve seen for Golang. It’s the perfect syntax reminder.
  • Go By Example: The next step up from a cheatsheet, Go By Example has small digestable topics perfect for those moments you want to see variations on a particular pattern.
  • Komand Tech Blog: Komand is a security automation suite built using Golang. They’re taking an open approach and sharing what they learn, including their most recent post Quick security wins in Golang (Part 1).
  • Awesome Go: My go to place to find out “Is there a Golang project for X?”. There’s almost always something useful.
  • Dave Cheney’s Blog: If there’s a blog to start with it’s Dave’s. He has been writing about Golang since the beginning and has some great lessons. For newbies I recommend starting with Resources for new Golang programmers and Five suggestions for setting up a Golang project.


  • The Go Programming Language: This was where I went after the Golang Tour and I still think the best formal resource for learning Go. It’s nicely paced, doesn’t assume too much or too little, and covers everything in the Golang core you’ll need to start building.
  • The Little Book of Go: As far as open source books go (yep, it’s 100% free as in beer) I think LBoG is as good as it gets. I keep it on my desktop and reference it often. If I were a student and the $30 price tag of GoPL scared me off this is a very close second.

Other Resources

Projects to Look at!

There are tons of interesting Golang projects out there on GitHub, especially around security:

I also know there’s something awesome coming out soon from Ryan Huber in Go. I don’t want to spoil the surprise, but I’d keep an eye out.

Companies Using Golang

Lots of security companies are making Golang part of their stack (and thus a good language to learn if you want to work for those companies):

And I’m sure more are looking into it. Hopefully you will be too.

CTI SquadGoals - Setting Requirements

Squad Goals

Requirements. The first part of the intelligence cycle and the most neglected. According to the appendix of Joint Publication 2-0: Joint Intelligence

intelligence requirement. 1. Any subject, general or specific, upon which there is a need for the collection of information, or the production of intelligence. 2. A requirement for intelligence to fill a gap in the command’s knowledge or understanding of the operational environment or threat forces.

Intelligence requirements (or just requirements) are key questions (as @cyint_dude calls them) that stakeholders (the CERT, leadership, etc) want the intelligence team to answer. Requirements set the entire priority of your intelligence cycle, what sources you collect from, what types of processing you need to do, and what methods of dissemination will meet the need. Where your team is focusing, aka Squad Goals. Lets get started…

Past Incident Based Requirements

Effective leadership is putting first things first. ~ Stephen Covey

I’ve said it a thousands times, but place #1 to start is your own incident management system. Build requirements based on incidents you’ve already investigated, every IP address, every piece of malware, every domain. Why do you think firms with consultants who do incident response always have good intelligence? It’s no coincidence, they set intelligence requirements based on the incidents they respond to and so should you. This isn’t ground breaking, so we’ll move on.

Business Plan Based Requirements

If there are two men, Rod and Rob, and you can only steal from one, which one would you choose? The answer is: Whichever one is a banker. ~ Jarod Kintz

The next place to gather requirement is by digging into your organizations business plan. Certain actors target resources in a way that you can rule them in or out of scope:

  • Are you a bank or financial institution? Financially motivated attackers probably want to attack you.
  • Are you doing energy exploration? A handful of attackers specialize in that.
  • Do you invent pharmaceuticals? Look for actors that try to steal intellectual property.

CrowdStrike illustrated this in their Uncover the Adversary map:

Crowdstrike Actor Map

So using this as an example (and CrowdStrike vernacular):

  • What about that bank? They’ll want to focus on Deep Panda, Impersonating Panda, and Singing Spider.
  • If you’re an oil company you should be on the look out for Cutting Kitten, Energetic Bear, and Ghost Jackal.
  • The firm with the new heart attack treatment? I’d be on the look out for Deep Panda, Union Spider, and Corsair Jackal.

These distinctions won’t always be perfect but they provide a solid basis to develop requirements. Your industry heavily defines your threat model and thus is a rich starting point to begin defining your intelligence requirements.

Geographic Based Requirements

I get to go to overseas places, like Canada. ~ Britney Spears

Geography is a tricky thing. The Internet lets us access information across the world yet at the same time it’s often an extension of real life. Attackers take advantage of geographic distribution to remain semi-anonymous, but in others they use computer network attack to project a different kind of power within a region.


The Naikon Group is a good example. Public reports say they’re focused on operations within South East Asia. Don’t live in South East Asia? Naikon is probably not interested to you. Living in Singapore? You should probably be paying attention. Even though an actor could strike across the world (and may do so) it would be sloppy to not consider location in defining intelligence requirements.

Aside: Before you say it I agree, VPNs are easy and hotpoints can be anywhere. Avoid assuming too much when it comes to the Internet and physical location.

Technology Based Requirements

Windows 2000: Designed for the Internet. The Internet: Designed for UNIX. ~ Unknown

It’s a common mistake to think of attackers as omnipotent. They have limits, budgets, time, etc. As a result most attackers focus on attacking certain kinds of networks. A Windows network running Active Directory is often harder to attack than a Linux network on LDAP (don’t knock MSFT, they’ve invested a lot in security over the last 15 years) but the fact is most attackers are probably more familiar and invested in attacking Windows and AD.

Nmap... because?

When developing intelligence requirements take the time to look around your infrastructure, what makes you special? Do you run lots of Industrial Control Systems? You should probably be paying attention to different attackers than a law firm running all Macs. Part of this is vulnerability management as well, understanding the flaws that are out there for the systems you protect, but it’s also understanding what attacks can exploit those vulnerabilities.

Vertical Based Requirements

I would rather walk with a friend in the dark, than alone in the light. ~ Helen Keller

One of the least appreciated approaches to forming intelligence requirements is asking others. Backs work with banks on similar criminal groups. A university could collaborate with similar universities by sharing information about incidents. Industry has been embracing this for years with the formation of Information Sharing and Analysis Centers such as FS-ISAC, the Retail Cyber Intelligence Sharing Center, and others. Whether formal or informal these groups provide the chance to learn from others and avoid making the same mistakes twice.

One Off Requirements

Reports that say that something hasn’t happened are always interesting to me, because as we know, there are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns – the ones we don’t know we don’t know. And if one looks throughout the history of our country and other free countries, it is the latter category that tend to be the difficult ones. ~ Donald Rumsfeld

Then there are the odd bits. They may come in when a higher up is watching CNN in the airport or they may come from a spooky government source. No matter what they will show up at some point and become part of your intelligence requirements, maybe as a one time addition, maybe on an ongoing basis. Don’t discount them.

Writing Strong Requirements

The best intelligence requirements are specific enough to direct answering the question. Here are some characteristics that lead to good intelligence requirements:

  • Singular: A strong requirement focuses on one question and only one question.
  • Atomic: The requirement should be specific to a particular fact or event.
  • Decision Centric: The requirement should lead to making a decision.
  • Timeliness: A requirement should capture the timeframe for usable intelligence.

It takes time and practice to develop good requirements.

Weak Requirement Strong Requirement
What actors might attack our company? Is X Actor active?
What indicators should we be using in our IDS? What are current file system indicators of Actor X?
Are there actors targeting the location we’re in? Is an actor specifically targeting Region Y?
What malware should we look for on our network? Are attackers using Locky against my vertical?
Will our organization get DDoS’d? Are there indicators of DDoS attack in the next 30 days?

Practice makes perfect, but working with others to sharpen your requirements writing is even better.

Just the Start

Even though they’re commonly missed the fact is developing good requirements sets the solid foundation for a good intelligence cycle. If you want to read more the CIAs Center for Intelligence Studies has a great article A Fresh Look at Collection Requirements and Army Field Manual 34-2 Collection Management to Support Commanders includes the appendix Developing Priority Intelligence Requirements. Read those and you’ll be on the right track.

osquery 101 - Getting Started

I admit it… I’m a fanboy. A straight up osquery fanboy.

Oh… what is osquery you ask? Good question there sport.

osquery allows you to easily ask questions about your Linux and OSX infrastructure. Whether your goal is intrusion detection, infrastructure reliability, or compliance, osquery gives you the ability to empower and inform a broad set of organizations within your company.

That’s how Facebook describes it. I’d say osquery is the most effective way available to monitor an OSX or Linux host for security. But that’s just me. Still not bad no matter which definition you prefer.

The big question everyone asks though is how do you get started with osquery and so without further ado:

Install osquery

This one is pretty simple. Just follow these handy instructions.

OSX Side Note: If you’re on OSX and a fan of Homebrew (as I am) you’ll want to skip Homebrew this time and install the package directly. Homebrew gets weird about running things as root later on, and that will be a pain. Safe yourself the pain.

Using osqueryi

osquery provides two main interfaces to the user: osqueryi and osqueryd (we’ll get to osqueryd in another post). osqueryi is a REPL (Read-Evaluate-Print Loop) similar to $ irb or $ python. This lets you input a command and get back an immediate response. We can use this for a few things:

  • One off, spot checking of specific system attributes.
  • Developing new queries.
  • Learning your way around osquery.

So lets give it a try. Get started by typing osqueryi at your terminal.

osqueryi Prompt

osquery… uh… queries

Now that you’re at the prompt you’re ready to start looking for things. osquery uses the sqlite query language for simplicity and descriptiveness. The most basic query format is:

> SELECT columns FROM table;

Don’t forget the semicolon “;” at the end of your query! You wouldn’t like it without the semicolon. In that it won’t do anything.

For example lets take a look at the OSX kernel extensions I have installed:

select from example

Ok, so that found the data I wanted, but a bit too much. Every OSX system has Apple written kernel extensions, which makes those less interesting. Using the sql concept of a where clause we can limit this just to non-Apple kernel extensions:

select from where example

Much more useful. Lets take it one step further and make it more readable with sorting:

select from where example

For those playing at home you can see I have 11 kernel extensions for 5 applications (and OSX itself). The final query that got this information is:

> SELECT name FROM kernel_extensions WHERE name NOT LIKE 'com.apple%' ORDER BY name;

Pretty straight forward, but by now the power should be evident. As you practice more you can express powerful and complex ideas using this syntax. For more on syntax take a look at the SELECT documentation for sqlite. This coupled with the osquery Tables reference will give you most of what you need. A couple of key, non-query, commands will help you get started:

Command Use
.help Gives you the osqueryi help dialogue
.schema [TABLE] Prints the table columns and data types
.exit It isn’t exit or quit or exit()

If you want some examples of what you can look for and how you can use the sqlite syntax take a look at the osquery Packs. Here are a few of my favorites:

Incident Response: open_sockets

> select distinct pid, family, protocol, local_address, local_port, remote_address, remote_port, path from process_open_sockets where path <> '' or remote_address <> '';

A typical SELECT FROM WHERE style query to find open network connections. Combine this with an IP blacklist to identify C2 activity.

Incident Response: startup_items

> select * from startup_items;

Startup Items is a common way for OSX malware to persist between restarts. Most users don’t even notice it’s there.

Vulnerability Management: chrome_extensions

> select chrome_extensions.* from users join chrome_extensions using (uid);

Chrome extensions have a scary amount of access, especially in web based workflows. They’re worth investigating.

Want to Learn More?

I do too. There’s a lot to osquery. It’s not a solution, it’s a framework. On one hand that makes it customizable, on the other that means many parts of using it are left to the user. It may be free, but that means it’s an investment of time rather than money.

Kevin Thompson and I spoke about osqueryi and more at BSidesDFW 2015:

I’ll explore some of these ideas from our presentation in the next post in this series: osquery 110 - Programmatic osqueryi.