Welcome to the start of this new series on building threat hunting tools with Python!
In this series I will be showcasing a variety of threat hunting tools which you can use to hunt for threats, automate tedious processes, and extend to create your own toolkit! The majority of these tools will be simple with a focus on being easy to understand and implement. This is so that you, the reader, can learn from these tools and begin to develop your own. There will be no cookie-cutter tutorial on programming fundamentals, instead this series will focus on the practical implementation of scripting/programming through small projects. It is encouraged that you play with these scripts, figure out ways to break or extend them, and try to improve on their basic design to fit your needs. I find this approach the best way to learn any new programming language/concept.
Before we delve into any technical details let’s consider why we should develop our own threat hunting tools in the first place.
The Why
So why would you need your own tools when a simple Google search will reveal tons of free and paid options?
When you first start out in cybersecurity you don’t need to know how to build things. There are plenty of free and paid tools you can use, and your company should provide training on the tools it has chosen to utilise. However, I believe there is a point that everyone reaches in their cybersecurity career (or any technical career) where they graduate beyond being given toys to play with and, instead, are ready to make their own toys. It’s a natural next step for anyone who wants to progress in a technical career. But why?
This question can be answered in four parts:
- Everything has an API nowadays
- You have a specific need for a tool that just isn’t being met
- You need to tailor or extend an existing tool to your needs
- You want to gain a deeper understanding of something
Everything has an API Nowadays
To begin with, the simplest reason to create your own tools is so that you can interact with a pre-existing tool or web service’s API.
Nowadays everything has an API from your favorite Internet site to your smart lights at home, and security tools are no exception. Your SIEM solution will have an API, the EDR platform running on your company’s machines will have an API, and the threat intelligence sites you use to add context to an IOC all have APIs. As security professionals we are surrounded by a world that is designed to be programmable and it would be a missed opportunity to not take advantage of it!
- If you need to run a report every morning to check for outbound connections to new IP addresses, you could use your SIEM’s API to schedule, run, and email you this report every morning before you even get out of bed.
- If you need to check to see if one of hundreds of IOCs are present in your environment, you could use your EDR’s API to loop through each IOC in a matter of minutes.
- If you need to check if an IOC is actually malicious or not, you could use your favorite threat intelligence provider’s API to check hundreds within minutes.
APIs provide us with the opportunity to automate our workflow and save countless hours manually searching, pressing buttons, or waiting for reports to generate. In a hyper connected world it only makes sense to learn how to interact with them through the power of scripting.
Create new tools to fulfill your need
Sometimes you will search high and low for a tool to fulfill a specific need and all you will find is something beyond your budget or nothing at all. In these cases you will need to create something custom to fulfill your needs.
An example might be running threat hunt queries. A company I previously worked at was using an EDR solution to run threat hunting queries. This solution had an API but it was behind a paywall. This made it difficult to automate running threat hunting queries because every time you wanted to run one you had to go through the web GUI. As a workaround I put together a Python script that used browser automation to loop through all the threat hunting queries I wanted to run.
It logged into the EDR solution, opened a new browser tab, and automatically typed out the query into the search bar and ran it. Then it would open a new tab and type out the next query, and so on. All I had to do was cycle through the browser tabs to see if any results had been returned to investigate. This saved hours of time manually typing out and running queries.
Creating a tool from scratch may sound like a daunting process but using a scripting language like Python can make it incredibly simple. Python has thousands of pre-build packages you can import to achieve a staggering amount of functionality.
- If you want to make web requests then Python will have a module for that.
- If you want to parse CSV then Python will have a module for that.
- If you want to interact with a database then Python will have a module for that.
- And so on…
This lets you quickly develop almost limitless functionality in a matter of hours because you don’t need to re-implement the associated complexities of said functionality, just import a module to do that for you. For a better idea of the vast amounts of modules available check out the Python Package Index. Once you learn a few programming basics you will be able to use Python’s package library to achieve almost anything you want!
Tailor existing tools to your own needs
Perhaps you don’t need to create a brand new tool to get a job done and there is already an existing tool that does 90% of the job for you. In that case, you may want to extend that tool to get that final 10% done. Some security tools offer users the ability to extend their functionality through scripting. A couple of well known examples are Burp Suite and Immunity Debugger.
- Burp Suite is a set of tools used for web applications testing that have been integrated into a unified GUI application. It includes a wide range of functionality for performing this testing but it also provides an extensibility API which can be used to add additional functionality. Extensions can be written in Java, Python or Ruby, and can be shared with other users on Burp Suite’s App Store (BApp Store). For instance, if you wanted to bypass 403 (access forbidden) error messages you could write an extension that allowed you to automatically do that (like this one).
- Immunity Debugger is a powerful software debugger that is used in the cybersecurity industry for writing exploits, analyzing malware, and reverse engineering binary files. One of it’s unique selling points is a large and well supported Python API for easy extensibility. This Python API allows users to automate many of the tedious functions of searching through assembly code. Perhaps the most well-known extension is mona.py, which allows users to automate and speed up specific searches while developing exploits for the Windows platform.
There are plenty of other examples of tools that allow a user to add additional functionality themselves, through scripting, or by importing this functionality using an add-on (similar to browser extensions). For instance, Burp Suite has the BApp Store which contains Burp extensions written by Burp Suite users to extend Burp’s capabilities.
Some knowledge of scripting is vital for interacting with existing tool’s APIs and making the most of a tool’s feature set.
The learning experience
The best reason to start creating your own tools is to learn. Developing your own cybersecurity tools can be an incredibly valuable learning experience. It can help you gain a deeper understanding of how security works by providing hands-on experience with the low-level details that underpin technologies and techniques.
Take port scanning as an example. This is a reconnaissance technique whereby an attacker will send network packets to each port on a target system to see which ones are open. We can easily do this by using a tool like Nmap, however this hides away many of the low-level details involved in port scanning.
- How is my machine connecting to the target machine?
- What packets are being sent to elicit a response?
- How do I know if a port is open or not?
These questions are implementation details that are hidden away by Nmap. If we want answers to these questions then a good learning activity is to build your own port scanner and discover the implementation details for yourself. This will require us to have a deep understanding about networking (e.g. sockets, the TCP 3-way handshake, etc.), as well as being able to build something practical using a scripting/programming language. As such, our understanding of port scanning goes from the theoretical to the practical level.
At this point we could look at how other people have implemented port scanners by delving into open source tools (usually on GitHub) to see how they solved the problems we faced. This broadens our knowledge further as we will see different problem solving approaches and we can then apply these approaches to future problems we may face. We have gone from creating a port scanner to learning new ways to tackle complex problems!
This is why building your own tools is so important in progressing your technical career. It provides added value to your everyday job whilst allowing you to delve into avenues of learning you would have completely missed by using off-the-shelf tools!
The How
Now we have a solid understanding of why it is important to create our own tools for threat hunting we can shift our focus to understanding how we can go about doing this.
In my opinion there are three steps to achieving this:
Step #1: Gain an understanding of why you should bother to learn how to create your own tools
We have briefly discussed this above, and hopefully you have some motivation to begin, but to truly understand the power of creating your own tools you should investigate some of the advantages that even a basic scripting language can offer you over manual work. Once you gain an appreciation for the capabilities you can develop, once you harness the power of computing and automation your motivation for learning how to achieve this will increase tenfold.
For instance, if you find that writing a Python script can save you 2 hours of work a day (2 hours you could spend with loved ones or playing Call of Duty) you will want to know how you can do this.
Step #2: Learn the fundamentals of a basic scripting language
Once you have sufficient motivation for achieving the how, you should next try to learn the fundamentals of a basic scripting language like Python. This language is great for automating basic tasks like system administrator work, web scraping, and creating command line tools, whilst offering an easy to understand syntax that abstracts much of the harder to understand programming concepts (e.g. memory management).
I would advise avoiding languages like C++, Rust, Go, etc. at this stage as it’s important to gain an appreciation for programming without overwhelming yourself with abstract concepts that will provide little immediate value. Instead, look at taking a Python course so you can learn the fundamentals and understand the basic building blocks of programming. Then you can jump straight to getting value for your efforts!
Step #3: Build, build, build!
Once you have an understanding of the basics (data types, control structures, loops, etc.) you should dive into creating your own projects to see how these basic blocks can come together to create something of value. You don’t need to build these projects from scratch or know every implementation detail to begin building a project. You can start by extending an existing project to make it your own; change a variable’s name, add another command line option, try adding some additional functionality that will be helpful to you. When you feel like you are ready to create your own project simply write a brief outline of what you will need to implement and tackle it step by step.
If you get stuck then use Google or online documentation to get by. It is important to remember that no one has an entire project figured out before they start (especially if it’s a complex project), they learn as they go and, in the process, get better at programming. Then the next project they tackle they’ll do it a little better or finish it a little faster. Learning to program is an iterative process that continues by taking small steps and acquiring knowledge on the job.
The rest of this series focuses on step 3 of the how. There will be many threat hunting tools showcased along the way that you can use yourself in your daily hunts or learn from to get better at programming by tailoring them to your own needs.
All code will be hosted on GitHub for you to have easy access.
Hopefully, you can draw inspiration from these tools to create your own threat hunting tools and share these with the cybersecurity community.
Till next time, stay frosty and keep on hunting!
Discover more in the Python Threat Hunting Tools series!
