Python Threat Hunting Tools: Part 3 — Interacting with APIs

Welcome back to this series on building threat hunting tools!

In this series, I will be showcasing a variety of threat hunting tools that you can use to hunt for threats, automate tedious processes, and extend to create your own toolkit! The majority of these tools will be simple, focusing on being easy to understand and implement. This is so that you, the reader, can learn from these tools and begin to develop your own. 

There will be no cookie-cutter tutorial on programming fundamentals, instead, this series will focus on the practical implementation of scripting/programming through small projects. It is encouraged that you play with these scripts, figure out ways to break or extend them, and try to improve on their basic design to fit your needs. I find this the best way to learn any new programming language/concept and, certainly, the best way to derive value!

In this installment, we will be looking at APIs and learning how we can create our own tool to interact with an API to check if an Indicator of Compromised (IOC) is malicious.

What are APIs?

An API, or Application Programming Interface, is a set of protocols, routines, and tools for building software applications. It defines a set of rules that allow software applications to interact with each other to exchange data and services. They can be considered a bridge between two applications that allows them to communicate with each other in a standard way. 

APIs enable developers to access specific functionality or data from a service or application without requiring access to the underlying code. This makes it possible to integrate different software applications, databases, and services, as well as to build new applications on top of existing ones.

The most common APIs you are likely to interact with are web APIs, which allow you to interact with a web-based application in a programmable manner. APIs can be public (which anyone can use) or private (which are only accessible by authorized parties). Again, most of the time you will use APIs that will require some form of authorization (username & password, token, etc.) as they will relate to an individual account or tenant and be private.

That said, in this demonstration, we will be making use of a public API that is provided by the team over at Maltiverse. Maltiverse is a threat intelligence platform that provides a comprehensive view of potential cyber threats. It aggregates data from various sources such as malware analysis reports, open-source intelligence, and proprietary threat intelligence feeds to create a centralized database of threat information. 

The platform offers a range of tools and features to help security professionals and researchers analyze threats and respond to them effectively. These include a threat intelligence feed, a threat hunting platform, a malware analysis sandbox, and a visualization tool for exploring and analyzing threat data. 

The platform aims to help organizations detect and respond to cyber threats quickly and effectively by providing real-time threat intelligence and analysis. It is regularly used by security operations teams, incident responders, threat hunters, and security analysts to gain insights into potential threats and take proactive measures to mitigate them. They have a free and paid API that security professionals can use to check if an IOC is malicious or not. We will be using the free version in this demonstration.

The Problem

As a threat intelligence analyst (or any member of the SOC), you are inundated with IOCs that you need to quickly analyze and determine if they represent a threat:

• You will see a machine reach out to an IP address — is that IP address malicious?
• You will see an uncommon file hash on a machine — is this a piece of malware?
• You will see a user visit an uncommon domain — is this domain malicious?

These are questions that we have to answer on a daily basis as security professionals and it can become a very time-consuming process if these IOCs stack up to be in the hundreds or even thousands. It would be great if we could quickly check if an IP address in a terminal window. To solve this problem we can make use of a bit of Python scripting and the Maltiverse API.

The Solution(s)

The Maltiverse API offers us a way to check if an IOC is malicious or not in a programmable manner. There are two ways we can do this. Firstly, we can use the Maltiverse Python module to interact with the API through methods this module exposes. Secondly, we can directly query the Maltiverse web API with the Python requests module. I will demonstrate both through a tool that takes in an IP address and quickly determines if it is malicious or not.

Maltiverse Python Module

The team at Maltiverse was gracious enough to release a Python library for their web-based API. This allows us to easily make requests to their IOC search engine using the Python methods their module exposes. The module can be found on GitHub and installed using Python’s pip package installer with the command:

pip install maltiverse

First, we need to import the required modules we will use in this script. This includes the ipaddress module (which you can also install using pip), which we will use to validate IP addresses provided to us by a user, as well as the Maltiverse class from the aforementioned maltiverse package. This class gives us access to the Maltiverse API by instantiating an object of this class.

import ipaddress
from maltiverse import Maltiverse

# instantiate object to interact with Maltiverse API
api = Maltiverse()
# api = Maltiverse(auth_token="...")

Notice here I create the api object without passing any parameters to the Maltiverse class. This is because we will be using the free version of the API. If you have access to the paid version you will need to create an authorization token on the Maltiverse website and pass this to the class as auth_token. The code for this is commented out in the example above.

Next, we will create an infinite while loop to get input from the user using the built-in input() method in Python. This input is saved to the ip variable and sanitized using the Python string method .strip() to remove whitespace. If the input is equal to the string quit then we exit the loop and the script terminates.

while True:
 # get user input
 ip = input("Please enter IP to search for (or type exit to quit): ")
 # clean user input
 clean_ip = ip.strip()
 
 # check if the user wants to exit
 if clean_ip == "quit":
     print("Goodbye.")
     break

If the user entered an IP address, we need to validate this to make sure it is a legitimate IP address, otherwise, our script will throw an error. To do this, we will use a try / except block which is a way of dealing with errors (exceptions) in Python. 

The code under the try block will run and if the error specified in the except block header is returned then the except block will run. If another error is thrown then the default Python error behavior will occur (the script will terminate and the error will be printed to standard output), while if no error is generated then the script will continue. 

Here we test if the user input is an IP address by using the ipaddress module’s .ip_address() method. This will return a ValueError if the IP address is not valid, which will be caught by our except statement and handled accordingly. An error message will be printed and the script will return to the start of the while loop using a continue statement. If no error is raised then we continue with our script.

# validate the IP address
try:
 ipaddress.ip_address(clean_ip)
except ValueError:
 print(f" - {clean_ip} is not a valid IP address. Please try again...\n")
 continue

If the IP address is valid then we can move on to using the Maltiverse API to determine if it is malicious or not by using the api object’s .ip_get() method. This method takes an IP address (as a string value) and returns data about the said IP address in JSON format as shown with 110.189.222.98 (source).

{
   "address": "No.31 ,jingrong street,beijing\n100032",
   "as_name": "AS4134 Chinanet",
   "asn_cidr": "110.184.0.0/13",
   "asn_country_code": "CN",
   "asn_date": "2009-05-11 00:00:00",
   "asn_registry": "apnic",
   "blacklist": [
       {
           "count": 1,
           "description": "Mail Spammer",
           "first_seen": "2017-11-30 12:39:45",
           "last_seen": "2017-11-30 12:39:45",
           "source": "Blocklist.de"
       },
       {
           "count": 1,
           "description": "Malicious Host",
           "first_seen": "2020-01-28 00:45:48",
           "last_seen": "2020-01-29 07:10:12",
           "source": "CIArmy"
       },
       {
           "count": 1,
           "description": "Malicious Host",
           "first_seen": "2020-01-29 00:18:08",
           "last_seen": "2020-02-19 06:34:10",
           "source": "Alienvault Ip Reputation Database"
       },
       {
           "count": 1,
           "description": "Mail Spammer",
           "first_seen": "2020-03-22 11:18:35",
           "last_seen": "2020-03-22 11:18:35",
           "source": "Barracuda"
       },
       {
           "count": 5,
           "description": "Scanning IPs",
           "first_seen": "2020-01-26 09:01:00",
           "last_seen": "2020-02-01 10:26:00",
           "source": "IBM X-Force Exchange"
       },
       {
           "count": 32,
           "description": "Spam",
           "first_seen": "2017-07-14 06:44:00",
           "last_seen": "2020-03-21 07:52:00",
           "source": "IBM X-Force Exchange"
       }
   ],
   "cidr": [
       "110.184.0.0/13"
   ],
   "classification": "malicious",
   "country_code": "CN",
   "creation_time": "2017-11-30 12:39:45",
   "email": [
       "[email protected]",
       "[email protected]"
   ],
   "ip_addr": "110.189.222.98",
   "location": {
       "lat": 30.6667,
       "lon": 104.0667
   },
   "modification_time": "2020-02-19 06:34:10",
   "registrant_name": "CHINANET Sichuan province network\\nData Communication Division\\nChina Telecom",
   "tag": [
       "mail",
       "spam"
   ],
   "type": "ip"
}

We can extract data from the information that is returned using Python key indexing. The key we are most interested in is named "classification" as this will contain data related to whether the IP address is malicious or not. We can access the value stored at this key with the code result['classification'] as shown.

# query Maltiverse API for data about the result
result = api.ip_get(clean_ip)

# check if the 'classification' key is present
try:
  # print the 'classification' of the IP address
  print(f"\\n=> The IP address {clean_ip} has been identified as {result['classification']} by Maltiverse\\n")
except KeyError:
  # no classification available from Maltiverse
  print(f"\\n - The IP address {clean_ip} cannot classified by Maltiverse\\n")

This code uses a try / except block again to test if the classification key is present in the data. If it is then the value it holds is printed (using f strings), if it is not and a KeyError exception is raised then a different message is printed. The infinite while loop will then continue to ask for user input and we can look up another IP address.

The full code can be found on my GitHub under maltiverse_ip_lookup-v1.py and looks like:

import ipaddress
from maltiverse import Maltiverse

# instantiate object to interact with Maltiverse API
api = Maltiverse()
# api = Maltiverse(auth_token="...")

while True:
   # get user input
   ip = input("Please enter IP to search for (or type exit to quit): ")
   # clean user input
   clean_ip = ip.strip()
   # check if the user wants to exit
   if clean_ip == "quit":
       print("Goodbye.")
       break
       
   # validate the IP address
   try:
       ipaddress.ip_address(clean_ip)
   except ValueError:
       print(f" - {clean_ip} is not a valid IP address. Please try again...\n")
       continue

   # query Maltiverse API for data about the result
   result = api.ip_get(clean_ip)

   # check if the 'classification' key is present
   try:
       # print the 'classification' of the IP address
       print(f"\n=> The IP address {clean_ip} has been identified as {result['classification']} by Maltiverse\n")
   except KeyError:
       # no classification available from Maltiverse
       print(f"\n - The IP address {clean_ip} cannot classified by Maltiverse\n")

Python Requests Module

The code for this example is very similar to before however, instead of using the Maltiverse Python module to interact with the Maltiverse API we will need to use the requests module. The Python requests package is the typical way you will use Python to interact with web APIs if said API does not offer a Python package. 

To use this module we need to import it and the json module, because the Maltiverse API (and most other web APIs) will return results in JSON format. We also need to create a variable to hold the URL we will be sending our requests. This is the API endpoint which when queried will return JSON data.

import ipaddress
import json
import requests

# URL to send Maltiverse API requests related to IP addresses to
url = 'https://api.maltiverse.com/ip/'

The next bit of code is the same as before; setting up an infinite while loop, getting user input, cleaning the input, checking if the user wants to quit, and validating the IP address using the ipaddress module.

while True:
 # get user input
 ip = input("Please enter IP to search for (or type exit to quit): ")
 # clean user input
 clean_ip = ip.strip()

 # check if the user wants to exit
 if clean_ip == "quit":
     print("Goodbye.")
     break

 # validate the IP address
 try:
     ipaddress.ip_address(clean_ip)
 except ValueError:
     print(f" - {clean_ip} is not a valid IP address. Please try again...\n")
     continue

Now we need to use the requests module’s .get() method to send a request to the API endpoint (URL) with the IP address we want to get data about. We can do this using Python string concatenation (url + ip). This will return a JSON object that we need to parse in order to get the relevant data. 

This is done by using the json module’s .loads() method to transform this JSON object into a Python dictionary object that we can query using key indexing as before. We specifically want the text attribute of this JSON object so we only create a dictionary containing the relevant data (no metadata).

# query to Maltiverse API for a response (returned as a JSON object)
response = requests.get(url + ip)
# parse the JSON object to just get the relevant text data
esult = json.loads(response.text)

Finally, the last part of this script is the same as the previous version where we check if a classification can be provided by Maltivese for the user-provided IP address and, if so, print this out. If not, we print an error. Either way, we start the while loop again and ask for user input.

# check if the 'classification' key is present
try:
  # print the 'classification' of the IP address
  print(f"\n=> The IP address {clean_ip} has been identified as {result['classification']} by Maltiverse\n")
except KeyError:
  # no classification available from Maltiverse
  print(f"\n - The IP address {clean_ip} cannot classified by Maltitverse\n")

The full code can be found on my GitHub under maltiverse_ip_lookup-v2.py and looks like:

import ipaddress
import json
import requests

# URL to send Maltiverse API requests related to IP addresses to
url = 'https://api.maltiverse.com/ip/'

while True:
   # get user input
   ip = input("Please enter IP to search for (or type exit to quit): ")
   # clean user input
   clean_ip = ip.strip()

   # check if the user wants to exit
   if clean_ip == "quit":
       print("Goodbye.")
       break

   # validate the IP address
   try:
       ipaddress.ip_address(clean_ip)
   except ValueError:
       print(f" - {clean_ip} is not a valid IP address. Please try again...\n")
       continue

   # query to Maltiverse API for a response (returned as a JSON object)
   response = requests.get(url + ip)
   # parse the JSON object to just get the relevant text data
   result = json.loads(response.text)

   # check if the 'classification' key is present
   try:
       # print the 'classification' of the IP address
       print(f"\n=> The IP address {clean_ip} has been identified as {result['classification']} by Maltiverse\n")
   except KeyError:
       # no classification available from Maltiverse
       print(f"\n - The IP address {clean_ip} cannot classified by Maltiverse\n")

These simple scripts demonstrate how to interact with a web API, either through a Python module or directly using the requests module. Interacting with APIs is fundamental for returning context about an IOC or automating a process, and you will find them incredibly useful in your day-to-day work as a cybersecurity professional. 

Try to improve the scripts provided here by expanding them to include other IOC types (hostnames, URLs, file hashes) or by creating an IOC list and passing this into the script instead of relying on user input.

Next time in this series we will look at using browser automation to interact with the web when an API is not available. Till then, happy hunting!

Discover more in the Python Threat Hunting Tools series!