We are happy to announce the completion of the IntelMQ 3.0 milestone. The addition of 2 developers to the IntelMQ development team in October 2020 allowed us to work on major IntelMQ 3.0 changes - new features as well as architectural changes. The most important new features are:

• Domain-based workflow (Capability to use domain-based data)
• Simplified configuration management
• Easier bot development via "Mixins"

The development was based on a thorough collection of user requirements by former CERT.at employee Aaron Kaplan. This requirements collection included personal interviews in the CERT-community, workshops at CERT-conferences as well as architectural reviews. Our previous IntelMQ release, IntelMQ 2.3.0, was an intermediate step, and brought, for example, Docker-support, the IntelMQ-API and the revised documentation page.

Domain based workflow

The introduction of various new IntelMQ bots (also know as "Experts") removed IntelMQ's previous limitation to IP address-based data. Versions prior to IntelMQ 3.0 supported almost no domain-based workflows, most operations and look-ups could only be performed with IP addresses. For example, if the domain "example.com" was used as a command and control server, IntelMQ had to resolve example.com to an IP address and add that to the event. Any further processing was based on this IP address. To overcome this limitations, we added some new components in IntelMQ 3.0 that address the different requirements for processing domain-based data. These include:

• Querying the HTTP return status of websites
• Examining website's content (for example to detect whether a website has been defaced)
• Taking a screenshot of a web page using LookyLoo
• Querying contact information for a domain

The important "experts" bots for domain-based workflows that where added in IntelMQ 3.0 at a glance:

• HTTP StatusCode Expert: Checks websites' response status code with a given value
• HTTP Content Expert: Checks websites' content for a given string
• LookyLoo Expert: Instructs a LookyLoo-instance to create a screenshot of the given website and adds a screenshot URL
• RDAP Expert: Checks the given domain against public or user defined RDAP services.^[1]
• uWhoisd: Queries a uWhoisd WHOIS-proxy instance and saves the whois-response. This expert was contributed by Raphaël Vinot (CIRCL).
• Trusted Introducer Lookup Expert: Checks which CERT is responsible for the domain in the Trusted Introducer directory.
• Tuency Expert: Queries a tuency Contact database to enrich the event with constituency and notification information (e.g. abuse contact for a domain) based on configured rules. A future blog post will discuss the release of Tuency in greater detail.

For additional approaches for looking up abuse contact information, please have a look at our guide on abuse-contact look-ups.

IntelMQ Enhancement Proposal (IEP)

An IEP is a document, like an RFC or a PEP (Python Enhancement Proposal) that describes the proposed changes in detail, including examples and implications. These texts are available in a separate GitHub repository certtools/ieps, which can also be used to work on them collaboratively. Our idea behind introducing the IntelMQ Enhancement Proposal (IEP) process was to discuss major changes publicly to reach a consensus within the community. The discussion in the community takes place on the developer's mailinglist. This process allows us to collect feedback, insight and proposals from the whole community. It's up to the community to decide if a proposal can be adopted or rejected. We have published 4 IEPs so far and have two more in the pipeline:

IEP01: Configuration

In IntelMQ 3.0, we redesigned the configuration to simplify the maintenance of an IntelMQ instance. In the proposal IEP 001 we described common issues with the current JSON configuration and asked the community to vote for one out of three possible new data formats. The community decided on YAML, a choice we are very happy with because we believe it offers the best usability. This change also merged three different and overlapping configuration files (defaults.conf, pipeline.conf and runtime.conf) into one single runtime.yaml. Furthermore, it is now possible to store default parameter values inside each bot programmatically, which reduces the risk for errors when new parameters are added. This eliminates the need for the BOTS file, further reducing configuration complexity and development effort.

IEP02: Mixins

Mixins are a new component in IntelMQ to simplify the creation of new bots, by allowing the developer to use already existing templates to cover common tasks, such as Caches/States and HTTP Sessions.

IEP03: Multiple Values in the data format

In IntelMQ's Internal Data Format ("IDF"), one field can have only one value, for example a single IP address or domain. However, in practice an IoC (Indicator of Compromse) may consist of multiple fields, e.g. a domain could map to multiple IP addresses. Allowing multiple values - as specified by IEP03 - would address this issue. An alternative approach would be linking events by unique identifiers as specified in IEP04 (see below). IEP03 and IEP04 are mutually exclusive.

Data example without Multiple Values

  {
  "source.ip": "192.168.0.1",
  "source.observation": "2021-07-22T11:12:52+00:00Z",
  "extra.tags": "http-exploit",
  }

Data example with Multiple Values

  {
  "source.ip": ["192.168.0.1", "192.168.2.1"],
  "source.observation": "2021-07-22T11:12:52+00:00",
  "extra.tags": ["http-exploit", "ddos-amplification"],
  }

IEP04: Meta Information and Data Exchange

Meta Information provides context to IoCs and is useful when exchanging data across IntelMQ instances or visualizing event relationships. Based on previous work by the community and our own research, we proposed a possible solution in March 2021. The proposal included a detailed analysis of the problem, use-cases and an implementation suggestion. The community welcomed the proposal, but did not reach a consensus on the exact specification. IEP03 and IEP04 were also the topic of the first - virtual - IntelMQ Hackathon which took place on 22nd April 2021. The discussion of the proposal is still ongoing and a conclusion has not been reached yet. The biggest concern within the community is, that IEP04 would add a lot of complexity and would be considered a breaking change.

Example

{
  "__meta": {
    "UUID": "1ea60f56-b67b-61fc-829a-0242ac130003",
    "Version": 1,
  },
  events: [
    {
      "source.ip": "192.168.0.1/32",
      "source.observation": "2021-07-22T11:12:52+00:00",
    }
  ]
}

Various other new bots and improvements

An Aggregate Expert is now included in IntelMQ by default, allowing users to aggregate matching events within a configured time span. Aggregation means that single events will be discarded and the count of matching events in the time span is kept. The Threshold expert provides similar functionality, which passes an event when a configured threshold count is reached within the configured time span. This expert is a contribution by Karl Johan Karlsson (liu.se).

n6 Integration

IntelMQ 3.0 integrates seamlessly with n6, an automated incident handling tool similar to IntelMQ, including bi-directional conversion of data formats. For more information, please see our documentation.

Final note

A detailed list of all changes can be found in the release notes, the full documentation is available at intelmq.readthedocs.io. As IntelMQ is a community project, every contribution is welcome. If you want to know more about "How to create a bot", please check out our Developer's Guide.

1. [1] RDAP is the successor of WHOIS using JSON as machine-readable data format.

This blog post is part of a series of blog posts related to our CEF Telecom 2018-AT-IA-0111 project, which also supports our participation in the CSIRTs Network.