Today we released the newest IntelMQ version 2.3.0 along with its companion tools - the IntelMQ Manager and the new IntelMQ API.

This version comes with significant changes, being an important intermediate step for the 3.0 release scheduled for summer 2021.

We moved the documentation to a new home, see our previous blog post "IntelMQ offers tutorial lessons and a new documentation page". The new documentation page also integrates the documentation for the Manager and API, ending the times of distributed documentation. It further features overviews of the integration possibilities with n6 and MISP.

Using a combination of Vagrant and Ansible, end-to-end tests enhance our quality management, which now consists of extensive unit-tests, packaging tests, spelling & styling check and as well as security analyses.

Numerous "bots" (IntelMQ's plug-able components) have been added or gained new significant features:

• CZ.nic HAAS and PROKI Parsers, by Filip Pokorný and Edvard Rejthar (CSIRT.CZ)
• ESET Collector and Parser, by Mikk Margus Möll (CERT.EE)
• Kafka Collector, by Birger Schacht (CERT.at)
• Key-Value Parser, by Karl-Johan Karlsson (Linköping University)
• Request Tracker Output, by Marius Urkis (NRDCS.LT)
• Shadowserver Reports API and JSON Parser, by Birger Schacht (CERT.at)
• Splunk Saved Search Expert, by Karl-Johan Karlsson (Linköping University)
• Threshold Expert, by Karl-Johan Karlsson (Linköping University)
• Shadowserver CSV & JSON Parser: Support for the feeds MSRDPUDP, Vulnerable-HTTP, Sinkhole DNS and fixes for existing feed mappings, by Sebastian Waldbauer and Sebastian Wagner (CERT.at)
• HTTP collector: PGP signature check functionality, by sinus-x
• Several Experts (1, 2, 3, 4): Integrated local database update mechanisms, by Filip Pokorný (CSIRT.CZ)

All new changes can be read in the change log. If you are upgrading, please also have a look at the news file. If you get started, have a look at our documentation which contains an introduction and detailed information on the installation.

The new IntelMQ API and overhauled IntelMQ Manager back-end

Version 2.3.0 comes with a new API, a feature which has often been requested for IntelMQ. The API actually originates from the IntelMQ Manager:

Thanks to SUNET-funding, the contributing company Intevation rewrote the back-end of the IntelMQ Manager in Python. Python is the main language used in the IntelMQ projects, but until the rewrite PHP was used for the backend. As part of the revamp, the URLs have been changed to better match those of a proper programming interface. Additionally, Intevation added optional authentication directly into the API.

CERT.at then further split the IntelMQ Manager's back-end off into the IntelMQ API. Therefore it's now possible to run the Manager and the API on different hosts.

Docker

IntelMQ 2.3.0 is the first release with an official Docker image available at Dockerhub under certat/intelmq-full. Using Docker is the simplest way of getting started with IntelMQ as of now. But as it is brand-new, we consider it as beta currently.

The container consists of IntelMQ with all optional dependencies, including the Manager and the API, whereas Redis and nginx are ran in separate containers.
Some configuration variables are passed to the containers using environment variables. This functionality is new in IntelMQ as well, but not yet available for all configuration settings. IntelMQ 3.0 will be able to use arbitrary parameters from the environment.

The installation instructions contain details about the set-up process.

Shadowserver Reports API

Shadowserver is an internationally active and altruistic organisation scanning the Internet for vulnerable devices every day and sinkholing various botnets. They provide the data free-of-charge to CERTs worldwide which are able to act upon the Threat Intelligence data. The vast amount of data is split into different report types, whereas one report by report type is provided per day. A report only contains the data which is relevant for the recipient.

The traditional data transmission manner are e-mails with CSV data files, either directly as attachment or - if very big - linked for download via HTTPS. Since October 2020, Shadowserver also provides an HTTP API. IntelMQ supports this API as data collection since this version and allows all IntelMQ users (with an active Shadowserver cooperation) to get Shadowservers's reports directly and without detours into the processing pipeline. Aligned with IntelMQ's concept of separating data collection and parsing, IntelMQ has two separate components for the Shadowserver Reports API support:

• The Shadowserver Reports API Collector needs to be configured with the API credentials, the relative time-frame and optionally a list of report types. If no list of report types is given, all available reports are downloaded. The collector keeps track which reports have already been downloaded, so the collector can be executed frequently without downloading data multiple times. The format of the downloaded data is JSON, as opposed to the data provided via e-mail, which is CSV.
• The Shadowserver JSON Parser uses the same field-mappings as CSV. But as opposed to the Shadowserver CSV parser, which maps the columns of CSV-files to IntelMQ's internal fields, the JSON parser does the same for JSON dictionaries. Both Shadowserver parsers are able to detect the report type based on the file name, which was recorded by the collector in the first place.

Shadowserver is actively looking for sponsors to keep up the great value they provide to the IT security community. Please consider becoming a sponsor.

This blog post is part of a series of blog posts related to our CEF Telecom 2018-AT-IA-0111 project, which also supports our participation in the CSIRTs Network.