Crawler

(Deutsche Fassung)

Web-Crawler in general

CERT.at - the National Computer Emergency Response Team of Austria - is searching for vulnerable software-installations within our constituency. The intent is to help the people responsible to close the security-holes.

To achieve this, we have implemented an automatic crawler (like the common searchengines use) which visits every .at-domain.
Because there are a great many at-domains we currently only retrieve the starting page of each domain. Should we expand on this service, this might change in the future.

The crawler identifies itself as CERT.at-Crawler/0.9 (+http://www.cert.at/about/crawler/content.html).

How does it work?

Currently the crawler tries to retrieve the following files:

  • robots.txt (this states what automatic crawlers are allowed to retrieve, see http://en.wikipedia.org/wiki/Robots.txt for details).
  • a sitemap, if it was mentioned in robots.txt and not forbidden
  • the start-page, if not forbidden in robots.txt

Usual hostnames like "www" and "shop" are also tried.

Web-Crawler for MS14-066 (WinShock)

CERT.at - the National Computer Emergency Response Team of Austria - is searching for vulnerable software-installations within our constituency. The intent is to help the people responsible to close the security-holes. To achieve this, we have implemented an automatic crawler (like the common searchengines use) which visits IP addresses within Austrian IP ranges.

The crawler identifies itself as CERT.at-Crawler/1.0 (+http://www.cert.at/about/crawlerms14066/content.html).

How does it work?

Currently the crawler tries to retrieve/verify the following:

  1. (retrieve) Index-/Entry-Page
  2. (verify) if Internet Information Services for Windows Server (IIS)
  3. (verify) supported SSL/TLS-Ciphers

Frequently asked questions

How has the crawler found my domain, I just registered it?

NIC.at, the registry for .at-domains, kindly provides us with a list of all .at-domains. This information is confidential, of course, and we will not give it to third parties.

In case we find problems, we'll work with the contacts provided during registration (admin-c, tech-c) and the owner to fix them.

I don't want CERT.at accessing my domain, how can I prevent that?

The best option is to make an appropriate entry in the sites' robots.txt-file, such as:

User-agent: CERT.at-Crawler*
Disallow: /

For details, please see http://en.wikipedia.org/wiki/Robots.txt.

There's a technical problem with your crawler, help!

Don't hesitate to contact us! See our contact page for details.