Cloud Discovery

If you have arrived here, it's probably because something called 4N6-CloudDiscovery visited your website. This page explains what it is, what it does, and how to slow it down or stop it.

What it is

4N6-CloudDiscovery is a tool run by 4N6 Ltd, a Cyber Essentials certification body. Given a domain name, it works out which cloud services an organisation appears to use, and records the evidence for each one.

It is used by 4N6, our partners, other Cyber Essentials certification bodies, IT support companies and other organisations, for reasons such as:

  • helping an applicant for Cyber Essentials complete the questionnaire accurately, and helping assessors check that nothing has been missed
  • finding services that were signed up to and forgotten (legacy services) or signed up to by staff without approval (shadow IT)
  • helping organisations and their IT support companies understand their dependencies, so they can tidy up their estate and reduce risk
  • testing and improving the tool itself

It looks only at what any visitor to the site could see. It does not log in to anything.

What it does, and what it does not do

It does

Reads public records for the domain

Things like DNS, certificate transparency logs, registrar (RDAP/WHOIS) and IP ownership.

Loads pages of your website in a real browser as a visitor would

It uses Chromium (the browser engine inside Google Chrome and Microsoft Edge) so that services loaded by scripts show up. By default it loads up to 40 pages on your main site.

Records which other hosts each page loads from, links to or embeds

Things like analytics, chat, payments, forms, fonts, maps, video and so on.

Reads the sitemap and follows links from the home page

It starts with the pages closest to your homepage and works through until the limits are reached or when it has checked all pages, whichever comes first.

Clicks "accept" on a cookie banner

Many services only load after consent has been given.

Checks subdomains it finds reading each one's own robots.txt first

Just like the searches on your main website but by default we limit this to up to 10 pages.

Looks for subdomains whose DNS points at a service that no longer exists

These could be taken over by someone else.

Reports what it found

Supplies the findings along with the evidence, to the person who asked for the scan.

It does not

Scan ports or probe servers for vulnerabilities

Attempt to log in, guess passwords, or open anything behind a login

Download files

We won't download PDFs, other documents, or fetch the content of images, media and fonts.

Try to find hidden pages

The only guessed addresses are a few common contact page URLs (such as /contact), and only when the site has no contact link we could find.

Attempt to submit forms

The only exception is where they appear to be part of a cookie consent process.

Click through transaction pages

Things like a web shop, booking  page or account sign-up steps, or pages that sign a visitor out.

Contact addresses that DNS says are private (internal) ones

Change, damage or store anything on your site

All these icons are completely free for commercial use.

Publish results

They go to whoever commissioned the scan.

How we identify ourselves

Every request carries a User-Agent like this, which ends with our name and a link to this page:

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0 Safari/537.36 4N6-CloudDiscovery/1.0 (+https://4n6.io/cloud-discovery)

Requests come from cloud services operated by AWS or OVH.

If you cannot match a visit to this page, or you think it is not us, tell us: [email protected]

How politely it behaves

Pace. It requests two pages at a time by default, and normally manages somewhere around ten pages a minute. A scan is limited by default to 40 pages of the main site, 10 per subdomain, and 30 minutes duration.

robots.txt. We read robots.txt before crawling, on the main site and on each subdomain, and follow it. If a group names 4N6-CloudDiscovery it applies to us and the * group does not. Otherwise the * group applies. Where rules overlap, the longest (most specific) rule wins, and Allow beats Disallow if they are equal, as in RFC 9309.

The one exception is where the organisation, or someone acting on its behalf such as its IT support company, has given us permission to ignore robots.txt rules. We record who gave that permission, which may be disclosed on request. Any specified crawl delay is still respected in that case.

Crawl-delay. If your robots.txt has a Crawl-delay, it is treated as the time to leave between pages. If we request two pages together, we wait twice the delay before the next pair, measured from when the pair was requested. If that would take more than 30 seconds, we request fewer pages at once, down to one page at a time. A delay of more than 30 seconds is treated as 30 seconds, as many large search engines do.

Being told to slow down. If your site answers HTTP 429 (too many requests) or 503, we put the page back and wait before trying again: 15 seconds, then 30, 60, 120 and 240, never more than 5 minutes, or longer if your Retry-After header asks for it. After the first such response we request one page at a time, and ease off only gradually as pages succeed. A page is tried four times at most. If five waves in a row are refused, the scan stops. Pages already read are still reported, and the report says the scan was cut short.

Time limit. Whatever the delays, a scan stops starting new pages when its time budget (30 minutes by default) runs out.

Slowing us down or blocking

Ask us nicely, in robots.txt. To keep us out of everything:

User-agent: 4N6-CloudDiscovery
Disallow: /

To keep us out of part of the site, and slow us down on the rest:

User-agent: 4N6-CloudDiscovery
Disallow: /private/
Crawl-delay: 10

Note that a group naming us replaces the * group for us, so repeat any * rules you want us to follow in the same group.

Rate limit us. We treat a 429 or 503 as a request to slow down, so a rate limit that returns one, ideally with a Retry-After header, is respected and is the most reliable way to control our pace.

Block us. You can block the User-Agent token 4N6-CloudDiscovery in your firewall, WAF or CDN. A block is simply a failed request to us. We will not try to get round it.

Is blocking a problem? Not for us. If the scan is for a Cyber Essentials application, your assessor may ask you to allow it.

If you did not ask for this

Someone, perhaps your IT support company or a certification body, will have asked 4N6 or one of our partners to scan your domain. If you did not expect it and do not want it, let us know your domain name and we will add it to our do-not-scan list which is checked before every scan.

What we keep

Results are provided only to whoever commissioned the scan. They're also kept in our log files until they expire, typically around 6 months after the scan takes place. We may use the results to help improve the tool.

We do not sell the scan results and do not make them publicly available. They contain only details about your website and services, and do not contain personal data unless this has been provided publicly by you (e.g staff profile pages, client case studies).

Contact

Our Cyber Essentials team created this tool and can be contacted via the following methods: