InteliwertBot

InteliwertBot is the web crawler of Inteliwert, operated by Bandpey GmbH in Munich. It reads the websites of companies named in public procurement notices, to collect public evidence about what they do. It identifies itself with this user agent:

InteliwertBot/1.0 (+https://inteliwert.com/bot)

InteliwertBot visits a company’s website only when a public procurement notice names that website. It reads the Impressum to confirm that the website belongs to the company the notice names, then a few pages from a fixed list:

• Impressum or legal notice
• About us
• Services and products
• Certificates and quality
• References and projects

It stays on the website’s own domain and fetches at most 8 pages per website. It never fetches contact, career, team or people pages, never logs in, never fills in a form and never runs scripts.

• From the Impressum: the company’s legal name, its register entry and its VAT ID.
• The certifications and prequalifications the website names, such as ISO 9001 or PQ-VOB, each with a short quote.
• The link to the company’s LinkedIn page.
• The web address of the company’s logo.

It reads no contact details and nothing about people. The fetched pages stay private. Inteliwert uses at most short quotes from them, each with a link to its page and the date it was checked, and shows a certification as self-declared on the company’s website.

Before it fetches a page, InteliwertBot reads the website’s robots.txt and follows the group for InteliwertBot, or else the group for all crawlers. When robots.txt cannot be read, it fetches no page and tries again later.

It honours machine-readable reservations of text and data mining under section 44b of the German Copyright Act (Urheberrechtsgesetz, UrhG): the tdm-reservation HTTP header and meta tag, the file /.well-known/tdmrep.json, and noai in the X-Robots-Tag header or the robots meta tag. A reserved page is neither stored nor read.

• One request every 2 seconds per domain.
• At most 8 pages per website, 2 MB per page and 10 seconds per request.
• A website is checked again about once a month.

To keep InteliwertBot away from your whole website, add this to your robots.txt:

User-agent: InteliwertBot
Disallow: /

To keep it away from part of your website, disallow that path instead. A text and data mining reservation, as above, also keeps your pages from being read.

For questions, corrections or to have your company’s data removed, write to hello@inteliwert.com.