Usefull links

The Internet of Bots analyzes automated bots and looks at how a website can best be structured. This process results in a lot of data, in which a part of the data consists of useful links. These links often point to systems that enable users to employ or improve on bots. The Internet of Bots has categorized these to the following categories


Webmaster Tools

What can you do as a webmaster in order to ensure that your website is defined and crawled correctly? There are several tools that can help analyse the structure of your website(s) and offer feedback on possible improvements. The Internet of Bots tries to collect and enlist the best of them:

Language subtag lookupCheck the validity of language tags
Online XML-SitemapGet an automated sitemap with this online tool
Google WebmastersAdd your domains to Google Webmasters
UptimeDo a Domain Health Check for your website

Call to Crawl

Do you have a website and do you want bots from search engines to come and crawl your content? The following anchors result in a pages where you can submit your own domain to be crawled.

Logo Google Google Get your sited indexed by Google
Logo Bing Bing Get your sited indexed by Bing (and Yahoo)
Logo Yandex Yandex Get your sited indexed by Yandex (needs login)
Logo Exalead Exalead Get your sited indexed by Exalead
Logo Wotbox Wotbox Get your sited indexed by Wotbox
Logo Website-datenbank Website-datenbank Get your sited indexed by Website-datenbank
Logo Findx Findx Get your sited indexed by Findx

Bots for You

There are several tools with which you can crawl the internet for yourself. Are there certain types of websites you want to analyse? Do you want to employ bots for yourself? The following suggestions might get you started.

Crawler4jAn open source web crawler (using Java)
ScrapyAn open source web crawler (using Python)
SEOstats An open source web crawler (using PHP)
SqlmapAn open source web crawler (using SQL)
BUbiNGAn open source web crawler