Usefull links
The Internet of Bots analyzes automated bots and looks at how a website can best be structured. This process results in a lot of data, in which a part of the data consists of useful links. These links often point to systems that enable users to employ or improve on bots. The Internet of Bots has categorized these to the following categories
Webmaster Tools
What can you do as a webmaster in order to ensure that your website is defined and crawled correctly? There are several tools that can help analyse the structure of your website(s) and offer feedback on possible improvements. The Internet of Bots tries to collect and enlist the best of them:
| Language subtag lookup | Check the validity of language tags |
| Online XML-Sitemap | Get an automated sitemap with this online tool |
| Google Webmasters | Add your domains to Google Webmasters |
| Uptime | Do a Domain Health Check for your website |
Call to Crawl
Do you have a website and do you want bots from search engines to come and crawl your content? The following anchors result in a pages where you can submit your own domain to be crawled.
 |
Get your sited indexed by Google | |
 |
Bing | Get your sited indexed by Bing (and Yahoo) |
 |
Yandex | Get your sited indexed by Yandex (needs login) |
 |
Exalead | Get your sited indexed by Exalead |
 |
Wotbox | Get your sited indexed by Wotbox |
 |
Website-datenbank | Get your sited indexed by Website-datenbank |
 |
Findx | Get your sited indexed by Findx |
Bots for You
There are several tools with which you can crawl the internet for yourself. Are there certain types of websites you want to analyse? Do you want to employ bots for yourself? The following suggestions might get you started.
| Crawler4j | An open source web crawler (using Java) |
| Scrapy | An open source web crawler (using Python) |
| SEOstats | An open source web crawler (using PHP) |
| Sqlmap | An open source web crawler (using SQL) |
| BUbiNG | An open source web crawler |