User Agents

What is a user agent?

When a user wants to visit a website, it is necessary to access it with a web browser. The web browser is in this case called the user agent. It is the software that is acting on behalf of the user in order to open the website. The user agent always communicates with the website and identifies itself by its name. An Android Tablet using Firefox might identify itself as:

Mozilla/5.0 (Android 4.1.1; Tablet; rv:50.0) Gecko/50.0 Firefox/50.0

Every browser has its own standard name it gives to a visiting website. It is possible to change this name and when a bot visits a website, it often gives its own user agent, often containing a URL and/or an email address. When a bot from Google comes to visit a website, it might identify itself as:

Googlebot/2.1 (+http://www.googlebot.com/bot.html)


How is it used?

  1. Every browser has its own standard user agent. You are currently visiting this website with user agent: CCBot/2.0 (https://commoncrawl.org/faq/)
  2. Users and bots can change the user name that is given to a website. An online tool for changing a user name is:
  3. Every webserver registers the visiting user agents. These can often be accessed using the metrics from the back end of a server.
  4. Webmasters can use several tools in order to access or register the visiting user agents for themselves. Two of the most popular means are through the PHP function string$user_agent or the Javascript function javascript:alert(navigator.userAgent)

Why is it important?

  • The user agent is one of the variables a website uses in order to recognize who is visiting.
  • Webmasters can use the user agents to check what type of users are visiting a website. With this information, it is for example possible to approach the number of visiting bots and/or the number of real users. This is exactly what Google Analytics and the server statistics do.
  • Automated user agents are expected to follow the rules established in the robots.txt file.
  • It is possible to authorize a website and exclude certain user agents from visiting.

What can I do to check if my own user agent?

Your web browser has told its name to The Internet of Bots. The name and meaning of your current user agent are:

CCBot/2.0 (https://commoncrawl.org/faq/)

Meaning: