Scrapy

  • For a Public crawler
  • Called Scrapy
  • From Uruguay
  • By an unknown organization
  • Gets a score of 48%

What is Scrapy?

Scrapy is a bot for website Scrapy. Scrapy is a piece of software that everyone can install in order to crawl websites for themselves. It is an open source and collaborative framework for extracting data from websites (in a fast, simple, yet extensible way). They advertise with two simple steps: Build and run your web spiders and Deploy them to Scrapy Cloud or use Scrapyd to host the spiders on your own server. Since everyone can use Scrapy, it is not clear what is done with the data that is being collected when this crawler visits your domain. The creators and the main maintainers of Scrapy operate the website scrapinghub.com, where they offer their services.


BotRank for Scrapy (48%):

The Internet of Bots has evaluated Scrapy against 50 different checkpoints, of which 24 have been confirmed as being positive. These checkpoints evaluate the transparency and occurrence of a bot and don´t necessarily say something about its quality. The BotRank is calculated as [24*2=] 48%. The details of the bot and the BotRank can be seen below. For more information on how the BotRank is made, you can visit the page Botrank.

User Agent(s) (3/5):

  1. Distinguishable: Yes
  2. Botname: Scrapy
  3. Email: Not mentioned
  4. Version: 1.3.0
  5. Mozilla: Not mentioned

Whois (4/5):

  1. Public: Visit Whois
  2. Organization: Not specified
  3. Country: Uruguay
  4. City: Montevideo
  5. Street: Yes

Weblinks (3/5):

  1. User agent: Visit Scrapy
  2. Crawler: Visit Scrapy
  3. Homepage: Visit Scrapy
  4. Query: Not available
  5. Adding: Not available

Usage (2/5):

  1. Recommended: No
  2. Category: Public crawler
  3. Free query: No
  4. Register: No
  5. Logo: Yes

Occurrence (6/15):

  1. Has visited during 6 of the 15 control months

Webdepth (6/15):

  1. Has visited 6 of the 15 control sites