|
|
1. |
All About Search Indexing Robots and Spiders
-
Search Tools Consulting explains how the search engine programs called "robots" or "spiders" work, and reviews related sites.
|
|
|
2. |
HTTP User Agent Index
-
An alphabetical list of user agents and the deployer behind them, compiled by Christoph Rüegg.
|
|
|
3. |
List of Robot Agent Strings
-
A list from PGTS of Web robots with the identifying data they leave in Web site logs.
|
|
|
4. |
Psychedelix List of User-Agents
-
Andreas Staeding's large list of search engine spiders, similar Web robots, and Web browsers: their web-log identification and links to their originators.
|
|
|
5. |
Robots, Spiders and Other User Agents: a Resource for WebMasters
-
José Luis Pellicer's searchable database of robots, spiders and other user agents for programs that surf the web.
|
|
|
6. |
Search Engine IP Addresses
-
Lists IP addresses of search engine spiders. Can be searched by IP address. Also links to resources on spiders.
|
|
|
7. |
Search Engine Robots and Other User Agents
-
John A. Fotheringham presents data in tabular form on the robots sent by search engines and other sites to read and index Web pages: their origins, names and IP addresses.
|
|
|
8. |
Spider Track
-
Tracks and displays real time hits and IP addresses of MSNbot, Yahoo's Slurp, and Googlebot. Provides aggregate hits of each spider over the last 30 days.
|
|
|
9. |
ASAP Consulting: Robots/Crawlers DB
NEW!
-
Searchable list of robot agent strings with their description and links to home pages.
|
|
|
10. |
Bots vs Browsers
-
This large database lists user agents in categories and distinguishes between robots and browsers.
|
|
|
11. |
Robot IP Address
-
Brian Dunnintg provides a list of all the major search engine robot IP addresses, by full class C only.
|
|
|
12. |
Robotstxt.org
-
Information on the robots.txt Robots Exclusion Standard and other articles about writing well-behaved Web robots.
|
