User-agent: (www.hotbot.com), collecting Web statistics
Disallow: /
User-agent: publish maps, on and of the World wide web.
Disallow: /
User-agent: to be used mainly by consultants, and web masters to create and
Disallow: /
User-agent: "Hazel's Horde". The information is publicly available and
Disallow: /
User-agent: (Australian/New Zealand content ONLY).
Disallow: /
User-agent: - Clear Lake.
Disallow: /
User-agent: 1996
Disallow: /
User-agent: 2.4.2 browser (also written in C)
Disallow: /
User-agent: n) of the Vision Home Page
Disallow: /
User-agent: Alex Zavatone 3/29/96
Disallow: /
User-agent: Based Software Engineering Program at the Research Institute
Disallow: /
User-agent: California, Irvine, in 1993. Presented at the First
Disallow: /
User-agent: Can be used to describe a set of pages to fetch, and to
Disallow: /
User-agent: Check URLs modified since a given date.
Disallow: /
User-agent: Clark)
Disallow: /
User-agent: Collects WWW pages for both InfoSeek's free WWW search and
Disallow: /
User-agent: Effective date of the data posting is to be
Disallow: /
User-agent: Frontier, a scripting system for Macs
Disallow: /
User-agent: HTML document and the absolute url. A search engine provides
Disallow: /
User-agent: HTML objects that can be found. Also, Harvest allows users
Disallow: /
User-agent: Hence - Job Robot.
Disallow: /
User-agent: HÂkan Ard– and myself.
Disallow: /
User-agent: In the User-agent field $TYPE is set to 'Mapper' for Web searcher
Disallow: /
User-agent: Intended to seek out sites of potential "career interest".
Disallow: /
User-agent: International WWW Conference in Geneva, 1994.
Disallow: /
User-agent: Macromedia on a hungover Sunday as a proof of concept. -
Disallow: /
User-agent: NT.
Disallow: /
User-agent: Searcher is intended to perform the list of WWW sites of
Disallow: /
User-agent: Society.
Disallow: /
User-agent: Spry is refusing to give any comments about this
Disallow: /
User-agent: Sunderland
Disallow: /
User-agent: The resulting database is used by the search engine
Disallow: /
User-agent: They work in an onion format hopping from spot to spot one
Disallow: /
User-agent: This Robot traverses the net and creates a searchable
Disallow: /
User-agent: This code has not been looked at in a while, but will be
Disallow: /
User-agent: WWW sites of mit.edu, com, org, etc... domain)
Disallow: /
User-agent: Written by Anders Ard–, Mattias Borrell,
Disallow: /
User-agent: accordingly
Disallow: /
User-agent: algorithm to find WWW pages.
Disallow: /
User-agent: all pages within n links (for some small
Disallow: /
User-agent: an appropriate hyper-text link.
Disallow: /
User-agent: analysis of the Web versus GopherSpace (from the Veronica
Disallow: /
User-agent: and 'StAlone' for Web analyzer.
Disallow: /
User-agent: and FTP servers and downloads them into database searchable
Disallow: /
User-agent: and Search Engine, plus distributed operation under
Disallow: /
User-agent: and accessed by clients - is run by the end user or archive
Disallow: /
User-agent: and generate statistics. Localised South Pacific Discovery
Disallow: /
User-agent: and interacting with both the webserver and MX servers of
Disallow: /
User-agent: and tells you when they have changed.
Disallow: /
User-agent: and to generate statistics. The ArchitextSpider collects
Disallow: /
User-agent: and validate links. Used for indexing the .dk top-level
Disallow: /
User-agent: announced.
Disallow: /
User-agent: applications, including resource discovery, link validation,
Disallow: /
User-agent: at a time. The User-agent field contains a coded ID
Disallow: /
User-agent: at http://www.fi/search.html.
Disallow: /
User-agent: at the WWW94 Conference in Chicago.
Disallow: /
User-agent: automatic abstract generation
Disallow: /
User-agent: be blocked via robots.txt using this ID.
Disallow: /
User-agent: behind the HP firewall
Disallow: /
User-agent: both the raw data and original source code.
Disallow: /
User-agent: browsing.
Disallow: /
User-agent: by automatic converters such as LaTeX2HTML or WebMaker. At
Disallow: /
User-agent: capture the crawled information. If the robot sees a
Disallow: /
User-agent: catalog relations among URLs and to support a special
Disallow: /
User-agent: collect approximately 10k html documents for testing
Disallow: /
User-agent: collection of Java classes which can be used in a variety of
Disallow: /
User-agent: commercial search. Uses a unique proprietary algorithm to
Disallow: /
User-agent: contain Shockwave movies. It is the first bot or spider
Disallow: /
User-agent: continuing its crawl. Since this is a personal spider, it
Disallow: /
User-agent: copy document trees. Designed as a tool for retrieving web
Disallow: /
User-agent: data) as well as indexing.
Disallow: /
User-agent: database from the Finnish (top-level domain .fi) www servers.
Disallow: /
User-agent: database of Web pages. It stores the title string of the
Disallow: /
User-agent: database, as well as link validation.
Disallow: /
User-agent: decided not to release this robot to the
Disallow: /
User-agent: default).
Disallow: /
User-agent: delayed browser or as a mirroring tool. It cannot jump from
Disallow: /
User-agent: designed to maintain the cache generated by the emacs emacs
Disallow: /
User-agent: desired domain (for example it can perform list of all
Disallow: /
User-agent: development.
Disallow: /
User-agent: diff, and extensive link management facilities). All
Disallow: /
User-agent: diff/patch).
Disallow: /
User-agent: distributed workload and supports context queries. Intended
Disallow: /
User-agent: document trees, and generate statistics. Webwalk is easily
Disallow: /
User-agent: document. Known to work on U*ixes and Windows
Disallow: /
User-agent: documentation. Catch discovery.com without waiting! A fully
Disallow: /
User-agent: documents (keep them in sync with the original document via
Disallow: /
User-agent: domain as well as other Danish sites for aDanish web
Disallow: /
User-agent: ever been run remotely.
Disallow: /
User-agent: excerpts.
Disallow: /
User-agent: exists that is integrated into the Tübingen Mosaic
Disallow: /
User-agent: extensible to perform virtually any maintenance function
Disallow: /
User-agent: fast, but never has more than one request per site
Disallow: /
User-agent: files recursively using HTTP protocol.It can be used as a
Disallow: /
User-agent: focus on a particular site. No instance of the robot should
Disallow: /
User-agent: for Computing and Information Systems, University of Houston
Disallow: /
User-agent: for a random link generator. Also is used to research the
Disallow: /
User-agent: formula, properties etc.)
Disallow: /
User-agent: francophone. Uses the Accept-Language tag and reduces demand
Disallow: /
User-agent: from anywhere
Disallow: /
User-agent: from completion. I'm planning to limit the depth at which it
Disallow: /
User-agent: gathered into different relational databases, known as
Disallow: /
User-agent: given set of pages on one or more servers. It reports links
Disallow: /
User-agent: growth of specific sites.
Disallow: /
User-agent: have more than one outstanding request out to any given site
Disallow: /
User-agent: host per minute.
Disallow: /
User-agent: identify the most popular and interesting WWW pages. Very
Disallow: /
User-agent: identifying the instance of the spider; specific users can
Disallow: /
User-agent: ignores off-site links, so does not stray from a list of
Disallow: /
User-agent: indexing/ressource discovery, fulltext search, text
Disallow: /
User-agent: information for the Excite and WebCrawler search engines.
Disallow: /
User-agent: information related to Israel or Israeli
Disallow: /
User-agent: is typical of robots. Pauses 1 second between requests (by
Disallow: /
User-agent: level at a time over the internet. The information is
Disallow: /
User-agent: links from a given starting point (with some controls to
Disallow: /
User-agent: links that looks like a hot list. The search can be by key
Disallow: /
User-agent: lists and depth and count limits. Therefore, Harvest
Disallow: /
User-agent: local server
Disallow: /
User-agent: localsites. External GETs will be added, but these will be
Disallow: /
User-agent: maintain an archive or mirror. Is not run by a central site
Disallow: /
User-agent: maintainer
Disallow: /
User-agent: mirroring, etc. It currently limits itself to one visit per
Disallow: /
User-agent: model of the web to guide intelligent, directed searches for
Disallow: /
User-agent: month. It will honor the /robots.txt file at that
Disallow: /
User-agent: navigation aid.
Disallow: /
User-agent: of Hong Kong
Disallow: /
User-agent: of Informix(tm) Database and WN 1.11 serversoftware for
Disallow: /
User-agent: of their own site as well as to other sites, perhaps with
Disallow: /
User-agent: on Web sites connected with mathematics and statistics. It
Disallow: /
User-agent: one site to another.
Disallow: /
User-agent: only the directory and subdirectories of it's start
Disallow: /
User-agent: operational web robot for NT/95 today, most UNIX soon, MAC
Disallow: /
User-agent: option of the find(1) command. Webwalk is usually used
Disallow: /
User-agent: options
Disallow: /
User-agent: outstanding at any given time. Has been refined for more
Disallow: /
User-agent: owners to add the url to the searchable database.
Disallow: /
User-agent: pages in batch mode without the encumbrance of a browser.
Disallow: /
User-agent: perform mirroring, and generate statistics. Uses combination
Disallow: /
User-agent: prevent it going off-site or limitless)
Disallow: /
User-agent: problem. At the end of its completion, I hope to publish
Disallow: /
User-agent: product uses the robot to manage a site (site copy, site
Disallow: /
User-agent: product, it acts as a personal spider to work with a browser
Disallow: /
User-agent: project
Disallow: /
User-agent: project conducted by three undergraduate at the University
Disallow: /
User-agent: project.
Disallow: /
User-agent: provides a much more controlled way of indexing the Web than
Disallow: /
User-agent: public
Disallow: /
User-agent: published map, it will return the published map rather than
Disallow: /
User-agent: remote sites.
Disallow: /
User-agent: retrieval and discovery in the WWW, using a finite memory
Disallow: /
User-agent: retrieved
Disallow: /
User-agent: returned. The TkWWW Robot is described in a paper presented
Disallow: /
User-agent: robot
Disallow: /
User-agent: round-robin fashion, with multiple processes.
Disallow: /
User-agent: running slowly. WebLinker is meant to be run locally, so if
Disallow: /
User-agent: servers here.
Disallow: /
User-agent: servers specified initially.
Disallow: /
User-agent: services. Uses a unique, incremental, very fast proprietary
Disallow: /
User-agent: similar content.
Disallow: /
User-agent: sleeps between requests, and the next version will use HEAD
Disallow: /
User-agent: specific information needs
Disallow: /
User-agent: specific collections, rather than to locate and index all
Disallow: /
User-agent: spider. Packaged with a full GUI in the CyberPilo Pro
Disallow: /
User-agent: spruced up for the Emacs-w3 2.2.0 release sometime this
Disallow: /
User-agent: statistics.
Disallow: /
User-agent: than a year.
Disallow: /
User-agent: that local statisistical info can be used in artificial intelligence programs.
Disallow: /
User-agent: that may be logically related. The Robot returns a list of
Disallow: /
User-agent: the Hong Kong and China WWW domain . It is a research
Disallow: /
User-agent: the boolean AND & OR query models with or without filtering
Disallow: /
User-agent: the indexing of the Scandinavian Web
Disallow: /
User-agent: the moment it works at full speed, but is restricted to
Disallow: /
User-agent: the stop list of words. Feature is kept for the Web page
Disallow: /
User-agent: time.
Disallow: /
User-agent: to be a complete updated resource for Israeli sites and
Disallow: /
User-agent: to check if the entire document needs to be
Disallow: /
User-agent: to control the enumeration several ways, including stop
Disallow: /
User-agent: to facilitiate context-based navigation. The WebMapper
Disallow: /
User-agent: to retrieve data from all available sources on the internet.
Disallow: /
User-agent: tomorrow.
Disallow: /
User-agent: validate HTML, and generate statistics
Disallow: /
User-agent: validate links, validate HTML, perform mirroring, copy
Disallow: /
User-agent: versions can create publishable NetCarta WebMaps, which
Disallow: /
User-agent: w3 mode (N*tscape replacement) and to support annotated
Disallow: /
User-agent: web pages so
Disallow: /
User-agent: which involves web traversal, in a way much like the '-exec'
Disallow: /
User-agent: which returned an error code
Disallow: /
User-agent: will be free for the browsing at www.greenearth.com.
Disallow: /
User-agent: will be hitting sites while I finish debugging it.
Disallow: /
User-agent: will be launched from multiple domains. This robot tends to
Disallow: /
User-agent: will probe, so hopefully IAgent won't cause anyone much of a
Disallow: /
User-agent: with structure queries (substructure, fullstructure,
Disallow: /
User-agent: word or all links at a distance of one or two hops may be
Disallow: /
User-agent: written in Shockwave. The bot was originally written at
Disallow: /
User-agent: you see it elsewhere let the author know!
Disallow: /
User-agent: your hard disk on a TV-like schedule. Catch w3
Disallow: /
User-agent:
Disallow: /
User-agent: Indexing in webdew is done by manually.
Disallow: /
User-agent: data collection for webdew indexing service.
Disallow: /
User-agent: statistics reflects the priority of WWW server
Disallow: /
User-agent: top pages last modified date data. Collected
Disallow: /
User-agent: It supports HTTP, File, FTP, and Mailto schemes.
Disallow: /
User-agent: can also be used to mirror whole or partial sites.
Disallow: /
User-agent: can be used to up-load pages to the site or to
Disallow: /
User-agent: modify existing pages and URLs within the site. It
Disallow: /
User-agent: site, builds HTML reports describing the site, and
Disallow: /
User-agent: web sites. It builds a database detailing the
Disallow: /
User-agent: processing Laboratory, KDD R&D Laboratories, 1996-1997
Disallow: /
User-agent: 1995/1996
Disallow: /
User-agent: Malaysia and other countries.
Disallow: /
User-agent: University in 1995.
Disallow: /
User-agent: University of Washington for finding personal Homepages.
Disallow: /
User-agent: specified user.
Disallow: /
User-agent: the growth in the web.
Disallow: /
User-agent: to WWW pages within New Zealand. Now also used in
Disallow: /
User-agent: . It schedules visits
Disallow: /
User-agent: HTTP HEAD. Robot runs twice a week. Under HTTP 5xx
Disallow: /
User-agent: Lisa Search service. The robot manually launch
Disallow: /
User-agent: Robbie is still under development and runs several
Disallow: /
User-agent: Sites are visited in the order in which references
Disallow: /
User-agent: The robot runs once a month, more or
Disallow: /
User-agent: Visits sites in random order.
Disallow: /
User-agent: accordingly.
Disallow: /
User-agent: and de-htmlized text. Also support server-side and
Disallow: /
User-agent: and detects and converts several different Vietnamese
Disallow: /
User-agent: and index and will not generally index pages with
Disallow: /
User-agent: and visits sites in a random order.
Disallow: /
User-agent: any two-minute period.
Disallow: /
User-agent: are found, but no host is visited more than once in
Disallow: /
User-agent: by our own search engine. This robot auto-detect the
Disallow: /
User-agent: character encodings.
Disallow: /
User-agent: client-side IMAGEMAP.
Disallow: /
User-agent: directory entries (http://www.abcdatos.com), checking
Disallow: /
User-agent: error responses or unable to connect, it repeats
Disallow: /
User-agent: every two minutes. It uses a subject matter relevance
Disallow: /
User-agent: followed, and the page that they refer
Disallow: /
User-agent: for the NetScoop search engine.
Disallow: /
User-agent: from them.
Disallow: /
User-agent: group of keywords.
Disallow: /
User-agent: in our database. These links are
Disallow: /
User-agent: include: date, url, title, total words, title, size
Disallow: /
User-agent: is downloaded to get some statistics
Disallow: /
User-agent: language (french, english & spanish) used in the HTML
Disallow: /
User-agent: less, and visits the first 10 pages
Disallow: /
User-agent: list of links from the search engines
Disallow: /
User-agent: listed in every search engine, for a
Disallow: /
User-agent: no Vietnam related content. Uses Unicode internally,
Disallow: /
User-agent: of the page that was registered to the music
Disallow: /
User-agent: ongoing basis.
Disallow: /
User-agent: page. Each database record generated by this robot
Disallow: /
User-agent: provided in the PlanetSearch service, which is operated by
Disallow: /
User-agent: pruning algorithm to determine what pages to crawl
Disallow: /
User-agent: randomly, but will not visit a site more than once
Disallow: /
User-agent: sites and tests for the integrity of all hyperlinks
Disallow: /
User-agent: specialty searching service.
Disallow: /
User-agent: specific WWW index for VietGATE
Disallow: /
User-agent: specified set of documents and update a database
Disallow: /
User-agent: temporary situation.
Disallow: /
User-agent: that Japanese collage students want to visit.
Disallow: /
User-agent: the Philips Multimedia Center. The robots runs on an
Disallow: /
User-agent: times a day, but usually only for ten minutes or so.
Disallow: /
User-agent: to external sites.
Disallow: /
User-agent: verification some hours later, verifiying if that was a
Disallow: /
User-agent: webpages for search engine called Fifi.
Disallow: /
User-agent: database for the WWW search engine "Verno".
Disallow: /
User-agent: data gathered in one minute from numerous web sites, or from
Disallow: /
User-agent: multiple web requests, and answers them by combining disparate
Disallow: /
User-agent: search engine, CLINKS.
Disallow: /
User-agent: the robots cache. Currently the only user is me.
Disallow: /
User-agent: which will be retrieved via an experimental cross-language
Disallow: /
User-agent: Wanderers, Brokers, and Bots" by the robot's author.
Disallow: /
User-agent: an earlier project at the University of Washington.
Disallow: /
User-agent: and training.
Disallow: /
User-agent: at the Faculty of Engineering, Tokushima University, Japan.,
Disallow: /
User-agent: evolved into a new creature. It still uses some support code
Disallow: /
User-agent: first published in the book "Internet Agents: Spiders,
Disallow: /
User-agent: from Harvest.
Disallow: /
User-agent: music center on November 4, 1997.
Disallow: /
User-agent: since Dec. 1996.
Disallow: /
User-agent: the OLLA system, which is a prototype system, developed
Disallow: /
User-agent: under DARPA funding, to support computer-based education
Disallow: /
User-agent: we began the creation of this robot.
Disallow: /
User-agent: we needed an automated tool. That's why
Disallow: /
User-agent: working in the directory maintenance.
Disallow: /
User-agent: (part of AOL).
Disallow: /
User-agent: (part of WorldLight Network).
Disallow: /
User-agent: The robot runs constantly, and visits sites in a random order.
Disallow: /
User-agent: The robot runs random day, and visits sites in a random order.
Disallow: /
User-agent: The robot runs weekly, and visits sites in a random order
Disallow: /
User-agent: at Nagoya University in 1998.
Disallow: /
User-agent: at the Tokyo in 1997
Disallow: /
User-agent: for the Entireweb.com search service operated by WorldLight.com
Disallow: /
User-agent: for the Yappo search service by k,osawa
Disallow: /
User-agent: for the dpsindia/lawvistas.com search service .
Disallow: /
User-agent: Funded by NEMOnline.
Disallow: /
User-agent: .hr domain.
Disallow: /
User-agent: Croatian web servers in December 1996.
Disallow: /
User-agent: GoLightWay.
Disallow: /
User-agent: It was done as result of desperate need for central index of
Disallow: /
User-agent: It will be used as a post-processing tool on documents created
Disallow: /
User-agent: Perspective Projects Group in 1996
Disallow: /
User-agent: Plumtree Server to index documents on the World Wide Web.
Disallow: /
User-agent: The TitIn is used to index all titles of Web server in
Disallow: /
User-agent: links in HTML documents.
Disallow: /
User-agent: organize, search & navigate Intranet sites and to validate
Disallow: /
User-agent: web pages which are then parsed and feed to the Knowledge Base. The
Disallow: /
User-agent: It mainly gathers pages written in Japanese.
Disallow: /
User-agent: search service operated by NTT Communications Corporation.
Disallow: /
User-agent: client may run this software one or few times every day, manually or
Disallow: /
User-agent: on user's frequently clicked pages in the past 31 days.
Disallow: /
User-agent: specified time.
Disallow: /
User-agent: Ottawa, Canada in 1996.
Disallow: /
User-agent: If an URL is to a page of interest (via CGI), then we access the
Disallow: /
User-agent: In order to not waste internet bandwidth with yet another crawler,
Disallow: /
User-agent: an advanced method for indexing the WWW documents. Uses libwww-perl
Disallow: /
User-agent: any other pages.
Disallow: /
User-agent: database, and copy document trees. Our primary goal is to develop
Disallow: /
User-agent: image files for watermarks but more focused on CGI Urls.
Disallow: /
User-agent: page to get the image URLs from it, but we do not crawl to
Disallow: /
User-agent: to provide us with a list of specific CGI URLs of interest to us.
Disallow: /
User-agent: we have contracted with one of the major crawlers/seach engines
Disallow: /
User-agent: It is an automated page-fetching engine. FetchRover can be
Disallow: /
User-agent: Its database can use any ODBC compliant database server, including
Disallow: /
User-agent: Microsoft Access, Oracle, Sybase SQL Server, FoxPro, etc.
Disallow: /
User-agent: product sold by Engineeering Software, Inc.)
Disallow: /
User-agent: used stand-alone or as the front-end to a full-featured Spider.
Disallow: /
User-agent: - gathering: gather data of original standerd TAG for Puu contains the
Disallow: /
User-agent: - maintenance: link validation
Disallow: /
User-agent: AKA FocusedCrawler
Disallow: /
User-agent: EuroSeek service.
Disallow: /
User-agent: Focused carwling on specific topic.
Disallow: /
User-agent: IDS (Information Discovery Service) and has gone retail.
Disallow: /
User-agent: Information gathering.
Disallow: /
User-agent: It runs when requested.
Disallow: /
User-agent: It supports the proprietary exclusion "Frequency: ??????????" in the
Disallow: /
User-agent: It works non-interactively, and can retrieve HTML pages and FTP
Disallow: /
User-agent: Mar 1997: crawl again;
Disallow: /
User-agent: More features are being added.
Disallow: /
User-agent: Product for selling.
Disallow: /
User-agent: Puu robot is used to gater data from registered site in Search Engin
Disallow: /
User-agent: RobotUserAgent may changed by the user.
Disallow: /
User-agent: The Pentone Group, Inc.
Disallow: /
User-agent: The product is constatnly under development.
Disallow: /
User-agent: This robot patorols based registered sites in Search Engin "straight FLASH!!"
Disallow: /
User-agent: Uses gammaFetcherServer
Disallow: /
User-agent: Web-related sites or products.
Disallow: /
User-agent: Wget is a utility for retrieving files using HTTP and FTP protocols.
Disallow: /
User-agent: clients.
Disallow: /
User-agent: end user or archive maintainer.
Disallow: /
User-agent: engine for the Russian republic of Udmurtia.
Disallow: /
User-agent: gulliver.northernlight.com
Disallow: /
User-agent: indicating number of milliseconds to delay between document requests. This
Disallow: /
User-agent: is called VDRF(tm) or Variable Document Retrieval Frequency. Note that
Disallow: /
User-agent: pages.
Disallow: /
User-agent: robots.txt file. Question marks represent an integer
Disallow: /
User-agent: sites, or for traversing the Web gathering data. It is run by the
Disallow: /
User-agent: supplied by clients. Designed to assist with editorial updates of
Disallow: /
User-agent: trees recursively. It can be used for mirroring Web pages and FTP
Disallow: /
User-agent: users can re-define the useragent name.
Disallow: /
User-agent: web pages for indexing and subsequent searching of the index.
Disallow: /
User-agent: "straight FLASH!!" for building anouncement page of state of renewal of
Disallow: /
User-agent: (1995-97) under supervision of Prof. Rik Belew
Disallow: /
User-agent: (JPG, MP3, MPG, etc)
Disallow: /
User-agent: (Travel and Tourist Info) and //www.golightway.com (Christian Sites).
Disallow: /
User-agent: (currently) not followed
Disallow: /
User-agent: (http://people.freenet.de/Muninn/)
Disallow: /
User-agent: *.worldlight.com
Disallow: /
User-agent:
Disallow: /
User-agent: AllThatNet search service operated by All That Net. The robot runs weekly,
Disallow: /
User-agent: Ask Jeeves in 2000.
Disallow: /
User-agent: Collective can wonder the web for days if required.
Disallow: /
User-agent: Contact Victoria Real for more information.
Disallow: /
User-agent: Directory.
Disallow: /
User-agent: Dridus' Web Cataloging Project, which is intended to catalog domains and
Disallow: /
User-agent: Entireweb.com, that was developed in Halmstad, Sweden during 1998-2000.
Disallow: /
User-agent: Frank of the Institut für Neue Medien, respectively.
Disallow: /
User-agent: Geneva
Disallow: /
User-agent: HTML validation robot run using a web page interface.
Disallow: /
User-agent: Have you ever wanted to have the email addresses of as many companys that
Disallow: /
User-agent: In full service April 1998.
Disallow: /
User-agent: In order to not waste internet bandwidth with yet
Disallow: /
User-agent: Indexing Search and Rescue sites.
Disallow: /
User-agent: Informatics Engineering Department, Institute of Technology Bandung, Indonesia.
Disallow: /
User-agent: Internet Programming book by Jamsa/Cope
Disallow: /
User-agent: It
Disallow: /
User-agent: It can parallel process hundreds of URLS's at a time. It runs on a sporadic basis
Disallow: /
User-agent: It takes four computers to run Raven Search. Scalable in sets of four.
Disallow: /
User-agent: It will support ROBOTS.TXT soon.
Disallow: /
User-agent: Its a none-comercial one-woman-project of Birgit Bachmann
Disallow: /
User-agent: Jan. 1998, WebQuest will run from time to time. Since then, it will run
Disallow: /
User-agent: Knowledge Base classifies the sites into any of hundreds of categories
Disallow: /
User-agent: Laboratories, NEC Corporation. Current robot (Version 1.0) is based
Disallow: /
User-agent: Link Validation, Load Time, HTML Validation and much much more.
Disallow: /
User-agent: Ltd. in Korea.
Disallow: /
User-agent: Medibot software.
Disallow: /
User-agent: Playsback slide show format of one text paragraph plus image from each site.
Disallow: /
User-agent: Plus it's free, you don't have to do any leg work just run the program and
Disallow: /
User-agent: Research Lab.
Disallow: /
User-agent: Robot runs everyday.
Disallow: /
User-agent: Runs constantly and visits site no faster than once every 90 seconds.
Disallow: /
User-agent: School of computing National University Of Singapore
Disallow: /
User-agent: Search Engine" (http://letsfinditnow.com). The robot runs every 30 days.
Disallow: /
User-agent: Search Engines. The robota are run intermittently and perform nearly
Disallow: /
User-agent: Seeks out cooking and recipe pages.
Disallow: /
User-agent: Several options exist to control whether sites are discovered and/or
Disallow: /
User-agent: So what?
Disallow: /
User-agent: State University.
Disallow: /
User-agent: The crawler and image database were written by Sevo Stille and Peter
Disallow: /
User-agent: The robot may become part of a commercial service (at which time it may be
Disallow: /
User-agent: The robot runs at irregular intervals and will only pull a start page and
Disallow: /
User-agent: The robot runs every day, and visits sites in a random order.
Disallow: /
User-agent: The robot will support the Robot Exclusion Standard by early December, 1996.
Disallow: /
User-agent: Those of you who may use this type of robot will know exactly what you can
Disallow: /
User-agent: TopicLink.com
Disallow: /
User-agent: URL is to an image, we may read the image, but we do not crawl to any other
Disallow: /
User-agent: URLs. If an URL is to a page of interest (ususally due to CGI), then we
Disallow: /
User-agent: University of Marseille in 1995-1996
Disallow: /
User-agent: University of Munich (LMU), started in 2000.
Disallow: /
User-agent: Victoria is used to monitor changes in W3 documents,
Disallow: /
User-agent: We send reports of bad ones to webmasters.
Disallow: /
User-agent: With all found url?s guaranteed to have your search terms.
Disallow: /
User-agent: Your a international distributer of "dried fruit" and you boss has told you
Disallow: /
User-agent: a database of pages relating to the Perl programming language.
Disallow: /
User-agent: a html file for your to view at any time even before the program has finished.
Disallow: /
User-agent: a portuguese-specific search engine. Now, we are developping new
Disallow: /
User-agent: a webpage, testing the links found on the page, evaluating your server
Disallow: /
User-agent: about the .au zone's web content.
Disallow: /
User-agent: access the page to get the image URLs from it, but we do not crawl to any
Disallow: /
User-agent: agent.
Disallow: /
User-agent: all-source procurement spider.
Disallow: /
User-agent: an australian ( .au ) zone search engine and make it commercially
Disallow: /
User-agent: analysed. AURESYS can found new server by IP incremental. It generate
Disallow: /
User-agent: and Intranet. It is based on SQL database and supports numerous
Disallow: /
User-agent: and a product for other sites. Jan '97 -- First data results published by
Disallow: /
User-agent: and address's etc from which I hope to build more accurate stats
Disallow: /
User-agent: and its performance.
Disallow: /
User-agent: and retrieving modules of simmany.
Disallow: /
User-agent: and tries to analyze the abundance of the logo metaphor in WWW
Disallow: /
User-agent: and visits sites in a random order.
Disallow: /
User-agent: another crawler, we have contracted with one of the major crawlers/seach
Disallow: /
User-agent: any one webserver.
Disallow: /
User-agent: around sites in japan(JP domain).
Disallow: /
User-agent: as UDMSearch) is an advanced search solution for large-scale websites
Disallow: /
User-agent: as freeware.
Disallow: /
User-agent: as testing continues. It is really several programs running concurrently.
Disallow: /
User-agent: available for advertising to cover the costs which are starting
Disallow: /
User-agent: based on the vocabulary used. Currently used by: //www.travel-finder.com
Disallow: /
User-agent: be very configurable.
Disallow: /
User-agent: been acquired from Tomi Officine Multimediali srl and it is next to
Disallow: /
User-agent: before. Has the penny droped yet, no well now you have the opertunity to
Disallow: /
User-agent: began in january 2004.
Disallow: /
User-agent: beta test. Seeking approval for public release.
Disallow: /
User-agent: both intranet and internet based.
Disallow: /
User-agent: broken or redirected links. Checks all off-site links using HEAD
Disallow: /
User-agent: builds an graph of its visit path.
Disallow: /
User-agent: but has taken on many new and exciting roles.
Disallow: /
User-agent: by Victoria Real Ltd. (voice: +44 [0]1273 774469,
Disallow: /
User-agent: characters (in URL and other places), and we just _had_ to have something up and going the next day...
Disallow: /
User-agent: classified fully automatically, full manually or somewhere in between.
Disallow: /
User-agent: collector, thus the name e-collector.
Disallow: /
User-agent: collects Web statistics for the Direct Hit Search Engine (available at
Disallow: /
User-agent: commerical Portal Juice search engine services.
Disallow: /
User-agent: continiously.
Disallow: /
User-agent: corporate design.
Disallow: /
User-agent: crawling the web and putting together a good search engine with very little
Disallow: /
User-agent: crawling.
Disallow: /
User-agent: current exhibitions.
Disallow: /
User-agent: customers can add to the
Disallow: /
User-agent: customized robot in various projects of COSMO Information & Communication Co.,
Disallow: /
User-agent: daily(for few hours and very slowly).
Disallow: /
User-agent: database engine and web server . Currently in Japanese.
Disallow: /
User-agent: database for MagPortal.com.
Disallow: /
User-agent: designed to crawl wml-pages. html is indexed, but html-links are
Disallow: /
User-agent: development - it runs in random intervals and visits site in a priority
Disallow: /
User-agent: directory of the german search engine Suchmaschine21.
Disallow: /
User-agent: distributed information retrieval
Disallow: /
User-agent: distributed over several hours. Since the robot does not actually
Disallow: /
User-agent: do with information, first don't spam with it, for those still not sure
Disallow: /
User-agent: domains.
Disallow: /
User-agent: driven order (.de/.ch/.at first, root and robots.txt first)
Disallow: /
User-agent: emulation community. Primary focus is information on emulation and
Disallow: /
User-agent: encoding (iso-8859-9 or windows-1254)
Disallow: /
User-agent: engine and goes out weekly.
Disallow: /
User-agent: engine.
Disallow: /
User-agent: engines to provide us with a list of specific URLs of interest to us. If an
Disallow: /
User-agent: especially old arcade machines. Primarily english sites will be indexed and
Disallow: /
User-agent: every month initially, and soon every week bandwidth and cpu allowing.
Disallow: /
User-agent: experimental crawls were done under various user agent
Disallow: /
User-agent: fax: +44 [0]1273 779960 email: victoria@pavilion.co.uk).
Disallow: /
User-agent: features.
Disallow: /
User-agent: filter parameters are met, downloads one picture and a paragraph of test.
Disallow: /
User-agent: find out who they are with an internet address and a person to contact in
Disallow: /
User-agent: follow links (aside from those returned from the major search engines
Disallow: /
User-agent: for Hometown Singles.
Disallow: /
User-agent: for a easy to use link checking program.
Disallow: /
User-agent: for building the BoxSea search engine indices.
Disallow: /
User-agent: for luxembourgish sites. It only indexes .lu domains and luxembourgish
Disallow: /
User-agent: format
Disallow: /
User-agent: formats
Disallow: /
User-agent: functionalities that cannot be found in any other on-line search engines.
Disallow: /
User-agent: gathering and indexing hypertextual, multimedia and executable file
Disallow: /
User-agent: gathers information from several sources: HTTP, Databases or filesystem. At
Disallow: /
User-agent: hardware resources.
Disallow: /
User-agent: high-quality business information.
Disallow: /
User-agent: http://search.falconsoft.com/
Disallow: /
User-agent: http://www.cydral.com/)
Disallow: /
User-agent: identical functions.
Disallow: /
User-agent: if the project warrants further development, i will turn it into
Disallow: /
User-agent: if you rise sales by 10% then he will bye you a new car (Wish i had a boss
Disallow: /
User-agent: in a database to work with. i hope to run a refresh of the crawl
Disallow: /
User-agent: including an audio player, etc.
Disallow: /
User-agent: index capabilities.
Disallow: /
User-agent: index of (user specified) lists of URLs. havIndex does not crawl -
Disallow: /
User-agent: indexed. havIndex does (optionally) save urls parsed from indexed
Disallow: /
User-agent: information in a defined order.
Disallow: /
User-agent: information of the sites registered my Search Engin.
Disallow: /
User-agent: intranet/small domain internet servers
Disallow: /
User-agent: is available at the Portuguese search engine Cusco http://www.cusco.pt/.
Disallow: /
User-agent: it finds for your search terms to ensure those terms are present, any positive urls are added to
Disallow: /
User-agent: its associated /.*logo\.gif/i (if any). It will be terminated once a
Disallow: /
User-agent: just an example.
Disallow: /
User-agent: language for the database of the Fireball search service.
Disallow: /
User-agent: like that), well anyway there are thousands of shops distributers ect, that
Disallow: /
User-agent: links from a page end then indexes the first and stores the latter for further
Disallow: /
User-agent: living in Hamburg, Germany.
Disallow: /
User-agent: locally overriding the security implemented
Disallow: /
User-agent: maintaining the database for the the HamRad search engine.
Disallow: /
User-agent: management tool to validate that HTTP links on a page are functional and to
Disallow: /
User-agent: march 1st, 1998 and after nine days had 70mb of compressed ascii
Disallow: /
User-agent: mopilot.com, a search engine for mobile contents; it is specially
Disallow: /
User-agent: names such as NutchCVS(boxsea)
Disallow: /
User-agent: news filtering for Lotus Notes in 1995.
Disallow: /
User-agent: number of links to follow specified by the user.
Disallow: /
User-agent: of IP address.
Disallow: /
User-agent: offline browsing.
Disallow: /
User-agent: on a list of accumulated visitor requests
Disallow: /
User-agent: on the prototype and has more functions.
Disallow: /
User-agent: online search engines and online databases, it will ignore web pages
Disallow: /
User-agent: only if they have their own domain. Sites are added manually on based on
Disallow: /
User-agent: originally called Sven.
Disallow: /
User-agent: other pages.
Disallow: /
User-agent: over republic of Udmurtia http://search.udm.net
Disallow: /
User-agent: pages are looked at for links. Pages are visited randomly to limit impact on
Disallow: /
User-agent: per day should be quite small, and these hits should be randomly
Disallow: /
User-agent: periodically, marking the ones that return errors for review.
Disallow: /
User-agent: produce various analysis reports to assist in managing a site.
Disallow: /
User-agent: products. The simmini is the Web products that make use of the indexing
Disallow: /
User-agent: program
Disallow: /
User-agent: project at the Technical University of Berlin in 1996 and was
Disallow: /
User-agent: project to demonstrate our development capabilities and to fill the need of
Disallow: /
User-agent: provide structured information to user.
Disallow: /
User-agent: rather it requires one or more user supplied lists of URLs to be
Disallow: /
User-agent: re-implemented by its developer in 1997 for the present owner.
Disallow: /
User-agent: registered site in "straight FLASH!!".
Disallow: /
User-agent: registered urls in the german language search-engine for kids.
Disallow: /
User-agent: release as service and commercial
Disallow: /
User-agent: requests and does not progress further. Designed to behave well and to
Disallow: /
User-agent: resource finding webspider
Disallow: /
User-agent: results published by Travel-Finder. Oct '96 -- Generalized and announced
Disallow: /
User-agent: robot runs weekly, and visits sites that have a useful korean
Disallow: /
User-agent: runs daily, and visits sites in a random order.
Disallow: /
User-agent: search engine
Disallow: /
User-agent: search engines work.
Disallow: /
User-agent: search engines. It is also implemented in several meta-search engines.
Disallow: /
User-agent: search service operated by NEC Corporation. The robot searches URLs
Disallow: /
User-agent: search service sites which will be in service by early 1998. Until the end of
Disallow: /
User-agent: sell or supply for example "dried fruit", i personnaly don't but this is
Disallow: /
User-agent: servers and calculates average number of current servers status. The robot
Disallow: /
User-agent: services. The robot runs daily and visits predefined sites in a random order.
Disallow: /
User-agent: significant number of samples has been collected.
Disallow: /
User-agent: sit back and watch your potential customers arriving.
Disallow: /
User-agent: site owners on missing links, images resize problems, syntax errors, etc.
Disallow: /
User-agent: sites added to the directory.
Disallow: /
User-agent: sites and index images
Disallow: /
User-agent: sites contained in the AustLII legal links database.
Disallow: /
User-agent: sites which ends with one of the following domains: "no", "se", "dk", "is", "fi"
Disallow: /
User-agent: somebody who search information. The database is structured to be
Disallow: /
User-agent: spider, Mattie was reborn 2002 Jul. 07 Sun. 03:47:29 -0500 GMT (R) as an
Disallow: /
User-agent: statistically
Disallow: /
User-agent: statistics...
Disallow: /
User-agent: submitions after they has been evaluated.
Disallow: /
User-agent: subsumed by some other, existing robot).
Disallow: /
User-agent: such as Lycos), it does not fall victim to the common looping problems.
Disallow: /
User-agent: tags). The indexing process can be started based on starting URL(s) or a range
Disallow: /
User-agent: testing focused crawling strategies.
Disallow: /
User-agent: that are relevant to user queries. Users are notified of any new or
Disallow: /
User-agent: that company just by downloading and running e-collector.
Disallow: /
User-agent: that lie about there content, and dead url's, it can be super strict, it searches each web page
Disallow: /
User-agent: the Verity robot (of that time) was unable to index sites with iso8859
Disallow: /
User-agent: the Walhello search engine
Disallow: /
User-agent: the newscan-online news search service operated by smart information
Disallow: /
User-agent: the robot started crawling from http://www.geko.net.au/ on
Disallow: /
User-agent: the simmany service operated by HNC(Hangul & Computer Co., Ltd.). The
Disallow: /
User-agent: them to a label bureau
Disallow: /
User-agent: there in other countries or the nearest town but have never heard about them
Disallow: /
User-agent: this moment, it's universe is the .pt domain and the information it gathers
Disallow: /
User-agent: to aid in building the database for HamRad Search - The Search Engine for
Disallow: /
User-agent: to give me access to a database of web content ( html / url's )
Disallow: /
User-agent: to help by the manual proof of registered Links.
Disallow: /
User-agent: to investigate the power of a search engine and web crawler
Disallow: /
User-agent: to mount up. --dez (980313 - black friday!)
Disallow: /
User-agent: to set the User-Agent.
Disallow: /
User-agent: updated pages. The robot runs daily, but the number of hits per site
Disallow: /
User-agent: urls (no content).
Disallow: /
User-agent: used to build a database for the void search service, as well as for link
Disallow: /
User-agent: using a customizable keyword algorithm. Only home pages are indexed, but all
Disallow: /
User-agent: validation, etc
Disallow: /
User-agent: validation.
Disallow: /
User-agent: web sites listed in http://dmoz.org/Adult/, in order to build a adult search
Disallow: /
User-agent: web-projects and generates knowledge bases in Javascript or an own
Disallow: /
User-agent: webcrawler, lycos or excite to get URLs, then visits sites. If the user's
Disallow: /
User-agent: what this type of robot will do for you then take this for example:
Disallow: /
User-agent: world's first wap-search engine
Disallow: /
User-agent: www.directhit.com and our partners' sites)
Disallow: /
User-agent: www.webindex.de search service operated by CyberCon. The robot ios under
Disallow: /
User-agent: you could be doing business with but you don't know who they are?, because
Disallow: /
User-agent: Googlebot
Disallow: /
User-agent: Mediapartners-Google
Disallow: /
User-agent: Adsbot-Google
Disallow: /
User-agent: Jeeves
Disallow: /
User-agent: Slurp
Disallow: /
User-agent: Yahoo-MMCrawler
Disallow: /
User-agent: modified-by: stanislav shalunov
Disallow: /
User-agent: modified-by: Nobuya Kubo, Hajime Takano
Disallow: /
User-agent: modified-by: Kerry B. Rogers
Disallow: /
User-agent: modified-by:
Disallow: /
User-agent: modified-by: Istvan Fulop
Disallow: /
User-agent: modified-by: Jef Poskanzer
Disallow: /
User-agent: modified-by: Giorgio Galeotti
Disallow: /
User-agent: modified-by: Dave Finnegan
Disallow: /
User-agent: modified-by: Kazunori Matsumoto
Disallow: /
User-agent: modified-by: toney@nwnet.net
Disallow: /
User-agent: modified-by: Sigfrid Lundberg
Disallow: /
User-agent: modified-by: Arto Sarle
Disallow: /
User-agent: modified-by: Hans de Graaff
Disallow: /
User-agent: modified-by: Jaakko.Hyvatti@www.fi
Disallow: /
User-agent: modified-by: Marc Langheinrich
Disallow: /
User-agent: modified-by: Massimiliano Pucciarelli
Disallow: /
User-agent: modified-by: Michael Newbery
Disallow: /
User-agent: modified-by: Paul Bourke
Disallow: /
User-agent: modified-by: TAMURA Kent
Disallow: /
User-agent: modified-by: Yoshihiko HAYASHI
Disallow: /
User-agent: modified-by: fielding@ics.uci.edu
Disallow: /
User-agent: modified-by: Doug Green
Disallow: /
User-agent: modified-by: "Reiji SUZUKI"
Disallow: /
User-agent: modified-by: 7 Feb 1997
Disallow: /
User-agent: modified-by: A.Y.Kiky Shannon
Disallow: /
User-agent: modified-by: ABCdatos
Disallow: /
User-agent: modified-by: Aimo Pieterse
Disallow: /
User-agent: modified-by: Anders Hedstrom
Disallow: /
User-agent: modified-by: Andrew Daviel
Disallow: /
User-agent: modified-by: Anil Peres-da-Silva
Disallow: /
User-agent: modified-by: Antoine Bajolet
Disallow: /
User-agent: modified-by: Antonio Provenzano
Disallow: /
User-agent: modified-by: Antti.Westerberg@mwd.sci.fi
Disallow: /
User-agent: modified-by: Axel Mueller
Disallow: /
User-agent: modified-by: Benjamin Benson
Disallow: /
User-agent: modified-by: Benjamin Franz
Disallow: /
User-agent: modified-by: Bill Dimm
Disallow: /
User-agent: modified-by: Bob Gray
Disallow: /
User-agent: modified-by: Bob Worthy
Disallow: /
User-agent: modified-by: BombJack mameadm@chaos.dk
Disallow: /
User-agent: modified-by: BoxSeaBot
Disallow: /
User-agent: modified-by: Brian B.
Disallow: /
User-agent: modified-by: Brian MacIntosh
Disallow: /
User-agent: modified-by: Britz Thibaut
Disallow: /
User-agent: modified-by: Bryan Ankielewicz
Disallow: /
User-agent: modified-by: Christopher Walsh and Adam Rutter
Disallow: /
User-agent: modified-by: Dan Ramos
Disallow: /
User-agent: modified-by: Daniel Austin
Disallow: /
User-agent: modified-by: Daniel Doubrovkine
Disallow: /
User-agent: modified-by: Daniel Matuschek
Disallow: /
User-agent: modified-by: Daniel Vilà
Disallow: /
User-agent: modified-by: David Fernández
Disallow: /
User-agent: modified-by: Dean Smart
Disallow: /
User-agent: modified-by: Detlev Kalb
Disallow: /
User-agent: modified-by: Dieter Kneffel, data@wap4.com
Disallow: /
User-agent: modified-by: Dimitri Khaoustov
Disallow: /
User-agent: modified-by: Dmitry Tkatchenko
Disallow: /
User-agent: modified-by: Dobrica Pavlinusic
Disallow: /
User-agent: modified-by: DownLoad Express Inc
Disallow: /
User-agent: modified-by: Fah-Chun Cheong
Disallow: /
User-agent: modified-by: Filipe Costa Clerigo
Disallow: /
User-agent: modified-by: Flavio Tordini
Disallow: /
User-agent: modified-by: Francois Pottier
Disallow: /
User-agent: modified-by: Frank Tore Johansen
Disallow: /
User-agent: modified-by: Geoff Duncan
Disallow: /
User-agent: modified-by: Geoff Hutchison
Disallow: /
User-agent: modified-by: Gregoire Welraeds
Disallow: /
User-agent: modified-by: Guti
Disallow: /
User-agent: modified-by: Hideyuki Ezaki
Disallow: /
User-agent: modified-by: Hiroyuki Shigenaga
Disallow: /
User-agent: modified-by: Hometown Singles
Disallow: /
User-agent: modified-by: Horace A. (Kicker) Vallas
Disallow: /
User-agent: modified-by: Hrvoje Niksic
Disallow: /
User-agent: modified-by: Ian Hicks
Disallow: /
User-agent: modified-by: Ignacio Cruzado Nu.o
Disallow: /
User-agent: modified-by: Ilse
Disallow: /
User-agent: modified-by: Innerprise
Disallow: /
User-agent: modified-by: JD
Disallow: /
User-agent: modified-by: Jeremy DeYoung
Disallow: /
User-agent: modified-by: Jorge Alegre
Disallow: /
User-agent: modified-by: Joseph A. Stanko
Disallow: /
User-agent: modified-by: KO
Disallow: /
User-agent: modified-by: Katsuo Doi
Disallow: /
User-agent: modified-by: Keith Jones
Disallow: /
User-agent: modified-by: Ken Wadland
Disallow: /
User-agent: modified-by: Kenji Kita
Disallow: /
User-agent: modified-by: Kenneth R. Churilla
Disallow: /
User-agent: modified-by: Kim Gam-Jensen
Disallow: /
User-agent: modified-by: Lars Eilebrecht
Disallow: /
User-agent: modified-by: Lawrence R. Hughes, Sr.
Disallow: /
User-agent: modified-by: MPRM Group Limited
Disallow: /
User-agent: modified-by: Mannina Bruno
Disallow: /
User-agent: modified-by: Marc Wils
Disallow: /
User-agent: modified-by: Marius Dahler
Disallow: /
User-agent: modified-by: Mark Otway
Disallow: /
User-agent: modified-by: Marki@baytsp.com
Disallow: /
User-agent: modified-by: Markus Hoevener
Disallow: /
User-agent: modified-by: Marty Anstey
Disallow: /
User-agent: modified-by: Matt
Disallow: /
User-agent: modified-by: Matt McKenzie
Disallow: /
User-agent: modified-by: Matt Weber
Disallow: /
User-agent: modified-by: Michael Goeckel (Michael@cybercon.technopark.gmd.de)
Disallow: /
User-agent: modified-by: Michael Jennings
Disallow: /
User-agent: modified-by: Mihai Preda
Disallow: /
User-agent: modified-by: Mike Blaszczak
Disallow: /
User-agent: modified-by: Mike Mulligan
Disallow: /
User-agent: modified-by: Mike Thompson
Disallow: /
User-agent: modified-by: Mr. Matthias H. Gross
Disallow: /
User-agent: modified-by: Neal Krawetz
Disallow: /
User-agent: modified-by: Neil Mansilla
Disallow: /
User-agent: modified-by: Owen Lydiard
Disallow: /
User-agent: modified-by: Pat Morin
Disallow: /
User-agent: modified-by: Philip Lenir, MerzScope lead developper
Disallow: /
User-agent: modified-by: Raven Group
Disallow: /
User-agent: modified-by: Ross Mellgren
Disallow: /
User-agent: modified-by: Roy Bryant
Disallow: /
User-agent: modified-by: Sander Steffann
Disallow: /
User-agent: modified-by: Sandra Groth
Disallow: /
User-agent: modified-by: Sat, 20 Oct 2001 04:00:00 GMT
Disallow: /
User-agent: modified-by: Sebastien Ailleret
Disallow: /
User-agent: modified-by: Sevo Stille
Disallow: /
User-agent: modified-by: Stefan Fischerlaender
Disallow: /
User-agent: modified-by: Stefan R. Mueller
Disallow: /
User-agent: modified-by: Steve DeJarnett
Disallow: /
User-agent: modified-by: Terry Dexter
Disallow: /
User-agent: modified-by: Thomas Gimon
Disallow: /
User-agent: modified-by: Tom Aman
Disallow: /
User-agent: modified-by: Tom Tanaka
Disallow: /
User-agent: modified-by: Tomoaki Nakashima (naka@kinet.or.jp)
Disallow: /
User-agent: modified-by: TopicLink Spider Team
Disallow: /
User-agent: modified-by: UptimeBot team
Disallow: /
User-agent: modified-by: Yo Okumura
Disallow: /
User-agent: modified-by: Yong Cao
Disallow: /
User-agent: modified-by: Youngsik, Lee
Disallow: /
User-agent: modified-by: asidirop@csi.forth.gr
Disallow: /
User-agent: modified-by: bot@void.be
Disallow: /
User-agent: modified-by: bowen@hotwired.com
Disallow: /
User-agent: modified-by: brian d foy
Disallow: /
User-agent: modified-by: brian@smithrenaud.com
Disallow: /
User-agent: modified-by: brucep@ask.com
Disallow: /
User-agent: modified-by: chris@gestalt.sewanee.edu
Disallow: /
User-agent: modified-by: cydral@cydral.com
Disallow: /
User-agent: modified-by: friedman@cs.washington.edu (Marc Friedman)
Disallow: /
User-agent: modified-by: googlebot@google.com
Disallow: /
User-agent: modified-by: grabber@directhit.com
Disallow: /
User-agent: modified-by: harada@graco.c.u-tokyo.ac.jp
Disallow: /
User-agent: modified-by: jamieson@mit.edu
Disallow: /
User-agent: modified-by: jvandal@9bit.qc.ca
Disallow: /
User-agent: modified-by: maxim@cs.sunysb.edu
Disallow: /
User-agent: modified-by: msnbot@microsoft.com
Disallow: /
User-agent: modified-by: nick@webthing.com
Disallow: /
User-agent: modified-by: noto@isl.ntt.co.jp
Disallow: /
User-agent: modified-by: olly@muscat.co.uk
Disallow: /
User-agent: modified-by: pjspider@portaljuice.com
Disallow: /
User-agent: modified-by: psbot@picsearch.com
Disallow: /
User-agent: modified-by: schwartz@imaginon.com
Disallow: /
User-agent: modified-by: slurp@inktomi.com
Disallow: /
User-agent: modified-by: steves@avs.dec.com
Disallow: /
User-agent: modified-by: support@thesoftwareobjects.com
Disallow: /
User-agent: modified-by: tbray@textuality.com
Disallow: /
User-agent: modified-by: tech@krstarica.com
Disallow: /
User-agent: modified-by: techbot@techaid.net
Disallow: /
User-agent: modified-by: victoria@pavilion.co.uk
Disallow: /
User-agent: modified-by: webmaster@verticrawl.com
Disallow: /
User-agent: modified-by:Aniruddha Choudhury
Disallow: /
User-agent: modified-by:C. Fenijn
Disallow: /
User-agent: modified-by:Dean Smart
Disallow: /
User-agent: modified-by:Jerry Walsh
Disallow: /
User-agent: modified-by:Joseph Reynolds
Disallow: /
User-agent: modified-by:Karen Ng
Disallow: /
User-agent: modified-by:Leung Hok Peng and Dr Hsu Wynne
Disallow: /
User-agent: modified-by:Marcus Andersson
Disallow: /
User-agent: modified-by:Mike Davis
Disallow: /
User-agent: modified-by:Seiji Sasazuka & Takahiro Ohmori
Disallow: /
User-agent: modified-by:TaeYoung Choi
Disallow: /
User-agent: modified-by:Torsten Kaubisch
Disallow: /
User-agent: modified-by:Watanabe Takashi
Disallow: /
User-agent: modified-by:Zoltan Milosevic
Disallow: /
User-agent: modified-by:moget@goo.ne.jp
Disallow: /
User-agent: modified-by:tkac
Disallow: /
User-agent: modified-by:toka@navi.ocn.ne.jp
Disallow: /
User-agent: modified-by:webdocs
Disallow: /
User-agent: modified-date: Thu, 26 Apr 2001 02:55:21 GMT
Disallow: /
User-agent: modified-date: Jan 1, 1997
Disallow: /
User-agent: modified-date: Fri, 11 Apr 1997 19:08:02 GMT
Disallow: /
User-agent: modified-date:
Disallow: /
User-agent: modified-date:
Disallow: /
User-agent: modified-date:
Disallow: /
User-agent: modified-date: Fri, 6 Sep 1996 10:00:00 GMT
Disallow: /
User-agent: modified-date: Sun, 04 May 2003 10:15:00 GMT
Disallow: /
User-agent: modified-date: Wed, 04 Dec 1996 21:30:11 GMT
Disallow: /
User-agent: modified-date: Mon, 07 Jun 2004 14:50:01 GMT
Disallow: /
User-agent: modified-date: July 9, 1997
Disallow: /
User-agent: modified-date: Mon, 2 June 1997 18:00:00 JST
Disallow: /
User-agent: modified-date: Thurs, 15 Aug 1996
Disallow: /
User-agent: modified-date: Wed Jun 26 13:58:04 MET DST 1996
Disallow: /
User-agent: modified-date: 1996-06-25
Disallow: /
User-agent: modified-date: Fri Aug 9 17:06:56 1996.
Disallow: /
User-agent: modified-date: Fri Feb 9 00:11:22 1996.
Disallow: /
User-agent: modified-date: Fri Jan 19 05:08:15 1996.
Disallow: /
User-agent: modified-date: Fri Jan 19 21:50:32 1996.
Disallow: /
User-agent: modified-date: Fri Jun 23 16:30:42 FRE 1995
Disallow: /
User-agent: modified-date: Fri June 28 14:00:00 1996
Disallow: /
User-agent: modified-date: Fri Mar 8 16:03:04 1996
Disallow: /
User-agent: modified-date: Fri Mar 29 20:06:12 1996.
Disallow: /
User-agent: modified-date: Fri May 31 02:10:39 1996.
Disallow: /
User-agent: modified-date: Fri May 5 15:47:55 1995
Disallow: /
User-agent: modified-date: Fri May 5 16:09:18 1995
Disallow: /
User-agent: modified-date: Fri May 5 17:48:48 1995
Disallow: /
User-agent: modified-date: Fri, 16 Nov 2001 08:30:00 GMT
Disallow: /
User-agent: modified-date: June 27 1996.
Disallow: /
User-agent: modified-date: Mon Apr 29 08:52:25 1996.
Disallow: /
User-agent: modified-date: Mon Feb 5 02:49:32 1996.
Disallow: /
User-agent: modified-date: Mon Feb 19 00:28:37 1996.
Disallow: /
User-agent: modified-date: Mon Jan 22 22:09:19 1996.
Disallow: /
User-agent: modified-date: Mon Jul 1 07:30:00 GMT 1996
Disallow: /
User-agent: modified-date: Mon Jun 24 17:20:44 PDT 1996
Disallow: /
User-agent: modified-date: Mon May 6 17:41:29 1996.
Disallow: /
User-agent: modified-date: Mon May 13 03:19:17 1996.
Disallow: /
User-agent: modified-date: Mon May 8 09:31:19 1995
Disallow: /
User-agent: modified-date: Mon Nov 27 21:30:11 1995
Disallow: /
User-agent: modified-date: Sat Apr 27 01:20:15 1996.
Disallow: /
User-agent: modified-date: Sat Jan 6 20:58:44 1996
Disallow: /
User-agent: modified-date: Sat Jan 27 09:21:40 1996.
Disallow: /
User-agent: modified-date: Sat Jan 27 10:31:43 1996.
Disallow: /
User-agent: modified-date: Sat Jan 27 21:02:20 1996.
Disallow: /
User-agent: modified-date: Sat Mar 23 20:12:39 1996.
Disallow: /
User-agent: modified-date: Sat Mar 30 00:55:40 1996.
Disallow: /
User-agent: modified-date: Sat May 6 08:11:58 1995
Disallow: /
User-agent: modified-date: Sun Feb 18 02:02:49 1996.
Disallow: /
User-agent: modified-date: Sun Jul 2 15:27:04 1995
Disallow: /
User-agent: modified-date: Sun May 19 08:13:06 1996.
Disallow: /
User-agent: modified-date: Sun May 28 01:35:48 1995
Disallow: /
User-agent: modified-date: Thu Feb 29 00:39:49 1996.
Disallow: /
User-agent: modified-date: Thu Mar 7 14:21:55 1996.
Disallow: /
User-agent: modified-date: Thu May 18 04:47:02 1995
Disallow: /
User-agent: modified-date: Tue Apr 23 19:23:55 1996.
Disallow: /
User-agent: modified-date: Tue Jan 9 18:55:55 1996
Disallow: /
User-agent: modified-date: Tue Jul 11 09:29:45 GMT 1995
Disallow: /
User-agent: modified-date: Tue Jun 18 19:16:31 1996.
Disallow: /
User-agent: modified-date: Tue Jun 25 07:44:00 1996
Disallow: /
User-agent: modified-date: Tue Jun 25 10:03:36 1996
Disallow: /
User-agent: modified-date: Tue Mar 12 15:52:25 1996.
Disallow: /
User-agent: modified-date: Tue May 16 00:57:42 1995.
Disallow: /
User-agent: modified-date: Tue May 23 17:51:39 1995
Disallow: /
User-agent: modified-date: Tue May 9 15:13:12 1995
Disallow: /
User-agent: modified-date: Tue Oct 3 01:10:26 1995
Disallow: /
User-agent: modified-date: Tue, 25 Jun 96 11:40:07 +1200
Disallow: /
User-agent: modified-date: Tues, 25 Jun 1996
Disallow: /
User-agent: modified-date: Wed Apr 24 13:23:42 1996.
Disallow: /
User-agent: modified-date: Wed Feb 21 02:57:42 1996.
Disallow: /
User-agent: modified-date: Wed Jan 10 08:23:00 1996
Disallow: /
User-agent: modified-date: Wed Jan 10 23:19:08 1996.
Disallow: /
User-agent: modified-date: Wed Jan 10 23:56:22 1996.
Disallow: /
User-agent: modified-date: Wed Jul 26 13:36:32 1995
Disallow: /
User-agent: modified-date: Wed May 29 14:47:01 1996.
Disallow: /
User-agent: modified-date: Wed Nov 15 09:51:59 PST 1995
Disallow: /
User-agent: modified-date: Wed Oct 4 06:54:31 1995
Disallow: /
User-agent: modified-date: Tue Jun 17 09:24:58 EST 1997
Disallow: /
User-agent: modified-date: 04/26/1999
Disallow: /
User-agent: modified-date: 05-08-2001
Disallow: /
User-agent: modified-date: 06/13/1997
Disallow: /
User-agent: modified-date: 1/15/2001
Disallow: /
User-agent: modified-date: 14 March 2004
Disallow: /
User-agent: modified-date: 2000-3-28
Disallow: /
User-agent: modified-date: 22 Jul 1998
Disallow: /
User-agent: modified-date: 25.5.97
Disallow: /
User-agent: modified-date: 6-27-98
Disallow: /
User-agent: modified-date: Dec. 1st, 1996
Disallow: /
User-agent: modified-date: Fri Aug 14 03:37:56 CEST 1998
Disallow: /
User-agent: modified-date: Fri Feb 28 13:57:43 PST 1997
Disallow: /
User-agent: modified-date: Fri Jan 17 15:20:08 EST 2003
Disallow: /
User-agent: modified-date: Fri Mar 13 10:03:32 EST 1998
Disallow: /
User-agent: modified-date: Fri Mar 31 15:03:12 GMT 2000
Disallow: /
User-agent: modified-date: Fri Nov 13 14:08:01 EST 1998
Disallow: /
User-agent: modified-date: Fri, 07 May 1998 17:00:00 GMT
Disallow: /
User-agent: modified-date: Fri, 10 Jan 1997.
Disallow: /
User-agent: modified-date: Fri, 11 May 2001 17:28:52 GMT
Disallow: /
User-agent: modified-date: Fri, 13 March 1997 16:31:00
Disallow: /
User-agent: modified-date: Fri, 13 Sep 2002 00:36:13 GMT
Disallow: /
User-agent: modified-date: Fri, 14 May 2004 19:58:52 GMT
Disallow: /
User-agent: modified-date: Fri, 16 Oct 1998 17:28:52 JST
Disallow: /
User-agent: modified-date: Fri, 17 Apr 1998 21:44:00 GMT
Disallow: /
User-agent: modified-date: Fri, 17 Jan 2001 12:00:00 GMT
Disallow: /
User-agent: modified-date: Fri, 18 Dec 1998 23:37:40 GMT
Disallow: /
User-agent: modified-date: Fri, 18 Jul 1996 12:34:21 GMT
Disallow: /
User-agent: modified-date: Fri, 19 Sep 2003 08:57:52 GMT
Disallow: /
User-agent: modified-date: Fri, 2 Nov 2001 04:55:00 PST
Disallow: /
User-agent: modified-date: Fri, 20 Apr 2001 17:00:00 GMT
Disallow: /
User-agent: modified-date: Fri, 20 Jan 2001 02:22:00 EST
Disallow: /
User-agent: modified-date: Fri, 20 Oct 2000 14:58:40 GMT
Disallow: /
User-agent: modified-date: Fri, 22 Nov 1996 16:45 GMT
Disallow: /
User-agent: modified-date: Fri, 23 Jan 1998 16:09:00 MET
Disallow: /
User-agent: modified-date: Fri, 23 Jul 2004 11:58:00 PST
Disallow: /
User-agent: modified-date: Fri, 23 Jun 2000 14:33:52 MESZ
Disallow: /
User-agent: modified-date: Fri, 24 Nov 2000 00:00:00 GMT
Disallow: /
User-agent: modified-date: Fri, 25 Mar 2000 17:28:52 GMT
Disallow: /
User-agent: modified-date: Fri, 26 Feb 1999 12:00:00 GMT
Disallow: /
User-agent: modified-date: Fri, 26 Jun 1998
Disallow: /
User-agent: modified-date: Fri, 27 Jun 2001 00:53:12 CST
Disallow: /
User-agent: modified-date: Fri, 27 Oct 2000 09:08:06 GMT
Disallow: /
User-agent: modified-date: Fri, 30 Aug 1996 00:00:00 GMT
Disallow: /
User-agent: modified-date: Fri, 30 Jun 2000 19:02:52 JST
Disallow: /
User-agent: modified-date: Fri, 4 Dec 1998 0:0:0 GMT
Disallow: /
User-agent: modified-date: Fri, 5 Dec 1997 12:00:00 GMT
Disallow: /
User-agent: modified-date: Fri, 9 Apr 1999 11:45:00 GMT
Disallow: /
User-agent: modified-date: Friday March 05, 2004
Disallow: /
User-agent: modified-date: July 09, 2000 17:43 GMT
Disallow: /
User-agent: modified-date: July 28, 2000
Disallow: /
User-agent: modified-date: July 30th, 2001
Disallow: /
User-agent: modified-date: June 23, 2003
Disallow: /
User-agent: modified-date: May 11, 2001
Disallow: /
User-agent: modified-date: Mo, 13 Mar 2000 14:00:00 GMT
Disallow: /
User-agent: modified-date: Mon Feb 23 11:26:08 1998
Disallow: /
User-agent: modified-date: Mon Jul 22 1998
Disallow: /
User-agent: modified-date: Mon, 1 Jul 1996 14:30:00 GMT
Disallow: /
User-agent: modified-date: Mon, 11 Aug 1997 00:00:00 GMT
Disallow: /
User-agent: modified-date: Mon, 11 Nov 1996 06:00:44 MET
Disallow: /
User-agent: modified-date: Mon, 12 Jul 1999 17:50:30 GMT
Disallow: /
User-agent: modified-date: Mon, 13 Dec 1999 21:50:32 GMT
Disallow: /
User-agent: modified-date: Mon, 13 Jan 1997 10:41:00 EST
Disallow: /
User-agent: modified-date: Mon, 14 Jan 2002 08:02:23 GMT
Disallow: /
User-agent: modified-date: Mon, 16 Sep 1996 14:08:00 PDT
Disallow: /
User-agent: modified-date: Mon, 20 Oct 1997 16:44:29 GMT
Disallow: /
User-agent: modified-date: Mon, 21 Apr 1997 18:00:00 JST
Disallow: /
User-agent: modified-date: Mon, 21 Jun 1999 14:00:00 GMT
Disallow: /
User-agent: modified-date: Mon, 22 May 2000 12:28:52 GMT
Disallow: /
User-agent: modified-date: Mon, 22 May 2000 15:47:30 GMT
Disallow: /
User-agent: modified-date: Mon, 25 Aug 2003 09:00:00 GMT
Disallow: /
User-agent: modified-date: Mon, 26 May 1997 15:55:02 EEST
Disallow: /
User-agent: modified-date: Mon, 29 Dec 1997 15:30:00 GMT
Disallow: /
User-agent: modified-date: Mon, 30 Nov 1998 08:00:00 GMT
Disallow: /
User-agent: modified-date: Mon, 4 Jan 1999 14:30:00 GMT
Disallow: /
User-agent: modified-date: Mon, 5 Aug 1996 14:35:08 GMT
Disallow: /
User-agent: modified-date: Mon, 6 Jun 2004 08:25 +1 GMT
Disallow: /
User-agent: modified-date: Mon, 6 Sep 1999 10:28:52 GMT
Disallow: /
User-agent: modified-date: Mon, 9 Feb 2004 11:51:10 GMT
Disallow: /
User-agent: modified-date: Monday, 19 July 1999, 13:46:00 PDT
Disallow: /
User-agent: modified-date: Sat, 10 Jul 1999 00:05:40 GMT
Disallow: /
User-agent: modified-date: Sat, 16 Oct 1999 19:40:00 GMT
Disallow: /
User-agent: modified-date: Sat, 17 Aug 1996 12:00:00 GMT
Disallow: /
User-agent: modified-date: Sat, 18 Aug 2001 00:38:52 GMT
Disallow: /
User-agent: modified-date: Sat, 18 Dec 1998 14:26:00 EST
Disallow: /
User-agent: modified-date: Sat, 19 Jun 2004 20:25:00 GMT+1
Disallow: /
User-agent: modified-date: Sat, 19 March 2004 21:19:03 GMT
Disallow: /
User-agent: modified-date: Sat, 2 Nov 1996 00:08:18 GMT
Disallow: /
User-agent: modified-date: Sat, 20 Oct 2001 04:00:00 GMT
Disallow: /
User-agent: modified-date: Sat, 23 May 1998 17:22:00 GMT
Disallow: /
User-agent: modified-date: Sat, 8 Feb 1997 01:10:00 CET
Disallow: /
User-agent: modified-date: Sun, 21 Nov 2001 20:01:19 GMT
Disallow: /
User-agent: modified-date: Sun, 25 Mar 2001 18:49:52 GMT
Disallow: /
User-agent: modified-date: Sun, 27 Jun 1999 09:00:00 GMT
Disallow: /
User-agent: modified-date: Sun, 3 Nov 1996 11:55:00 GMT
Disallow: /
User-agent: modified-date: Sun, 31 Aug 1997 02:28:52 GMT
Disallow: /
User-agent: modified-date: Sun, 6 Apr 1997 10:00:00 GMT
Disallow: /
User-agent: modified-date: Sun, 6 Jun 1999 13:25:33 GMT
Disallow: /
User-agent: modified-date: Sun, 8 Apr 2001 13:06:54 CET
Disallow: /
User-agent: modified-date: Thu Dec 10 14:01:13 MET 1998
Disallow: /
User-agent: modified-date: Thu Jun 3 16:36:47 CEST 2004
Disallow: /
User-agent: modified-date: Thu Mar 20 19:09:56 JST 1997
Disallow: /
User-agent: modified-date: Thu Mar 29 21:00:07 PST 2001
Disallow: /
User-agent: modified-date: Thu Sep 6 17:50:32 BST 2001
Disallow: /
User-agent: modified-date: Thu Sep 19 18:01:46 MET DST 1996
Disallow: /
User-agent: modified-date: Thu, 03 Apr 1997 21:49:50 EST
Disallow: /
User-agent: modified-date: Thu, 03 Jan 2000 16:00:00 GMT
Disallow: /
User-agent: modified-date: Thu, 09 Jan 2001 17:28:52 GMT
Disallow: /
User-agent: modified-date: Thu, 12 Dec 1996 16:06:42 MET
Disallow: /
User-agent: modified-date: Thu, 13 Dec 2001 23:28:23 EET
Disallow: /
User-agent: modified-date: Thu, 17 Dec 1998
Disallow: /
User-agent: modified-date: Thu, 19 Sep 1996 07:02:26 GMT
Disallow: /
User-agent: modified-date: Thu, 20 Jul 2000 22:38:00 GMT
Disallow: /
User-agent: modified-date: Thu, 21 Nov 1996 20:30 GMT
Disallow: /
User-agent: modified-date: Thu, 25 Jul 1996 16:00:52 PDT
Disallow: /
User-agent: modified-date: Thu, 25 Mar 1999 15:00:00 GMT
Disallow: /
User-agent: modified-date: Thu, 29 May 2003 01:00:00 GMT
Disallow: /
User-agent: modified-date: Thu, 30 Oct 1997
Disallow: /
User-agent: modified-date: Thu, 9 Jan 1997 22:57:28 EST
Disallow: /
User-agent: modified-date: Thu, Jan 23 1997 23:09:00 GMT
Disallow: /
User-agent: modified-date: Thus, 22 Dec 1999
Disallow: /
User-agent: modified-date: Tue Apr 7 16:25:05 MET DST 1998
Disallow: /
User-agent: modified-date: Tue Jul 13 03:50:06 GMT 1999
Disallow: /
User-agent: modified-date: Tue Mar 28 11:30:09 CEST 2000
Disallow: /
User-agent: modified-date: Tue, 04 Mar 1997 16:11:40 GMT
Disallow: /
User-agent: modified-date: Tue, 11 Aug 1998 17:28:52 GMT
Disallow: /
User-agent: modified-date: Tue, 11 May 2004 17:45:00 CET
Disallow: /
User-agent: modified-date: Tue, 17 Jun 2004, 11:50:30 GMT
Disallow: /
User-agent: modified-date: Tue, 20 Aug 1996 15:45:11
Disallow: /
User-agent: modified-date: Tue, 21 Aug 2001 10:55:38 CEST 2001
Disallow: /
User-agent: modified-date: Tue, 21 May 1997 17:11:00 GMT
Disallow: /
User-agent: modified-date: Tue, 22 Dec 1998 00:22:00 PST
Disallow: /
User-agent: modified-date: Tue, 27 Jun 2000, 11:17:50 EDT
Disallow: /
User-agent: modified-date: Tue, 27 June 2000
Disallow: /
User-agent: modified-date: Tue, 28 Aug 2001 21:40:47 GMT
Disallow: /
User-agent: modified-date: Tue, 28 Mar 2000 16:00:00 GMT
Disallow: /
User-agent: modified-date: Tue, 3 Mar 1999 08:15:00 PST
Disallow: /
User-agent: modified-date: Tue, 31 Mar 1998 01:02:00 GMT
Disallow: /
User-agent: modified-date: Tue, 4 Mar 1997 20:00:00 GMT
Disallow: /
User-agent: modified-date: Tue, 6 Mar 2001 02:15:00 GMT
Disallow: /
User-agent: modified-date: Tuesday, 18 Feb 1997 06:02:47 GMT
Disallow: /
User-agent: modified-date: Wed Jun 23 17:00:00 EST 1999
Disallow: /
User-agent: modified-date: Wed, 5 Feb 1997 19:00:00 GMT
Disallow: /
User-agent: modified-date: Wed, 05 May 1998
Disallow: /
User-agent: modified-date: Wed, 08 Oct 1997 00:09:52 GMT
Disallow: /
User-agent: modified-date: Wed, 09 Jun 1999 10:43:18 GMT
Disallow: /
User-agent: modified-date: Wed, 10 Oct 1996 13:15:00 GMT
Disallow: /
User-agent: modified-date: Wed, 11 Jun 1997 03:58:40 GMT
Disallow: /
User-agent: modified-date: Wed, 11 Sep 1997 02:00:00 GMT
Disallow: /
User-agent: modified-date: Wed, 12 Sept 2001
Disallow: /
User-agent: modified-date: Wed, 13 Jan 1999 17:18:59 GMT
Disallow: /
User-agent: modified-date: Wed, 13 May 1998 17:28:52 GMT
Disallow: /
User-agent: modified-date: Wed, 16 Apr 1997 20:50:00 GMT
Disallow: /
User-agent: modified-date: Wed, 17 Jan 2001 11:52:00 EST
Disallow: /
User-agent: modified-date: Wed, 21 Apr 1999 16:00:00 GMT
Disallow: /
User-agent: modified-date: Wed, 21 Feb 2001 03:36:39 GMT
Disallow: /
User-agent: modified-date: Wed, 21 Jan 2001 12:16:00 GMT
Disallow: /
User-agent: modified-date: Wed, 22 Apr 1998
Disallow: /
User-agent: modified-date: Wed, 22 Mar 2000 14:10:49 GMT
Disallow: /
User-agent: modified-date: Wed, 28 Jan 1998 17:32:52 GMT
Disallow: /
User-agent: modified-date: Wed, 3 Jul 1996 15:30:00 +0100
Disallow: /
User-agent: modified-date: Wed, 3 Jun 1998 12:00:00 GMT
Disallow: /
User-agent: modified-date: Wed, 5 Mar 1997 17:35:16 CST
Disallow: /
User-agent: modified-date: Wed, 5 Sep 2001 19:21:00 GMT
Disallow: /
User-agent: modified-date: Weekly
Disallow: /
User-agent: modified-date: mon, 27 Jul 2006 17:28:52 GMT
Disallow: /
User-agent: modified-date:08/08/1999
Disallow: /
User-agent: modified-date:1 july 2000
Disallow: /
User-agent: modified-date:11/21/96
Disallow: /
User-agent: modified-date:6 th November 2000
Disallow: /
User-agent: modified-date:August, 03, 2000
Disallow: /
User-agent: modified-date:Fri, 20 Mar 1998 18:34 JST
Disallow: /
User-agent: modified-date:Fri, 21 Jan 2000 10:15:49 GMT
Disallow: /
User-agent: modified-date:Fri, 21 Oct 1999 17:28:52 GMT
Disallow: /
User-agent: modified-date:Fri, 3 September 1999 17:00:00 PDT
Disallow: /
User-agent: modified-date:Mon 17 Jan 2000 10:00:00 EST
Disallow: /
User-agent: modified-date:Mon, 17 July 2000 11:05:03 GMT
Disallow: /
User-agent: modified-date:Mon, 27 Nov 2000 12:26:00 GMT
Disallow: /
User-agent: modified-date:Mon,25 Jan 2000 15:25:30 GMT
Disallow: /
User-agent: modified-date:Nov. 15, 2000
Disallow: /
User-agent: modified-date:September,10,1999 17:28 GMT
Disallow: /
User-agent: modified-date:Sun Mar 28 14:39:38
Disallow: /
User-agent: modified-date:Thu, 30 Mar 2000 18:40:37 GMT
Disallow: /
User-agent: modified-date:Thu, 6 Dec 2001 01:55:00 GMT
Disallow: /
User-agent: modified-date:Thursday, 24 Apr 1997 20:00:00 GMT
Disallow: /
User-agent: modified-date:Tue, 3 Nov 1998 10:09:02 EST
Disallow: /
User-agent: modified-date:Tue, 30 Dec 1997 09:27:20 GMT
Disallow: /
User-agent: modified-date:Tue. 10 Nov 1998 20:00:00 GMT
Disallow: /
User-agent: msnbot
Disallow: /
User-agent: psbot
Disallow: /
User-agent: robot-availability: binary
Disallow: /
User-agent: robot-availability: none
Disallow: /
User-agent: robot-availability:
Disallow: /
User-agent: robot-availability:
Disallow: /
User-agent: robot-availability: none
Disallow: /
User-agent: robot-availability: source
Disallow: /
User-agent: robot-availability: none
Disallow: /
User-agent: robot-availability: binary
Disallow: /
User-agent: robot-availability: data
Disallow: /
User-agent: robot-availability: none
Disallow: /
User-agent: robot-availability: none (at the moment)
Disallow: /
User-agent: robot-availability: at the moment, none...source when developed.
Disallow: /
User-agent: robot-availability: free service and more extensive commercial service
Disallow: /
User-agent: robot-availability: - none
Disallow: /
User-agent: robot-availability: 24/7
Disallow: /
User-agent: robot-availability: Binary
Disallow: /
User-agent: robot-availability: Commercial as part of search engine package
Disallow: /
User-agent: robot-availability: None
Disallow: /
User-agent: robot-availability: None, yet
Disallow: /
User-agent: robot-availability: Protected by Password
Disallow: /
User-agent: robot-availability: available now
Disallow: /
User-agent: robot-availability: binary
Disallow: /
User-agent: robot-availability: binary
Disallow: /
User-agent: robot-availability: binary as bundled software
Disallow: /
User-agent: robot-availability: binary, source
Disallow: /
User-agent: robot-availability: bulk data gathered by robot available
Disallow: /
User-agent: robot-availability: data
Disallow: /
User-agent: robot-availability: data, binary, source
Disallow: /
User-agent: robot-availability: data, source on request
Disallow: /
User-agent: robot-availability: free and commercial services
Disallow: /
User-agent: robot-availability: no
Disallow: /
User-agent: robot-availability: none
Disallow: /
User-agent: robot-availability: service
Disallow: /
User-agent: robot-availability: source
Disallow: /
User-agent: robot-availability: source (GPL), mail me for customization
Disallow: /
User-agent: robot-availability: source (commercial)
Disallow: /
User-agent: robot-availability: source, binary
Disallow: /
User-agent: robot-availability: source, binary, data
Disallow: /
User-agent: robot-availability: source, data
Disallow: /
User-agent: robot-availability: source;data
Disallow: /
User-agent: robot-availability: via web page
Disallow: /
User-agent: robot-availability:Executible
Disallow: /
User-agent: robot-availability:None
Disallow: /
User-agent: robot-availability:Open Source
Disallow: /
User-agent: robot-availability:Program is shareware
Disallow: /
User-agent: robot-availability:binary
Disallow: /
User-agent: robot-availability:binary&source
Disallow: /
User-agent: robot-availability:data
Disallow: /
User-agent: robot-availability:none
Disallow: /
User-agent: robot-availability:none
Disallow: /
User-agent: robot-availability:not yet
Disallow: /
User-agent: robot-availability:source
Disallow: /
User-agent: robot-availability:source, binary
Disallow: /
User-agent: robot-availability:source;data
Disallow: /
User-agent: robot-cover-url: http://www.openfind.com.tw/
Disallow: /
User-agent: robot-cover-url: http://netplaza.biglobe.or.jp/
Disallow: /
User-agent: robot-cover-url: http://www.pentone.com
Disallow: /
User-agent: robot-cover-url:
Disallow: /
User-agent: robot-cover-url:
Disallow: /
User-agent: robot-cover-url:
Disallow: /
User-agent: robot-cover-url: http://webdew.rnet.or.jp/
Disallow: /
User-agent: robot-cover-url: http://www.acme.com/java/software/Acme.Spider.html
Disallow: /
User-agent: robot-cover-url: http://www.otthon.net/search
Disallow: /
User-agent: robot-cover-url: http://www.sygol.com
Disallow: /
User-agent: robot-cover-url: http://www.cutternet.com/products/webcheck.html
Disallow: /
User-agent: robot-cover-url: http://www.ub2.lu.se/NNC/projects/NWI/the_nwi_robot.html
Disallow: /
User-agent: robot-cover-url: http://mlc.kddvw.kcom.or.jp/CLINKS/html/clinks.html
Disallow: /
User-agent: robot-cover-url: http://www.nwnet.net/technical/ITR/index.html
Disallow: /
User-agent: robot-cover-url: http://140.190.65.12/~khooghee/index.html
Disallow: /
User-agent: robot-cover-url: http://Snark.apana.org.au/James/GetURL/
Disallow: /
User-agent: robot-cover-url: http://comics.scs.unr.edu:7000/top.html
Disallow: /
User-agent: robot-cover-url: http://cs6.cs.ait.ac.th:21870/pa.html
Disallow: /
User-agent: robot-cover-url: http://deweb.orbit.de/
Disallow: /
User-agent: robot-cover-url: http://esperantisto.net
Disallow: /
User-agent: robot-cover-url: http://fang.cs.sunyit.edu/Robots/tkwww.html
Disallow: /
User-agent: robot-cover-url: http://funnelweb.net.au
Disallow: /
User-agent: robot-cover-url: http://harvest.cs.colorado.edu
Disallow: /
User-agent: robot-cover-url: http://hplyot.obspm.fr/~dl/robo.html
Disallow: /
User-agent: robot-cover-url: http://isserv.tas.ntt.jp/chisho/titan-e.html
Disallow: /
User-agent: robot-cover-url: http://js.stir.ac.uk/jsbin/jsii
Disallow: /
User-agent: robot-cover-url: http://lycos.cs.cmu.edu/
Disallow: /
User-agent: robot-cover-url: http://nhse.mcs.anl.gov/
Disallow: /
User-agent: robot-cover-url: http://nzexplorer.co.nz/
Disallow: /
User-agent: robot-cover-url: http://osiris.sunderland.ac.uk/sst-scripts/simon.html
Disallow: /
User-agent: robot-cover-url: http://phoenix.cs.hku.hk:1234/~jax/w3rui.shtml
Disallow: /
User-agent: robot-cover-url: http://rbse.jsc.nasa.gov/eichmann/urlsearch.html
Disallow: /
User-agent: robot-cover-url: http://schiele.organik.uni-erlangen.de/cactvs/spider.html
Disallow: /
User-agent: robot-cover-url: http://sequent.uncfsu.edu/~micah/pioneer.html
Disallow: /
User-agent: robot-cover-url: http://tronche.com/W3M2
Disallow: /
User-agent: robot-cover-url: http://wsk.eit.com/wsk/dist/doc/admin/webtest/verify_links.html
Disallow: /
User-agent: robot-cover-url: http://www-personal.engin.umich.edu/~yunke/scripts/churl/
Disallow: /
User-agent: robot-cover-url: http://www-swiss.ai.mit.edu/~ptbb/SG-Scout/SG-Scout.html
Disallow: /
User-agent: robot-cover-url: http://www.blacktop.com.zav/bots
Disallow: /
User-agent: robot-cover-url: http://www.cern.ch/WebLinker/
Disallow: /
User-agent: robot-cover-url: http://www.cs.colorado.edu/home/mcbryan/WWWW.html
Disallow: /
User-agent: robot-cover-url: http://www.cs.colostate.edu/~sonnen/projects/nomad.html
Disallow: /
User-agent: robot-cover-url: http://www.cs.indiana.edu/elisp/w3/docs.html
Disallow: /
User-agent: robot-cover-url: http://www.cs.washington.edu/research/ahoy/
Disallow: /
User-agent: robot-cover-url: http://www.di.uminho.pt/wc
Disallow: /
User-agent: robot-cover-url: http://www.empirical.com/
Disallow: /
User-agent: robot-cover-url: http://www.excite.com/
Disallow: /
User-agent: robot-cover-url: http://www.federated.com/~tim/webvac.html
Disallow: /
User-agent: robot-cover-url: http://www.fi/search.html
Disallow: /
User-agent: robot-cover-url: http://www.geocities.com/SiliconValley/3086/iagent.html
Disallow: /
User-agent: robot-cover-url: http://www.greenearth.com/
Disallow: /
User-agent: robot-cover-url: http://www.htdig.org/
Disallow: /
User-agent: robot-cover-url: http://www.ibm.com/%7ewebmaster/
Disallow: /
User-agent: robot-cover-url: http://www.ics.uci.edu/WebSoft/MOMspider/
Disallow: /
User-agent: robot-cover-url: http://www.idc.ac.il/Sandbag/
Disallow: /
User-agent: robot-cover-url: http://www.ifi.uio.no/~janl/w3mir.html
Disallow: /
User-agent: robot-cover-url: http://www.inf.utfsm.cl/~vparada/webcopy.html
Disallow: /
User-agent: robot-cover-url: http://www.info.waseda.ac.jp/search-e.html
Disallow: /
User-agent: robot-cover-url: http://www.infoseek.com
Disallow: /
User-agent: robot-cover-url: http://www.infoseek.com/
Disallow: /
User-agent: robot-cover-url: http://www.intercom.com.au/wombat/
Disallow: /
User-agent: robot-cover-url: http://www.ius.cs.cmu.edu/cgi-bin/vision-search
Disallow: /
User-agent: robot-cover-url: http://www.jubii.dk/robot/default.htm
Disallow: /
User-agent: robot-cover-url: http://www.maths.usyd.edu.au:8000/jimr/pe/Peregrinator.html
Disallow: /
User-agent: robot-cover-url: http://www.maxum.com/phantom/
Disallow: /
User-agent: robot-cover-url: http://www.mib.org/~ucsdcrawl
Disallow: /
User-agent: robot-cover-url: http://www.micrognosis.com/~ajack/jobot/jobot.html
Disallow: /
User-agent: robot-cover-url: http://www.mit.edu/people/mkgray/net/
Disallow: /
User-agent: robot-cover-url: http://www.netcarta.com/
Disallow: /
User-agent: robot-cover-url: http://www.neva.ru/monster.list/russian.www.html
Disallow: /
User-agent: robot-cover-url: http://www.onramp.net/proquest/resume/robot/robot.html
Disallow: /
User-agent: robot-cover-url: http://www.ontv.com/
Disallow: /
User-agent: robot-cover-url: http://www.python.org/
Disallow: /
User-agent: robot-cover-url: http://www.roverbot.com/
Disallow: /
User-agent: robot-cover-url: http://www.sitetech.com/
Disallow: /
User-agent: robot-cover-url: http://www.specter.com/users/janos/specter
Disallow: /
User-agent: robot-cover-url: http://www.spry.com/wizard/index.html
Disallow: /
User-agent: robot-cover-url: http://www.starnet.it/pgp
Disallow: /
User-agent: robot-cover-url: http://www.tricon.net/Comm/synapse/spider/
Disallow: /
User-agent: robot-cover-url: http://www.univ-paris8.fr/~loic/weblayers/
Disallow: /
User-agent: robot-cover-url: http://www.urlabs.com/
Disallow: /
User-agent: robot-cover-url: http://www.vuw.ac.nz/~newbery/Katipo.html
Disallow: /
User-agent: robot-cover-url: http://www.win.tue.nl/bin/fish-search
Disallow: /
User-agent: robot-cover-url: http://www.winsite.com/pc/win95/netutil/wbmiror1.zip
Disallow: /
User-agent: robot-cover-url: http://www.xs4all.nl/~graaff/checkbot/
Disallow: /
User-agent: robot-cover-url: http://www.greenpac.com/inspector/
Disallow: /
User-agent: robot-cover-url: (forthcoming)
Disallow: /
User-agent: robot-cover-url: (none)
Disallow: /
User-agent: robot-cover-url: ftp://gnjilux.cc.fer.hr/pub/unix/util/wget/
Disallow: /
User-agent: robot-cover-url: http://Search.Aus-AU.COM/
Disallow: /
User-agent: robot-cover-url: http://crrm.univ-mrs.fr
Disallow: /
User-agent: robot-cover-url: http://darsun.sit.qc.ca
Disallow: /
User-agent: robot-cover-url: http://dmoz.org/
Disallow: /
User-agent: robot-cover-url: http://esculapio.cype.com
Disallow: /
User-agent: robot-cover-url: http://euroseek.net/
Disallow: /
User-agent: robot-cover-url: http://fouineur.9bit.qc.ca/
Disallow: /
User-agent: robot-cover-url: http://gazz.nttrd.com/
Disallow: /
User-agent: robot-cover-url: http://gestalt.sewanee.edu/ic/
Disallow: /
User-agent: robot-cover-url: http://ilker.ulak.net.tr/EvliyaCelebi
Disallow: /
User-agent: robot-cover-url: http://image.kapsi.net/
Disallow: /
User-agent: robot-cover-url: http://informant.dartmouth.edu/
Disallow: /
User-agent: robot-cover-url: http://irobot.mame.dk/
Disallow: /
User-agent: robot-cover-url: http://kichijiro.c.u-tokyo.ac.jp/odin/
Disallow: /
User-agent: robot-cover-url: http://kvasir.sol.no/
Disallow: /
User-agent: robot-cover-url: http://marunaka.homing.net/straight/
Disallow: /
User-agent: robot-cover-url: http://members.tripod.com/poppisearch
Disallow: /
User-agent: robot-cover-url: http://mopilot.com/
Disallow: /
User-agent: robot-cover-url: http://mysearch.udm.net/
Disallow: /
User-agent: robot-cover-url: http://naragw.sharp.co.jp/myweb/home/
Disallow: /
User-agent: robot-cover-url: http://opensource.or.id/projects.html
Disallow: /
User-agent: robot-cover-url: http://orbsearch.home.ml.org
Disallow: /
User-agent: robot-cover-url: http://oscar.lang.nagoya-u.ac.jp
Disallow: /
User-agent: robot-cover-url: http://para.inria.fr/~ailleret/larbin/index-eng.html
Disallow: /
User-agent: robot-cover-url: http://pauillac.inria.fr/~fpottier/mac-soft.html.en
Disallow: /
User-agent: robot-cover-url: http://people.freenet.de/Muninn/eyrie.html
Disallow: /
User-agent: robot-cover-url: http://perlsearch.hypermart.net/
Disallow: /
User-agent: robot-cover-url: http://phpdig.toiletoine.net/
Disallow: /
User-agent: robot-cover-url: http://pisuerga.inf.ubu.es/lsi/Docencia/TFC/ITIG/icruzadn/cover.htm
Disallow: /
User-agent: robot-cover-url: http://post.mipt.rssi.ru/~billy/search/
Disallow: /
User-agent: robot-cover-url: http://profitnet.bizland.com/
Disallow: /
User-agent: robot-cover-url: http://ravensearch.tripod.com
Disallow: /
User-agent: robot-cover-url: http://sappho.csi.forth.gr:22000/
Disallow: /
User-agent: robot-cover-url: http://search-info.com/
Disallow: /
User-agent: robot-cover-url: http://search.falconsoft.com/
Disallow: /
User-agent: robot-cover-url: http://search.msn.com
Disallow: /
User-agent: robot-cover-url: http://simmany.hnc.net/
Disallow: /
User-agent: robot-cover-url: http://stage.perceval.be (under developpement)
Disallow: /
User-agent: robot-cover-url: http://theautochannel.com/~mjenn/bw.html
Disallow: /
User-agent: robot-cover-url: http://valet.webthing.com/
Disallow: /
User-agent: robot-cover-url: http://vancouver-webpages.com/VWbot/
Disallow: /
User-agent: robot-cover-url: http://web.cps.msu.edu/~dexterte/isl/packrat.html
Disallow: /
User-agent: robot-cover-url: http://web.tiscali.it/_flat
Disallow: /
User-agent: robot-cover-url: http://www-a2k.is.tokushima-u.ac.jp/search/index.html
Disallow: /
User-agent: robot-cover-url: http://www-cse.ucsd.edu/users/fil/agents/agents.html
Disallow: /
User-agent: robot-cover-url: http://www.1klik.dk/omos/
Disallow: /
User-agent: robot-cover-url: http://www.MagPortal.com/
Disallow: /
User-agent: robot-cover-url: http://www.NationalDirectory.com/addurl
Disallow: /
User-agent: robot-cover-url: http://www.ObjectsSearch.com/
Disallow: /
User-agent: robot-cover-url: http://www.abcdatos.com/
Disallow: /
User-agent: robot-cover-url: http://www.altavista.com/
Disallow: /
User-agent: robot-cover-url: http://www.araykoo.com/
Disallow: /
User-agent: robot-cover-url: http://www.ask.com
Disallow: /
User-agent: robot-cover-url: http://www.atomz.com/help/
Disallow: /
User-agent: robot-cover-url: http://www.austlii.edu.au/
Disallow: /
User-agent: robot-cover-url: http://www.baytsp.com/
Disallow: /
User-agent: robot-cover-url: http://www.blinde-kuh.de/
Disallow: /
User-agent: robot-cover-url: http://www.blue.lu/
Disallow: /
User-agent: robot-cover-url: http://www.bmtmicro.com/catalog/tton/
Disallow: /
User-agent: robot-cover-url: http://www.boxsea.com/crawler
Disallow: /
User-agent: robot-cover-url: http://www.canadiancontent.net/
Disallow: /
User-agent: robot-cover-url: http://www.christcrawler.com/search.cfm
Disallow: /
User-agent: robot-cover-url: http://www.cienciaficcion.net/
Disallow: /
User-agent: robot-cover-url: http://www.computingsite.com/robi/
Disallow: /
User-agent: robot-cover-url: http://www.crawlpaper.com/
Disallow: /
User-agent: robot-cover-url: http://www.cs.washington.edu/research/projects/ai/www/occam/
Disallow: /
User-agent: robot-cover-url: http://www.cusco.pt/
Disallow: /
User-agent: robot-cover-url: http://www.cybercon.de/Motor/index.html
Disallow: /
User-agent: robot-cover-url: http://www.cyberspyder.com/cslnkts1.html
Disallow: /
User-agent: robot-cover-url: http://www.cydral.com/
Disallow: /
User-agent: robot-cover-url: http://www.desertrealm.com
Disallow: /
User-agent: robot-cover-url: http://www.diggit.com/
Disallow: /
User-agent: robot-cover-url: http://www.digimarc.com/prod_fam.html
Disallow: /
User-agent: robot-cover-url: http://www.digital-integrity.com/robotinfo.html
Disallow: /
User-agent: robot-cover-url: http://www.dridus.com/~rmm/dwcp.php3
Disallow: /
User-agent: robot-cover-url: http://www.engsoftware.com/fetch.htm
Disallow: /
User-agent: robot-cover-url: http://www.fireball.de
Disallow: /
User-agent: robot-cover-url: http://www.foi.hr/~dpavlin/titin/
Disallow: /
User-agent: robot-cover-url: http://www.gammasite.com
Disallow: /
User-agent: robot-cover-url: http://www.goodlookingcooking.co.uk
Disallow: /
User-agent: robot-cover-url: http://www.googlebot.com/
Disallow: /
User-agent: robot-cover-url: http://www.hamrad.com/search.html
Disallow: /
User-agent: robot-cover-url: http://www.hav.com/
Disallow: /
User-agent: robot-cover-url: http://www.hometownsingles.com
Disallow: /
User-agent: robot-cover-url: http://www.ianett.com/parasite/
Disallow: /
User-agent: robot-cover-url: http://www.imaginon.com
Disallow: /
User-agent: robot-cover-url: http://www.infoseek.de/
Disallow: /
User-agent: robot-cover-url: http://www.inktomi.com/
Disallow: /
User-agent: robot-cover-url: http://www.inm.de/projects/logogif.html
Disallow: /
User-agent: robot-cover-url: http://www.innerprise.net
Disallow: /
User-agent: robot-cover-url: http://www.instrumentpolen.se/gcreep/index.html
Disallow: /
User-agent: robot-cover-url: http://www.jacksonville.net/~dlxpress
Disallow: /
User-agent: robot-cover-url: http://www.javabee.com
Disallow: /
User-agent: robot-cover-url: http://www.kensaku.org/
Disallow: /
User-agent: robot-cover-url: http://www.kinet.or.jp/naka/tomo/wwwc.html
Disallow: /
User-agent: robot-cover-url: http://www.krstarica.com/
Disallow: /
User-agent: robot-cover-url: http://www.lisa.co.jp/voyager/
Disallow: /
User-agent: robot-cover-url: http://www.matuschek.net/software/jbot
Disallow: /
User-agent: robot-cover-url: http://www.matuschek.net/software/jobo/
Disallow: /
User-agent: robot-cover-url: http://www.mcw.aarkayn.org
Disallow: /
User-agent: robot-cover-url: http://www.metastatic.org/wlm/
Disallow: /
User-agent: robot-cover-url: http://www.mindpass.com/_technology_faq.htm
Disallow: /
User-agent: robot-cover-url: http://www.mnogosearch.org
Disallow: /
User-agent: robot-cover-url: http://www.mobrien.com/add_site.html
Disallow: /
User-agent: robot-cover-url: http://www.muscat.co.uk/euroferret/
Disallow: /
User-agent: robot-cover-url: http://www.nathan.de/nathan/software.html#TARANTULA
Disallow: /
User-agent: robot-cover-url: http://www.nederland.net/
Disallow: /
User-agent: robot-cover-url: http://www.netmechanic.com
Disallow: /
User-agent: robot-cover-url: http://www.newscan-online.de/
Disallow: /
User-agent: robot-cover-url: http://www.nihongo.org/jcrawler/
Disallow: /
User-agent: robot-cover-url: http://www.northernwebs.com/set/spider_view.html
Disallow: /
User-agent: robot-cover-url: http://www.oops-as.no/rix
Disallow: /
User-agent: robot-cover-url: http://www.otway.com/webreaper
Disallow: /
User-agent: robot-cover-url: http
Disallow: /
User-agent: *
Crawl-delay: 999.00
Disallow: /*.avi$
Disallow: /*.doc$
Disallow: /*.docx$
Disallow: /*.gif$
Disallow: /*.Gif$
Disallow: /*.GIF$
Disallow: /*.jpg$
Disallow: /*.Jpg$
Disallow: /*.JPG$
Disallow: /*.jpeg$
Disallow: /*.Jpeg$
Disallow: /*.JPEG$
Disallow: /*.pdf$
Disallow: /*.Pdf$
Disallow: /*.PDF$
Disallow: /*.ppt$
Disallow: /*.pptx$
Disallow: /*.wmv$
Disallow: /*.zip$
Disallow: /*.Zip$
Disallow: /*.ZIP$