Notice to webmasters
Notice to Webmasters - 2012 Australian domain harvest crawl
If you have linked directly to this page your site is being automatically crawled by the Internet Archive (www.archive.org) on behalf of the National Library of Australia. More details about this project can be found below.
If this crawling is having an impact upon the performance of your site, please email immediately:
Please note that the email address above is not clickable links. You will need to note the address and type it into your email client.
You can also forward a copy to the National Library of Australia project officer at: crawl@nla.gov.au
The National Library's whole Australian domain harvest project - 2012
Beginning in the week of the 27th February 2012 the National Library of Australia (NLA) is undertaking its seventh annual crawl and harvest of the Australia web domain. Test harvesting will commence on 17 February 2012. This web crawl is being conducted on behalf of the NLA by the Internet Archive based in San Francisco. This crawl has a target of harvesting 700,000 documents (unique files) from the .au domain for the NLA's archival collection of freely available websites and documents.
Previous crawls of the Australian web domain for the purpose of collecting archival content were conducted during February and March 2011, September and October 2009, July to September 2008, between August and September 2007, between August and September 2006 between June and July 2005.
For the purpose of these collections, the Australian web domain includes .au domain sites. In addition some sites identified by DNS lookup as having an IP address located in Australia may be included. Because of the scope of this project publishers and webmasters cannot be contacted in advance. However, the content harvested during the whole domain crawl will not generally be made available by the NLA without the permission of the content owners or other legal warrant. In this regard this archival project is distinct and differs from the PANDORA Archive which is a selective web archive the contents of which are collected and made accessible with the prior permission of the content owners. Content harvested from this domain crawl may at some time be included in the Internet Archive's own Wayback Machine collection. Should authors or publishers have objections to content being available from the Internet Archive collection, it may be removed from public availability in accordance the the Internet Archive's terms of use.
The National Library's harvest of Commonwealth Government web publications
Concurrent to the broad Australian domain harvest, the National Library in conjuction with the Internet Archive will be conducting a harvest of Australian Commonwealth Government web materials. The scope of this project is to establish an archival collection of freely available content from the websites of agencies und the Financial Management and Acountability Act 1997 (FMA Act) under the warrant of whole-of-government arrangements for the collection and preservation of government online content endorsed by the Secretaries' ICT Governance Board in May 2010.
Crawl timeframe
The National Library of Australia has contracted the Internet Archive to undertake the crawling (harvesting) of Australian websites for a period of time beginning in mid-January 2011 and lasting for around six weeks.
The crawl robot
The Internet Archive uses a sophisticated harvesting software called Heritrix (crawler.archive.org) developed specifically for the purpose of archiving web sites for long term preservation. Heritrix is designed to harvest material at a measured, adaptive pace that is unlikely to disrupt normal website activity.
Crawler robot identification
The crawl agent for this domain crawl will identify with the name: archive.org_bot.
(Note that this is a different identity to the crawler usually associated with the Alexa Internet Archive, i.e. ia_archiver)
Robots.txt compliance
The harvester is programmed to respect robot exclusion rules as specified in the robots.txt file and robots META tags (www.robotstxt.org/wc/robots.html) with the following significant exception:
Embedded page links with extensions .bmp .gif .jpeg .jpg .png .tif .tiff .css or .js will be crawled even if they are in directories disallowed in the robots.txt. The reason for this is to harvest a faithful and accurate representation of the page for the archival purpose of this crawl.
The special crawl agent "img+css-bot" will be used for these fetches, and the 'referrer' in your logs will show the containing URL.
Stop the crawl using robots.txt
If you wish to disallow the crawl robot please specify the agent mentioned above and disallow in your robots.txt file. i.e.
User-agent: archive.org_bot
Disallow: /
Please note that the crawler will re-check robots.txt on a 24 hour cycle. If you need to have the crawl stopped sooner than that please contact the crawl engineer at the address above or the crawl project officer at the National Library on the following email.
For further information contact the project officer at: crawl@nla.gov.au
