Skip to content

Skip to menu

Pandora : Australia's Web Archive National Library of Australia and Partners
Home >

Notice to webmasters

Notice to Webmasters - 2008 Australian domain harvest crawl

If you have linked directly to this page your site is being automatically crawled by the Internet Archive (www.archive.org) on behalf of the National Library of Australia. More details about this project can be found below.

If this crawling is having an impact upon the performance of your site, please email immediately:

and forward a copy to the National Library of Australia project officer at:

Please note that the email addresses above are not clickable links or able to be copied (to avoid spam they are images). You will need to note the address and type it into your email client.

The National Library's whole Australian domain harvest project - 2008 '1 billion document crawl'

Beginning around 18th July 2008 the National Library of Australia (NLA) is undertaking its fourth annual crawl and harvest of the Australia web domain. This web crawl is being conducted on behalf of the NLA by the Internet Archive based in San Francisco. This crawl has a target of harvesting 1 billion documents (unique files) from the .au domain for the NLA's web archive. It is the largest such crawl focused on the Australian web domain.

Previous crawls of the Australian web domain for the purpose of collecting archival content were conducted August and September 2007, August and September 2006 and June and July 2005.

For the purpose of these collections, the Australian web domain includes .au domain sites. In addition some sites identified by DNS lookup as having an IP address located in Australia may be included. Because of the scope of this project publishers and webmasters cannot be contacted in advance. However, the content harvested during the whole domain crawl will not generally be made available by the NLA without the permission of the content owners or other legal warrant. In this regard this archival project is distinct and differs from the PANDORA Archive which is a selective web archive the contents of which are collected and made accessible with the prior permission of Pthe content owners. Content harvested from this domain crawl may at some time be included in the Internet Archive's own Wayback Machine collection. Should authors or publishers have objections to content being available from the Internet Archive collection, it may be removed from public availability in accordance the the Internet Archive's terms of use.

How the gathering of your site is done

The National Library of Australia has contracted the Internet Archive to undertake the crawling (harvesting) of Australian websites for a period of time beginning on 18 July 2008. The crawl is expected to last in the order of twelve weeks.

The Internet Archive uses a sophisticated harvesting software called Heritrix (crawler.archive.org) developed specifically for the purpose of archiving web sites for long term preservation. Heritrix is designed to harvest material at a measured, adaptive pace that is unlikely to disrupt normal website activity.

The harvester is generally programmed to respect robot exclusion rules as specified in the robots.txt file and robots META tags (www.robotstxt.org/wc/robots.html). The crawl agent for this domain crawl will identify with the name "archive.org_bot" should you opt to disallow this agent.

However, in order to successfully obtain archival content inline links to images and stylesheets will be followed even if these refer to directories excluded in the robots.txt file. The special crawl agent "img+css-bot" will be used for these fetches, and the 'referrer' in your logs will show the containing URL.

The harvester uses a large number of seed URLs to initiate the crawl and it will follow links within the parameters of the specified Australian domain (as outlined above). The aim of the harvest is to collect all the publicly available content of the sites crawled, including all text, images, stylesheet files (CSS), JavaScript, audio, video, PDF etc. so that representation in the archive is as complete and accurate as is technically possible.

For further information contact the project officer at:

Please note that the email address above is not given as a link that is clickable or able to be copied (to avoid spam it is an image). You will need to note the address and type it into your email client.