Skip to content

Skip to menu

Pandora : Australia's Web Archive National Library of Australia and Partners
Home >

Notice to webmasters

Notice to Webmasters - 2009 Australian domain harvest crawl

If you have linked directly to this page your site is being automatically crawled by the Internet Archive (www.archive.org) on behalf of the National Library of Australia. More details about this project can be found below.

If this crawling is having an impact upon the performance of your site, please email immediately:

and forward a copy to the National Library of Australia project officer at:

crawl@nla.gov.au

Please note that the email addresses above are not clickable links. You will need to note the address and type it into your email client.

The National Library's whole Australian domain harvest project - 2009

Beginning on 10th September 2009 the National Library of Australia (NLA) is undertaking its fifth annual crawl and harvest of the Australia web domain. Test harvesting will commence on 4 September 2009. This web crawl is being conducted on behalf of the NLA by the Internet Archive based in San Francisco. This crawl has a target of harvesting 650,000 documents (unique files) from the .au domain for the NLA's web archive.

Previous crawls of the Australian web domain for the purpose of collecting archival content were conducted during July to September 2008, between August and September 2007, between August and September 2006 between June and July 2005.

For the purpose of these collections, the Australian web domain includes .au domain sites. In addition some sites identified by DNS lookup as having an IP address located in Australia may be included. Because of the scope of this project publishers and webmasters cannot be contacted in advance. However, the content harvested during the whole domain crawl will not generally be made available by the NLA without the permission of the content owners or other legal warrant. In this regard this archival project is distinct and differs from the PANDORA Archive which is a selective web archive the contents of which are collected and made accessible with the prior permission of the content owners. Content harvested from this domain crawl may at some time be included in the Internet Archive's own Wayback Machine collection. Should authors or publishers have objections to content being available from the Internet Archive collection, it may be removed from public availability in accordance the the Internet Archive's terms of use.

Crawl timeframe

The National Library of Australia has contracted the Internet Archive to undertake the crawling (harvesting) of Australian websites for a period of time beginning on 10th September 2008. The crawl is expected to last around six weeks.

The crawl robot

The Internet Archive uses a sophisticated harvesting software called Heritrix (crawler.archive.org) developed specifically for the purpose of archiving web sites for long term preservation. Heritrix is designed to harvest material at a measured, adaptive pace that is unlikely to disrupt normal website activity.

Crawler robot identification

The crawl agent for this domain crawl will identify with the name: archive.org_bot.

(Note that this is a different identity to the crawler usually associated with the Alexa Internet Archive, i.e. ia_archiver)

Robots.txt compliance

The harvester is programmed to respect robot exclusion rules as specified in the robots.txt file and robots META tags (www.robotstxt.org/wc/robots.html) with the following significant exception:

Embedded page links with extensions .bmp .gif .jpeg .jpg .png .tif .tiff .css or .js will be crawled even if they are in directories disallowed in the robots.txt. The reason for this is to harvest a faithful and accurate representation of the page for the archival purpose of this crawl.

The special crawl agent "img+css-bot" will be used for these fetches, and the 'referrer' in your logs will show the containing URL.

Stop the crawl using robots.txt

If you wish to disallow the crawl robot please specify the agent mentioned above and disallow in your robots.txt file. i.e.

User-agent: archive.org_bot
Disallow:
/

Please note that the crawler will re-check robots.txt on a 24 hour cycle. If you need to have the crawl stopped sooner than that please contact the crawl engineer at the address above or the crawl project officer at the National Library on the following email.

For further information contact the project officer at: crawl@nla.gov.au

Please note that the email address above is not clickable . You will need to note the address and type it into your email client.