Skip to content

Skip to menu

Pandora : Australia's Web Archive National Library of Australia and Partners
Home > Domain harvests

Australian web domain harvests

About the domain crawls

The National Library began conducting large scale harvests to complement the PANDORA selective web archiving in 2005. Since 2005, annual harvests have been conducted on behalf of the National Library by the Internet Archive in San Francisco. The scope of the harvests has been a broad crawl of the .au top level country domain along with some non .au links for resources located in Australia.

Table of the extent of the data collected for the Australian domain harvest collections

Domain Harvest
2005
2006
2007
2008
Unique files
185 million
596 million
516 million
1 billion
Hosts crawled
811,523
1,046,038
1,247,614
3,038,658
Size
6.69 TB
19.04
18.47TB
34.55 TB

 

Reports on the Australian domain crawls

Links to Australian domain harvests

Access to the Australian domain harvests is not currently available to the public. Researchers with a compelling reason for access to the content should direct enquiries to the Manager Web Archiving at the National Library. See the contact page for details.

Access for authorised users is via this reverse proxy link for Wayback Machine and full text search access. Crawl reports are also available from this page.