Australian web domain harvests
About the domain crawls
The National Library began conducting large scale harvests to complement the PANDORA selective web archiving in 2005. Since 2005, annual harvests have been conducted on behalf of the National Library by the Internet Archive in San Francisco. The scope of the harvests has been a broad crawl of the .au top level country domain along with some non .au links for resources located in Australia.
Table of the extent of the data collected for the Australian domain harvest collections
| Domain Harvest | 2005 |
2006 |
2007 |
2008 |
| Unique files | 185 million |
596 million |
516 million |
1 billion |
| Hosts crawled | 811,523 |
1,046,038 |
1,247,614 |
3,038,658 |
| Size | 6.69 TB |
19.04 |
18.47TB |
34.55 TB |
Reports on the Australian domain crawls
- The Australian web domain harvests: a preliminary quantitative analysis of the archive data. A report by Paul Koerbin on the size and nature of the content of the 2005, 2006 and 2007 Australian web domain harvests
- Report on the crawl and harvest of the whole Australian web domain undertaken during June and July 2005 by Paul Koerbin (October 2005; PDF document)
- An analysis of the content of AusCrawl 2005 by Chris Huston (January 2006; Word document)
- AusCrawl 2005 versus PANDORA an analysis report by Alexander Osborne (February 2006; HTML document)
- Report on the analysis of the 2005 Australian domain crawl by Paul Koerbin (March 2006; Word document)
Links to Australian domain harvests
Access to the Australian domain harvests is not currently available to the public. Researchers with a compelling reason for access to the content should direct enquiries to the Manager Web Archiving at the National Library. See the contact page for details.
Access for authorised users is via this reverse proxy link for Wayback Machine and full text search access. Crawl reports are also available from this page.
