AusCrawl 2005 versus PANDORA

Since PANDORA contains many websites that have been checked and manually gathered where necessary, it should be possible to make a rough estimate of how much of the Australian domain was collected by AusCrawl by comparing two archives. Two different methods were used in the comparison which was carried out by a small collection of computer programs written by Alex Osborne.

URL for URL comparison

A list of PANDORA URLs was collected from the url.map files scattered around the the PANDAS production filesystem. From these, only URLs from instances that had been gathered within the period of July 2004 to July 2005 were selected. The list was then manually pruned by removing non .au URLs and also other obviously invalid ones. This final list was then compared against AusCrawl's Heritrix crawl logs using a couple of small computer programs to determine the proportion of URLs that were gathered by AusCrawl.

Number of URLs tested:2121196
Successfully archived by AusCrawl (200, 3xx):94079744.35%
Inaccessible - missing, restricted or server error (4xx, 5xx):836973.94%
blocked by robots.txt rules:237261.11%
Unable to connect to server:50610.024%
Not encountered by AusCrawl, or queued when crawl was halted:106791550.34%

These figures were signifigantly lower that previous estimates, so they were investigated further. It turned out that there were several very large websites which had tens of thousands of missed URLs that somewhat severely skewing the numbers. The top 25 were:

HostnameURLs missed
longpaddock.qld.gov.au98772
austlii.edu.au65218
democrats.org.au60949
abc.net.au54430
csu.edu.au54382
home.vicnet.net.au33855
dpi.qld.gov.au32673
dairyaustralia.com.au30793
onlineopinion.com.au28408
jcu.edu.au27589
dpie.gov.au20452
health.gov.au18442
swingpatrol.com.au14948
deh.gov.au14676
mariners.records.nsw.gov.au14527
230battalion.org.au14463
statelibrary.vic.gov.au14252
thatsmelbourne.com.au13974
bom.gov.au12270
aph.gov.au12159
amcoz.com.au11642
randomhouse.com.au10982
olympics.com.au10948
auscert.org.au10933
thebladder.com.au9699

Average domain coverage

To try and address some of the problems with the URL for URL comparison, a different method was devised. Instead of comparing the total number of URLs gathered or missed by AusCrawl, a value for the percentage of each website covered would be calculated and then an average of all these values would be taken. This way, a few large websites would not dominate the results.

Also, at this point, Mark Triggs assisted in a collecting a more complete list of PANDORA URLs. Since it was infeasible to retrieve all the PANDORA logs from storage tapes for the given time period, a list was created by examining the headers that the HTTrack software adds to each of the HTML files it gathers. The header is not present in other files such as images and PDF documents so only HTML files were considered this time. Again, the URLs were limited to those within the .au domain namespace. Also, websites that AusCrawl was not able to retrieve at least two pages from were filtered out from the calculations.

Number of (HTML) URLs tested:2793058
Percentage of each website successfully crawled by AusCrawl:59.91%
Percentage of each website at least known about by AusCrawl:64.39%
Percentage of each website left in the queue when crawl was stopped:4.48%
Percentage of each website not encountered by AusCrawl:35.61%

A note on error

Since that are archived in PANDORA are included based on selection guidelines, PANDORA urls are not random selection of the Australian web domain and probably somewhat biased towards larger and more well-known websites. Also, the PANDORA websites were archived over a whole year, while AusCrawl just reflects six weeks, so a proportion of web pages that had been archived by PANDORA may have not have been online during AusCrawl.

Because of these sources of error, the determined values of success are likely to be lower than the "true" results. So it could be concluded that we are quite certain that at least 50-60% of the .au domain was successfully archived by AusCrawl.