Annual report to pandora partners 2004-05
Contents
1. Working with partners
2. Growth of the Archive
3. Development of the Archive infrastructure
4. Focus on users
5. The cost of web archiving
6. Preservation
7. International relations
7.1 Participation in the International Internet Preservation Consortium
7.2 Archiving Web Resources International Conference
8. Promoting the Archive
9. Concluding summary
Appendix 1 PANDORA Consultative Committee - a list of representatives
PANDORA, Australia's Web Archive, is a selective archive of Australian online publications and web sites, which is built collaboratively by the National Library, all of the mainland State libraries, the Northern Territory Library, the National Film and Sound Archive, the Australian War Memorial, and the Australian Institute of Aboriginal and Torres Strait Islander Studies (AIATSIS). This is a report to contributing partners on activities and developments in the 2004-05 financial year.
1. Working with partners
One of the strengths of the PANDORA Archive is the fact that it is based on the collaboration of ten cultural collecting agencies, each with their own selection guidelines. this give the Archive volume, richness, and depth that it could not have if it were built by just one agency.
1.1 Consultation mechanisms
The National Library consults with partners about strategies, policy, procedures and infrastructure through two committees, on which every partner is represented. The PANDORA Consultative Committee comprises staff at the manager level within partner agencies and discusses matters relating to management of the Archive, policy and infrasturcture. Appendix 1 provides a list of the representatives.
All staff within partner agencies who carry out tasks related to archiving in PANDORA form another committee, which discusses matters relating to procedures and infrasturcture and day to day problems encountered with archiving.
Both committees communicate via email discussion lists - pandoraconsult-l and pandora-l respectively. As required, teleconferences of both committees are organised by the National Library to enable partners to raise any concerns and the Library to seek input from partners on various matters concerning the Archive and its operation. A progress report on activities is circulated to partners beforehand. A teleconference of the PANDORA Consultative Committee was last held on 3 March 2005. A teleconference of operational staff was last held on 29 July 2004, with another being planned for 19 July 2005.
1.2 Reports
As mentioned under 1.1, the National Library sends progress reports on activities to partners in association with scheduled teleconferences. In addition, each month, a report on the growth of the Archive, usage statistics, and a summary of responses to the online PANDORA user survey form are sent to both email discussion lists. This report includes information about the ten most popular sites for the month and which agency has archived them.
On a bi-monthly basis, the Library compiles two list of instances archived by each partner agency - one list contains all instances archived during the period and the other details government publications only. These lists are put up on the PANDORA web site at http://pandora.nla.gov.au/newtitles/new_titles_reports.html and partners are advised of their availability via a message to the two email discussion lists. These lists of titles are excellent tools for managers to enable them to review easily both the number and nature of the titles being archived by their agency. Managers can also compare their own performance with partner agencies of similar size.
From this year onwards, the National Library will also supply an annual report of progress and activities to the chief Executive Officers of partner agencies.
1.3 Memorandum of Understanding
To date the business relationship between the National Library and each of its partners has been governed by an exchange of letters that took place between the Director General of the Library and the chief executive officer of each partner agency when it first commenced archiving. In line with other collaborative services such as LibrariesAustralia, PictureAustralia and MusicAustralia, the Library considers it useful to formalise the relationship with partners more explicitly through a Memorandum of Understanding. The text of the proposed MoU has been drafted and will be sent to the heads of partner agencies for comment in July 2005.
1.4 Quality survey
During 2003-04, as requested by partners through the PANDORA Consultative Committee, all partner agencies contributed to a benchmarking study which looked at a number of factors related to our web archiving programs, including quantity of items archived by each agency, the number of full time equivalent staff assigned to the associated tasks, and the way staff were organised. On completion of this study the quality aspect of archiving was an obvious omission and it was agreed to conduct a quality survey to supplement the benchmarking study. The National Library undertook to carry out the survey according to methodology and criteria agreed with the Consultative Committee.
The quality survey was carried out by identifying a sample of 15 per cent of titles archived by each partner, including the National Library, during the period March to June 2004. A maximum of 30 titles was examined for each partner. Titles were assessed for quality and completeness of archiving, whether appropriate notes about any technical difficulties or quality assurance decisions had been recorded in PANDAS, and whether the title entry page had been set up correctly. By the end of April, a detailed confidential report had been provided to each partner, including recommendations for correcting identified problems.
The most common problem identified was failure to create a link from the title entry page to the publisher's copyright statement, when this was available. but there were also more substantial problems, and sometimes it appeared that basic quality checking had not been carried out. In a couple of cases this meant that a title or instance (1) was too defective to be kept and needed to be removed from display in the Archive. In most cases remedial action could be taken.
The quality survey highlighted a few issues:
- There is tension between quality and quantity. Quality checking is the most time consuming aspect of archiving a title for PANDORA. All partners want to archive as many titles as possible with limited resources, but completeness of the material archived is an important factor in their future usefulness.
- Staff engaged in archiving may benefit from additional training in technical aspects.
- The procedures manuals and training documentation provided by the National Library could be developed further, and a policy statement is required.
1.5 Training
As web archiving becomes more and more technically complex, there is an ongoing need for all staff involved in it to continue to develop their skills. The quality survey also identified a need for training in some specific skills. From March to June the Library embarked on a program of training for its own web archiving staff. It also prepared and delivered to other partners in May and June some training material and exercises which could be carried out in the PANDAS training system. Assistance with the exercises was offered.
This training package was not very well received, with a minority of partners making use of the material. During its regular teleconference with operational staff in July, the Library will explore with partners whether this training can be delivered in a better way.
2. Growth of the Archive
| 30 June 2004 | 30 June 2005 | Growth 2004-05 | |
|---|---|---|---|
Titles |
6,320 |
8,892 |
2,572 (+ 40.6%) |
Instances |
12,449 |
17,947 |
5,498 (+ 44.1%) |
Gigabytes (2) |
668 |
884 |
216 (+ 32.3%) |
Usage (page views) |
4,398,148 |
5,390,459 |
992,311 (+ 22.5%) |
Note: The figures for Titles, Instances and Gigabytes are the totals in the Archive at the given date, and the differences between the figures for the two dates represent the growth that took place during 2004-05. The figures for Usage (page views) are the total usage during the financial year ending in the given date.
The Archive showed strong growth during 2004-05, both in terms of the number of titles and instances archived, and in its usage.
Government publications comprise approximately 43 per cent of the Archive.
3. Development of the Archive infrastructure
To keep pace with a rapidly changing web archiving environment the National Library is committed to ongoing development of the policy, procedures and technical infrastructure which support the Archive.
3.1 Development of PANDAS
PANDAS (the PANDORA Digital Archiving System) is web-based software developed by the Library to enable staff of partners to carry out all of the tasks involved in contributing selected online publications and web sites to PANDORA. (This does not include cataloguing, which is carried out in separate systems.)
During 2004-05 the Library has been working intensively on the development of PANDAS 3, with the principle aim of improving stability and maintainability. Developers are also taking the opportunity to incorporate some required enhancements while the code is being re-written, and the user interface is being redesigned to make it more intuitive and efficient. Deployment of the first release of PANDAS 3 into production is expected in late 2005, followed by release 3.1. The latter will include the capability to batch load metadata for online publications from trusted sources, for instance, from government department libraries, and to initiate automatic harvesting for selected resources.
Through the PANDORA Consultative Committee, partners have been encouraged to make suggestions for the development of PANDAS. Suggestions can be made at any time.
3.2 Archiving databases
Most web archiving programs of national libraries depend on the use of harvesting robots which are sent out to the URL of an identified online publication or web site. Navigating the html links within the resource, the harvester then brings the contents back to the web archive. However, there is an increasing volume of information being published on the Web that is inaccessible to harvesting robots. Publications that incorporate interactive search and browse interfaces communicating with a web server application, delivering content from a database, are one example. This has meant that this very valuable information has not been collected and preserved in any country to date.
One of the major obstacles to archiving database resources is that the archiving agency does not have licences to use the software on which the database depends for organisation and discovery of information. The software in any case becomes obsolete with passing time.
The Library identified a method for providing access to data supplied by publishers and, with a small grant from the International Internet Preservation Consortium (IIPC) (3), on behalf of the Deep Web Archiving Working Group, has developed software that provides a generic interface to archived databases, including a search facility. The tool will replicate search access to all the categories of data used by the publisher and, for the first time, enables PANDORA and other web archiving programs around the world to provide access to archived databases. This software is called Xinq (4) and in early 2005 was made available as open source software through the SourceForge software repository.
The Library has now received data from publishers for ten database publications and the generic interface has been generated for three of these. Two titles are expected to become publicly available in PANDORA in July. Partners have been informed that they can select databases, negotiate permission to archive, and pass the details to the National Library for processing. Two partners have supplied details on databases to the Library.
3.3 Search engine
Throughout 2004-05 the Archive continued to have a pressing need for an effective full text search engine. Search functionality based on free software had previously been installed on the PANDORA home page, but numerous complaints were received about its inadequacy, and it was removed in October 2004.
During the past nine months the Library has been actively searching for a suitable search engine, but those tested have not adequately addressed the size and complexity of the Archive. The Library is currently evaluating tenders for a search engine that would meet the needs of PANDORA and other Library services. In May, as an interim measure, a Google title search was installed on the PANDORA home page. This offers staff and users a more efficient alternative to the long browse titles lists. In addition, until the acquisition project is complete and the selected software is implemented and customised for PANDORA requirements, the open source Lucene full text search software has been installed and adapted to the needs of the PANDORA Archive Search. The entire text contents of the Archive have been indexed. The Search Help on the PANDORA home page provides a link to this demonstration search service and, even in its raw form, it is providing reasonable search access to the contents of the Archive.
3.4 Whole domain harvest
Despite the many advantages of selective archiving, there are recognised disadvantages. The Library has wished for a number of years to undertake a whole domain harvest to supplement the selective Archive. During 2004-05 we were able to allocate the funds to do such a harvest. In May 2005 the Library commenced a three month project to harvest the entire Australian web domain and contracted the Internet Archive to undertake the harvest to run continuously over a four week period commencing on 13 June. The Internet Archive has experience in doing large scale web crawls and has developed a harvesting robot called Heritrix, which is designed specifically for harvesting web sites for preservation purposes.
The Library provided lists of seed URLs to the Internet Archive to direct the harvest initially towards sites of highest priority, including government and higher education sites. A list of Australian domains that do not include .au in the URI was also provided.
The completed harvest will be analysed by the Internet Archive and the National Library in July. The Internet Archive will provide access to the contents for the Library and others authorised by it. As a result of this crawl we will have a better understanding of the size and scope of the Australian web domain. We will also have captured many sites that are beyond the scope and resources of our selective archiving operation.
Most of the archived resources cannot be made publicly accessible for Australian copyright reasons. As part of the analysis, however, consideration will be given to whether it might be technically and legally possible to provide public access to certain resources, for instance, Northern Territory publications which are now covered by legal deposit, NSW government publications which are subject to the Premier’s Department Memorandum, and Commonwealth government publications where the Library holds blanket permission for an agency domain.
Partners will be provided with a report on the whole domain harvest and would be consulted if it appears possible to provide access to any part of the harvest for which they take collecting responsibility.
4. Focus on users
4.1 PANDORA web site redesign
On 7 October the re-designed PANDORA website was implemented. The PANDORA web site is multi-purpose, providing access to the contents of the Archive to the general public, information about PANDORA and PANDAS to information professionals, and policy, procedures and training materials to partners.
The aim of redesigning the site was to
- Improve the organisation and discoverability of the information contained on the site;
- Bring the information about PANDORA and PANDAS up to date;
- Implement a more effective graphic design.
The new design has been well-received and the aims of the upgrade appear to have been met.
4.2 Online user survey form
An online user survey form was implemented with the new PANDORA web site in October and refinements were made in March to improve the usefulness of the information being received. Apart from demographic information telling us who is using the Archive, we are principally interested in knowing whether users are finding what they are looking for and, if not, what it is that they have not found. The survey form is now working well in this regard, and the results are reported at the end of each month to partners’ PANDORA operational staff and managers.
4.3 Usage analysis
In November an analysis of the usage of PANDORA during 2003-2004 was completed and a copy of the report was sent to partners’ PANDORA managers and operational staff. Although the tools available for analysing web usage are as yet not very sophisticated and the figures need to be interpreted cautiously, some interesting trends emerged.
- During each of the quarters of 2003-2004, there were fives archived titles that appeared consistently in the ‘top ten’. Four of these were no longer available on the publishers’ sites. The fifth of these contains freely available content provided by the publisher, which is not available on the publisher’s commercial site. The analysis therefore suggests that the Archive appears to be fulfilling its primary purpose of providing access to Australian online publications that have disappeared.
- There has been a steady increase in the use of PANDORA during 2002-2003 and 2003-2004.
- Approximately 60 per cent of use during 2003-04 was referred by search engines, and most of these visitors are likely to have visited without knowing what PANDORA is and not particularly wanting archived resources. However, a significant minority, 34 per cent, were either coming directly to PANDORA (and therefore presumably knew it by reputation or had visited it before), or were referred by ‘expert services’. These included the Library’s catalogue and its web site, other libraries’ web sites, indexing and abstracting services, and the web sites of Australian schools and universities. This suggests that one third of PANDORA visitors may be putting the Archive to the use for which it is intended. This was considered a satisfying result at this early stage of the development of the Archive.
5. The cost of web archiving
Anyone involved in archiving for PANDORA knows that the unit cost of acquiring online publications is high in comparison with the acquisition of printed books and serials, because online publications require much more staff time to process. Until recently the Library had no precise information about what this cost was. Early in 2005 the Collections Management Division completed costings of legal deposit activities for print serials and monographs, and for archiving Australian online publications.
The costing of web archiving activities examined the cost to the Library of acquiring an instance of a title within scope of its selection guidelines, cataloguing it, and adding it to the PANDORA Archive. It addressed the costs incurred by the National Library only and did not include partners’ costs for adding their titles. The costing ascertained that each archived instance cost $178.68. (This compared with $43.77 for a monograph, and $11.29 for an issue of a serial.) Staff costs comprise 94 per cent of this unit cost for online publications, with supplier costs and infrastructure maintenance and development making up the rest. Quality checking is the most expensive part of the process.
The cost to other partners of adding an instance to the Archive could vary from the above-mentioned costs, depending upon a number of factors including the salary levels at which staff are payed, and the experience and expertise of the staff involved. In addition, other partners do not incur system development and maintenance costs.
An article describing the methodology and detailed results of this costing exercise for online publications was published in RLG DigiNews (5) in June 2005.
6. Preservation
During the past year the National Library continued to monitor the file formats entering the PANDORA archive. Although the cocktail of formats continues to grow, the Library is not aware of significant current accessibility problems.
Based on the extensive risk assessment undertaken for PANDORA in the previous financial year, the Library has continued to manage the Archive’s data streams securely, aiming to protect it against data corruption or loss.
The National Library’s Archiving Web Resources International Conference in November 2004 confirmed that a number of potentially useful tools are being developed that may be applicable to PANDORA. These include tools for recognising and describing file formats, tools for verifying and normalising file formats, tools for extracting metadata from files, and tools for identifying dependencies for presenting different formats. The Library is particularly interested in mechanisms for identifying active risks, and mechanisms for drawing on all the relevant tools as they become available. The Library has commenced talks with the Distributed Systems Technology Centre (DSTC), and is working with Australian Partnership for Sustainable Repositories (6) partners to assess and develop relevant mechanisms. Testbeds of files generated from the PANDORA Archive will be used in this work.
7 International relations
7.1 Participation in the International Preservation Consortium
In 2003, because of the Library’s experience with web archiving, it was invited to become a foundation member of the International Preservation Consortium (IIPC). Membership comprises 11 national libraries and the Internet Archive. The main aim of the IIPC is to develop a suite of open source tools that all national libraries can implement to support their web archiving programs. The work of the Consortium is carried out through six working groups and during 2004-05 the Library has actively contributed to two, the Deep Web Archiving and the Content Management Working Groups. Progress with the former is reported in section 3.2, ‘Archiving Databases’.
The Content Management Working Group had identified that many libraries, and the Internet Archive itself, have a need for software that will enable staff without IT qualifications to undertake web archiving. PANDAS does just this, but it has been developed specifically to support the particular workflows of PANDORA partners and they do not necessarily suit the needs of many other libraries. Because of its experience with PANDAS, the Library was asked to write the work flow diagrams for a generic ‘curator tool’. The Library was pleased to participate in this task because it will have benefits for PANDAS development in the future. It should enable:
- the Library to move away from being the sole developer of this type of software and to share the task with others; and
- PANDORA partners to have ongoing access to a world class digital archiving management system that is interoperable with systems being used elsewhere in the world.
7.2 Archiving Web Resources International Conference
As reported elsewhere, the Library organised the Archiving Web Resources International Conference 9-11 November 2004, with associated events, including the Information Day on 12 November. Bringing together many experts in the field from Australia and overseas, the conference was very beneficial for the Library in fostering relations with others who are doing similar work and sharing knowledge about archiving and preservation.
7.3 Swiss Library Science Talks
Paul Koerbin, Supervisor of the Digital Archiving Section at the National Library, delivered the Swiss Library Science Talks at the Swiss National Library in Bern and at CERN in Geneva on 20 and 21 September respectively.
7.4 PANDAS Evaluation System
To address the enquiries from around the world that the Library receives about PANDAS from agencies wanting to establish their own web archiving programs, the Library has set up the PANDAS Evaluation System, which enables enquirers to have access to a demonstration version of the software, to assess its suitability for their own needs. Six agencies have had access to the software during 2004-05.
The National Library Board of Singapore has indicated that it will implement PANDAS version 3 when it becomes available. The UK Web Archiving Consortium (7) launched its publicly available archive based on the PANDAS software in May.
8. Promoting the Archive
8.1 PANDORA Fact Sheet
In April, the Library issued the first PANDORA Fact Sheet, which summarises key information about the Archive. It is intended to supplement other publicity material, including the printed PANDORA Brochure, and is available for use by all partners. It is being updated with the latest statistical information every month and the digital file is distributed to partners every month by email. It can be printed out in colour (preferably) or black and white. One partner has indicated its intention to send it with requests to publishers for permission to archive selected titles, which is an ideal use to make of it.
8.2 UNESCO Memory of the World
PANDORA was placed on the Memory of the World Australian Register at a ceremony at the State Library of Victoria on 10 August 2004. At the suggestion of the Australian selection committee, a nomination for the World Register was also submitted. The Library has been advised that this nomination was unsuccessful because the World Register selection committee had no selection criteria for digital resources.
An entry was also submitted for the Memory of the World Jikji Prize, which is offered by the Republic of Korea. The Prize, consisting of an award of US$ 30,000, will be given every two years to individuals or institutions that have made significant contributions to the preservation and accessibility of documentary heritage. The Library has been advised that PANDORA is on a short list of seven, out of an original 35 submissions.
8.3 Papers and articles
A number of papers and articles have been written or contributed to for the dual purpose of promoting our work and sharing what we have learned. Some key examples follow:
- Koerbin , Paul. The PANDORA Digital Archiving System (PANDAS) : A paper presented at the 4th International Web Archiving Workshop on 16 September 2004 in Bath UK. Available at http://www.nla.gov.au/nla/staffpaper/2004/koerbin2.html
- Meacham, Steve. ‘Editing our future’ in Sydney Morning Herald: Insight, Wednesday 4 May 2005, p. 18.
- Phillips, Margaret. ‘Selective archiving of web resources: A study of the costs at the National Library of Australia.’ in RLG DigiNews, vol. 9, no. 3, 15 June 2005. Available at http://www.rlg.org/en/page.php?Page_ID=20666#article0
- Phillips, Margaret. ‘What do we preserve? – the question for heritage libraries in a digital world’ in Library Trends vol. 53, no. 5, Summer 2005. In publication.
9. Concluding summary
The 2004-05 financial year has been a productive one for PANDORA and its partners. Some of the highlights include:
- A Memorandum of Understanding with partners was drafted, to be distributed to chief executive officers of partner agencies for comment in July 2005 (section 1.3);
- A quality survey of archived titles was undertaken and reports were distributed to partners (section 1.4);
- The Archive grew by 44 per cent (8) (section 2);
- Substantial progress was made on PANDAS version 3, which will provide greater stability, enhanced functionality and a re-designed user interface when deployed in late 2005 (section 3.1);
- Software to provide access to archived database publications was developed (section 3.2);
- A whole domain harvest of the .au domain commenced in June (section 3.4);
- The PANDORA web site was re-designed (section 4.1);
- An analysis of usage of the Archive was undertaken and a report distributed to partners (section 4.3);
- A costing of web archiving within the National Library was undertaken and an article on it was published in RLG DigiNews (section 5);
- The Library has continued to manage the Archive’s data streams securely, to protect it against data corruption or loss. (section 6);
- Active participation in the International Internet Preservation Consortium continued (section 7.1).
- The Archiving Web Resources International Conference was held 9-11 November (section 7.2);
- PANDORA was placed on the Australian Memory of the World Register (section 8.2).
End notes
1 An 'instance' is a single gathering of a title. It includes the gathering of a monograph that has been archived once only, the first gathering of a serial title or integrating title (for example, a web site that changes over time), and all subsequent gatherings.
2 This figure does not include the preservation and other master and back-up copies.
3 Information about the IIPC is available from its web site at http://netpreserve.org/about/index.php
4 More information about Xinq is available at http://www.nla.gov.au/xinq/
5 Phillips, M. E. ‘Selective archiving of web resources: A study of acquisition costs at the National Library of Australia.’ In RLG DigiNews, vol.9, no. 3, 15 June 2005. Available at http://www.rlg.org/en/page.php?Page_ID=20666#article0
6 An Australian Research Council funded project in partnership with the Australian National University, the University of Queensland, the University of Sydney, and the Australian Partnership for Advanced Computing.
7 The UK Web Archiving Consortium web site is available at http://www.webarchive.org.uk/
8 Calculated on number of instances added.
Appendix 1
PANDORA Consultative Committee – list of representatives
Australian Institute of Aboriginal and Torres Strait Islander Studies - Pat Brady, Collection Manager, AIATSIS Library
Australian War Memorial - Mal Booth, Head, Research Centre
National Film and Sound Archive - Matthew Davies, Manager, Collection Development
National Library of Australia - Margaret Phillips, Director, Digital Archiving (Chair of Committee)
Northern Territory Library and Information Service - Ann Ritchie, Assistant Director
State Library of New South Wales - Jim Tindall, Senior Librarian, Collection Services
State Library of Queensland - Helen Thiselton, Manager, Publications
State Library of South Australia - Tony Leschen, Manager, Collection Development
State Library of Victoria - Liz Jesty, Manager, Collections Management
State Library of Western Australia - Brian Stewart, Manager, Battye Published Collections
