Frequently asked questions about PANDORA
Questions relating to organisation
- What is PANDORA?
- Where is PANDORA based?
- When did work on PANDORA begin?
- Who builds the PANDORA Archive?
- How is the Archive funded?
- How do participants contribute to the Archive when they are remote from the National Library?
Questions relating to collection
- What is the scope of the Archive? Do you have selection guidelines?
- How frequently do you capture web sites?
- Do you attempt to collect all levels of each web site?
- Do you archive external links?
- Do you receive publications directly from the publishers? If so, how?
- Why did the National Library of Australia choose to take a selective approach to archiving?
- Do you archive the software (i.e., the plugins) or just the text and images?
- How large is the Archive?
- Why does the National Library build PANDORA when there is the Internet Archive that archives the whole Web?
- Are publications in the Archive subject to legal deposit?
- What kind of resources does the Archive contain?
Questions relating to description and resource discovery
- How are resources in PANDORA described (resource discovery metadata)?
- What other kinds of metadata are kept for items in PANDORA?
Questions relating to access
- Who can have access to the Archive?
- How do I gain access to the PANDORA Archive?
- How does PANDORA cater for commercial publications?
Technical questions
- Does PANDORA use persistent identifiers?
- What is the technical infrastructure for PANDORA?
- How can I find out more about the PANDORA Digital Archiving System (PANDAS)?
- How does PANDORA attempt to deal with dynamic sites, such as databases and other Deep Web sites?
Questions about preservation
What is PANDORA?
PANDORA, Australia's Web Archive, is the name used by the National Library of Australia for the Archive of Australian online publications and web sites that it builds in conjunction with nine other participating collecting agencies . It is also the policies, guidelines, procedures and technical infrastructure that supports the Archive.
PANDORA is an acronym that encapsulates our mission - Preserving and Accessing Networked DOcumentary Resources of Australia.
Where is PANDORA based?
PANDORA is based at the National Library of Australia, Canberra, Australia.
It is stored centrally on a server at the Library at http://pandora.nla.gov.au/index.html
When did work on PANDORA begin?
Work began on defining selection guidelines in January 1996. The first titles were archived in October of that year. For more information about PANDORA's history, see History and achievements
Who builds the PANDORA Archive?
The Archive is built by the National Library of Australia and nine other participating agencies .
All of the participants allocate staff to the tasks of selecting, archiving and cataloguing titles
Within the National Library, the Web Archiving Section is responsible for administering the Archive, supporting other participants, providing input to the ongoing development of the PANDORA Digital Archiving System (PANDAS), as well as the central tasks of selecting, archiving and cataloguing titles.
Approximately 11 Full Time Equivalent (FTE) operational staff from all partner agencies perform the tasks necessary to get titles into the Archive. An additional 2.25 FTE staff from all partner agencies perform managerial and administrative functions.
How is the Archive funded?
All of the participants fund their contributions to the PANDORA Archive from their ongoing operational budgets. This includes the National Library, which bears the expense of administering the Archive, storing and preserving it, and developing and maintaining the technical infrastructure.
No additional funding has been received from government to support this essential, though costly, activity.
How do participants contribute to the Archive when they are remote from the National Library?
All participants undertake the tasks necessary to archive titles in PANDORA using web-based software developed by the National Library. The PANDORA Digital Archiving System (PANDAS) enables all the workflows and processes to be carried out at work stations remote from the National Library.
What is the scope of the Archive? Do you have selection guidelines?
PANDORA is a selective archive of significant Australian online publications and web sites considered to be of long-term research value. Each of the participants publishes selection guidelines, which define the portion of the Australian web domain for which it will take responsibility.
The National Library aims to archive titles of national significance, while the State libraries aim to archive those of State and regional significance. The National Film and Sound Archive takes responsibility for sites relating to music and film; the Australian War Memorial archives sites relating to Australian military history; and the Australian Institute of Aboriginal and Torres Strait Islander Studies archives publications and web sites of our Indigenous peoples.
The Archive contains only a small proportion (estimated at less than one per cent) of the Australian web domain.
To be selected for archiving in PANDORA, a significant proportion of a work should:
- be about Australia ; or
- be on a subject of social, political, cultural, religious, scientific or economic significance and relevance to Australia and be written by an Australian author; or
- be written by an Australian of recognised authority and constitute a contribution to international knowledge.
It may be located on either an Australian or an overseas server. Australian authorship or editorship alone is insufficient grounds for archiving. Significant content is the pre-eminent factor determining selection.
In addition to the selective Archive that has been the focus of attention since 1996, the National Library undertook a harvest of the entire Australian web domain for six weeks during June and July 2005. The Library contracted the Internet Archive to do the work and 185 million unique documents were harvested from 811,523 hosts, and 6.69 terabytes of raw data were collected. In the absence of legal deposit provisions for online publications, it is uncertain at this stage what access can be provided to these documents.
How frequently do you capture web sites?
It depends on the nature of the web site or publication, particularly its publication schedule, the value of the content, and how stable and long living the site is likely to be. Monographs need to be captured once only. There is at least one serial that is archived on a weekly basis. In the case of certain events, such as the Sydney Olympics, sites may be archived on a daily basis. Decisions are made on a case by case basis.
At this stage the PANDORA Digital Archiving System (PANDAS) does not enable harvesting of a given item more than once per day.
Do you attempt to collect all levels of each web site?
It depends on the web site. Often we would collect the entire web site. In the case of a very large web site, for example, the web site of a government department, we may select only a part of a site containing information about a particular initiative or program. Sometimes we select only the publications from a very large site, for instance, an e-journal or newsletter, or scientific or technical reports.
Do you archive external links?
No, we archive only the links that belong the directories of the site itself. We do not archive links that lead to other sites. This is mainly because we do not have permission to archive the other sites.
Do you receive publications directly from publishers? If so, how?
Usually publications are obtained by using a harvesting robot that downloads copies of the publications from the publishers' web sites. (This is done with the permission of publishers.) In some circumstances, however, publishers email their titles to us. Occasionally, when a site is large or particularly complex we will ask the publisher to send it to us on disk.
Why did the National Library of Australia choose to take a selective approach to archiving?
There are two main approaches taken by national libraries to collecting and preserving their online documentary heritage:
- the selective approach taken by the national libraries of Australia, Canada and Japan; and
- the whole domain snapshot approach of national libraries including Sweden, Norway, Finland, Iceland and Austria.
There are advantages and disadvantages to both approaches and for a discussion of what these are see Sections 3-5 (pages 10-12) of Collecting Australian Online Publications. The ideal situation would, perhaps, be a selective archive supplemented by periodic snapshots of the entire domain. The National Library of Australia undertook its first whole domain harvest over six week period in June and July 2005.
Other libraries are adopting hybrid approaches. The Library of Congress' MINERVA contains both a selective archive and thematic archives that are a comprehensive snapshot on a particular theme, such as the US election of 2000 and the events of September 11, 2001. In Denmark, the two national libraries, (The Royal Library and the State and University Library) have a pilot project with a muti-pronged approach that consists of three different types of harvests: whole domain harvest of the .dk domain four times a year; selective, high-quality harvest of about 80 sites; and event harvesting of two or three events per year. The Bibliothèque nationale de France is attempting to program a robot to archive both automatically and selectively those resources likely to be of research value.
The Library chose the selective approach because it enables it to achieve six important objectives:
- Each item in the archive is quality assessed and functional to the fullest extent permitted by current technical capabilities;
- A gathering schedule can be individually tailored for each selected title, taking into account its publication schedule or the frequency with which the web site changes. This enables the content gathered to be as comprehensive as possible;
- Each item in the archive can be fully catalogued and therefore can become part of the national bibliography;
- Each item in the archive can be made accessible immediately, owing to the fact that permission to make publications available to the public via the web have been negotiated with the publishers;
- The 'significant properties' of individual resources and classes of resources within the archive can be analysed and determined. (These are the attributes that convey the full meaning and intellectual content of an item and enable it to be experienced as the author intended.) This enhances our knowledge of preservation requirements and enable strategies for preservation to be put into place.
- Sites that are inaccessible to harvesting robots can be identified and archived using other methods, by arrangement with the publisher.
Do you archive the software (i.e., the plugins) or just the text and images?
The plugins and other software are not being archived in PANDORA. This is a task that is beyond the resources of the National Library of Australia. We are very interested in work of initiatives such as the Global Digital Format Registry and see this as an area for collaboration with other institutions.
How large is the Archive?
Check the Statistics page for up to date figures on the size of the Archive.
Why does the National Library build PANDORA when there is the Internet Archive that archives the whole Web?
The Internet Archive is an important resource. It attempts to capture the entire Web over and over again to record for posterity as much of the Web as possible.
However, because it is an archive that is built from automatically harvested snapshots it lacks the advantages of a selective archive. For instance, there is no quality assurance applied to any of the resources. It would be impossible in an archive the size of the Internet Archive. The result is that many documents are incomplete or lack functionality. This is not acceptable for Australian publications of national significance. PANDORA participants assess the quality of each title harvested before it is added to the Archive.
The Internet Archive harvests without copyright permission from publishers. This means that their harvesters must follow robot.txt rules. If the site says 'don't index me', it will not be gathered. This means important resources will be missed. Commercial publications, dynamic sites, including those structured as databases (part of the Deep Web), will all be missed by the Internet Archive's crawlers. PANDORA participants negotiate with publishers to include these categories of publications and web sites in the Archive.
Because the Internet Archive attempts to archive the entire Web every two months, it tends to skim over the top and does not collect everything that is available. For instance, in November 2004 during its usual sweep of the Web, the Internet Archive harvested approximately 20 million documents from the Australian web domain. However, during a six week period under contract to the National Library of Australia, when it focused exclusively on the Australian web domain, it harvested 185 million unique documents.
The PANDORA Archive is a carefully developed collection of Australian web resources that are catalogued and become a part of the national bibliography in line with the recommendations of the International Federation of Library Associations (IFLA). A lot of effort is put into creating multiple pathways for resource discovery - through participant's catalogues, through lists on the PANDORA home page, and via a search engine on the home page. Search engines such as Google and Yahoo! also index PANDORA to the title level. It is essential that Australians are easily able to discover their documentary heritage.
The Internet Archive is an important source of online information for Australian researchers who will find publications and web sites there that are not in the PANDORA Archive. However, it cannot stand instead of PANDORA, Australia's Web Archive. The National Library, the State libraries and other cultural collecting institutions have responsibilities for the preservation of our cultural heritage. Preservation of online publications requires active and sustained intervention into the indefinite future. The preservation of significant Australian documentary heritage cannot be left to an overseas organisation over which we have no control, no matter what their present fine and altruistic intentions may be.
Are publications in the Archive subject to legal deposit?
No, legal deposit legislation in Australia does not yet cover online publications. All participants negotiate with publishers for permission to copy titles into the Archive.
What kind of resources does the Archive contain?
PANDORA contains a wide range of publications and Web sites. High priority is placed on collecting government publications and academic e-journals. In addition there are many other types of sites. The following are just a few examples in a few categories:
Cultural activity
Community concerns
Scientific standards and research
- Asbestos: Code of Practice for the Safe Removal of Asbestos
- Qualitative Research Journal
- Strategic Issues for Australian Gene Technology
Politics and government
Indigenous peoples
- Native Title Conference, held Adelaide University, July 2001
- Reconciliation Australia
- Zero Tolerance Policing: Its Background and Implications for Aboriginal People
Sport
People
There are sites of well known as well as 'ordinary' Australians:
- Ian Thorpe (sportsman)
- Pauline Hanson (politician)
- Kate Ceberano (musician)
- Albert Chapman (mineralogist)
- Bob Buick (Vietnam veteran)
- Nancy Crick (euthanasia campaigner)
- Trishan Ponnamperuma (11 year old)
A variety of online formats
PANDORA contains multimedia and dynamic formats, as well as text like documents.
- Ngapartji's virtual writers in residence was archived during 1998 and 1999 and includes some early multimedia. It is no longer available from the publisher's site.
- Official 1998 Mardi Gras netcast documents the first netcast of the Mardi Gras in Sydney. Some of the links do not work, but the videos of the netcast are there. It is no longer available from the publisher's site.
- Bangarra Dance Theatre contains beautiful video clips of some of the company's dances.
- OnSecure is a dynamically generated database site. In the Archive it is stored as static pages.
How are resources in PANDORA described (resource discovery metadata)?
Titles in the Archive receive full level cataloguing, which is undertaken according to the PANDORA Archive Electronic Cataloguing Manual.
Principal cataloguing tools used are:
- AACR2, 2nd ed., 2002 revision, Chapter 9, "Computer files" and relevant CSBs;
- CONSER Cataloguing Manual, Module 31, Remote Access Electronic Serials (Online Serials);
- OCLC Cataloguing Internet Resources: A Manual and Practical Guide;
- OCLCCataloguing Electronic Resources: OCLC-MARC Coding Guidelines, revised July 1999;
- Library of Congress Guidelines for the Use of Field 856, revised March, 2003;
- Library of Congress Guidelines for Coding Electronic Resources in Leader/06, revised December 2003;
- Library of Congress Use of Fixed Fields 006/007/008 and Leader Codes in CONSER Records;
- Library of Congress MARC21 Format for Bibliographic Data : Field List;
What other kinds of metadata are kept for items in PANDORA?
Administrative metadata is created and kept in the PANDORA Digital Archiving System (PANDAS) for all publications and web sites in PANDORA. This includes contact details for the publisher, date that permission to archive was granted, frequency of harvesting, and any restrictions that have been placed on access. For further detail see the PANDAS Manual.
Some preservation metadata is collected automatically by PANDAS, though this is currently inadequate for long-term preservation needs. The National Library plans to make enhancements to the PANDAS software, including increasing the range of preservation metadata it collects and provide a user interface to enable managers to access the metadata more easily.
Who can have access to the Archive?
Anyone in the world with an Internet connection can have access to the PANDORA Archive. Most of the titles are freely available to all.
There is a very small proportion of titles which are restricted for commercial reasons, or because their content is sensitive. Most of the commercial publications are available for access via a single PC in the reading room of the National Library.
How do I gain access to the PANDORA Archive?
Access to the contents of the PANDORA Archive is available from the PANDORA Home Page. A number of access paths are available:
- an alphabetical list of titles on the PANDORA Home Page;
- a list of titles by subject on the PANDORA Home Page;
- a search facility on the PANDORA Home Page; and
- hotlinks from catalogue records for each title or collection in the National Bibliographic Database (Kinetica), the National Library's online catalogue, and other participants' online catalogues.
Commercial search engines such as Yahoo! and Google also index the Archive to the title level.
How does PANDORA cater for commercial publications?
The PANDORA Digital Archiving System (PANDAS) enables collection managers to restrict access to titles archived in PANDORA. Titles can be restricted for a period of time, such as one year from date of archiving; restrictions can be set to expire on a set date, such as 1 August 2006; or access can be controlled by password.
Participants negotiate with publishers to determine an appropriate period for restriction, usually the period during which the item is commercially viable. During the period of restriction, most publishers permit us to make the title available on a limited basis, on a single PC in the main reading room of the National Library or other participants. Electronic copying and email are disabled on these PCs, though a print copy can be made.
Only a very small proportion of titles in the Archive are restricted. Most are freely available to anyone with an Internet connection.
Does PANDORA use persistent identifiers?
Yes. The National Library is committed to providing persistent access to items in PANDORA and its other digital collections. To this end in 2000 it commissioned a consultant to advise on the most advantageous approach to persistent naming and resolution. As a result of the consultant's report, Report on a Consultancy conducted by Diana Dack for the National Library, May 2001, a persistent identification scheme and a resolution system were implemented.
When a publication or web site is archived in PANDORA, the Digital Archiving System (PANDAS) automatically allocates a unique persistent identifier to it, and this is recorded towards the bottom of the title entry page. For example, see Safety Science Monitor, "To cite this page use http://nla.gov.au/nla.arc-13139."
As well as providing a persistent identifier at the title level, the system also creates one for all of the component parts, for instance, for an article within a issue of an electronic journal, or for an image or a table within a web site. The persistent identifier for any part of a title that a researcher may wish to cite can be ascertained by using the citation service. This is available towards the bottom of every title entry page, just under the persistent identifier for the title.
To obtain the persistent identifier for any part of title, with the part that you want the persistent identifier for open in your browser, highlight and cut the PANDORA URL from the address box and paste it into the text box for the citation service. Click on the Go button. The persistent identifier for the part will then be displayed.
The persistent identifier will always point to the resource it identifies and it can be cited with confidence that the link will never break.
What is the technical infrastructure that supports PANDORA?
For a detailed description of the PANDORA technical infrastructure, see the paper by Steve McPhillips.
How can I find out more about the PANDORA Digital Archiving System (PANDAS)?
The PANDORA Digital Archiving System (PANDAS) has been developed by the National Library of Australia to support its work flows, processes and metadata requirements, as well as those of other participants, in building a selective web archive. It is web-based software that enables collection managers in various geographic locations to contribute to a central archive. The PANDAS Manual provides information about PANDAS functionality and the workflows that it supports.
How does PANDORA attempt to deal with dynamic sites, such as databases and other Deep Web sites?
The primary method for collecting copies of Australian online publications and adding them to the archive is via the harvesting robot that is part of the PANDORA Digital Archiving System (PANDAS). To access a target site, harvesting robots need to be able to navigate HTML links. An increasing number of publications and web sites are structured as databases, or have other interactive or dynamic content that harvesting robots can not deal with. Typically these sites require the entry of terms into search text boxes, or the selection of options from a drop down box.
In 2003, the National Library of Australia commenced a research project as part of its membership of the International Internet Preservation Consortium (IIPC) and leadership of the Deep Web Working Group. As a result, the National Library developed the Xinq tool which provides a generic interface to data deposited with the Library by publishers. Xinq has been made available open source through SourceForge.
Is there a long-term preservation policy for the contents of the PANDORA Archive?
Yes. The National Library of Australia is strongly committed to ensuring long-term access to all its digital collections, including the PANDORA Archive. To this end it:
- has formulated a Digital Preservation Policy
- has undertaken a risk assessment of its digital collections, with special focus on PANDORA; and
- continues active involvement in research both in-house and in conjunction with other organisations, including OCLC (the Online Computer Library Center), and the International Internet Preservation Consortium (IIPC).
