PANDORA Business Process ModelThe National Library of Australia is developing an electronic archive called PANDORA to provide long term preservation of and access to significant Australian online publications. A dedicated online repository will be established. Electronic objects contained in it will be maintained on an ongoing basis and converted to new formats as standards demand.
1.1 PANDORA objectives
1.2 Service model and data modeling
1.3 Online electronic publications
1.5 The Electronic Unit
1.6 Components of the Business Process Model
1.7 Relationship to other NLA systems
4.1 Addition of new publications to the archive
4.2 Enabling access to the archive
4.3 Updating the archive
4.4. Modifying a publication in the archive
4.5 Deleting a publication
4.6 Management of user access screens
5.1 PURL resolver service
5.2 Standards for metadata
5.3 Administrative metadata
5.4 Standard storage formats
5.5 Software archive
5.6 File structure for the digital objects within the PANDORA archive
The PANDORA project has, as its basic objectives, to:
- identify, select and catalogue significant Australian online publications using established guidelines
- arrange with the publishers of these publications to capture a copy of their publication for preservation in the National Library's electronic archive
- make the information available to users in line with fair dealing provisions
- take into account the publisher's commercial interests with regard to provision of access to remote users
- preserve the "look-and-feel" of online electronic publications
- update the information in the archive on an ongoing and systematic basis, while maintaining date stamped previous "editions"
- convert publications in the archive to new formats as standards change.
There are wider ramifications of the PANDORA project. Associated developments include:
- contribution to discussions on extending copyright and legal deposit to online publications
- implementation of a system of permanent naming for Australian publications to overcome the location problems associated with unsupported links
- implementation of a system of describing archived documents based on the Dublin Core attributes, to make online searching for information more efficient
- cooperation with other Australian libraries to establish a national digital archive.
Work on PANDORA has been proceeding at two levels:
- establishing a proof of concept archive
- establishing the specifications and operation of the final production archive.
In order to proceed with the design of the full archive system, a service model was prepared to illustrate how the archive might operate. This model is given at Attachment A.
A small team worked further to develop a business process model based on the service model. The PANDORA Business Process Model (BPM) was derived by a National Library of Australia (NLA) team consisting of Wendy Smith [PANDORA project manager], Margaret Phillips [Manager, Australian Electronic Unit], Leanne McKinnon [Senior Librarian, Electronic Unit], Deb Woodyard [Preservation Services Branch] and Debbie Campbell [PANDORA I.T. Research Officer] during a series of workshops in the first half of 1997. Input was also gratefully received from Judith Pearce, Margaret Kennedy, Richard Stone and Jasmine Cameron.
Workshops were held to discuss selection processes, ephemera, technical evaluation, filing, management of the archive, permanent naming, copyright/access and fair dealing, archiving and report generation.
PANDORA does not sit in isolation from the rest of the Library's business. There are many developments within the Library which have a close relationship to PANDORA - the Systems Environment Review Group (SERG) report, the Networked Services Project (particularly NSP Stage 2), the MetaWeb project and the proposed PURL Resolver Service. SERG has provided an overarching architectural framework within which the PANDORA infrastructure should reside.
The PANDORA archive will hold a variety of electronic publications, some of which currently emulate print form. The archive also holds types of publications which do not have any print equivalents, such as organisational and personal sites. The currently identified types of publications can be classified as follows:
- monographs - with a fixed content as in a traditional print publication
- cumulative or evolving monographs, whose contents change over time
- regular serials - issues appear sequentially in traditional print publication patterns
- evolving serials, whose contents change over time
- newspapers sampled in snapshots
- organisational and personal sites.
Over time, electronic types may change to reflect enhanced use of the online environment.
While most Australian online publications are currently available free of charge, some electronic publications lodged in the Library's collection have (or will have) an associated fee, for example, a pay-per-view requirement, a subscription, or a "purchase" price per issue. The Library does not intend to pass these fees on to the user.
Under the Copyright Act of 1968, electronic publications are NOT subject to legal deposit in the National Library of Australia. If changes to the Copyright Act lead to legal deposit for electronic materials, this will have some impact on the operation of the archive, particularly with respect to negotiation with publishers.
In 1995, the National Library made available to all publishers a Voluntary Deposit Deed for tangible publications (publications on discrete physical media such as CD-ROMs and diskettes), in lieu of Legal Deposit provisions. A similar Voluntary Deposit Deed has been drafted by the National Library for publishers of online publications, and is available by request from the Electronic Unit (at firstname.lastname@example.org).
The Electronic Unit was set up in April 1996 to assist in the development and implementation of policy and procedures for ensuring long term access Australian online publications. It is responsible for all aspects of the management of these publications, including identification, selection, negotiation with publishers, cataloguing and archiving.
The Business Process Model is divided into 5 components, identified as:
- the user access component
- the publication selection component
- the archive management component, including copyright management
- the report generation component
- the filing infrastructure - for the day to day management of correspondence files.
These components comprise the PANDORA archive's business processes.
The PANDORA archive sits within a broad framework of other Library applications:
- A National Bibliographic Utility (NBU) record may contain more than one electronic location, , but only one will be a link to the PANDORA archive.
- An OPAC record may contain more than one electronic location, but only one can be selected at any one time. A publisher may or may not provide a link back to the NLA site.
- Currently, NBU records are catalogued with URLs only. When PURLs are deployed in the future, an NBU search request will also be diverted through the PURL resolver.
Metadata may be recorded in the archive management facility at any time throughout the identification, selection, evaluation and archiving processes.
The Electronic Unit regularly scans the Internet to identify publications in scope for preservation by the Library. Suggestions also come from other members of staff. Other triggers - such as the request for an allocation of an International Standard Serial Number (ISSN) or a Cataloguing In Publication (CIP) from a publisher - also bring publications to the attention of the Electronic Unit.
When a publication is identified, the first task is to check the publication's details. In order to determine suitability for archiving, the following questions are asked:
- has the publication has already been assessed ?
- is there a print or other version already available in the NLA collection?
- does the publisher produce an equivalent print-based publication?
The process of checking the details will result in:
- highlighting the publication as having potential for preservation, and assessing it against the SCOAP guidelines
- rejecting the publication for preservation, based on the answers to one or more of the above questions.
The results of checking the details will be recorded in PANDORA's management facility, to ensure that the status of any publication is known at any time. Information will also be made available to other staff and agencies.
Electronic Unit staff assess each publication against the SCOAP guidelines.
There are three possible outcomes:
- the publication is accepted for archiving
- the publication is rejected
- the publication does not contain sufficient information to make a decision. It is placed on hold while the publisher is contacted or the site is monitored.
The appropriate decision is recorded in the management facility. Where a publication is rejected, the reasons for its rejection are noted.
If the publication is selected for preservation, the following actions occur:
Where a unique identifier already exists for a publication, for example, an ISBN or an ISSN, it will be noted in the publication's catalogue record and in the archive management facility. In addition, each version, edition, or issue of a publication will be assigned a Persistent Universal Resource Locator (PURL) and an identifier internal to the archive. The former will permit some permanency of the location of the publication and the latter will allow archive managers to quickly retrieve a digital object from the archive. Its use may be extended in the future to distinguish PANDORA publications from those held by other archives in the distributed national model.
New identifiers, such as the Digital Object Identifier (DOI) and handles, can also be noted when they are provided with an publication's metadata.
Descriptive metadata are noted for the publication. These include:
- identifier, such as ISBN or ISSN.
- frequency of publication
- life date
- other notes, for example, if a subscription is payable; or any restricted access conditions.
For more detail, please refer to the Logical Data Model.
Negotiation with the publisher is conducted to seek permission for archiving, to ascertain publishing intentions (for example, whether content is deleted) and to set initial access levels to the publication. At this point, if the publisher does not respond, then the publication is filed for future follow-up action.
To date, no outright refusals have been encountered, but if this should occur, the information would be noted.
Any communications between the publisher and the National Library are filed in the Library's registry file system.
Electronic Unit staff create a record on the NBU for the publication and include the descriptive metadata. Cataloguing is based on the AACR2, Library of Congress and ABN Standards Committee guidelines. The cataloguing rules permit the PURL to be placed in the 856 tag and in the TED field. It is not classed as a holding and placed in the 984 tag, as there is no concept of a holding for electronic records (although PANDORA changes this). Intention to archive is placed in the 583 (NTA) tag. PANDORA-specific metadata will not be recorded in the cataloguing record.
For serials, the Web Server team in the Systems Projects Branch is notified of the addition of the publication to the Library's collection, in order to include it in the Australian Journals Online site on the NLA's Information Server (http://www.nla.gov.au/oz/ausejour.html).
After a publication has been identified and selected for preservation within the National Library, a technical evaluation is carried out.
Prior to contact with the publisher , the following metadata are recorded about the publication. This metadata may be augmented during the publisher negotiation process:
- publisher's (P)URL
- current issue, and whether it is an evolving serial
- date of evaluation
- back issues, including details of archived material or previous issues, and any changes, for example whether back issues "disappear" from the publisher's site
- copyright provisions, i.e., note the publisher's copyright statement and whether it remains with the publisher or individual authors
- other access conditions requested by the publisher
- format of the contents, for example, if a document has been written in HTML or another text-based markup language, this information may be available in the underlying source code; in file extensions or by scanning
- file size
- type of access (generally via the World Wide Web)
- links. The publication is explored to understand its structure.
- upward links, i.e., does the publication depend on a link to a higher level ? and whether there is useful information at the higher level which should be captured
- email - (online) address
- contact details including postal address, telephone number, contact names
- proposed gathering schedule
- other metadata.
If the publication is a serial, some additional access information may also need to be captured. For example:
- whether the publication could be sent regularly to an NLA e-mail address
- any password for the publisher's site which is sent by the publisher to the NLA permitting the publication to be downloaded from the external site
- if the publisher has agreed to notify the NLA when each issue is posted (although this is less likely to be the case for PANDORA which intends to archive issues less often than the frequency of distribution of each issue).
This component comprises functions relating to users gaining access to the PANDORA archive.
The likely users of PANDORA have been identified as:
- government departments
- policy makers
- private enterprise
- the curious public (via the Web)
- serious researchers
- publishers (to check on the status of their publications)
- other libraries, both Australian and international
- NLA staff, including:
- Information Services
- Technical Services, including the Electronic Unit
- Systems Projects
- Document Supply Service
- Preservation Services.
The PANDORA service model indicates that the only necessary entry point for user access to the PANDORA archive is via the WebPAC (the Online Public Access Catalogue's Web-based interface). However, it has also been suggested that access should also be provided through other Web browser services including the NLA's information server. To accommodate both strategies, the infrastructure will be very carefully considered. It is expected that a range of access points will eventually be provided by the Networked Services Project infrastructure.
The user will be presented with a standardised title entry page for each publication which will have, where possible, features in common with the Library's WebPAC or other standard Library screens.
The requirements or special needs of these users above and beyond what the OPAC offers as a search tool will also need to be met. For example:
- Technical Services will need access to the PANDORA archive in order to obtain information not accessible to the user in relation to the management of the electronic archive
- Preservation Services will maintain metadata for tangible items in the Library's collection
- in addition, Preservation Services will require information about the archive's contents on a semi-regular basis in order to make decisions about preserving the physical manifestation of each electronic publication which has been archived
- publishers who need to see a list of all of their electronic publications which have been archived.
- direct access to the archive may be supported for users coming from the publisher's site to PANDORA via the publisher's hot link if it is installed.
Access to the PANDORA archive will be free of charge to the end user. However, there will be occasions when it is necessary to restrict access to some materials in the archive, either to all users or to some specific groups of users. There are three main categories of material that will be liable to have restrictions imposed on access:
- material which has a current commercial value to the publisher
- material which is subject to exclusive rights for a specified duration, at the request of the publisher and
- material which has some temporary legal embargo.
In addition, the following requirements will be met:
- publications which are not commercially viable will be accessible free of charge to both internal and external users
- fee-based publications, whether acquired by Voluntary Deposit or subscription, may be temporarily available to in-house readers only
- publications which are not commercially viable will be accessible free of charge to both internal and external users
- any publisher-imposed restriction on access to a publication will be negotiated with the publisher to achieve a standardised approach to access restrictions in the archive.
- any restriction imposed on access to a publication will be described in its title entry page
- where a publisher insists on a fee for access to a commercially viable publication or a specified version or issue of it, the reader will be directed to the publisher's site.
Negotiations will be conducted with publishers to achieve standard conditions of access. Because the primary purpose of the PANDORA archive is to ensure long term access to publications, it is not a major concern if a publisher wishes to impose short term restrictions while the title is available on the publisher's site.
Should a publisher require restricted access to a publication, or a version or issue of it, a standard three year period of restriction will be applied. If necessary, a longer standard period of five or ten years could be negotiated. (If individual publishers cannot agree to the Library's conditions, their material may be excluded from the PANDORA archive. This will depend in part on the nature of the publisher's conditions and the importance of the title.)
Where the Library negotiates voluntary deposit of a publication, we would expect to be able to provide immediate access to internal users, and would negotiate a three year (preferably) or five or ten year restriction on external users while the item is still available on the publisher's site.
Once a publication has been archived, it would not be withdrawn from the archive in the event of the publisher withdrawing permission, except for extenuating legal circumstances.
To enable the archive management facility to manage automatic release from restriction, an access profile will be determined for each publication's issue, edition or version, based on the publisher's requests and the NLA's preferred conditions.
|unrestricted publications (1)||partial commercial restriction (2)||full commercial restriction (3)||full|
|- no NLA subscription||NLA subscription||- no NLA subscription||NLA subscription|
|internal user||access to all issues||access to all issues||access to all issues||no access for 3 (or 5 or 10) years||access to all issues||no access|
|external user||access to all issues||no access for 3 (or 5 or 10) years||no access for 3 (or 5 or 10) years||no access for 3 (or 5 or 10) years||no access for 3 (or 5 or 10) years||no access|
1. Unrestricted publications are most likely to be those freely available on the Internet. Some fee-based titles may also fall into this category.
2. Publications under partial commercial restriction are likely to be those available on the Internet on payment of a fee to the publisher. The Library will encourage publishers to deposit titles and to permit free access to both internal and external users. It is anticipated, however, that the majority will permit free access to internal users only.
Where an online title is likely to be in strong demand for the provision of information services, the Library may also subscribe to or purchase the right to access the publisher's site.
3. Full commercial restriction on a title is likely to occur when, for a specified period of time, a publisher requires that no access is given to current issues/versions in the archive, in order to protect vested commercial interests. If access to a title is considered important for the provision of information services, the Library may subscribe to or purchase the right to access the publisher's site.
4. Fully restricted access for legal reasons could apply in a situation, for instance, where a publication is the subject of a court case or is banned.
Other types of access, such as a "reproduction" capability within PANDORA for printing, downloading, faxing and saving to diskette, are considered to be beyond the scope of PANDORA's role. This will be provided by default to external users within their browser software and will be provided to internal Main Reading Room users by a combination of the browser software and the mechanisms used for the physical format electronic materials.
The major activities of archive management fall under the following headings:
- adding new publications to the archive
- enabling access to publications in the archive
- updating publications in the archive
- modifying publications in the archive
- deleting publications from the archive.
The initial tasks required of the archive manager are to:.
- create a catalogue record on the NBU
- file all related correspondence (see section 7 for more detail)
- check the PURL resolver service to ensure new publications have been registered (see part 5.1 for more detail)
- collect the publisher's metadata (see part 5.2 for more detail)
- create other PANDORA archive metadata (see part 5.3 for more detail)
- determine the most appropriate file structure for the object and its versions (see part 5.6 for more detail)
- activate the harvesting software. Currently, the URL indicating the location of the title of the publication is provided to the harvesting software. If an error occurs during the harvesting process, a report is produced for the Electronic Unit by IT Facilities Branch (see part 4.7.2 for more detail)
- create the title entry page (see part 4.6.1 for more detail)
- inform the publisher of the new link's availability.
Tasks allowing the publications in the PANDORA archive to be made accessible to the reader include:
- adding a new publication to the search engine's indexes
- capturing the software that supports the viewing of the publication, for example, plug-ins, decompression utilities and placing it into a repository (see part 5.5 for more detail)
- testing the captured software with the publication
- converting publications from non-standard to standard data formats if required (see part 5.4 for more details)
- setting the publication's access profile.
The updating tasks form an ongoing workload for the Electronic Unit staff. These tasks may include:
- tracking missing issues of e-journals
- adding new volumes and issues to a captured electronic journal
- adding new "editions" or versions to a captured monograph
- adding new versions to a captured serial
- converting the digital objects if non-standard (see part 5.4 for more detail)
- updating the metadata (for both internal use and reader use) to reflect the addition of new content or other information about the object
- reviewing the subsequent snapshots of captured digital objects against original snapshots held in the archive
- updating the title entry page details, for example, to advise that links to payment options or a publisher's e-mail have been de-activated.
The modifying tasks form an ongoing workload for the Electronic Unit staff and are necessary to assure the quality and completeness of the content and framework of each captured digital object (where each publication is comprised of many discrete digital objects).
These may include:
- de-activating payment functions, such as an online subscription
- removing links for corresponding with the publisher (unless the publisher can supply the Library with a link free copy of the publication)
- removing duplicated content (i.e. content which may have been harvested twice) by deleting single URLs which represent only a part of each digital object
- de-activating pull-down prompts for software plug-ins so that the version in the repository is always accessed
- deleting unwanted or extraneous parts of publications.
Deletion is defined as removing whole publications from the archive. This may be necessary for several reasons, including:
- harvesting errors
- legal judgments which mandate deletion
- discovery of a print version of a preserved electronic object. Preference is given to the print version in the Library's collection.
Associated tasks include:
- removing all content from the PANDORA archive under the title-level PURL (by deleting its URLs)
- updating the metadata in the PURL resolver service to de-activate the publication's PANDORA PURL (the publisher's PURL will remain)
- updating the metadata in the PANDORA archive management facility to reflect the status of the digital object.
A title entry page is created by Electronic Unit staff for each publication accessed via the WebPAC (the Web-based interface to the Online Public Access Catalogue), and is the first screen the reader will see after selecting a publication. However, where a publication is accessed from another Library system, the reader will be taken straight to the publication. The reader will be made aware that they are in the PANDORA archive by the presence of a banner.
A sample title entry page is included below. Subsidiary screens, such as displays of error messages, will follow the standard screen design of the title entry page.
For each publication, the following information is to be displayed where appropriate:
Preserving and Accessing Networked Documentary Resources in Australia
This title is archived by the National Library of Australia as part of a project
to ensure long term
access to Australia's documentary heritage online
Australian Humanities Review
Gathering Schedule: This title is archived at three monthly intervals.
The publisher's site may
provide more current information.
Holdings in the PANDORA archive:
Home Page last updated on Jul 1 07:55
Current Issue last updated on Jul 1 07:55
1997, Issue 1, March last updated on Jun 26 22:43
1996, Issue 4, December last updated on Jun 26 22:43
1996, Issue 3, September last updated on Jun 26 22:37
1996, Issue 2, July last updated on Jun 26 22:36
1996, Issue 1, April last updated on Jun 26 22:37
Copyright in all manifestations of this publication, both within the PANDORA
archive and on the
publisher's site, is held by the author and/or publisher of the publication. For information about
copyright ownership in each case, please refer to the publisher's copyright statement, usually found
within the publication, or contact the author or publisher.
Permission to copy or quote extracts from a publication must be sought from
the copyright owner
when it is proposed to copy or quote a portion of the publication exceeding that permitted by the
author's or publisher's copyright statement. If the permitted use is not specified, you are asked to
note the fair use provisions set out in sections 40-43 of the Copyright Act 1968.
Where the publisher's or author's copyright statement grants more limited rights
than those permitted
by the fair dealing provisions of the Copyright Act, the fair dealing provisions will prevail.
Publisher's copyright statement
Internet publications can change frequently and the appearance and functionality
of the version that
has been gathered in this archive may not be exactly the same as is currently available on the
publisher's site. While every attempt has been made to capture the publication as it is on the
publisher's site, a number of factors, including software limitations, may mean that this has not been
The site's internally linked documents only have been archived and links to
other documents and sites
may not be active.
- PURLs must be displayed for all archived material, in order to suppress the associated URL, and to conform to the permanent naming conventions espoused by the National Library. Suppressing the URL display will allow the level of access of a document to be checked, in order to allow conformance to deposit provisions. Cataloguing on the NBU and the OPAC must permit the addition of more than one (P)URL.
- the order of display of the NLA's archive PURL, the publisher's (P)URL, and any mirror site on the title entry page is yet to be determined. However, it is assumed that the publisher's URL appears first in the catalogue record, to encourage readers to visit that site.
- a mirror site is set up by a publisher, for example "Yoni" has its main site in the United States, and an Australian site. One site may be more up-to-date than another. It is also possible that when the PANDORA archive grows larger, the Library may wish to set up mirror sites.
- the publisher's contact details, and any charging incurred by the Library for access to any item, will not be made known to the user.
- for details about the gathering schedule, see part 4.7 for extra detail.
- information will be provided for the user as a brief introduction to the purposes and scope of the archive, and general navigational hints.
- messages may be sent to the system administrator if there are any difficulties regarding access to documents, for example, if the displayed document title page does not appear to match the title initially selected. This scenario is possible if an incorrect (P)URL has been stored as part of the document's metadata.
- holdings may be provided on the title entry screen or on additional screens, depending on the amount of information available for the publication. Over time, it may be necessary to summarise lists of issues chronologically.
- the Table of Contents details are provided by the Electronic Unit and may appear as Volume 1 Issue 10 etc. They are to appear in reverse chronological order, i.e., most recent last (at the end of the Table of Contents). If feasible, it is intended that this information will be automatically updated.
- the title entry page will need to indicate (if true) that not all of the holdings are present when the Library decides to archive only a sample. A corresponding general statement of intent could be provided in the NBU/OPAC record. The selection process will also need to indicate whether the document has been captured for informative and/or look & feel purposes.
- Articles in comprehensively indexed electronic titles (such as those held by the Australian Public Affairs Information Service (APAIS)) will be accessible at the title entry page level, not at article level. However, some article level access will be made available for APAIS articles indexed in journals not archived by PANDORA. An APAIS Miscellaneous Article title entry page will be created in a manner similar to that used for ephemera collections (see Attachment D for more information).
New functionality may be considered at a later date, for example:
The gathering schedule, or time interval between each successive capture, must be managed in its own right.
The gathering schedule shows the date of capture for each electronic document and should give details of frequency, for example daily, weekly, monthly and the date of last gathering.
The gathering schedule does not necessarily match the date of issue of any electronic publication or its objects. PANDORA will capture publications at intervals appropriate to each publication. For example, a journal issued monthly may be captured quarterly or annually. PANDORA must also provide the capability to capture a publication on a one-off occasion such as the demise of that publication or when the publisher advises the Electronic Unit that it is about to undergo a change in its presentation.
It is anticipated that the reader will access the publisher's site for the latest issue of any journal and for any issues not yet captured by PANDORA. To encourage this, the publisher's URL will be displayed above PANDORA's in the catalogue record and on the title entry page in the archive.
Publishers will be encouraged to advise the Library of the availability of new versions, changes to publishing intentions (including demise) and other circumstances, such as changes in schedule. When this occurs, it will be necessary to conduct a manual override of the automated gathering schedule.
For some publications, a one-off capture will be necessary. Examples include Commonwealth Government election results, and selected ephemera. Some online newspapers will be captured on a 9-monthly basis, in order to ensure a wider variety of material.
Currently, the PANDORA archive relies on HARVEST to capture publications on a regular basis.
Publications are harvested in all of the formats provided by the publisher, and will be harvested in full each time. In order to overcome "misrepresentations" of digital objects, it was decided to re-capture all of the site's embedded links and not edit them to impose meaning. All snapshots of a publication will be made available to the reader via the title entry page.
The Electronic Unit performs a check on the captured results, to ascertain whether the harvested digital object is present and correct. If it passes quality control tests, then a link to this new archived object is inserted into the archive via the title entry page.
Some general considerations for the management of the archive - some with wider implications for the Australian library community - are discussed below.
Many PANDORA functions will use the service transparently to the user and the manager. This is described in more detail in the documentation for the service, for example on the OCLC Internet site
When a publisher's site disappears, the PURL resolver service can redirect the (P)URL to PANDORA's archived copy of the digital object or the NBU/OPAC catalogue record may be updated. Disabled links, caused by disappearing URLs and other events, will be brought to the attention of Electronic Unit staff by periodically running a software "spider" against both the PANDORA archive and the PURL resolver database.
Information describing the electronic publications, (which will allow easy access to them and management of them in the PANDORA archive) needs to be captured as smoothly as possible. One means of attaining this is by using a standard set of metadata.
Where feasible, this information will be managed under the Dublin Core standard for metadata. Using this standard gives the potential for automated capture of the information. Attachment C gives some preliminary results comparing the automated Dublin Core generators of OCLC and UKOLN.
The Dublin Core holds the following metadata. Attributes marked 'mandatory' should always be supplied by the publisher.
|creator||author or editor of the publication||Optional|
|format of contents*||for example, for text, whether HTML, SGML, PDF, or other; for images, whether JPEG (jpg extension), GIF, TIFF (tif extension), PDF, etc||Mandatory|
|identifier||for example, ISBN, ISSN, URL||Optional|
|issue(s)||if a serial||Optional|
|publisher contact details||Optional|
|frequency of issue||Optional|
*Online at http://purl.oclc.org
Several options for automated capture are currently under review by the PANDORA team.
the decisions regarding standards for this attribute form one of the cornerstones of successful operation for PANDORA
Administrative and preservation metadata may need to be captured at any time during the maintenance of a digital object. This metadata is in addition to any required under Dublin Core attributes.
Digital object formats support full text, images, text as image, audio and video. There is not one particular standard format which presides over the others at the moment. However, the two major contenders for text based information are the standard storage formats of PDF and SGML - like formats. Publishers are likely to reflect the breadth of choice available by selecting formats which meet their strategic needs and budgets. In addition, they may use several formats within the one publication, for example, a home page in HTML and full text in PDF.
In order to fully support this variety, where several formats seem to peacefully co-exist, and to continue to keep a true representation of electronic materials available to readers, publications are captured in more than one format where they are provided. If a publisher releases a document in HTML only, a PDF version will not be created to provide alternative access for current users. However, the Library does have a responsibility to experiment with conversion to alternative formats.
Where a publisher creates a publication in a format which is deemed to be non-standard, options for the Library are to:
- negotiate with the publisher to provide a copy of the publication in a standard format
- ask for a copy of the software which created the publication
- convert the publication to one of the archive's standard formats.
The issue of whether to convert every object to one or more standard formats for the archive was discussed as follows:
|FOR CONVERSION||AGAINST CONVERSION||KEEPING MORE THAN ONE FORMAT|
|the content of any given object is the only significant reason for its capture||it is also highly desirable to retain the 'look and feel' of objects, in order to convey a true overview of the state of electronic publishing throughout history||FOR
provides true future-proofing against the loss of any one format ( where an object is currently published in more than one format, they are all captured)
the cost of storage, although currently inexpensive, may become costly depending on what platforms the archive is forced to migrate to over time*
|it will be cheaper to maintain and manage all objects in the same format||objects may become boring - there is a possible loss of creativity and functionality||readers are not required to use one browser only - a range of browsers will already be available in the archive||managers of the archive will need to monitor the future of more than one standard|
|it is a form of future-proofing provided the right single format is chosen||maintenance and management of the conversion process will be costly||* where a publisher offers a single non-standard format, then the Library will convert it to one of the archive's standard formats|
|migration costs will be reduced||a conversion path which is valid over time will need to be identified, created and modified, in order to protect objects from technological obsolescence||where a publisher offers several formats, the Library should retain the most commonly used or standard formats, in order to preserve the integrity of the object's structures (and hence the integrity of embedded links), and migrate these formats as changes in standard require it|
The introduction of a new object standard will not always require conversion of publications, as some standards will be backwardly compatible. The decision regarding conversion will be arrived at on a publication-by-publication basis, but will loosely conform to the following groupings:
In order to maintain the full functionality of the archived publications, a database of software will need to be established and updated as needed.
An example is the several versions of the Adobe Acrobat software. Any modified or additional software will need to be captured, approved, made virus-free, and validated before placing in a software repository. It can be obtained automatically by the downloading software. Ideally, the digital object will automatically invoke the repository version when the reader requests it.
The structure of digital objects varies from publisher to publisher. Each publisher creates a unique set of file naming conventions to bring together the disparate parts of a publication. For an example, see Attachment B.
The PANDORA archive must achieve three goals with its file structure:
- provide correct access to publications at all times
- deal with inconsistencies between publishers' file structures
- manage situations where URLs are reused by the publisher.
The file structure for electronic monographs, journals, and home pages are often modest variations of the following models. Both of the models are based on the assumption that the highest level entry point (or top level URL) is represented by a title.
The home page is used in the archive to describe the first entry point to the publication. For example,
for a journal:
Within this file structure, there are variations within publications. For example, a home page and the journal's table of contents or index may be the same digital object. The archived issues may be re-edited by the publisher so that the content is arranged differently over time.
For a monograph:
Monographs may evolve in this environment, for example, a publisher issues a "book" and then adds its reviews or a comments column. In this situation, some digital objects will need to be captured and added to the archive.
While it would be worth capturing this as a significant change, it would not be worth activating Harvest to capture minor single word changes within the text, or an enhanced image, and bringing both editions to the attention of the reader. The administrative overheads of monitoring such small changes are deemed too expensive to sustain.
For an evolving serial, some publishers may restructure their publications by adding or deleting features, columns, reviews etc. Issues may be archived and re-arranged, for example, by resorting and grouping articles in sorted order by author. These activities may change the file structure significantly. The Library will cope with this by requesting new versions at each gathering.
Three sets of users will require reports from the PANDORA management facility:
- PANDORA managers, who will require daily activities to be monitored and reported, including Electronic Unit staff and the IT Services Branches
- other NLA managers, who will require statistical summaries, such as corporate staff and Preservation Services Branch
- other interested parties such as State Libraries and publishers.
Several reports are currently produced in the prototype environment:
|review list||selected publications,
|on demand, held on the ILMS and produced by ILMS staff member||Electronic Unit|
|status of publications||a summary of the status of actions and publisher negotiations for each publication||shared on G drive, produced by various staff||Electronic Unit|
|technical evaluation sheet (form)||a report on each publication||as required, produced by EU manager||Electronic Unit|
|comparison report (form)||compares the captured digital object with the object at its original site||as required, produced by EU staff||Electronic Unit|
|online title list (on PANDORA search page)||a list of the publications held in the archive, shown online||as required||Electronic Unit|
|technical report||shows the space usage of each URL, the number of URLs and files||as required, produced by IT or EU staff||Electronic Unit|
|statistics||as above, plus new &
modified catalogue records, plus hours spent scanning
collected as created
|once per month, produced by EU staff||Technical Services management report, plus the Annual Report|
|publications list||a list of publications held in the archive||produced regularly by the EU staff||State Libraries|
All of these reports or forms must be incorporated into and improved by the PANDORA management facility. Additional reports will also be required, including:
- various archive reports to reflect the status of the archive, for example the number of new archived publications in a given timeframe, the number of publications which have had information changed, the overall number of publications, the backlog of publications which have been captured but linked to access points
- metadata reports, particularly for metadata sent in by publishers
- various reports for preservation which must contain the metadata to allow decision-making to occur regarding the preservation action to be taken. Analysis by formats, age, software used etc by publication will be required
- a list of publications belonging to a particular publisher
- technical reports such as:
- time taken by the capture process
- number of URLs/files captured and space utilised
- lost URLs
- number and list of valid and invalid URLs for each publication
- system back-up failure
- total space usage by archive
- traditional database software reports
- transaction log activity, including counts of the number of times each publication has been accessed (may be analysed further by location of browser). The transaction log could also track the number of users rejected by PANDORA through access restrictions
- PURL resolver lists.
It was agreed that wherever possible, as none of the reports need to be published, most of the information can be made available online. This includes the statistics made available to State Libraries. A few report templates can be created and maintained online for interested parties, so that they do not require access to the PANDORA management facility.
As well as "end-result" reports, there are several PANDORA management activities which will require the digital objects at two sites to be compared simultaneously. These activities occur at various points in the maintenance process. Although not an all-inclusive list, the following activities indicate the kind of functionality required:
- comparing the captured object with the original object at the publisher's site
- comparing the first archived version with the second archived version (both held in PANDORA)
- comparing details held in the current select/reject list with the object at the publisher's site
- comparing the pre- and post-migration version of any object where there has been a change in standard.
There is on-going communication between the Library and the publisher which needs to be captured as a true record of PANDORA digital object creation. This is essential for future action and evidentiary purposes. While not all correspondence need be captured (trivial information may be excluded), it will be necessary to determine what is meant by meaningful correspondence. In this context, it is therefore appropriate to capture details of digital objects rejected for archiving in PANDORA, so that this decision may be re-examined, if necessary, in the future.
As most of the communication is conducted via e-mail, it was thought appropriate to capture it in electronic form. However, as it contains a formal corporate record of communication, it does not need to form a part of the PANDORA archive per se, provided a linking mechanism exists between it and the publication to which it refers.
Communication may take place between the Library and publishers on at least the following occasions:
- during the identification stage, to determine whether a print version of an object may already exist
- during the technical evaluation stage, to determine the publisher's intentions regarding file structure, cost, copyright, publication pattern
- during the cataloguing stage, for example to determine the assignment of an ISBN or ISSN and other matters
- during reporting.
Additional communications with the appropriate agencies can also be captured during the archiving stage, to determine what archives the publisher intends to maintain.
Other communications, such as suggestions by in-house staff for archivable publications can also be captured and linked to the appropriate PANDORA digital object.
A trial took place with TRIM Enterprise, which can store electronic communications in a registry-style file structure. Several possibilities were considered within TRIM's electronic file structure:
- maintaining a separate electronic registry file for each publisher
- maintaining a separate electronic registry file for each publisher's publication
- assigning appropriate file naming conventions to the electronic files. A PANDORA identifier has been assigned, as well as chronological information
- keeping guidelines for file creation accessible. It was proposed that the first file in the PANDORA series, PAN97/1-1, be used for this purpose.
Issues addressed included:
- if creating a separate file for each publication, how is it possible to link each publication with the same publisher ? This may only be done using the contact details attribute in the TRIM file record.
- when a publisher issues information pertaining to all of the publications simultaneously, how can this be stored with each publication ? Either the information is copied "manually" onto each publication's file or a new separate file is created and cross-referenced back to each publication's file. It would be necessary to remember that details about any particular publication could exist in 2 files - a general file, and a specific publication file. However, searching by 'Record View Pane' retrieves cross-referenced files.
- Each TRIM file record has a notes field. Can information about the publisher be stored here by transferring it automatically from PANDORA, instead of re-keying ? This is currently not available, but is planned for future versions.
- When the title of a journal changes, how would this be reflected in the file structure ? The file's title can be changed, and all folios preserved under the new title. It was also recommended that new files be created each year.
- After a file is created for a particular publication, and a PANDORA number is assigned in TRIM, are there any electronic means of advising the PANDORA archive management facility of this number ? No, it is only achievable via cutting and pasting.
- What reporting facilities will be available from TRIM for PANDORA managers ? None in the short term - a request for reports on PANDORA files will need to be raised with the Records Management Unit. Online searching will be available.
- Does the TRIM software cater for the year 2000 ? Yes, it is already 21st century compatible.
While filing is often the last activity considered when keeping track of a publication, there is merit in making the creation of a TRIM file for a publication the first activity in the process of identifying digital objects for preservation, to reflect the fact that publisher contact is the first step in acknowledging the significance of any particular digital object.
- the initial identification of material which has potential to be selected for archive. Currently the methods used are serendipity and systematic browsing of "What's New" sources
- the allocation of an ISBN or ISSN
- the cataloguing of electronic material on the National Bibliographic Utility, although the Australian Electronic Unit will have input into the guidelines
- the construction of the user's search query, which is managed by the WebPAC
- payment of money for usage (for example, micro-charging), or for articles which have a subscription or viewing fee. It is proposed that this will be managed by Information Services Branch
- any form of off-line delivery. It is not necessary to provide printing, faxing, or downloading within the archive. External users will be able to do so from their browsers, internal users will be able to do so under the auspices of the Library's Information Services Branch
- Library copying provisions for Document Supply Services.
|archiving||the process of downloading online publications, storing them and managing access to them within the PANDORA archive. See also Preservation.|
|electronic publication||an electronic publication is a manifestation of a digital object. The publication can be a single digital object or it may consist of a hierarchical structure of digital objects. Lower level objects (such as graphics and sounds) may be stored with the object (which then remains a single electronic file, for example, a MS Word document) or as separate objects (for example, a Web page, which stores graphics separately).|
|evolving serial||a publication that sometimes refers to itself as a 'zine', a 'magazine' or a 'journal', but has no discrete issues. Text is added, often on an irregular basis. Sometimes earlier segments disappear.|
|fair dealing||copying rights which individual users may apply to any publication. For electronic publications, the ability to read and browse without paying individual transaction fees.*|
|Legal Deposit||the requirement binding on all publishers and enforceable by law to deposit with one or more designated institutions a copy or copies of their printed publications.**|
|ISBN||International Standard Book Number, which uniquely identifies a monograph.|
|ISSN||International Standard Serial Number, which uniquely identifies a serial.|
|preservation||the process of initiating strategies and undertaking activities to ensure that archived publications will remain viable and accessible in the long term. See also Archiving.|
|PURL||a Persistent Universal Resource Locator is an URL. It points to a service which accepts the PURL as input and takes the searcher to another location (URL).|
|URL||A Universal Resource Locator points to specific locations of digital objects. Subject to change without notice.|
*National Strategy for Provision of Access to Australian Electronic Publications: A National Library of Australia Position Paper @ http://www.nla.gov.au/policy/paep.html
**National Strategy for Provision of Access to Australian Electronic Publications: A National Library of Australia Position Paper @ http://www.nla.gov.au/policy/paep.html
|Title entry||http://www.lib.latrobe.edu.au/AHR/||no extension|
|Title page||Cassandra Pybus||/books/WR/cass.html|
|Current Issue, December 1996||/current.html|
|The Demidenko Affair Update||/emuse/demidenko/demiupdate.html|
|Marilyn Lake reviews 'states ....||/archive/Issue-Dec-1996/lake.html|
|Ann Curthoys reviews 'Worlding....||/archive/Issue-Dec-1996/curthoys.html|
|(c) 1996 All Rights Reserved||/copyright.html|
|Contents -||Update register||/add.html|
|Issue 12/96||Bob Connell ....||/archive/Issue-Dec-1996/connell.html|
|Marilyn Lake reviews 'states ....||/archive/Issue-Dec-1996/lake.html|
|Ann Curthoys reviews 'Worlding....||/archive/Issue-Dec-1996/curthoys.html|
|Contents -||Send your response|
|Helen Darville aka Helen... - Update||/emuse/demidenko/demiupdate.html|
|Bob Connell Politics ....||/archive/Issue-Dec-1996/connell.html|
|Inga Clendinnen's ... Fellow Sufferers||/archive/Issue-Sept-1996/clendinnen.html|
|Lori Askeland (USA)||/emuse/History/askeland.html|
|Richard Rorty (USA)||/emuse/History/rorty.html|
|Dennis Altman's... On Global||/archive/Issue-July-1996/altman.html|
|David Halperin (Australia/USA)||/emuse/Globalqueering/halperin.html|
|Gary Dowsett (Australia)||/emuse/Globalqueering/dowsett.html|
|Michael Tan (Philippines)||/emuse/Globalqueering/tan.html|
|Donald Morton (USA||/emuse/Globalqueering/morton.html|
|Christopher Lane (USA)||/emuse/Globalqueering/lane.html|
|Fran Martin (Australia)||/emuse/Globalqueering/martin.html|
|Chris Berry (Australia)||/emuse/Globalqueering/berry.html|
|A response from Dennis Altman||/emuse/Globalqueering/altresp.html|
|Henry Reynolds After Mabo What||/archive/Issue-April-1996/Reynolds.html|
|Humphrey McQueen's ..Professions||/archive/Issue-Sept-1996/mcqueen.html|
|Contents - Good oil||Ninth Irish-Australia Conference *||/goodo/irish.html|
|Contents -||Updates register||/add.html|
|Issue 1 April 1996 **||/archive/Issue-April-1996/home.html|
|Issue 2 July 1996**||/archive/Issue-July-1996/home.html|
|Issue 3 September 1996**||/archive/Issue-Sept-1996/home.html|
* Another 6 conference references in similar format on site, + 2 offsite. The rest of the Good oil is all offsite - 21 additional offsite links
** these follow the same format as the current issue with each article having a separate URL in the format /archive/issuedate/author.html
NB The archived articles have the authors of feedback articles quoted in the first paragraph - i.e. the content of articles themselves has been updated with value added information since the time of their original publication.
This seems to imply that the whole title site can be overwritten at each capture - as long as no archived material is deleted!
Some of the issues/article are white text on a black background, later ones are black on a white background.
|Title page||Journal Philosophy||/aejnep-m.htm|
|Information for Contributors||/aejnec-m.htm|
|Volume 1 - No. 1||/vol1-1/vol1-1.htm|
|Volume 2 - No. 1||/vol2-1/vol2-1.htm|
|Volume 2 - No. 2 - Under construction||/vol2-2/v2-2con.htm|
|Peter Cleasby, School of Nursing and Health Administration, email@example.com|
|Contents||Volume 1. Number 1||/vol1-1/vol1-1.htm|
|Early australian nursing... Part 1: Scholars, Kathryn Roberts||/vol1-1/ajn1.htm|
|Early australian nursing... Part 2: Scholarship , Kathryn Roberts||/vol1-1/ajan2.htm|
|Early australian nursing ...: Part 3: Relationships between..., Kathryn Roberts||/vol1-1/ajan3.htm|
|Learning together: Using Simulation... Moya Conrick, Anne Dunne & Jan Skinner||/vol1-1/ajan4.htm|
|AEJNE Home Page (title entry)|| Next Story (editorial)|
|Contents||Volume 2 - No. 1||/vol2-1/vol2-1.htm|
|Clinical Decision Making: Issues in Teaching, Moya Conrick||/vol2-1/mc21.htm|
|Nurse-Academics and the Instructional Planning Process, Kathryn L. Roberts||/vol2-1/kayr2-1.htm|
|Meeting the Learning ..., Mrs Lisa McKenna||/vol2-1/lmck21.htm|
|Health in Australia: Sociological ... Reviewed by Kerry Mahony & Dennis McIntyre||/vol2-1/bkrev1.htm|
|On Moral Medicine - Theological Perspectives ..., Reviewed by Bruce C Wearne||/vol2-1/v2-1br2.htm|
|AEJNE Home Page || Next Story|
|Contents||Volume 2 - No. 2 - Under construction||/vol2-2/v2-2con.htm|
|Contents (Under Construction)|
|Using a computerised clinical ... Julianne Cheek, David Gillham and Patricia Mills||/vol2-2/v2-2jc.htm|
|The Clinical Portfolio, Lynette J. Stockhausen||/vol2-2/v2-2ljs.htm|
|AEJNE Home Page || Next Story (first article in issue)|
The six electronic publications selected for the prototype PANDORA archive were subjected to the online Dublin Core generators set up by UKOLN at http://www.ukoln.ac.uk/cgi-bin/dcdot.pl and OCLC at http://orc.rsch.oclc.org:9016/cgi-bin/dcget.pl. A table summarising the results follows.
The following titles were analysed:
|Australian Humanities Review||http://www.lib.latrobe.edu.au/AHR/|
|Coastal Bibliography Series||http://www.erin.gov.au/portfolio/ahc/hera/coast.html|
|Flight of ducks||http://www.cinemedia.net/FOD/FOD0001.html|
The following points indicate how the results of the two alternative processes compare:
- Format and Identifier (URL) - both systems correctly picked up the document URL and the html format. Neither identified any other formats - e.g. image formats. The PANDORA archive management facility will need to keep track of all formats used in every publication.
- Title - UKOLN correctly picked up the title for all 6 publications, OCLC missed one.
- Date - UKOLN picked up a date of creation or last modified date for 5 out of 6 publications. It gave no date for the other publication. OCLC only gave the date that the DC generator was run.
- Publisher - UKOLN picked up all publishers (or maybe host servers?), OCLC missed on 2. However, OCLC did assign a publisher address attribute to the other 4 publications.
- Subject - for each publication, OCLC assigned rather a strange grab bag of five subject headings (mostly highly irrelevant), based on Dewey classification . OCLC admits that their results are very preliminary. For 5 titles, UKOLN assigned a large number of mostly relevant headings, which appear to be generated from scanning lower level documents below the title URL. Australian Humanities Review was the odd one out - the only subject heading given being the publication designer.
- Creator, description - for 1 title (Flight of Ducks), UKOLN assigned a creator and a description attribute. The author of this publication is very WWW aware, and so had probably already set up his own metadata.
- Onscreen editing - UKOLN allows for onscreen editing of the search results.
At this stage the results from UKOLN are equal or better to those of OCLC for most DC attributes for the purposes of the PANDORA project. This conclusion is based on a very small sample, and on the understanding that development of both systems is ongoing.
Given the superior results obtained from Flight of Ducks, education of publishers is a key element in obtaining good results.
Ephemera has been considered as a separate category to other online electronic publications, because of its different characteristics.
As for print ephemera, electronic ephemera is selected in two ways
- identified topical issues that receive special attention for a finite period of time.
There are two approaches to preserving and describing electronic ephemera - by category and individually. The category approach is applied to electronic documents selected as part of special projects to collect items relating to particular topics. Topics may be determined by the Electronic Unit or by the Manager, Australian Collection Development, and often include targetted collecting of print ephemera as well. Electronic publications selected for these projects are preserved and described at the category level, i.e. euthanasia, gun control.
The individual approach applies to ephemeral sites selected on their own merit, rather than because they are part of a special project. These sites are preserved and described individually, unless it is likely that a number of other ephemeral sites on the same subject will be archived over time.
All electronic ephemera will be archived electronically. However, until the stability of the archive can be assured, documents of a transient nature are instead to be printed out and added to the print ephemera collection.
A lower level of publisher negotiation is undertaken for ephemeral items. The policy is to inform the publisher that we wish to archive the item and to proceed unless we hear otherwise from the publisher.
The Business Process Modeling team held a special workshop on ephemera to ascertain how it could be managed under the PANDORA model. The problem to be solved centred around the fact that access to items in the PANDORA archive is provided via the catalogue record in the OPAC, but individual catalogue records are not provided for ephemeral items.
As a result of the workshop, it was decided to upgrade the treatment of electronic ephemera. The workshop provisionally concluded that, unlike print ephemera, a full NBU record should be created for each item of electronic ephemera (We later modified this decision). This approach is consistent with the growing importance of ephemera as a research tool, and the responsibility of the Library to make its collection nationally available. The opportunity could also be taken to enhance the visibility of the print collection, by referring users to it from the electronic collection.
The workshop also concluded that the practice of printing out some electronic ephemera and filing it with print ephemera should be discontinued. Once PANDORA is fully operational, all electronic ephemera should be preserved electronically.
To test the principle that full NBU records should be created for individual items, a cataloguing trial was conducted. Five items were catalogued at each of the following levels:
- document level
- site level
- subject level
Cataloguing at the document level did not always achieve an adequate result as small single ephemeral documents often do not have meaningful authors and their titles often appear out of context. One example was a one page news item about euthanasia in Australia on the site for the Voluntary Euthanasia Society of London. The main entry for the document was 'World summary' which is not a helpful access point.
Cataloguing at the site level proved to be useful for sites with substantial and significant content. Although identified as part of an ephemera project, many of these sites would, in fact, qualify to be selected for preservation in their own right when assessed against the SCOAP guidelines, and would be eligible for a full catalogue record in any case. Because of their significance, it is likely that we would also want to gather them on a regular basis and they therefore would not, technically (as defined by the SCOAP guidelines) be part of the ephemera collection. There may be some items in this category that we would want to gather once only.
Cataloguing at the subject level mirrors the treatment currently given to print ephemera and proved the most successful approach for small single documents.