AusSI
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Internet Indexing
IntroductionWe are, without doubt, in the information age. Information technology pervades our work, our homes and our recreation. Each day we use this technology to turn on our heating, cook our meals, produce reports, communicate with others, for entertainment and to obtain information. Traditional publishing is changing in response to the new technology. Conferences convened by the Australian National Scholarly Communications Forum (3) in the past 3 years have described the move to electronic publishing, particularly on the Internet, by the academic/research and government sectors. This move has had an impact on all the links in the information chain - from the writing of ideas (now done on word processors), to editing and peer review (through word processing and electronic mail), publication (desktop publishing for print and electronic out put) and in locating or finding publications (electronic indexes). The changes in information technology have an impact on the steps or strategies used to find information for a research inquiry. Research tools such as abstracts and indexes are now online, on CD ROM and on the Internet. Automation of information sources has occurred in two ways. Firstly existing abstracting and indexing services have made their information available electronically. This has enabled easier, quicker searching. Secondly new tools are being developed, such as Internet indexes, which automatically generate a database of information. Retrieval of information from the Internet is much more complex than for publications in print form. This complexity has stimulated the production of automated indexes. Attempts to index Internet information using standard library cataloguing or database indexing methods, seen in Cyberstacks and Netfirst (an OCLC service) are yet to be as successful as automated indexes, partly because of the sheer volume of material to be indexed, and also because of the newness of the medium. The challenges to researchers and information specialists to find information are being addressed through innovative solutions, both automated and manual. This paper demonstrates that Internet indexing or searching is not yet fully solved. Search tools are critical to effectively accessing information on the Internet, but need to be used with care. A comparison of 7 major indexes (commonly called search engines) finds great differences in search results and effective access to information. Overall trends and issues for the future development of Internet search engines are then discussed. Why is the Internet important?The Internet is the most rapidly growing medium of communication and publication. The Internet Domain Survey, of January 1996 found almost 9.5 millions hosts were connected to the Internet, an increase of approximately 142% on the figure for July 1995. Academic, research and government sectors now publish on the Internet. The survey also found that:
Estimates of those with access to the Internet range up to 65 million people (Four Corners, February 1995 estimated 35 million, Compuserve and America Online Figures, were quoted as 65 million in In, around and about, September 1995). The original Internet community comprised the university and major research sectors. (AARNET, the Australian Internet segment now owned by Telstra was originally a consortium of the Australian Vice Chancellors Committee and CSIRO). Schools, libraries, corporations and to a increasing extent the general public have joined the Internet. All these groups are using the Internet to publish and to search for information (or "surfing" the "sea" of information on the Internet) Publishing via the Internet, whether by electronic mail or making information available through gopher, ftp or world wide web has the advantage of being:
Many of the traditional publishing and distribution processes we have relied upon for consistency and control are becoming less important as electronic publishing increases. Internet publishing of scholarly and non-scholarly material is changing forever the way we organise and seek material. A critical issue in this new publishing medium remains over how, with such a vast number of Internet documents, relevant material can be found. Those publishing on the Internet wish to communicate with an audience and be read, as they did in print. The location of information is thus an important issue to those publishing as well as information seekers. What is an Internet index?The Internet consists of many files across the world. There are over 22 million World Wide Web pages, millions of gopher files, applications and databases. To provide a finding service "Internet indexes" or search engines, as they are often known, have developed. These play much the same role as traditional indexes used to find books in the library, or journal articles, conference papers and other publications. Internet indexes all offer different methods of access (use of search terms) and different information about the files or "publications. The same process is used for the collection and distribution of information as a traditional printed index. Major differences exist in the production of the database used in the index, and its access. The process for construction of traditional and Internet indexes is:
The terminology used by Internet indexes/search engines can be confusing. Common usage of some key terms is:
In this paper the term Internet Indexes is used to describe the search service available, rather than search engines. Index indicates a process of selection, storage and retrieval using the search engine. Internet indexing challengesThe Internet cannot be considered to be just another step in the history of indexing. It offers huge challenges and needs a very different approach to indexing to enable effective information retrieval. The need for quality access to information has increased, because of the number of searchers. There is also a need for tailored search tools for specific subjects. These tools save searchers time and provide access to reliable, quality information including evaluative comments from professional indexers, or what Internet Indexing services now called "professional web reviewers". Indexing the Internet offers many challenges:
Internet documents have only two absolute features. The first is a location (where they are stored and can can be found) and the second is content. There are no standards that require authors or title to be used, nor a requirement that heading information include title or sub title information. Internet indexing is quite different to journal article indexing where this information is clear. Evaluation of indexesWhy do we need to evaluate indexes? There is a multiplicity of indexes for print and electronic information which have been used by researchers and librarians for many years. They have been used because they provide an effective and efficient mechanism for finding information on specific subjects. In commencing to research a topic or inquiry the best search strategies are selected (see Basch). Indexes are used to obtain the best citations (or information lists) as efficiently as possible. Information found needs to be relevant and accurate to enable futher refinment of the search or to enable the researcher to obtain the information sources listed in the results. Indexes are finding aids for publications. Different indexes are used for different subjects. Different search strategies are also used. By evaluating indexes it is possible to find which are best suited for different types of research, as well as the best way to use them. These decisions have to be made for the use of traditional printed and automated indexes, and apply equally to Internet indexes. Traditional indexes, whether in print, CD ROM or online, have been evaluated in terms of:
This is quite a comprehensive list, but on a day to day basis probably the most important criteria used in deciding which index to use are:
All of these criteria apply to Internet indexes, together with additional factors. Debates rage across the Internet over indexing. There is a very wide range of opinion on what should be done to ensure that information can be found through Internet indexes. At one extreme there are those who propose a completely new approach to information discovery or "mining" through a completely automated system. They argue that the only human components of information discovery will be the humans creating information on the Internet. In order to fully automate this process creators will be required to provide good quality information on title, authorship, publication date and subjects of their Internet publications. Without this information the ability of any Index to provide an excellent search result is limited. With high quality information personal robots could be used to find information on a regular (perhaps daily) basis according to personal profiles. This approach is based on a philosophy that "everyone can get what they want" and that information gathering is a long term activity gradually building each individual's knowledge. It presumes that some form of quality control can be built into the Internet to enable an easier detection of relevant information. At the other end of the spectrum are the indexing and searching professionals who see their role is changing, but becoming more important in the "information mine" of the Internet. An example of the application of traditional indexing procedures (a very time consuming approach) to the Internet is Cyberstacks starting with a selection of subject areas, such as agriculture. This approach has its difficulties in application too, mostly due to the volatile nature of the web where indexing a resource can really be like pinning jelly to the wall, as it may be there today and gone, or completely changed tomorrow. Not only can the resource's name, contents and location change regularly, but its accessibility and format change easily as well. If the Internet is growing, becoming an increasingly important research publishing format and more accessible, how can we determine what to use for each inquiry? A set of criteria to evaluate Internet indexes is proposed below, based on Internet index use and feedback from research skills students at the University of Canberra.
To test the proposed criteria, it has been applied to some actual search engines. In order to evaluate the usefulness of the Index three specific searches were also undertaken:
All searches were undertaken in the last week of March 1996 and results are valid only for this time.
* see note 4 Internet mythsThe Internet is clearly becoming an increasing important information source. Using the Internet productively for information can only be achieved through finding the relevant information. Indexes are the key to this process. The indexes have not yet solved the problem of how to find quality information. Many factors indicate that finding Internet information is at a very early stage of development. Some are outlined below. 1) Anyone can do itWhile the basic Internet software is becoming more sophisticated it can still be difficult to use. There can be problems in accessing sites on the Internet, and to search well takes a great deal of time. Searching can be complex and require the use of a number of Internet indexes, as well as a degree of serendipity. Libraries can expect an increasing role as intermediaries in the search for information from the Internet. 2) All the information you need is on the 'netMany major organisations connected to the Internet provide a wealth of publications and information. Many others however have only selected information available at present. Any inquiry for "published" information needs to be evaluated to see what format is the most appropriate. Care needs to be taken to make sure that essential information, which may be in print, is not missed in the search. 3) You can find resources on the 'netAs is clear from the comparison of the 8 Internet Indexes, there is a need to determine which index is the most appropriate for a search. Criteria of contents, scope, search techniques, interface design, accessibility and format of results are the most relevant. Finding resources is complicated but sites being unavailable or moving. 4) Researchers will use the 'net all the timeMany researchers are more than fully occupied undertaking research and policy work. There is a need for good searchers to build up skills in Internet searching. While many who have the Internet at their desktop will do their own searching, an Information Centre or specialist searcher can provide a cost effective result (especially in terms of searching time). 5) Indexers will be obsoleteWhile Internet Indexes are predominantly automated, the last 12 months has seen a trend to using people to evaluate sites. Excite, Lycos, Magellan and Yahoo now all provide searchable site descriptions or keywords allocated by "professional web reviewers". While these are limited by short descriptions, or American use of terms, they are a growing service. ConclusionsAutomated Internet indexes are:
The sheer volume of data in a relatively uncontrolled form is like providing the front door key to the whole of the Library of Congress to our clients without a catalogue or floor map. Adding the current search tools is in some ways like giving them a torch and saying "go for it". Limitations of Internet indexes:
There are personal preferences in using Internet Indexes, just as there are in using online and printed indexes. An ideal Internet index should:
The Internet resembles jelly in its constantly moving contents. Indexes for the Internet are also changing quickly. The information structure of traditional indexes, including clear statements about authorship, title and publication details do not exist in Internet files. Some have suggested the usd of metadata at larger "pins" to hold the jelly to the wall, but while publishing on the Internet remains uncontrolled it is unlikely that standards such as metadata can be made mandatory. New solutions for indexing have involved a mix of automated and human effort. The flexibility of the new indexes is stretched by the volume and range of files, including some creative attempts by authors to ensure they are indexed under terms not necessarily related to their files. For Internet Indexes to be even more useful more specific searching needs to be available, i.e. Boolean operators, phrase searching and an indication of contents of the individual index. Connections between professional indexes and searchers and those creating Internet indexes will ensure that Internet indexes develop to become more useful than ever before.
BibliographyAdams, Douglas (1979) The Hitchhikers Guide to the Galaxy, Pan Books, London Basch, Reva (1994) Secrets of the super searchers, Eight Bit Books, Wilton, CT BELNET (1996) A selection of Internet search tools, http://www.mtm.kuleuven.ac.be/Services/search.html 1 April 1996 Courtois, Martin, William Baer, Marcella Stark (Nov 1995). "Cool tools for searching the Web - a performance evaluation" Online Magazine, 19(6) : pp.14-32 Eliot, T S (1936) Collected poems 1909-1936, Faber & Faber, London Lane, Nancy (1989) Techniques for student research: a practical guide, Longman Cheshire, Melbourne Lebedev, Alexander (1996). Best search engines for finding scientific information in the Net, http://www.chem.msu.su/eng/comparison.html 1 April 1996 Mulvaney, J and Steele, C. (eds) (1993) Changes in scholarly communication patterns : Australia and the electronic library, Canberra, Australian Academy of the Humanities Network Wizards (1996) Internet domain survey: January 1996, http://www.nw.com/zone/WWW/report.html 26 March 1996 Seidelman, R. In, Around and about, September 1995 Stanley, Tracey (1996). "Alta Vista vs. Lycos", Ariadne on the Web, Issue 2, http://ukoln.bath.ac.uk/ariadne/issue2/engines/ 1 April 1996 University of Michigan School of Information and Library Studies (1996) Matrix of WWW Indices. A comparison of Internet indexing tools, 1 April 1996 Winship, Ian (1995) World Wide Web searching tools - an evaluation, 1 April 1996 © 1996 Roxanne Missingham. To be printed in the AusSI Newsletter and LASIE 27(3):32-42. |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||