USING EDUCATION INDEXES TO MAP RESEARCH TRENDS
(Paper Ref No. HOL99850)
Allyson Holbrook (The University of Newcastle)
Margaret Findlay (ACER)
Sebastian Misson (ACER)
Paper presented at the joint AARE - NZARE Conference, Melbourne, 29 November - 2 December 1999.
ABSTRACT
USING EDUCATION INDEXES TO MAP RESEARCH TRENDS
Allyson Holbrook (The University of Newcastle)
Margaret Findlay (ACER)
Sebastian Misson (ACER)
Joint AARE-NZARE Conference 29 November - 2 December 1999
Every educational researcher is familiar with the use of education indexes to locate research publications on selected topics, however the Australian Education Index (AEI) and the Bibliography of Education Theses in Australia (BETA) constitute a valuable resource in another way as well. The electronic version of the AEI contains publications with a research emphasis that spans almost two decades. This information can be used to map educational research from many angles, for selected periods, whether by level of schooling, publication type, discipline area or research method. The approach can also be global (all records) through to selectively fine-grained. Educational research is notoriously difficult to classify because the field of education draws on multiple disciplines, separately and in combination. The descriptor groupings used in the index have been developed and refined over many decades and are extremely comprehensive. Each item is indexed using a number of descriptors to capture the essential elements of importance in the article. Such information can be used to assist in devising coding frameworks for classifying research information other than publications, for example grants topics. The cumulative weight of experience contained in the Thesaurus of Education Descriptors contributes coding depth to classifications based on scant information such as titles. This paper primarily examines the potential of the AEI as a tool to map and monitor educational research. The authors provide a range of analyses that identify the thrust of research publications in Australia between 1984 and 1997. Key findings pertain to differences in the thrust of theses compared to other publications and to the overall stability in the pattern of research at the global or macro level. The authors also identify the strengths and weaknesses of the AEI with respect to its immediate use in monitoring research trends.
USING EDUCATION INDEXES TO MAP RESEARCH TRENDS
Allyson Holbrook (The University of Newcastle)
Margaret Findlay (ACER)
Sebastian Misson (ACER)
Joint AARE-NZARE Conference 29 November - 2 December 1999
(paper ref no. HOL99850)
In Australia, as in other developed and developing nations educational research has grown into a large-scale enterprise. Most of this research takes place within universities in faculties, departments or schools devoted to teacher education. The research tends to be applied in nature, a substantial proportion of it is carried out by postgraduate students and there are several thousand educational research publications produced by Australian researchers annually. Given the scale and complexity of the enterprise internationally one of the most pressing concerns about educational research at this time is the lack of information about research direction, quality and achievements. This concern is evident within and outside the research community and is commonly expressed in terms of the need to determine research impact on the school sector and teaching and learning and policy making in particular ( McGaw 1997, Rudduck & McIntyre 1998, NERPPB 1998).
Before much headway can be made with the investigation of 'impact', intensive mapping is required to determine the thrust and extent of research. Moreover, mechanisms need to be developed to facilitate the ongoing and systematic investigation of research trends and the accumulated research findings in areas of key educational importance. This is where educational indexes, with slight modifications, may prove extremely useful. Every educational researcher is familiar with the use of education indexes to locate research publications. Yet the Australian Education Index (AEI) and the Bibliography of Education Theses in Australia (BETA) also constitute a unique and versatile resource for other purposes. The AEI database is the only source of comparable, longitudinal data on educational research output and thrust in the Australian context, although this is not the purpose for which it was designed. The first section of this paper provides background on the indexes, the remainder explores selected aspects of the database in order to illustrate its potential uses in research monitoring and mapping.
Background to the AEI, BETA and their uses
The Australian Education Index was commenced in 1957 by the Australian Council for Educational Research as a printed quarterly index to journals relevant to education and published in Australia. The AEI is one of many indexes internationally, among them the ERIC index compiled by the Educational Resources Information Centre USA, and the British Education Index. Each index tends to have its own national focus. The AEI is available in every major academic library in Australia and in many school and education department libraries.
The AEI is still produced in print - as two annual publications, the Australian Education Index and the Bibliography of Education Theses in Australia. The AEI is one of the largest Australian indexing services, indexing 4000-5000 items each year. Figure 1 shows the trend in the number of records indexed for both the AEI and BETA for selected years of publication.

Figure 1 Items indexed for selected years of publication
In 1979, the AEI database was computerised. It now comprises more than 95,000 indexed records and is available:
The AEI and BETA are two separate collections merged into one database. In this paper they are usually treated as separate collections.
The education librarians responsible for the index draw on a core of more than 180 periodicals and regularly published materials, plus they attempt to incorporate books, reports, resources, conferences, papers and electronic documents that are published in Australia, about Australian education or by Australian educational researchers. In short the core includes ALL Australian education journals and those foreign journals (many of which were once Australian owned) with a substantial Australian presence on the editorial board. The indexers may choose not to index every item in a journal, for example editorial material or book reviews. They locate articles published outside Australia if these articles fall within their core education periodical search. On the whole the periodicals and materials accessed do not overlap to any great extent with other nationally based indexes such as ERIC.
Searching and indexing
The AEI databases can be searched using the search engines of the host vendors of the database, for example, SilverPlatter software is used by RMIT Publishing for Informit Online and AUSTROM and Dialog searching software is used for International ERIC. In both cases, searching by keyword is available across all the records of the database or by limiting the search to the separate fields for author, title, institution, year of publication, source, subject, document type, abstract. Advanced searching allows for the combination of keywords and fields using the boolean operators, AND, OR and NOT.
Indexing of all documents, except for theses, is undertaken with the actual items in hand. The Cunningham Library of ACER provides support to the indexing team by collecting the documents. After indexing, the documents are incorporated into the library collection and retained permanently. Theses are handled differently - title pages and abstracts for the theses are collected with the help of the libraries of the awarding universities and form the basis for indexing.
Documents (referred to as items later in this paper) are given full bibliographical details (author, title, publisher, year of publication, source, ISBN, ISSN, abstract). The subject content is indexed using descriptors drawn from the Australian Thesaurus of Education Descriptors (currently the 2nd edition, 1996). Subject indexing is undertaken 'primarily in order that the user may effectively retrieve specified information'. 'Two basic indexing rules take precedence over all others', namely, 'index only what is in the document' and at the 'level of specificity of the document' or source. Hence the indexers do not index on the basis of 'implications, applications, projected uses, future directions' or the like (Miller and Findlay, 1996, p. xiii). On average about five descriptors are allocated to each document, although this may range from three to eight depending on the complexity and availability of descriptors, and the subject scope of the document
The Australian Thesaurus of Education Descriptors (Miller and Findlay, 1996) has been developed by the Australian Council for Educational Research (ACER) as a controlled vocabulary for Australian education. The first edition was modelled on the Thesaurus of ERIC Descriptors developed by the Educational Resources Information Center of the U.S. Department of Education. Both thesaurii group the terms in 41 broad subject groups under nine main categories. The second edition of ATED contains 5,350 terms and provides links to broader, narrower and related terms. It also provides scope notes for guidance of the use of many terms. ACER is committed to the maintenance of the thesaurus and is currently working towards a third edition in 2000.
Certain Thesaurus terms are used to tag documents with an indication of the 'level of education' that is being dealt with in the document, for example, primary, secondary, TAFE, and/or the specific age level, e.g. children, adolescent, adults. Also terms are assigned to indicate the population group, for example, Migrants, Aborigines, Rural youth as well as terms to reflect the methodology used in the research discussed in the documents, where appropriate.
In addition to descriptors, the indexers assign 'identifiers' which are proper names or concepts not at the time represented by approved descriptors. Many identifiers become descriptors in future editions of the thesaurus, e.g. 'lifelong education' became a descriptor in the mid-1980s, 'feminist criticism' more recently. For the most comprehensive of searches for a particular area of interest, a person would use descriptors, identifiers, and a text search of the abstracts in the databases
An indication of the type of document is also shown, e.g. Journal article, Thesis, Book, Report.
A sample record from the AEI database on Informit Online is presented in Figure 2.
Documents are also assigned a single broad subject category based on a UNESCO typology. Therefore summaries of the subject matter of the indexed sources can be reported in two forms, as 'Broad Subject Categories', and in 'Descriptor groups' based on the terms from the Australian Thesaurus of Education Descriptors. These summaries will be discussed more fully later in this paper.
|
Title |
The historical investigation of other than formal education: a comparative framework. |
|
Author |
Holbrook-A; Barker-R |
|
Source |
Melbourne Studies in Education v.39 n.1 p.81-104 May 98 |
|
Publication Year |
1998 |
|
Abstract |
This article has developed out of the first phase of a project that aims to identify, group and compare a wide range of educational activities and agencies in a specific industrial region of new South Wales between 1900-1990, with particular emphasis on occupational forms of education. The article explores what is meant by `other' education, with particular emphasis on occupational forms of education. The article explores what is meant by `other' education, and describes the analytical framework the authors have devised to compare educational intent, practices and strategies, of these other educational agencies and providers. |
|
Descriptors |
Educational-history; Nonformal-education; Historiography-; Comparative-analysis; Nonschool-educational-programs; Adult-education; Vocational-education; |
|
Subject Category |
Educational history |
|
Notes |
Refereed article |
|
Publication Type |
Journal articles (080) |
|
Availability |
ISSN 00766275 |
|
Document Number |
90321 |
Figure 2 Sample record with descriptors
Types of documents indexed
An indication of the type of document is given with each record, e.g. journal article,
thesis, book, report. The mix of publications in the AEI is evident in Table 1. Table 1
Figure 3 Distribution of Publications 1992-1997
and Figure 3 provide a picture of the total records or items indexed for the years 1984, 1989, 1992, 1997 and 1992-97 inclusive. It would be unwise to look to the data to
Table 1 Publication types, selected categories in percentages for selected years
|
PUBLICATION TYPE |
1984 |
1989 |
1992 |
1997 |
1992-97† |
|
Books Collected Works General Proceedings Serials Individual Papers Creative works Theses Theses Doctoral level Masters level Practicum Papers Guides General Classroom-Learner Classroom-Teacher Non-classroom Historical Materials Information Analyses Book/product reviews Journal articles Australian based Foreign based Legislative/regulatory mats Non print media (includes Computer programs) Numerical data Opinion papers Policy Statements Government Bodies Other Reference Materials General Bibliographies Directories Vocabs/classifications Reports General Descriptive Evaluative Research Conference papers/speeches Tests/Questionnaires Translations TOTAL CODES* TOTAL RECORDS/ITEMS |
97 58 83 4 97 - 14 72 286 - 41 18 60 30 3 62 - 1807 143 12 20 14 71 11 13 2 22 5 1 239 159 101 47 828 5 1 4426 4131 |
32 35 47 1 302 - - 133 369 2 54 18 63 29 2 40 - 1346 159 7 5 17 14 81 33 1 14 16 2 214 66 56 108 965 - - 4231 3957 |
36 58 65 7 399 1 - 95 238 - 20 21 83 51 1 35 - 1162 187 - 5 30 39 37 13 1 5 14 - 157 91 63 163 1569 1 1 4648 4407 |
79 45 38 3 700 - - 190 240 - 23 4 49 55 3 24 1 1756 275 5 75 20 10 9 17 2 4 7 1 81 24 44 109 1287 6 - 5186 4980 |
274 291 426 36 3537 1 1 713 2038 - 132 77 425 292 43 137 2 8037 1048 19 135 160 136 177 96 5 32 64 5 560 548 323 857 10007 28 15 30713 29571 |
† The information in the table is complete for the period that ended 20 July 1999 and incorporates the total items in AEI and BETA
* An item or record can be assigned more than one document type, hence the number o assigned codes exceeds total records
.
indicate hard and fast trends in types of publications, as changes are likely to reflect revised policies in document collection over the longer term. The various on-line versions of the database are updated continuously, which will explain some variations in totals throughout the paper. Where possible we have used the latest version possible (i.e. July 1999) in the analyses. A very small amount of data published in 1997 is expected to continue to arrive, however it is highly unlikely to change the overall pattern for 1997.
Aggregated data such as that for 1992-1997 probably gives the best overall estimate of the mix of document types in recent times. There are some points of note regarding the number and type of records indexed. Journal articles and conference or meeting papers dominate the collection (See Figure3). There are only a relatively small number of books (see Table 10) but collected works including book chapters contribute 14% of the total.
There was a peak in publications in 1994 and indications of a rise in 1997 (See Figure 1). These changes appear to reflect changes in Federal government higher education policy, namely research support for new researchers in the aftermath of amalgamation, and more recently the introduction of the outcomes focussed research quantum publications audit (RQPA) process. This would also seem to tie in with the more pronounced growth in conference papers in particular. Similar trends to the AEI are reflected for BETA, although with a lag of a year, possibly suggesting a relationship between theses and other publications.
.
In 1997 there was a growth of articles in foreign journals (Table 1). This can essentially be explained by the take-over of some Australian journals by foreign publishing houses.
Finally, despite fluctuations, the number of items indexed has, on average, tended toward a slight increase between 1984 and 1997 (see Table 1 and Figure 1)
Comparison of AEI and BETA across broad subject categories
A summary of the subject matter of indexed items is reported in the form of 'Broad Subject Categories'. The full summary for AEI and BETA for selected years is attached at the end of the paper. Only one category or topic is assigned to each item. The result is a profile of the main topic areas dealt with in education publications. Figure 4 provides a representation of the distribution of topics for AEI and BETA for the combined years 1994-97. It is important to bear in mind that the majority of publications are journal articles and conference papers.
The inner circle represents the AEI and the outer circle represents BETA. For both AEI and BETA a large proportion of publication is in the area of traditional curriculum subjects, but the most striking finding is the difference between AEI and BETA with respect to psychology. Significantly more theses are concerned with educational psychology as the main topic than other publications (most of which are journal articles and conference papers).
Figure 4 Distribution of broad subject categories AEI & BETA

Figure 5 Proportion of AEI and BETA devoted to foundation studies
Educational research is a complex system, and very little is known about how the system operates. Nonetheless a routine analysis, such as in Figure 4, provides a glimpse into the dynamic. What seems to be indicated is that two key elements of the system (PhD students and academic staff) are operating in different ways within discipline areas. Such a finding prompts other questions: Do some areas of

Figure 6 Distribution of total foundation subjects

Figure 7 Distribution of total foundation subjects
research remain stable while others change? To what extent are the different topics and publications related? Will a change in one aspect have any significant impact on another? What factors are most likely to bring about change and is it possible to target change? A study of the indexes may well assist in the modelling of the system.
The broad subject categories can be usefully combined in various ways to gain a perspective on areas of particular interest. One obvious combination is the 'foundation disciplines' in education. Another useful grouping the set of categories concerned with the operation of educational institutions. Both of these are elaborated below.
Exploring the foundation disciplines using broad subject categories
Professional research associations, especially subject associations, would be examples of groups interested in information about the research output within foundation disciplines. They may need such information to support arguments for funding specific research or to determine their programs and policies. There may also be concern more generally about the strength of some foundation disciplines in education, compared to others.
Figure 5 shows the proportion of AEI and BETA devoted to foundation studies for selected years and the period 1992-1997. The four categories are educational history, sociology, psychology and a combined policy and philosophy grouping. At the time of writing the 1997 database is as complete as it ever likely to be. Hence the trend that shows a decline in theses in the foundation disciplines in 1997 is likely to provide cause for concern, especially if there is a desire to maintain the significant role played by theses in contributing to psychological research.
Figures 6 and 7 take the analysis further and exhibit the relative distribution of publications (or research)between the foundation disciplines. For BETA historical research accounts for a greater proportion of foundation studies than in education publication more generally, particularly in 1992. The dominance of psychology is generally evident in both the AEI and BETA with one exception in each case. There is a bulge in philosophy and policy studies in BETA in 1992 and one is also evident in 1989 in the AEI. This finding also suggests a link between completed theses and other research publication over time, yet the pattern is not in evidence across all the disciplines. All the foundation disciplines exhibit fluctuation over time, but the aggregated data suggests a tendency toward the same balance in the AEI and BETA in the medium term.
Monitoring such developments and disseminating the results in simple graphic form (such as that above) would provide a useful service to the educational research community.
Exploring the operation of educational institutions
Figure 8 refers to the AEI only. If one were to ask the question what is the main thrust of education publication, then the answer would be the operation of educational institutions (and as will be shown later - schools). The majority of educational publications fall into the combined category of management and administration, educational personnel, teaching and teaching materials, and curriculum and subjects. By far the largest proportion of the publications is focussed on curriculum in one way or another. In addition the greatest variation over time is in the combined curriculum groupings. There was a small rise in management and administration literature between 1992 and 1997 but a more substantial rise in the literature on teaching that levelled out by 1997.
It is inherently interesting that 'research' is such a small category (see attached table), albeit a growing one. Give the explosion of interest among academics in interpretative and action research in recent years one might expect that more publication would be devoted to research issues. Could it be that the forum for such publications is outside Australian journals, or is it that method is subsumed in content? This is one instance where the broad subject categories are clearly not helpful enough, hence the need for other forms of reporting, and more sophisticated searches.

Figure 8 Educational operations for selected years
Education levels
It was mentioned earlier that where it is pertinent, individual items are tagged with a 'level' of education. The level may directly equate with institution or age. For many articles the category 'school' applies without specific reference to age level. For example the descriptors 'university' and 'adult learner' applied to two separate items would clearly place both outside the sphere of school age education. By contrast, an item or article about science teaching may not specify a level of education but it may be obvious in other ways that it is school teaching rather than TAFE or university, so it would be allocated to the 'school' group. By means of merging a range of such descriptors and taking care to eliminate overlap it is possible to obtain a levels analysis of the whole database of some 94,000 records. Of these, 20,000 entries do not have level information because it is not appropriate, e.g. the articles may be about society outside schools, or relate to education in general. Table 2 provides a picture of educational publication that has a strong focus on school systems, school students and personnel, and relatively little attention to education outside that sphere, especially at the pre-school level. This distribution is more clearly represented in Figure 9.
Table 2 Number and percentage of records by level, whole database.
|
Level |
Whole database |
% |
|
Early Childhood/Pre-school |
2633 |
3.5 |
|
Primary (Years 1-6) |
16993 |
22.4 |
|
Secondary (Years 7-12) |
15247 |
20.1 |
|
School Generic (Years 1-12)* |
12252 |
16.1 |
|
Postcompulsory Education |
1465 |
1.9 |
|
Higher/College Education |
14966 |
19.7 |
|
TAFE/Vocational Education |
6849 |
9.1 |
|
Adult/Community Education |
5523 |
7.3 |
|
Tot. Discrete Level Records ** |
75955 |
100 |
|
TOTAL whole database (approx) |
94,000 |
|
* not nominated as specifically primary or secondary school
** About 1% of remaining records could be attributed a level using different search mechanisms. The remainder covered such areas as research methods, studies of institutions such as family, church, youth, government, etc
In recent years greater research attention has been directed to vocational and adult education by research funding bodies. An additional search using combinations of terms for selected years produced evidence of a pronounced growth in the number of publications about vocational and lifelong education in the mid 1990s, but only a very small increase in the proportion of total publications devoted to the area. Such information, produced regularly, could signify whether or not targeted research funding was having an impact.

Figure 9 Percentage of records by level, whole database.
Analyses based on educational descriptors
The descriptors from the Australian Thesaurus of Education Descriptors that constitute the substantive indexing framework for the AEI and BETA provide a more robust, coherent, holistic and multidimensional framework for describing trends in educational research, than the Broad Subject Category groupings. The forty-one descriptor groupings are best used to map the thrust of education publication and to identify deeper trends. Together with the identifers they allow for intensive exploration of a clearly defined, established research area, for example 'gifted and talented'.
Table 3 AEI and BETA by descriptor groups for 1992-97
|
AEI |
% |
BETA |
% |
TOTAL |
% |
|
|
LEARNING & DEVELOPMENT 1.1 Learning & Perception |
5363 |
3.8 |
705 |
4.0 |
6068 |
3.7 |
|
1.2 Indiv. Develt. & Characteristics |
6949 |
4.9 |
1486 |
8.5 |
8435 |
5.2 |
|
PHYS. & MENT. CONDITIONS 2.1Disability & Special Education |
1019 |
0.7 |
140 |
0.8 |
1159 |
0.7 |
|
2.2 Counselling |
1759 |
1.2 |
230 |
1.3 |
1989 |
1.2 |
|
2.3 Other Health & Well-being |
2281 |
1.6 |
584 |
3.3 |
2865 |
1.8 |
|
ED. PROCESSES & STRUCT. 3.1 Internal Educational Processes |
20707 |
14.2 |
3222 |
18.4 |
23929 |
14.7 |
|
3.2 Social & Philosophical Views |
23312 |
16.3 |
2350 |
13.5 |
25662 |
15.8 |
|
3.3 Curric. Organisation & levels |
17867 |
12.5 |
1772 |
10.1 |
19639 |
12.1 |
|
CURRICULUM AREAS 4.1 English |
1170 |
0.8 |
158 |
0.9 |
1328 |
0.8 |
|
4.2 Mathematics |
6794 |
4.8 |
996 |
5.7 |
7790 |
4.8 |
|
4.3 Science |
3989 |
2.8 |
318 |
1.8 |
4307 |
2.7 |
|
4.4 LOTE |
1997 |
1.4 |
237 |
1.4 |
2234 |
1.38 |
|
4.5 Study of Socy & the Environt |
938 |
0.7 |
177 |
1.0 |
1115 |
0.7 |
|
4.6 Technology |
2459 |
1.7 |
331 |
1.9 |
2790 |
1.7 |
|
4.7 Arts |
894 |
0.6 |
78 |
0.5 |
972 |
0.6 |
|
4.8 Health & Physical education |
984 |
0.7 |
217 |
1.2 |
1201 |
0.7 |
|
4.9 Other |
713 |
0.5 |
158 |
0.9 |
871 |
0.5 |
|
4.10 Prof & Technical Preparation |
568 |
0.4 |
59 |
0.3 |
627 |
0.4 |
|
HUMAN SOCIETY 5.1 Social Processes & structures |
8694 |
6.1 |
1129 |
6.5 |
9823 |
6.1 |
|
5.2 Social Problems |
3705 |
2.6 |
290 |
1.7 |
3995 |
2.5 |
|
5.3 Human geography & cultures |
2439 |
11.7 |
247 |
1.4 |
2686 |
1.7 |
|
SOCIAL/ECONOMIC ENTERPRISE 6.1 Government & politics |
3019 |
2.1 |
176 |
0.9 |
3195 |
2.0 |
|
6.2 Economics and Finance |
2664 |
1.9 |
121 |
0.7 |
2785 |
1.7 |
|
6.3 Labour, employt & workplace |
6468 |
4.5 |
502 |
2.9 |
6970 |
4.3 |
|
INFORMATION/ COMMUNICATION SYSTEMS 7.0 Info & Communications |
8888 |
6.2 |
538 |
3.0 |
9426 |
5.8 |
|
RESEARCH 8.1 Research and theory |
5475 |
3.8 |
563 |
3.2 |
6040 |
3.7 |
|
8.2 meths, instruments, & analytic techniques |
4068 |
2.9 |
608 |
3.5 |
4676 |
2.9 |
|
FACILITIES /EQUIPMENT 9.0 facilities & equipment |
1319 |
0.9 |
81 |
0.5 |
1400 |
0.9 |
|
TOTAL |
142502 |
17475 |
163977 |

Figure 10 AEI and BETA by descriptor groups 1992-97
Only the 1992-97 data is reported in Table 3, although a year by year analysis is possible. The broad descriptor group analysis above flagged a difference between AEI and BETA in the psychology area. As one might expect, given that more than one descriptor is allocated to each record, the difference is mediated in the main descriptor group analysis but is still evident. For example, the main thrust of theses is in the individual and development characteristics area - a psychology area. Other differences worth noting for the period are the greater proportion of theses in the 'other health and well-being' and 'internal educational processes', groups. Furthermore theses tend more toward the examination of educational activity and proceses rather than areas more concerned with context, such as studies of society.
It can be seen in Figure 10 that by far the greatest proportion of items in both AEI and BETA are in the educational processes and structures and curriculum areas. In our use of the curriculum areas we departed somewhat from the main descriptor groups usually reported for the AEI and reorganised them to represent the Key Learning Area structure in Australia. This exercise indicated that it was possible to reconfigure areas to meet the needs of the educational community at a particular time.
One finding with reference to curriculum is the emphasis on maths, science and English, relative to other KLAs (i.e. an imbalance in the attention paid the range of KLAs).
In Table 3 the group 4.10 in the curriculum areas represents technical and professional preparation subjects such as teacher education. Teacher induction tends to fall within 3.1. Identification of all the descriptors associated with teacher education and induction, and the creation of a separate data base (as has already been achieved for Aboriginal education) would allow a finer-grained analysis of the trends in this area.
Developing coding frameworks for publication titles based on the AEI
During 1999 the AEI was used to devise a verifiable coding framework for data restricted to titles alone. The Australian Thesaurus of Education Descriptors formed the basis for the coding training and detailed notes explaining what was, and was not, to be included in a category were devised. The categories were restricted to the main nine descriptor groups and sub-groups with minor modification. Two of the authors became proficient in coding titles of publications, grants, theses and other research documents to 90% agreement using this framework.
In recent years there have been other attempts to devise codes for mapping research publications but the task has proved problematic mainly because educational research is multidimensional (i.e. at various levels, using different methods, and often multi-focussed) (Kerr et al. 1998). Without a whole article or abstract and the opportunity to use multiple codes, it is difficult to produce an effective mapping device. Moreover, for mapping and monitoring purposes, the breadth of the literature has to be tapped. Mapping research is of interest on an international scale, yet in other nations there would not seem to be a move to use existing indexes in this task. The approach is to develop new coding frameworks at considerable cost of effort and funds.
Our recourse was to use the AEI to develop a coding framework in the knowledge that we would also be examining the database separately. Moreover we would use the AEI to validate our coding. What became apparent but was not initially planned was that if the two sets of data for 1997 (the AEI and the faculty data) were closely matched, this would indicate that faculty data need not be collected again in the forseeable future.
How representative is the AEI database?
At this stage it is pertinent to ask is the AEI database representative of educational research activity and publication in Australia? The report referred to above has addressed this question in detail.
Faculty publications documentation (including theses) was collected from all but one of the universities with faculties of education (or equivalent) in Australia for the year 1997. From such information it could be deduced that the AEI under-represents the number of books produced by academics, but is very thorough in journal article collection. In addition the AEI indexed virtually all the theses completed. With respect to conference papers, more than half of the papers produced by education academics in that year were indexed.
Despite the above mentioned differences in the number and distribution of records, in terms of the main descriptor groups, the pattern for AEI and faculty data was almost identical. It is unlikely other databases could be cross-checked in this way to establish representativeness. In the case of Australia it proved possible because all faculties could be approached and the system, while large, is nonetheless small enough to be comprehensively examined.
The results of the mapping exercise for 1997 in Australia indicate that the AEI is already a more than adequate and reliable mapping tool.
There are small differences between faculty data and the AEI that indicate its history as an index, for example in the AEI there is more information on library and data base management than one finds in Australian faculties of education. In addition because of the connection with the Cunningham library at ACER, ACER's own reports and publications are more comprehensively indexed than might have been the case if the index was located elsewhere.
It must be recognised that the AEI is a national database. If articles are published in journals overseas it is always possible they will fall outside the record collection parameters. If an area were to be comprehensively mapped then recourse to other indexes may be necessary, but for general indications of trends in educational research (with the exception of medical education which is primarily picked up in Psyclit) (NBEET 1996) the AEI is already sufficient.
A proportion of articles about Australian education by Australian or other authors are duplicated across national indexes. For 1997 a comparative analysis was undertaken between ERIC and the AEI. The pattern of descriptors obtained was very similar with one exception, namely the area of politics and economics. The general conclusion drawn from that finding was that those who publish outside major Australian journals would seem to be concentrating on topics that are more global and contextual in nature. Bearing in mind the articles represented a specialist sub-set of ERIC records, another interesting finding is the closeness of the match, suggesting high inter-indexer reliability, with associated ramifications for cross-validation of emerging trends.
Comparison between records
In education a perennial concern is effective dissemination of research information.
There are at least three main conduits of dissemination: between researchers and those with a research interest such as professional associations, between researchers and policy makers, and between researchers and teachers. Concentrating here on the
dissemination of research information between researchers it is possible to note changes in publication types with reference to particular topics that may be of use in mapping, cross checking and predicting research trends.

Figure 11 Main publications learning and development descriptor group

Figure 12 Main publications curriculum areas descriptor group
Figures 11 and 12 compare conference papers and journal articles for the main descriptor groups of 'learning and development' and 'curriculum areas' because previous analyses have shown these to be interesting areas. It is generally known in education that PhD students will produce papers more often than journal articles (the latter tend to follow the submission of the thesis). It will be recalled that there was a relative decline in the discipline area of psychology in 1992. This trend would appear to be represented in the conference papers, whereas journal articles remain constant. In curriculum areas where there is relatively little difference between AEI and BETA there is no great difference between conference papers and journal articles. Once again this points to a need to better understand the dynamics of educational research, and conference papers would appear to be an indicator that could be used to monitor research direction, particularly of PhD research. Most PhD students are practising teachers and administrators, who bring specific practical problems to bear on their research (which is nonetheless largely within the foundation disciplines especially psychology). Therefore the direction taken by conference papers may have other implications with respect to what those students perceive as pressing needs in education, and also have some bearing on other forms of dissemination.
The distribution of PhD Research between institutions
Earlier in this paper the point was made that institutional data can not be extracted from the database except for completed thesis research. One use of institutional information is to see if there are particular concentrations of research activity. A particular interest is whether one or more universities are responsible for the learning and development thrust that is the main difference between AEI and BETA. The data were examined by institution, by state, by location and by date of establishment (i.e. in terms of pre and post 1986).
There was an overwhelming uniformity about the pattern of research topics across institutions. Whether the universities were old or new the pattern of main descriptor groups follows the pattern for the AEI and BETA generally. This suggests most universities offer students the opportunity of the entire field of educational study. A perusal of individual universities located three that showed more activity in 'learning and development' and one in the group 'physical and mental conditions'. The new universities show fewer total completions between 1992-1997 (888 as against 1847). After allowance for size the old universities produce about one and two thirds more theses than the new ones. There is no difference in the pattern of research theses by location but the metropolitan universities produced a significantly greater number of theses after allowing for postgraduate numbers.
To sum up, few universities exhibit areas of research student topic concentration or depart from the norm with respect to the pattern of main descriptor groups. Obviously this analysis could have been far more detailed and could be wider ranging if the institutional affiliation of all authors was included at the point of indexing. Moreover, there are other useful ways of combining and reporting institutional data than used here. If a number of institutions was receiving grant funding in allied areas of research it would be possible to monitor the thrust of the publications over time to see if the emphasis was retained via postgraduate students. By contrast the staff with the expertise may relocate. The stability of the pattern of expertise around Australia is one aspect of research that would need to be monitored if new priorities for research occurred on a regular basis, e.g. every 5 or 7 years.

Figure 13 Literacy and Numeracy Search for 1984-1997
An example of a search concerned with a specific issue
There is much criticism within education of a tendency to follow fads, and arguments that not enough research is being undertaken in areas of perennial importance. The pattern documented so far in this paper has been pointing to sustained effort in certain areas of research. In addition there are indications emerging from the analysis suggesting that there are identifiable forces that shape the subject thrust of research activity in education across the board. One of these is characteristics of research students. This would appear to be a deep-rooted influence tied in with professional and institutional factors. Other forces may bring about change in the pattern. Prioritised research funding is one and technological and conceptual innovation is another, but it remains to be seen if the changes are permanent. There is obviously the need for debate about directions and priorities in research, but this debate should be informed by knowledge about the system. Consistent attention to mapping will assist in unravelling cause and effect with respect to influences on research.
If we look at research publications at a macro level we see stability, but it is possible that the aggregate data might mask shifts of emphasis over time. One way to overcome this is to thoroughly map specific areas of interest at a micro level.
Subject competencies, especially literacy and numeracy, receive a great deal of attention in policy and practitioner literature and in the popular press. In the 1989 Common and Agreed Goals for Schooling in Australia the goals of development of 'skills of English literacy, including skills in listening, speaking, reading and writing, [and] skills of numeracy and other mathematical skills' were specified. By determining the amount of publications in the area an idea can be obtained of the degree of research activity as well. Even more accurate prediction of the latter would be possible if each AEI record was tagged with some sort of research indicator. At present only a small proportion of AEI records have a descriptor or identifier that indicates a particular research methodology, hence it is not possible to be accurate about the number of records that are reporting research findings. This may be a useful move in the future that would also assist in more accurate mapping of methodological change, indicate possible skills shortages, and imbalances in research approach.
The searches depicted in Figure 13 were undertaken for the previously mentioned study of research impact. The searches employed descriptors, identifiers and text searches, some more comprehensive than others. Such searches require the involvement of experts in the area and the experts in turn need to become familiar with the search terms available. Given some 'new' terms may not yet be used by the indexers, text searches may be necessary.
The outcome for each separate search strand is quite different and suggests a response to different stimuli. For example, between 1989 and 1992 there is a jump from just over 200 to just under 400 articles about literacy. By 1996 publications are at 1989 levels followed by another steep rise in 1997. It would seem reading and writing teaching follows much the same pattern, but what is clearly evident is that reading and writing constitutes a strong theme in the literacy literature generally and this has been the case for the entire period.
Whereas literacy has a clear relationship with teaching reading and writing, it would seem maths teaching and numeracy have a different relationship. Moreover numeracy has attracted less research than maths teaching per se. It must be cautioned that these micro searches will have inherent flaws in the choice of search terms. In this case there was an attempt to be exclusive rather than inclusive with respect to the terms used. There is need for clear and expert guidelines to be set up for database development along such lines.
Conclusion and discussion
Educational research is a substantial enterprise involving collaboration at the local, national and international levels. Projects and publications have proliferated since the 1970s, and research approaches have become more varied. Moreover most of the research has been independent, non-specified and small scale. As a result it has become very difficult to obtain a clear grasp of what is going on, how and if the various activities link together, and what progress has been made. The time has come for more systematic data collection about research, and this is reflected in recent government exercises that seek to determine the extent and impact of educational research.
The electronic version of the AEI contains information about educational publication with a research emphasis. The database spans almost two decades. This paper, in order to focus on the potential of the AEI to map and monitor research output, has presented data that describes the pattern of research in Australia for selected years. The main emphasis has been the general thrust of research publications. The findings indicate stability in research activity at the macro level (i.e. across main descriptor groups), but there is some pronounced fluctuation and evidence of responsiveness to undetermined factors at the micro level. There is a striking difference in thrust between thesis and other publications, and what would appear to be an associated difference between conference and journal publications. Theses would appear, on the surface at least, to be both applied and tied strongly to foundation disciplines, especially psychology. The greater proportion of all Australian research is at the school level and in applied areas.
Clearly the AEI can be used to obtain a profile of research activity. Given the scope of the database it can also be used to develop scenarios and other predictions that will assist in the development of educational research in Australia. These uses would require expert input from researchers as well as indexers. Even at this stage however, it has been demonstrated that the two established forms of describing the data in the index, namely the broad subject categories and the descriptor groups, provide a comprehensive and accurate picture of the main elements of Australian activity.
The addition of institutional affiliation of author(s) during indexing would improve the usefulness of the AEI as a mapping tool. This would be possible, although a little more complex for multiple authors from different institutions. With such information it would be possible to explore trends in collaboration, to more easily and accurately locate pockets of expertise and to determine changes in the concentration of expertise over time.
Another way to enhance the AEI would be more attention to the recording of specific research method, or at least an indication if a publication was reporting research findings or not (although this would appear to be the case in most instances). The AEI (in the same way as all indexes) is a product of its history. When the indexes began the dominant research method was quantitative. Over the years this tended to become the 'default' method for indexers, and so it is interesting that so few papers based on quantitative design are indicated as such in earlier years. It was mostly those publications that focused on methods or methodology that were indexed adequately with respect to method. In recent years publications have tended to be indexed with a method, nonetheless the process would greatly benefit from expert attention to help define the field of methods more adequately for mapping purposes.
Clearly it is important that the main work of the indexers does not become unwieldy if the use of the AEI for mapping proceeds. Each item is at present assigned, on average, five descriptors although this may range from three to eight depending on complexity and availability of descriptors and the scope of the publication. Under the circumstances there is considerable scope to add a small quotient of extra elements of information for all items.
To sum up, the mechanisms to monitor educational activity using the AEI are essentially already in place or relatively easy to put in place. Two forms of reporting the data are already established. Further mechanisms and methods of reporting would need to be developed, however, as would a plan for specialised searches and intensive mapping of particular areas. The parameters for such searches would have to be devised by the appropriate expert group in conjunction with the indexing staff.
One of the most exciting possibilities raised by this extended use of the AEI is finding out more about the system dynamic in education research. Some glimpses of the dynamic have been discovered during the course of analysing the database for this paper. Policy changes may have unintended consequences for research. Some areas may be resistant to change, others may be very vulnerable, or responsive, for example to a stimulus such as funding. More extensive and sophisticated information about the system of research would seem to be a necessary component of any overall plan to address and improve the impact of educational research in education.
|
Broad subject CATEGORIES |
1984 AIE |
BETA |
1989 AEI |
BETA |
1992 AEI |
BETA |
1997 AEI |
BETA |
1992-7 AEI |
BETA |
|
|
Education (general) |
2 |
1 |
- |
- |
1 |
- |
26 |
- |
41 |
- |
|
|
Educational history |
45 |
5 |
29 |
12 |
78 |
8 |
27 |
6 |
307 |
54 |
|
|
Educational research |
64 |
4 |
51 |
5 |
141 |
3 |
191 |
5 |
834 |
19 |
|
|
Comparative education |
2 |
- |
15 |
- |
4 |
- |
8 |
16 |
45 |
26 |
|
|
Ed Philosophy, policy,development |
313 |
26 |
364 |
58 |
279 |
26 |
219 |
26 |
1450 |
154 |
|
|
Educational sociology |
209 |
12 |
116 |
18 |
139 |
10 |
221 |
18 |
1005 |
111 |
|
|
Educational planning & admin. |
119 |
15 |
110 |
31 |
156 |
10 |
134 |
16 |
1151 |
73 |
|
|
Educational Facilities |
14 |
- |
8 |
1 |
1 |
- |
8 |
- |
38 |
1 |
|
|
The teaching profession |
154 |
9 |
208 |
27 |
231 |
27 |
207 |
21 |
1410 |
179 |
|
|
Educational personnel |
15 |
5 |
1 |
3 |
23 |
6 |
40 |
7 |
152 |
33 |
|
|
Educational management |
144 |
4 |
103 |
2 |
121 |
11 |
142 |
4 |
872 |
90 |
|
|
Educational psychology |
607 |
126 |
417 |
141 |
513 |
102 |
583 |
60 |
3326 |
664 |
|
|
Study methods |
- |
- |
7 |
2 |
7 |
- |
13 |
- |
39 |
3 |
|
|
Teaching |
49 |
6 |
102 |
13 |
198 |
15 |
302 |
36 |
1193 |
143 |
|
|
Teaching materials & equipment |
266 |
13 |
272 |
26 |
342 |
12 |
284 |
19 |
2015 |
92 |
|
|
Curriculum |
146 |
12 |
110 |
20 |
143 |
11 |
127 |
8 |
884 |
43 |
|
|
Curriculum subjects - basic |
121 |
7 |
140 |
6 |
196 |
11 |
316 |
19 |
1195 |
108 |
|
|
Curriculum subjects - voc & prof. |
155 |
8 |
81 |
12 |
162 |
15 |
116 |
7 |
1250 |
83 |
|
|
Curriculum subjects -trad discip. |
629 |
68 |
483 |
48 |
546 |
46 |
675 |
49 |
3987 |
360 |
|
|
Educational systems & institutions |
34 |
4 |
30 |
5 |
105 |
1 |
32 |
6 |
342 |
22 |
|
|
School systems & institutions |
105 |
6 |
79 |
6 |
50 |
1 |
96 |
11 |
405 |
35 |
|
|
Higher education systms & insts |
211 |
4 |
216 |
4 |
274 |
5 |
251 |
33 |
1817 |
90 |
|
|
Out-of-school education |
5 |
- |
1 |
- |
3 |
- |
4 |
2 |
19 |
3 |
|
|
Adult education |
53 |
4 |
63 |
1 |
74 |
1 |
101 |
14 |
537 |
44 |
|
|
Continuing, community & mass ed. |
20 |
1 |
23 |
1 |
42 |
1 |
23 |
7 |
199 |
23 |
|
|
Educ. Specific categories of student |
343 |
18 |
195 |
27 |
199 |
16 |
218 |
24 |
1473 |
114 |
|
|
Students |
22 |
1 |
41 |
4 |
25 |
2 |
89 |
9 |
235 |
22 |
|
|
Library & Info services |
279 |
13 |
495 |
17 |
43 |
4 |
97 |
7 |
708 |
53 |
|
|
TOTAL |
4126 |
372 |
3760 |
490 |
4096 |
344 |
4550 |
430 |
26929 |
2642 |
|
REFERENCES
Kerr, D. with Lines, A., MacDonald, A. & Andrews, L. (1998). Mapping educational research in England: An analysis of the 1996 Research Assessment Exercise for Education. National Foundation for Educational Research Report May 98/25. Bristol: Higher Education Funding Council for England.
Miller, E & Findlay, M. (1996). Australian Thesaurus of Education Descriptors. Melbourne:ACER.
McGaw, B. (1997). Joint author in Australian Research Council Reviews of Grants Outcomes No.24 Education 1989-1993. Report of a panel to evaluate the ARC large grant funding scheme education 1989-1993. Canberra: NBEET.
National Educational Research Policy and Priorities Board U.S. Department of Education (1998). National Directions in Education Research Planning. Washington: NERPPB.
National Board of Employment Education and Training (1996). Psychological Science in Australia. Canberra:AGPS.
Rudduck, J. & McIntyre, D. (eds) (1998) Challenges for Educational Research, London: Paul Chapman Publishing Ltd.