Australia's Universities
News
About AVCC
AVCC Policies amp; Activities
For Students
For University Staff
Related Organisations
Site Map
Search
Home

Publications

 
Australian Vice-Chancellors' Committee
(ACN 008 502 930)


Graduate Careers Council of Australia
(ACN 008 615 012)

Standard Recommended Methodology for the Graduate Careers Council of Australia's

Graduate Destination Survey and Course Experience Questionnaire

July 1996
Canberra

Contents


Introduction

The aim of the Graduate Destination Survey (GDS) is to collect information about the activities of graduates after the completion of their degrees and also to gather, (via the Course Experience Questionnaire, or CEQ), their opinions of their courses. The responses provide valuable feedback to universities, and their staff and current students. They also provide information for prospective students to assist them in making informed career decisions, and careers advisers can integrate the survey results into their counselling practices.

In order to improve the quality of the data gathered in the GDS (including those gathered via the CEQ) it is vital that all universities conform as closely as possible to a standard survey form and methodology. The issues outlined and discussed in this document represent standard recommended methodological practice aimed at ensuring a high level of quality in the management of the major aspects of the survey process. Some major non-methodological issues relevant to the project are also discussed.

In so far as the CEQ is a component of the GDS, the points discussed in this document generally cover both instruments.

Survey Management

Each year the GCCA issues an updated Manual of Operating Procedure, Coding and Data Punching Instructions which encapsulates the main standard management and methodological issues.

The conduct of the GDS is overseen by the GCCA's Survey Management Group (SMG) which is made up of representatives from the Australian Vice-Chancellors' Committee (AVCC), the Department of Employment, Education, Training and Youth Affairs (DEETYA), university survey managers and careers advisers, graduate employers, and the GCCA.

Institutional Survey Managers

Each university is asked to appoint a 'survey manager', a member of the university's staff who acts as the liaison with the GCCA and conducts the GDS for the university and on behalf of the GCCA. Of 53 survey managers in 1995 (some multi-campus universities have different managers for each campus), 23 are based in institutional careers services, with the remainder being spread across statistics and/or planning units, and other administrative areas.

The staff member appointed to the position of survey manager should be of sufficient seniority in the university to ensure the work is properly conducted in a manner that meets the recommendations outlined in this document, and to be in a position to communicate issues of significance to the university's top management.

The GCCA arranges State-based meetings of survey managers as often as possible in order to encourage institutional input into the research work. It also maintains regular contact with survey managers through its Newsletter, emailed circulars to survey managers, and general mail-outs related to particular issues.

Individual survey managers are free to contact the GCCA with comments on the survey, and those survey managers working in careers services can also ensure input to SMG discussions by contacting the members of the SMG who are also National Association of Graduate Careers Advisers (NAGCA) members.

Questionnaire

Decisions regarding the content and layout of the GDS and CEQ forms are made within the SMG. However, advice is often sought from people outside the SMG (including the AVCC, NAGCA, institutional survey managers and survey research specialists) when the need arises. Ad-hoc comment is also welcomed by the SMG and is always given serious consideration.

All universities should use the standard GCCA questionnaire. Deviations from the standard questionnaire should be approved by the GCCA's Survey Management Group.

There have been concerns raised regarding the overuse of the CEQ instrument or instruments like it, with reports of students completing such surveys at the end of each year, and in some cases at the end of each unit (or equivalent). Such additional uses should be considered with caution, as they can lead to respondent fatigue and result in a lowering of the response rate to all such surveys.

Supply of Questionnaires

The GCCA supplies bulk questionnaires, which are ordered by survey managers and carry the university's name and a GCCA institutional code number. The GCCA takes all possible steps in getting survey forms to universities in an efficient and timely manner, however survey managers should ensure that they allow 4-6 weeks between ordering their forms and their mail-out date/s.

Structure of the Questionnaire

The GDS form has developed from a relatively rudimentary questionnaire in the late seventies to a well-refined set of questions in recent years. The major requirement that the questionnaire be standard enough to suit the needs of every higher education institution in the country has kept the examination of complex issues out of the survey, and this might be considered one of the survey's many strengths. It is a straightforward document with few problem areas, particularly in terms of questions.

The GDS form asks respondents about their just completed qualification, work in their final year of study, their major activity on the referent date, details about their employment and/or further study, and demographic/equity information.

The SMG seeks to meet the needs of users in fine-tuning current questions and coding practices or adding new questions where it is considered useful (the case of the new equity questions is an example here). It is also understood that there is always room for improvement in terms of layout and presentation. The questionnaire must be kept reasonably attractive to look at and easy to read, and advancing technology can always bring with it the requirement for changes (for instance, an eventual move to scannable forms).

The CEQ instrument is derived from the work of Professor Paul Ramsden, and has been tested separately by the Australian Council for Educational Research (ACER) and Professor Ramsden. It is expected that refinements to the CEQ will occur from time to time, in discussion with interested parties and only after rigorous testing.

Survey Population

The target population for the GDS is the group of people who had completed requirements for a higher education qualification in the previous calendar year, including overseas or international students. For example, the 1995 GDS included information relating to those who had completed requirements for a qualification in the calendar year 1994.

This includes all students who completed the study requirements for award of a bachelor degree or three-year diploma, a graduate or postgraduate diploma, a graduate certificate, or a higher degree.

Students continuing with honours courses which include an additional year of study, and who have thus met the requirements for the award of a pass bachelor degree, should also be sent a survey form. This is done because some universities graduate such people while others do not, and in order to maintain a consistent approach, all are included. It is also felt that the decision to undertake an honours year is important in terms of education and career, and that this factor should be tracked. Thus in this survey the term 'graduate' may be used to include some honours students who technically remain 'graduands'.

Students completing a two-year associate diploma should also be included where possible, although their responses are reported separately from degree or degree-equivalent respondents. Students completing two-year undergraduate certificate courses will be included in this category.

In instances where graduates have completed the first half of a combined degree, they should receive a survey form. They should also receive a survey form when they complete the second half of that combined degree. If a university graduates combined degree students only after they have completed both parts of their qualification, they should receive one survey form only.

Methodology

Initial Mail-out of Questionnaires

Some universities do a single mail-out of questionnaires aimed at arriving with graduates on or around the referent date. However, most universities distribute the questionnaires to coincide with mail-outs to graduands concerning their graduation ceremonies, and this decision is generally made to reduce postage costs and increase efficiency. This approach can result in some graduates getting their survey form a number of weeks before the referent date. Either process is considered acceptable by the GCCA, and the instructions on the front page of the survey form tell respondents that if they receive the form before the referent date, they should either:

. hold the form until the referent date and then fill it in and return it; or

. if they are reasonably certain of what they will be doing on the referent date, fill it in and return it.

Obviously, if the respondent receives the form after the referent date, they can easily fill it in as at that date.

The GCCA recommends that universities either supply a pre-addressed reply-paid envelope for the return of the form, or make other arrangements such as establishing a 'free-post' address.

Follow-up of Non-respondents

Survey Managers should keep track of the return of forms by respondents by, for instance, crossing their names off a master mailing list (a few have started using bar codes and scanners to do this work). The GCCA recommends at least two follow-up mailings to non-respondents.

A number of universities perform the first follow-up by sending out a postcard sized reminder, without a questionnaire. The GCCA recommends this practice, as it helps to cut the university's costs of doing follow-ups (e.g., in terms of stuffing envelopes and printing follow-up covering letters).

The majority of universities perform their first follow-up by re-mailing the questionnaire together with an explanatory covering letter and reply-paid envelope. This is also acceptable.

Second and subsequent follow-ups are conducted by re-mailing the questionnaire together with an explanatory covering letter and reply-paid envelope.

Some universities conduct additional (third and subsequent) follow-ups by telephone. This can be a relatively labour intensive process, and universities need to consider its cost-effectiveness and the difficulty, most particularly, of completing the CEQ in this manner.

It has been argued by some that this distribution of GDS forms some weeks before and after the referent dates constitute a weakness. Certainly, in the best of all possible worlds, in which no aspects of research methodology are problematic, this would be so. However, given the size and nature of the task and the available technology, the SMG believes that the method as it currently stands is simplest and most cost-efficient.

Information From Sources Other Than the Graduate

In an effort to maximise their response rates, most universities gather information from sources other than the graduate. The GCCA believes that (a) accurate and relevant information compatible with the GDS should not be ignored, and (b) universities deserve every assistance in achieving recommended response rates.

Student administration records are often used to collect data about graduates who went on to further full-time study at the university from which they had just graduated (including pass to honours people). These people must be sent a survey form so that their CEQ responses can be collected. Survey managers are advised, after the follow-up procedure has been completed, to 'create' responses for non-responding students undertaking further study by using the student records to complete the necessary questions. These include; field and level of study just completed, further full-time study indicator, details of further study undertaken, and some demographic data. In these cases, the CEQ will necessarily remain blank.

Relatives can sometimes provide accurate information about graduates' activities, and this is seen as acceptable by the GCCA. Some survey managers also gather data from their university's schools and faculties which maintain a systematic record of where their graduates go. This can be used to fill in information when the graduate has not responded to the GDS, and when the alternate source is considered accurate by the survey manager.

As a point of information, in the GDS-equivalent survey run in the UK, obtaining destination information from relatives is considered a normal part of the process.

In general, the GCCA believes that the graduate should be the primary source of information. However, the GCCA accepts that, where it is not possible to obtain information directly from the graduate, it may be useful to obtain it from other sources, as long as the information is accurate. In some cases, the accuracy of such information might be subject to the survey manager's judgement. In all cases, the CEQ section must remain blank.

Survey Referent Dates

Course completions may occur at any time of the year, however the bulk of completions take place in mid-year or at the end of the year. In recognition of this, the GDS is conducted in two 'waves' with separate referent dates: October 31 and April 30. The GCCA supplies survey forms noting the relevant referent date for each wave.

October 31

The first wave of any given GDS is a survey of those who complete the requirements for their qualification in the first half of the calendar year. The referent date for that part of the GDS is 31 October (so, for example, those who completed in the first half of 1994 were surveyed as at 31 October 1994).

The October survey is conducted (initial distribution and follow-up mailings) from early September through to early December. Returned forms should be bundled and labelled clearly and kept for processing with those returned from the following April wave.

April 30

The main referent date of the GDS (in terms of the number surveyed) is April 30 (i.e., the second wave) and includes those who completed the requirements for their qualification in the second half of the previous year (so, for example, those who completed in the second half of 1994 were surveyed as at April 30 1995). Data from the first wave are combined with those from the second wave and the results are reported as being the 'destinations of graduates in the first months after the completion of their degrees'.

This second wave is generally conducted between late February through to early July (initial distribution and follow-up mailings).

Data Preparation

Coding

Some universities prefer to code their own data using the Manual of Coding and Punching Instructions while others make use of the GCCA's Central Coding service. This facility sees the GCCA coding a university's forms free of charge in return for that university achieving a reasonable response rate (see comments on response rates, below), and arose out of a recognition that some survey managers were not being resourced in a way to allow adequate completion of the survey process.

Currently, around half of the forms returned in the GDS are coded by the GCCA, with many others being spot-checked by the GCCA's coders to ensure compatibility. The Central Coding process generally runs between late August and late October.

In 1996, the GCCA adopted the full set of six-digit DEET Field of Study codes. These codes are also used by universities in their data collection processes, and the change was made to allow universities to use information in their student datasets to 'pre-fill' some questionnaire fields or to check graduates' responses. Care should be taken in using this process as some universities only code field of study to a very broad level and graduates' responses can often represent more detailed (and more useful) information.

Data Entry

The majority of forms are also keyed at GCCA cost. Some universities key their own data, and send a disk to the GCCA for processing. Data files can also be returned electronically (e.g., via e-mail). The GCCA is currently examining the viability of optically scannable forms.

Universities keying their own data should ensure that the file returned conforms to the format outlined in the Manual of Coding and Punching Instructions.

Returning Forms or Disks to the GCCA

Universities returning GDS data on disk to the GCCA for processing should ensure they are delivered by the first week of October. Those returning coded survey forms should get them back by mid-September at the latest, and those returning uncoded forms should get them back by the first week of September.

Data Processing

At the GCCA, all GDS data are initially processed using SPSS PC+ (Statistical Package for the Social Sciences). This involves the formatting of data files to GDS specifications, the cleaning of the data (e.g., to remove out-of-range answers or provide default answers where necessary), and the production of Summary Tables which give basic destination information broken down by level of qualification and sex. These Summary Tables are returned to universities for their immediate use and comment.

The data cleaning and cross-tabulating programs are available to all universities, on request, for their information and use. Universities can also request that their data be returned to them in various formats, including SPSS, spreadsheet, and database compatible files. These, along with output files, can be transferred via electronic means.

After the production of Summary Tables, institutional data are sent to a computing agency (the University of Melbourne's Information Technology Service (ITS)), where a further set of tables is produced, and where the national data are stored. This second set of tables (referred to as Table Fours) showing more detailed destination by field of study and sex, is distributed a few weeks after the Summary Tables. The time lag is to allow survey managers to report on any anomalies found in the Summary Tables before processing goes too far.

Each university receives its own results, its State's results, and national results. Institutional data processing usually runs between mid-to-late September and early November.

Anticipating the needs of universities, provision has been made to allow universities to add additional course and/or faculty codes to their questionnaires. Universities are encouraged to use these course identifiers to allow them to analyse their data in ways more specific to their own course mix, going beyond the broader field of study analysis necessary in the production of national results.

When collection and processing of institutional data are completed, they are combined into a national file and the national tables are produced by ITS. A media release and (in recent years) a four-page summary are prepared, and results are usually released in the first or second week of December. The aim is to make the main results available in the calendar year in which they are collected.

Production of the national report then continues through December into early February, with a finished report usually available in March.

Response Rates

Response rates are of constant interest and concern to the GCCA and institutional survey managers, and the GCCA argues that given the size and nature of the task, current national response rates which are in the low to mid sixties are reasonable (albeit with room for improvement). The AVCC, in its Code of Practice for the Public Disclosure of Data from the Graduate Careers Council of Australia's Graduate Destination Survey and Course Experience Questionnaire (see comments below regarding this document), states that a national response rate of at least 70% is achievable and desirable for the GDS and CEQ.

The level of national response rate has fluctuated between the low seventies and low sixties since 1978, with a high of 72.0% in 1988 and a low of 62.2% in 1994.

Some argued that the additional CEQ questions (25 in all) would pull response rates down. However, a comparison of the mean response rates for the group of universities whose survey forms carried the CEQ in 1993 with those whose forms did not, showed no statistical difference. Further, the previous year's response rate (1992's 67.5%), in which no CEQ was used, also represented a fall (down from 1991's 68.4%).

In addition, the variability between the highest and lowest response rate (75.5 percentage points in 1994) and the variability between the response rates of those multi-campus universities which survey their graduates on a by-campus basis, all suggests that low response rates are related to differences in methodology at campus level and not to the questionnaire itself.

Survey managers returning low response rate datasets often complain of their lack of resources with which to complete the work, and the GCCA continues to argue in all available forums that the GDS should be resourced to an adequate level.

In agreement with the AVCC, the GCCA has been arguing that an institutional response rate of at least 70% is desirable and achievable. At the same time, it is recognised that some universities, such as those with highly mobile students and graduates and/or with a higher proportion of overseas students, will face greater difficulties in achieving such levels.

Ultimately, the GCCA has no power to raise university's resourcing of their GDS and thus no power to influence their response rates. The solution to a great deal of the variability in institutional response rates lies with those universities returning low response rates to give greater recognition to the value of 'outcomes' data from the GDS and greater support to their own survey managers.

The validity of low response rate datasets obviously suffers, and the addition of low response rate datasets to the national data also acts to reduce its validity. Higher response rates will result in greater validity, but the users of any survey dataset have to work with the knowledge that there is a degree of uncertainty about whether their figures are representative of the whole survey population. To date, comparison of Department of Employment, Education and Training course completion figures and GDS response figures (based on field of study and sex) suggest that GDS responses are representative of the survey population. Further work on establishing the profile of non-respondents is planned.

In the GCCA's view, the answer to concerns about low response rates is for the universities returning them to determine that they want to fix the problem. The key factors are as mentioned previously;

  • adequate resources in terms of staff time;
  • the cooperation of graduations offices to allow initial mail-outs with graduation ceremony information, which appears to be a relatively cost-efficient way of enhancing response rates;
  • the on-going monitoring of response rate throughout the period of the survey;
  • accurate and detailed mailing information; and
  • the efficient follow-up of non-respondents, involving at least two separate follow-up mailings.

The AVCC's Code of Practice notes that data for universities with response rates less than 50.0% should not be published.

Reporting GDS National Results

Due to funding constraints, the analysis and reporting of GDS results has been largely restricted to national aggregations of bachelor degree graduates (with a small group of three-year diplomates included). Additional funding from DEET now allows the publication of institutional reports, and a report of the destinations of postgraduates.

The annual GDS national report presents two initial tables of broad results by level of degree (from bachelor degree to doctorate) and then restricts itself to describing and analysing the results for bachelor degree graduates only. This includes a description of overall results including analysis based on factors such as sex, field of study, and that sub-group which was involved in the full-time labour force.

For the last four years, each report has examined additional issues, in an attempt to both react to and anticipate users' needs and interests. Examples of such additional issues have been: graduate mobility; credit transfers; graduates in part-time employment; overseas respondents; comparison of GDS results for those in part- and full-time attendance; initial nursing education; graduate salaries; graduates in the recession; arts, humanities and social sciences graduates; science graduates; and further full-time study. The discussion of these issues is often limited by space, and is generally restricted to use of GDS figures only (so do not represent wide-ranging reviews of the areas).

As always, users' comments are welcomed and taken into account (a number of the additional issues noted above arose from users' requests). The re-visiting of areas already covered when it might produce new information can always be considered.

Each report also carries a summary of results by aggregated field of study (40 in all) which includes basic destination figures and major areas of work undertaken. Also included are full destination details by disaggregated field of study (approximately 160 in all). Thus reports try to meet the needs of as many users as possible by including a discussion and summary of national results (for all users), simple (i.e., quick to use) tabular and graphical description at aggregated field of study level (e.g., for careers advisers, students), and more detailed results (e.g., for academics, educational researchers, government departments).

Reporting of GDS Institutional Results

The cooperation of universities makes it possible to produce a report of GDS results by university based largely on bachelor degree graduates aged less than 25. The format of this report includes an overview of the national results and institutional profiles, which combine contextual statistics (completions figures, response rates, etc.) a standard table showing destination by field of study, and standard graphics illustrating major facets of the figures. Universities supply text discussing their own results.

Code of Practice

As a result of problems arising with some universities and some areas of the media misinterpreting GDS and CEQ figures, the AVCC and GCCA have developed a Code of Practice for the Public Disclosure of Data from the Graduate Careers Council of Australia's Graduate Destination Survey and Course Experience Questionnaire regarding the uses of and precautions about interpretation of the survey results. The current Standard Recommended Methodology and the Code of Practice can be considered companion documents.

 

Back to Publications Index

[©] [AVCC Home Page]
Last WWW Update:04 Jul 2002
Questions/Comments/Broken Links: Webmaster