
Australian Vice-Chancellors' Committee
(ACN 008 502 930)
Graduate Careers Council of Australia
(ACN 008 615 012)
Standard Recommended Methodology for the Graduate Careers Council of
Australia's
Graduate Destination Survey and Course Experience Questionnaire
July 1996
Canberra
Contents
The aim of the Graduate Destination Survey (GDS) is to collect information about the
activities of graduates after the completion of their degrees and also to gather, (via the
Course Experience Questionnaire, or CEQ), their opinions of their courses. The responses
provide valuable feedback to universities, and their staff and current students. They also
provide information for prospective students to assist them in making informed career
decisions, and careers advisers can integrate the survey results into their counselling
practices.
In order to improve the quality of the data gathered in the GDS (including those
gathered via the CEQ) it is vital that all universities conform as closely as possible to
a standard survey form and methodology. The issues outlined and discussed in this document
represent standard recommended methodological practice aimed at ensuring a high level of
quality in the management of the major aspects of the survey process. Some major
non-methodological issues relevant to the project are also discussed.
In so far as the CEQ is a component of the GDS, the points discussed in this document
generally cover both instruments.
Each year the GCCA issues an updated Manual of Operating Procedure, Coding and Data
Punching Instructions which encapsulates the main standard management and
methodological issues.
The conduct of the GDS is overseen by the GCCA's Survey Management Group (SMG) which is
made up of representatives from the Australian Vice-Chancellors' Committee (AVCC), the
Department of Employment, Education, Training and Youth Affairs (DEETYA), university
survey managers and careers advisers, graduate employers, and the GCCA.
Institutional Survey Managers
Each university is asked to appoint a 'survey manager', a member of the university's
staff who acts as the liaison with the GCCA and conducts the GDS for the university and on
behalf of the GCCA. Of 53 survey managers in 1995 (some multi-campus universities have
different managers for each campus), 23 are based in institutional careers services, with
the remainder being spread across statistics and/or planning units, and other
administrative areas.
The staff member appointed to the position of survey manager should be of sufficient
seniority in the university to ensure the work is properly conducted in a manner that
meets the recommendations outlined in this document, and to be in a position to
communicate issues of significance to the university's top management.
The GCCA arranges State-based meetings of survey managers as often as possible in order
to encourage institutional input into the research work. It also maintains regular contact
with survey managers through its Newsletter, emailed circulars to survey managers, and
general mail-outs related to particular issues.
Individual survey managers are free to contact the GCCA with comments on the survey,
and those survey managers working in careers services can also ensure input to SMG
discussions by contacting the members of the SMG who are also National Association of
Graduate Careers Advisers (NAGCA) members.
Decisions regarding the content and layout of the GDS and CEQ forms are made within the
SMG. However, advice is often sought from people outside the SMG (including the AVCC,
NAGCA, institutional survey managers and survey research specialists) when the need
arises. Ad-hoc comment is also welcomed by the SMG and is always given serious
consideration.
All universities should use the standard GCCA questionnaire. Deviations from the
standard questionnaire should be approved by the GCCA's Survey Management Group.
There have been concerns raised regarding the overuse of the CEQ instrument or
instruments like it, with reports of students completing such surveys at the end of each
year, and in some cases at the end of each unit (or equivalent). Such additional uses
should be considered with caution, as they can lead to respondent fatigue and result in a
lowering of the response rate to all such surveys.
Supply of Questionnaires
The GCCA supplies bulk questionnaires, which are ordered by survey managers and carry
the university's name and a GCCA institutional code number. The GCCA takes all possible
steps in getting survey forms to universities in an efficient and timely manner, however
survey managers should ensure that they allow 4-6 weeks between ordering their forms and
their mail-out date/s.
Structure of the Questionnaire
The GDS form has developed from a relatively rudimentary questionnaire in the late
seventies to a well-refined set of questions in recent years. The major requirement that
the questionnaire be standard enough to suit the needs of every higher education
institution in the country has kept the examination of complex issues out of the survey,
and this might be considered one of the survey's many strengths. It is a straightforward
document with few problem areas, particularly in terms of questions.
The GDS form asks respondents about their just completed qualification, work in their
final year of study, their major activity on the referent date, details about their
employment and/or further study, and demographic/equity information.
The SMG seeks to meet the needs of users in fine-tuning current questions and coding
practices or adding new questions where it is considered useful (the case of the new
equity questions is an example here). It is also understood that there is always room for
improvement in terms of layout and presentation. The questionnaire must be kept reasonably
attractive to look at and easy to read, and advancing technology can always bring with it
the requirement for changes (for instance, an eventual move to scannable forms).
The CEQ instrument is derived from the work of Professor Paul Ramsden, and has been
tested separately by the Australian Council for Educational Research (ACER) and Professor
Ramsden. It is expected that refinements to the CEQ will occur from time to time, in
discussion with interested parties and only after rigorous testing.
The target population for the GDS is the group of people who had completed requirements
for a higher education qualification in the previous calendar year, including overseas or
international students. For example, the 1995 GDS included information relating to those
who had completed requirements for a qualification in the calendar year 1994.
This includes all students who completed the study requirements for award of a bachelor
degree or three-year diploma, a graduate or postgraduate diploma, a graduate certificate,
or a higher degree.
Students continuing with honours courses which include an additional year of study, and
who have thus met the requirements for the award of a pass bachelor degree, should also be
sent a survey form. This is done because some universities graduate such people while
others do not, and in order to maintain a consistent approach, all are included. It is
also felt that the decision to undertake an honours year is important in terms of
education and career, and that this factor should be tracked. Thus in this survey the term
'graduate' may be used to include some honours students who technically remain
'graduands'.
Students completing a two-year associate diploma should also be included where
possible, although their responses are reported separately from degree or
degree-equivalent respondents. Students completing two-year undergraduate certificate
courses will be included in this category.
In instances where graduates have completed the first half of a combined degree, they
should receive a survey form. They should also receive a survey form when they complete
the second half of that combined degree. If a university graduates combined degree
students only after they have completed both parts of their qualification, they should
receive one survey form only.
Initial Mail-out of Questionnaires
Some universities do a single mail-out of questionnaires aimed at arriving with
graduates on or around the referent date. However, most universities distribute the
questionnaires to coincide with mail-outs to graduands concerning their graduation
ceremonies, and this decision is generally made to reduce postage costs and increase
efficiency. This approach can result in some graduates getting their survey form a number
of weeks before the referent date. Either process is considered acceptable by the GCCA,
and the instructions on the front page of the survey form tell respondents that if they
receive the form before the referent date, they should either:
. hold the form until the referent date and then fill it in and return it; or
. if they are reasonably certain of what they will be doing on the referent date, fill
it in and return it.
Obviously, if the respondent receives the form after the referent date, they can easily
fill it in as at that date.
The GCCA recommends that universities either supply a pre-addressed reply-paid
envelope for the return of the form, or make other arrangements such as establishing a
'free-post' address.
Follow-up of Non-respondents
Survey Managers should keep track of the return of forms by respondents by, for
instance, crossing their names off a master mailing list (a few have started using bar
codes and scanners to do this work). The GCCA recommends at least two follow-up
mailings to non-respondents.
A number of universities perform the first follow-up by sending out a postcard sized
reminder, without a questionnaire. The GCCA recommends this practice, as it helps
to cut the university's costs of doing follow-ups (e.g., in terms of stuffing envelopes
and printing follow-up covering letters).
The majority of universities perform their first follow-up by re-mailing the
questionnaire together with an explanatory covering letter and reply-paid envelope. This
is also acceptable.
Second and subsequent follow-ups are conducted by re-mailing the questionnaire together
with an explanatory covering letter and reply-paid envelope.
Some universities conduct additional (third and subsequent) follow-ups by telephone.
This can be a relatively labour intensive process, and universities need to consider its
cost-effectiveness and the difficulty, most particularly, of completing the CEQ in this
manner.
It has been argued by some that this distribution of GDS forms some weeks before and
after the referent dates constitute a weakness. Certainly, in the best of all possible
worlds, in which no aspects of research methodology are problematic, this would be so.
However, given the size and nature of the task and the available technology, the SMG
believes that the method as it currently stands is simplest and most cost-efficient.
Information From Sources Other Than the Graduate
In an effort to maximise their response rates, most universities gather information
from sources other than the graduate. The GCCA believes that (a) accurate and
relevant information compatible with the GDS should not be ignored, and (b) universities
deserve every assistance in achieving recommended response rates.
Student administration records are often used to collect data about graduates who went
on to further full-time study at the university from which they had just graduated
(including pass to honours people). These people must be sent a survey form so that their
CEQ responses can be collected. Survey managers are advised, after the follow-up procedure
has been completed, to 'create' responses for non-responding students undertaking further
study by using the student records to complete the necessary questions. These include;
field and level of study just completed, further full-time study indicator, details of
further study undertaken, and some demographic data. In these cases, the CEQ will
necessarily remain blank.
Relatives can sometimes provide accurate information about graduates'
activities, and this is seen as acceptable by the GCCA. Some survey managers also gather
data from their university's schools and faculties which maintain a systematic record of
where their graduates go. This can be used to fill in information when the graduate has
not responded to the GDS, and when the alternate source is considered accurate by
the survey manager.
As a point of information, in the GDS-equivalent survey run in the UK, obtaining
destination information from relatives is considered a normal part of the process.
In general, the GCCA believes that the graduate should be the primary source of
information. However, the GCCA accepts that, where it is not possible to obtain
information directly from the graduate, it may be useful to obtain it from other sources,
as long as the information is accurate. In some cases, the accuracy of such
information might be subject to the survey manager's judgement. In all cases, the CEQ
section must remain blank.
Course completions may occur at any time of the year, however the bulk of completions
take place in mid-year or at the end of the year. In recognition of this, the GDS is
conducted in two 'waves' with separate referent dates: October 31 and April 30. The GCCA
supplies survey forms noting the relevant referent date for each wave.
October 31
The first wave of any given GDS is a survey of those who complete the requirements for
their qualification in the first half of the calendar year. The referent date for that
part of the GDS is 31 October (so, for example, those who completed in the first half of
1994 were surveyed as at 31 October 1994).
The October survey is conducted (initial distribution and follow-up mailings) from
early September through to early December. Returned forms should be bundled and labelled
clearly and kept for processing with those returned from the following April wave.
April 30
The main referent date of the GDS (in terms of the number surveyed) is April 30 (i.e.,
the second wave) and includes those who completed the requirements for their qualification
in the second half of the previous year (so, for example, those who completed in the
second half of 1994 were surveyed as at April 30 1995). Data from the first wave are
combined with those from the second wave and the results are reported as being the
'destinations of graduates in the first months after the completion of their degrees'.
This second wave is generally conducted between late February through to early July
(initial distribution and follow-up mailings).
Coding
Some universities prefer to code their own data using the Manual of Coding and
Punching Instructions while others make use of the GCCA's Central Coding service. This
facility sees the GCCA coding a university's forms free of charge in return for that
university achieving a reasonable response rate (see comments on response rates, below),
and arose out of a recognition that some survey managers were not being resourced in a way
to allow adequate completion of the survey process.
Currently, around half of the forms returned in the GDS are coded by the GCCA, with
many others being spot-checked by the GCCA's coders to ensure compatibility. The Central
Coding process generally runs between late August and late October.
In 1996, the GCCA adopted the full set of six-digit DEET Field of Study codes. These
codes are also used by universities in their data collection processes, and the change was
made to allow universities to use information in their student datasets to 'pre-fill' some
questionnaire fields or to check graduates' responses. Care should be taken in using this
process as some universities only code field of study to a very broad level and graduates'
responses can often represent more detailed (and more useful) information.
Data Entry
The majority of forms are also keyed at GCCA cost. Some universities key their own
data, and send a disk to the GCCA for processing. Data files can also be returned
electronically (e.g., via e-mail). The GCCA is currently examining the viability of
optically scannable forms.
Universities keying their own data should ensure that the file returned conforms to the
format outlined in the Manual of Coding and Punching Instructions.
Returning Forms or Disks to the GCCA
Universities returning GDS data on disk to the GCCA for processing should ensure they
are delivered by the first week of October. Those returning coded survey forms should get
them back by mid-September at the latest, and those returning uncoded forms should get
them back by the first week of September.
Data Processing
At the GCCA, all GDS data are initially processed using SPSS PC+ (Statistical Package
for the Social Sciences). This involves the formatting of data files to GDS
specifications, the cleaning of the data (e.g., to remove out-of-range answers or provide
default answers where necessary), and the production of Summary Tables which give basic
destination information broken down by level of qualification and sex. These Summary
Tables are returned to universities for their immediate use and comment.
The data cleaning and cross-tabulating programs are available to all universities, on
request, for their information and use. Universities can also request that their data be
returned to them in various formats, including SPSS, spreadsheet, and database compatible
files. These, along with output files, can be transferred via electronic means.
After the production of Summary Tables, institutional data are sent to a computing
agency (the University of Melbourne's Information Technology Service (ITS)), where a
further set of tables is produced, and where the national data are stored. This second set
of tables (referred to as Table Fours) showing more detailed destination by field of study
and sex, is distributed a few weeks after the Summary Tables. The time lag is to allow
survey managers to report on any anomalies found in the Summary Tables before processing
goes too far.
Each university receives its own results, its State's results, and national results.
Institutional data processing usually runs between mid-to-late September and early
November.
Anticipating the needs of universities, provision has been made to allow universities
to add additional course and/or faculty codes to their questionnaires. Universities are
encouraged to use these course identifiers to allow them to analyse their data in ways
more specific to their own course mix, going beyond the broader field of study analysis
necessary in the production of national results.
When collection and processing of institutional data are completed, they are combined
into a national file and the national tables are produced by ITS. A media release and (in
recent years) a four-page summary are prepared, and results are usually released in the
first or second week of December. The aim is to make the main results available in the
calendar year in which they are collected.
Production of the national report then continues through December into early February,
with a finished report usually available in March.
Response rates are of constant interest and concern to the GCCA and institutional
survey managers, and the GCCA argues that given the size and nature of the task, current
national response rates which are in the low to mid sixties are reasonable (albeit with
room for improvement). The AVCC, in its Code of Practice for the Public Disclosure of
Data from the Graduate Careers Council of Australia's Graduate Destination Survey and
Course Experience Questionnaire (see comments below regarding this document), states
that a national response rate of at least 70% is achievable and desirable for the GDS and
CEQ.
The level of national response rate has fluctuated between the low seventies and low
sixties since 1978, with a high of 72.0% in 1988 and a low of 62.2% in 1994.
Some argued that the additional CEQ questions (25 in all) would pull response rates
down. However, a comparison of the mean response rates for the group of universities whose
survey forms carried the CEQ in 1993 with those whose forms did not, showed no statistical
difference. Further, the previous year's response rate (1992's 67.5%), in which no CEQ was
used, also represented a fall (down from 1991's 68.4%).
In addition, the variability between the highest and lowest response rate (75.5
percentage points in 1994) and the variability between the response rates of those
multi-campus universities which survey their graduates on a by-campus basis, all suggests
that low response rates are related to differences in methodology at campus level and not
to the questionnaire itself.
Survey managers returning low response rate datasets often complain of their lack of
resources with which to complete the work, and the GCCA continues to argue in all
available forums that the GDS should be resourced to an adequate level.
In agreement with the AVCC, the GCCA has been arguing that an institutional response
rate of at least 70% is desirable and achievable. At the same time, it is recognised that
some universities, such as those with highly mobile students and graduates and/or with a
higher proportion of overseas students, will face greater difficulties in achieving such
levels.
Ultimately, the GCCA has no power to raise university's resourcing of their GDS and
thus no power to influence their response rates. The solution to a great deal of the
variability in institutional response rates lies with those universities returning low
response rates to give greater recognition to the value of 'outcomes' data from the GDS
and greater support to their own survey managers.
The validity of low response rate datasets obviously suffers, and the addition of low
response rate datasets to the national data also acts to reduce its validity. Higher
response rates will result in greater validity, but the users of any survey dataset have
to work with the knowledge that there is a degree of uncertainty about whether their
figures are representative of the whole survey population. To date, comparison of
Department of Employment, Education and Training course completion figures and GDS
response figures (based on field of study and sex) suggest that GDS responses are
representative of the survey population. Further work on establishing the profile of
non-respondents is planned.
In the GCCA's view, the answer to concerns about low response rates is for the
universities returning them to determine that they want to fix the problem. The key
factors are as mentioned previously;
- adequate resources in terms of staff time;
- the cooperation of graduations offices to allow initial mail-outs with graduation
ceremony information, which appears to be a relatively cost-efficient way of enhancing
response rates;
- the on-going monitoring of response rate throughout the period of the survey;
- accurate and detailed mailing information; and
- the efficient follow-up of non-respondents, involving at least two separate follow-up
mailings.
The AVCC's Code of Practice notes that data for universities with response rates
less than 50.0% should not be published.
Due to funding constraints, the analysis and reporting of GDS results has been largely
restricted to national aggregations of bachelor degree graduates (with a small group of
three-year diplomates included). Additional funding from DEET now allows the publication
of institutional reports, and a report of the destinations of postgraduates.
The annual GDS national report presents two initial tables of broad results by level of
degree (from bachelor degree to doctorate) and then restricts itself to describing and
analysing the results for bachelor degree graduates only. This includes a description of
overall results including analysis based on factors such as sex, field of study, and that
sub-group which was involved in the full-time labour force.
For the last four years, each report has examined additional issues, in an attempt to
both react to and anticipate users' needs and interests. Examples of such additional
issues have been: graduate mobility; credit transfers; graduates in part-time employment;
overseas respondents; comparison of GDS results for those in part- and full-time
attendance; initial nursing education; graduate salaries; graduates in the recession;
arts, humanities and social sciences graduates; science graduates; and further full-time
study. The discussion of these issues is often limited by space, and is generally
restricted to use of GDS figures only (so do not represent wide-ranging reviews of the
areas).
As always, users' comments are welcomed and taken into account (a number of the
additional issues noted above arose from users' requests). The re-visiting of areas
already covered when it might produce new information can always be considered.
Each report also carries a summary of results by aggregated field of study (40 in all)
which includes basic destination figures and major areas of work undertaken. Also included
are full destination details by disaggregated field of study (approximately 160 in all).
Thus reports try to meet the needs of as many users as possible by including a discussion
and summary of national results (for all users), simple (i.e., quick to use) tabular and
graphical description at aggregated field of study level (e.g., for careers advisers,
students), and more detailed results (e.g., for academics, educational researchers,
government departments).
Reporting of GDS Institutional Results
The cooperation of universities makes it possible to produce a report of GDS results by
university based largely on bachelor degree graduates aged less than 25. The format of
this report includes an overview of the national results and institutional profiles, which
combine contextual statistics (completions figures, response rates, etc.) a standard table
showing destination by field of study, and standard graphics illustrating major facets of
the figures. Universities supply text discussing their own results.
As a result of problems arising with some universities and some areas of the media
misinterpreting GDS and CEQ figures, the AVCC and GCCA have developed a Code of Practice for the Public Disclosure of Data from the Graduate
Careers Council of Australia's Graduate Destination Survey and Course Experience
Questionnaire regarding the uses of and precautions about interpretation of the survey
results. The current Standard Recommended Methodology and the Code of Practice
can be considered companion documents.
Back to Publications Index
[©] [AVCC Home
Page]
Last WWW Update:04 Jul 2002
Questions/Comments/Broken Links: Webmaster |