Introduction
Since 2004, when ICAO Doc 9835 (Manual on the Implementation of ICAO Language Proficiency Requirements) was first published, test developers and civil aviation authorities have relied solely on this document for information on how to design and develop or select tests which fulfil the ICAO LPR requirements. The second edition of Doc 9835 was published in 2010, providing more detailed testing guidelines found in Circular 318 (Language Testing Criteria for Global Harmonisation), published in 2009. However, these guidelines are still open to varying interpretations and focus broadly on best practice in test administration, design and development. The intention was that test developers would base their test development on ICAO Doc 9835 to apply a common standard. An important objective of the ICAO LPRs is that equivalence between different tests and the ICAO Levels these tests award is achieved.
In an ideal situation, each State would develop or select and implement its own testing system(s) in accordance with the ICAO LPRs. The standard implemented would ensure confidence of equivalence among States. In other words, ICAO Doc 9835 should provide the basis of a common framework for the implementation of tests and a means of promoting equivalence in the ICAO Levels awarded and the associated aviation English language skills and knowledge assessed by these tests. The objective was that a common and universal standard could be implemented globally, irrespective of which tests were used and in which States.
While ICAO Doc 9835 provides a common framework for the design, development, selection and administration of tests, the manual itself serves only as guidance material and was not intended to be highly prescriptive. Civil aviation authorities, airlines and ANSPs across the globe are now aware of the importance of language proficiency in safety and have put in place systems to assess their pilots and air traffic controllers. Language proficiency is now recognised in aviation operations. Language training is being implemented in response to the ICAO initiative on a global scale and awareness of the role of language and communication in aviation safety is improving. To this effect, the ICAO LPRs have achieved an important objective: to increase awareness of the role of and promote language proficiency for aeronautical communication.
Although the LPRs have been in place since the initial implementation deadline of 2008, it is now necessary to develop a means of improving standardisation and harmonisation of testing standards. Varying interpretations of the guidance material contained in ICAO Doc 9835 can lead to a range of different test designs and perspectives of what an effective ICAO LPR test needs to include. Harmonisation can only be achieved by reflecting on areas where ICAO LPR implementation can be strengthened and improved. Harmonisation begins with aligning test design to the real-world communicative requirements of the test takers and ensuring there is some common framework for how this is assessed – determined by the design of the test instrument. Achieving this harmonisation in test design means that different test instruments assess the same key features of language needed and assess how language skills are applied in aeronautical communication. Establishing a more common testing platform by aligning test instruments is a prerequisite for the standardisation of other aspects of LPR testing practices, including administration and rating standards. These guidelines aim to serve a starting point to assist in this process.
The design of a test instrument is the foundation of a testing system and affects all aspects of a testing system. This point can be easily overlooked in language testing settings, with attention often given to tests administration processes and testing personnel qualifications and skills rather than the design of the test. As a result, differences in testing standards and practices have arisen. This has introduced a number of challenges
that could undermine the long-term effectiveness of the ICAO LPRs and, compromise aviation safety. These challenges include the following:
Inadequate implementation of a uniform ICAO LPR standard.
A lack of equivalence between the ICAO Levels issued by different tests and States.
Variations in what language skills and language knowledge ICAO LPR tests assess and the extent to which these relate to the language, communicative contexts and proficiency levels needed for safe aeronautical communications.
Market forces favouring less effective tests over more effective tests.
The emergence and spread of less effective testing practices, which may not assess the full range of necessary language skills and/or award ICAO Levels at a lower standard.
Civil aviation authorities accepting tests or test results from outside their jurisdiction, irrespective of the effectiveness of those tests because of obligations to accept tests approved by other States or because of a lack of resources to evaluate the quality or impact of inferior tests, based on the assumption that if one State approves an LPR test for licensing purposes then that test meets an international standard.
A lack of confidence in the ICAO Levels awarded by different tests in other States.
A large number of tests have emerged that are based on less effective test designs which can impact on language training programmes and long-term attitudes towards language proficiency and the ICAO LPRs, and lead to threats to civil aviation safety.
This emerging range of LPR tests is a result of wide differences in test design. Too much variation may lead to differences that undermine the aims of the ICAO LPRs. Further, regulatory authorities do not yet have access to a common framework to evaluate LPR tests – a key initial step required to attempt to improve and harmonise LPR testing systems and standards. The Test Design Guidelines aim to provide framework to support regulatory authorities by providing the tools to allow them to evaluate the tests they select, approve or develop in order to improve ICAO LPR testing systems.
Who are the Test Design Guidelines for?
These guidelines have been developed to help civil aviation authorities and organisations involved in the design of LPR tests recognise and understand key issues related to the design of LPR tests and their impact on overall LPR testing practices, in the context of safety-critical aeronautical communication between pilots and air traffic controllers.
What do the guidelines cover?
A number of key criteria have been identified as critical elements in test design which influence the overall effectiveness of an ICAO LPR test. Each criterion plays a key role in how well a language proficiency test meets the objectives of the ICAO LPRs and how well the test performs overall in terms of validity, reliability and practicality. All the criteria are equally important in ICAO LPR best practice assessment. These criteria are core issues that shape the overall effectiveness and suitability of a testing system for the assessment of air traffic controllers and pilots for ICAO LPR licensing purposes.
Who developed the guidelines?
These LPR Test Design guidelines have been carefully researched and developed by ICAEA (International Civil Aviation English Association), supported by the ICAEA Research Group, who have many years of extensive academic and practical experience in both mainstream and ICAO LPR language testing.
ICAO LPR Test Design Guidelines Objective
These guidelines aim to provide clarification and explanation of key issues related to test design in order to limit stakeholder confusion or varying interpretations of ICAO Doc 9835. The guidelines provide in-depth explanations of why key criteria are critical to the effectiveness of a test. They aim to reduce markedly different interpretations of ICAO Doc 9835 that impact on ICAO LPR test design and which result in variations between testing systems. These guidelines are developed in response to the issues that have emerged since 2003, and that have caused such a divergence in LPR testing practices. The guidelines aim to help licensing authorities recognise key features of LPR test design in order to select, approve and audit tests and make recommendations for improvements.
What is a test instrument and how does it relate to the quality of a testing system?
A language test instrument is the tool or device which is administered during a language assessment to collect information about a person’s language skills and abilities to allow these to be measured. A test instrument is made up of sets of test tasks and test items (questions) constructed and assembled in a meaningful way and linked to test content and stimuli (recordings, videos and picture prompts) so that the language level(s), language knowledge and language skills the test is designed to measure, occur in a predetermined and controlled way. Test tasks and items may typically be a list of scripted questions asked by a trained interlocutor or a means of collecting test-taker responses and answers (e.g. on paper or a computer), or any combination of these.
The design of the test instrument is outlined in a test specifications document - the blueprint for the design of the test instrument and the development of all the test versions in the test bank. A test developer needs to develop test specifications as an important part in the early stages of a test development project. The design of the test instrument affects all aspects of the quality of a language testing system and is a fundamental requirement to best practice in language testing. The test instrument is the cornerstone to the quality of a testing system and determines its overall effectiveness – including the fairness and the validity of the interpretation of test results and how they are used. In fact, it is not possible for a test to achieve a good degree of validity if the test instrument is poorly designed.
Why is test instrument design so important?
The competent design of a test instrument (effectively ‘the test’) to assess language proficiency is central to the quality and overall effectiveness of a testing system. It is important to establish an equivalence between ICAO LPR tests which would facilitate harmonisation of LPR standards internationally. Identifying and highlighting the key elements, based on ICAO Doc 9835 and best practice in language assessment, provides clear parameters so that different tests can have more in common in terms of the language skills and language knowledge they assess. In simple terms, these guidelines aim to define baseline test instrument design elements that need to be included to allow effective comparisons to be made between tests. This can then facilitate international and inter-test standard setting.
Only once an ICAO LPR test instrument is well designed can it be possible to develop and implement a valid and reliable testing system. It is often incorrectly assumed that the quality of the raters determines the quality and effectiveness of a testing system. In fact, rating can only be effective if the test instrument is well designed, valid and reliable. It is the test instrument itself, not the interlocutor that determines the type, range and complexity of language as well as the adequate coverage of skills and contexts for communication.
The guidelines are framed around, but not exclusively based on, what is contained in ICAO Doc 9835. They aim to provide a common framework for stakeholders to analyse, evaluate and select LPR tests. By providing a clear explanation of best practice issues in test instrument design the guidelines aim to promote greater consistency and more uniform application of the ICAO LPR standards globally.
Test Design Criteria
The design of an LPR test instrument is the most fundamental factor that determines the quality of a testing system. And, because the design of the test instrument influences all aspects of LPR testing – from the delivery, administration, and rating to the interpretation of test scores – these guidelines focus on issues related to the design of the test instrument. Even with highly reliable and trained interlocutors and raters, a testing system remains inadequate and flawed if the test instrument is not designed effectively to meet specific criteria and standards in line with the ICAO LPR requirements and best practice in language testing.
The following criteria are essential for an effective LPR test.
