ADOS-2 Reliability Workshops: Frequently Asked Questions

Coding is a standardised part of the ADOS-2. Examiners are trained to understand and code items, in the same way, to look for the same evidence, and to follow the same coding conventions. However, this is not as straightforward as it sounds because of the wide range of different people who are assessed.

Many assessments are easy to administer and interpret regardless of who you assess, e.g., you follow the same procedure with everyone, and responses are right or wrong. This is not the case with the ADOS-2. Examiners administer the same tasks, use the same resources and prompts, but evidence for coding varies depending on what participants say and do. The examiner has to interpret what happens during an assessment through the ADOS-2 lens, which is not always easy. For example, they have to identify behaviours that might be sensory seeking, that are definitely sensory seeking, that might be sensory but are not the type that the ADOS-2 is looking for. 

It is easy for coding to “drift,” and reliability workshops are a way to minimise the risk of this happening. They provide opportunities for examiners to compare their codes to a coding consensus set by experienced ADOS-2 trainers.

To be reliable, coding needs to be at least 80% in agreement with the consensus. All examiners need to develop and maintain reliable coding to ensure that the ADOS-2 is valid. 

The initial ADOS-2 training introduces clinicians to the administration and coding of all assessment modules (modules 1-4 and the Toddler module). However, it takes time to develop these skills and become confident and competent in using the ADOS-2. 

For the ADOS-2 to be as robust as possible in the assessment process, coding must be at least 80% in agreement with the consensus coding, i.e., with codes agreed by ADOS-2 trainers, or by clinicians who are research reliable. Achieving this involves attending reliability meetings to watch and code videos of ADOS-2 assessments, and comparing coding to the consensus set by ADOS-2 trainers.

It is not mandatory, but it is strongly recommended as not attending these increases risk of drifting from standardised coding conventions.

Reliability workshops involve watching a recording of an ADOS-2 assessment that has been coded by trainers or clinicians who are research reliable and discussing coding to help participants calibrate their coding with the consensus codes. The discussions help understand and identify evidence for codes and provide opportunities to ask questions about administration as well as coding. They can be online or face-to-face.

ADOS-2 clinicians are strongly recommended to attend reliability workshops to become reliable, and to maintain reliability. Achieving this once, is not a guarantee that you will always be at least 80% in agreement with consensus codes. 

The manual does not recommend a certain number of times a year that clinicians should attend reliability. 

The difficulty of doing this is that you might reinforce ways of administering and coding the ADOS-2 that are not standardised. You are calibrating coding with each other as a team, which could increase risk of drifting from the consensus. Bringing in an external trainer and/or joining other teams can reduce risk of this.

Many people use all ADOS-2 modules in assessments, and they should be reliable in all of these. However, people who only use particular modules, e.g., they only assess adults, tend to focus on developing and maintaining reliability for these rather than for all of the modules. If you think you are likely to use all modules in the future, or you want to maintain coding skills for all modules, it is a good idea to attend reliability workshops for all of these. 

We assess such a wide range of different people, which makes it even more important to attend reliability workshops so that you can discuss different situations and develop your knowledge and understanding of administering and coding the ADOS-2. Achieving reliability for one assessment does not guarantee that coding all assessments will be reliable. This is something that all clinicians have to work at because of the varied nature of assessments. 

Reliable coding is something that all clinicians have to work towards and try to maintain. It can take time to achieve, and it can be inconsistent. Don’t be disheartened if you are finding it hard to get reliable. The most important learning is not whether a code is “right” or “wrong” but understanding why the consensus was this particular code. Reliable coding is underpinned by understanding, e.g., knowing what each item in the assessment is looking for, what is, and what is not, evidence for the codes. This develops over time with experience, but it requires work and a commitment to developing reliable coding. 

Yes, you will get a certificate of attendance for workshops. If you attend my workshops, you will also have all resources used during the session and access to the video for longer so that you can watch it again if you want to.