FLASH Marking evaluation frequently asked questions
1. What is a Randomised Controlled Trial (RCT)?
A type of scientific experiment used to trial how effective a particular intervention is.RCTs are commonly used in medical research (i.e. trialling new drug treatments) but are now being more widely used in education and social research too. They key characteristic is that people are allocated randomly to either an ‘intervention’ group (those who receive the intervention) or a ‘control’ group (those who do not receive the treatment/intervention).
Random allocation reduces bias and ensures that known and unknown characteristics which might affect the outcome of the trial are evenly distributed between groups. By using a control group any differences in the outcomes between the two groups can be causally attributed to the intervention.
2. What is the FLASH marking RCT trying to find out?
The FLASH marking RCT is seeking to find out whether the intervention (FLASH marking) has an impact on children’s attainment in English and English Literature. It is also looking to understand whether FLASH marking has an effect on English teachers’ workload (in terms of time spent on marking/feedback).
3. How will we be allocated to either the intervention or control groups?
Once your school has agreed to participate in the trial, the independent evaluators, Durham University, will randomly assign schools to either the intervention group (who will receive FLASH marking) or the control group (who will not receive FLASH marking treatment). This will be done using a simple computer programme. Once groups have been allocated there is no opportunity to change or ‘switch’.
4. What happens if my school is in the intervention group?
If your school is allocated to the intervention group then you will participate in receiving training into delivering the FLASH marking programme between (approx.) June 2018 and June 2020. Pupils who will be entering Year 10 in September 2018 and their English teachers will be the participants within each ‘intervention’ school. A series of three training sessions for the Subject Leader of English and one other member of the English dept.teachersis provided by the Meols Cop development team. English departments in participating schools are then required to implement the FLASH marking process for these pupils during their Year 10 and Year 11 English programme of study. Support from Meols Cop will be provided throughout.
As part of the evaluation, the team from Durham will select a small number of case study schools. They will visit these schools to find out more about how the intervention is working in practice. These visits will be arranged in agreement with the schools – nobody from the evaluation team will be showing up unannounced or ‘checking up’ on what schools are doing. The evaluation team will only be focusing on the intervention, not teachers’ performance.
There will also be a short teacher and pupil survey as part of the evaluation. The teacher survey will focus on issues of workload. The pupil survey will explore pupils’ views of the intervention and how it relates to their learning in English.
Schools will also be required to share some key data on participating pupils. This will support the analysis at the end of the trial. All data will be anonymised and can normally be easily compiled by schools’ IT/data managers from their existing school data management system.
5. What happens if my school is in the control group?
If your school is allocated to the control group then you will not be receiving the FLASH marking intervention. Your English teachers should continue marking and giving feedback to Key Stage 4 pupils in their normal way. This is often referred to as ‘business as usual’.
As part of the evaluation, we will still require you to share some key data on the pupils that will be forming the comparison group (i.e. Year 10 pupils in September 2018). This will support the analysis at the end of the trial. All data will be anonymised and can normally be easily compiled by schools’ IT/data managers.
The evaluation team will also select a small number of schools to visit in order to find out a little more about what ‘business as usual’ means in different schools. These visits will be arranged in agreement with the schools – nobody from the evaluation team will be showing up unannounced or ‘checking up’ on what schools are doing. The evaluation team will just be looking to establish what the existing practice is in control schools in order to provide a context for comparison with the intervention group. They will not be looking at teachers’ performance at all.
There will also be a short teacher as part of the evaluation. The teacher survey will focus on issues of workload.
6. Do I receive any incentive for participation in this trial?
Yes. If you are allocated to the intervention group your school will receive £700. If your school is allocated to the control group you will receive £1000.
Payment of these incentives is dependent on meeting certain conditions (i.e. attendance at FLASH marking training and providingand providing necessary data to the evaluation team).
7. What happens if we are allocated to the intervention group but then decide not to do the intervention?
It is vital that if you agree to participate in the trial that you are willing to deliver the intervention. If you are allocated to the treatment group then it is essential that you are able to deliver the intervention.
The evaluation uses an ‘intention to treat’ analysis. This means that all pupils that were randomised to the intervention group will be included in the final analysis, irrespective of whether the treatment was actually carried out or whether schools withdrew from the intervention.
As a result of this and as part of the agreement to participate in the project, we will require schools to provide necessary data at the beginning and end of the trial, whether or not they have delivered the intervention.
8. If my school agrees to participate, what happens next?
The FLASH marking team from Meols Cop High School and the evaluation team from Durham University will be in touch with further information and key documents that will need to be completed by your school. These will include a Memorandum of Understanding which will outline what each party (schools, the development team at Meols Cop and the evaluation team at Durham) agrees to do and provide as part of the trial. We will also provide ‘opt out’ consent forms for parents and details of what pupil information is required.
9. Is this ethical?
All evaluations funded by the Education Endowment Foundation adhere to strict ethical guidelines. In addition, the independent evaluators, Durham University, adhere to both their own University Code of Practice as well as the British Educational Research Association ethical guidelines.
No names or identifying features of schools, teachers or pupils are ever published in any reporting of the project. Parental ‘opt-out’ consent is sought in order for pupils’ data to be included within the analyses.
Sometimes people question the notion of having a ‘control’ group, believing that it is unfair that some pupils do not get to receive the intervention. We do understand that it can be disappointing not to be involved in receiving the intervention. However, we believe that in order to find out the most effective strategies for supporting pupils’ learning, it is vital that we conduct the most robust and high-quality research. Randomised Controlled Trials tend to be perceived as the ‘gold standard’ in scientific research and provide us with powerful information about the effectiveness of various teaching and learning approaches in schools. This information is essential for improving the outcomes and experiences for children in our schools, and so it is critical that we can conduct research with good comparison/control groups.
10. What is the time frame for this trial?
Recruitment of schools is now complete. The first FLASH marking training session will happen in June 2018. The delivery of the intervention to pupils will begin in September 2018 (for Year 10 pupils in English) and will run for two years, until the same pupils end their GCSE course in May/June 2020.
GCSE results will be used in the analysis of the trial and will be received by the evaluation team in Autumn term 2020. The evaluation report will follow in Spring 2021.