Last year, I wrote this post which shared how we had developed some analytics tools and data dashboards across Northampton Primary Multi Academy Trust. This was part of ongoing work in our trust to get better at using data we already have to describe, predict and ultimately intervene in the way we operate.
Since this post, we’ve had a lot of interest in how we’ve been developing analytics tools in the trust and so this is the first in a series of three blogs which I’ll interview the brains behind the analytics, self-confessed data geek, Matt Woodruff.
Matt is the founder and ‘Chief Data Scientist’ at Coscole Ltd. (now a part of Groupcall Ltd.) and I’ve been working with him for the last 3 years on this project within our trust.
One of the things I’ve learned from Matt is the power of predictive analytics. I believe that too much time is spent looking at pupil data as a rear view mirror in schools. This is often driven by a need to ‘know your data’ for accountability purposes rather than to help you think about what data can tell us about the future. If we think about the publication lag of documents like ASP (previously RaiseOnline), it’s crazy to suggest leaders should wait until November to find out about what happened in the past to a group of children who have already left the school. I think there’s more we could do to analyse the information we have about future cohorts of Year 6 to help adapt and tailor their provision whilst we still have time.
An example of how Matt’s brain works is our exchange over the title for this blog. I went for, ‘Matt and Tom’s Excellent (Analytics) Adventure’ whilst Matt suggested the catchy ‘A ‘Small Data’ Predictive Experiment using Machine Learning – Can MAT pupil level data generate reliable predictions for outcomes or identify pupils ‘at risk’?’.
Matt & Tom’s Analytics Adventure…
In this first interview, we go back about 3 years to a point in time where we were fumbling around in the dark for answers to a life without levels.
Me: Matt – when we first sat down with you, we had a ideas session where we outlined our vision of trying to bring together many pieces of pupil data in one place. What were your initial thoughts when you looked at the sea of post it notes which represented the many different pieces of data we wanted to bring together?
Matt: This is taking me back some time! It was at this stage that Coscole was finishing its direct commission with another MAT where we had spent three years building an approach to personalised learning, and in aggregating and visualising data. During this time I’d worked with multiple stakeholders from head office staff and directors of education, with data managers and school leaders as well as some pilot projects with teachers, students and parents. What was refreshing about beginning to work with NPAT is that you had, even back then, a good understanding of where there was challenge, and therefore where the opportunity was to improve: you were not setting out to boil the ocean; to have analytics be all things to all people.
I think like the mantra that ‘Exams are necessary, but not sufficient’ you imparted a view to me that said ‘Teacher Assessment is necessary but not sufficient’ in relation to understanding the whole profile of the pupil. You had already engaged with cognitive ability testing and understanding pupil attitudes to self and school, and were bringing in external assessment as standardised scores to provide a more rounded profile.
Of course it is natural at that stage you are inclined to pull your hair out – a growing MAT, albeit with a single MIS provider, with a growing need to make more effective use of data and to put it at the finger tips in an easily digestible manner to those that need it most – when your data comes from different providers, in different forms, at different times.
The ‘sea of post-it’ notes therefore represented the fact that as a Trust you had already embarked on the journey of understanding that there was a challenge in making effective use of data and saw the opportunity in the potential impact to improving outcomes if you could do it right. More than this – you’d moved down the road in determining exactly what you thought it important to capture, but also some things that you may have been doing by rote up until that point that you reassessed and decided it was not as important as you once thought it might be.
Me: There was a lot of planning and work that went on behind the scenes before we ever got close to inputting data. Aligning the MIS databases for across the trust was a big job and we spent a lot of time cleaning up our data as a trust which was an important but time consuming stage. Is this a normal part of the process with all schools you work with?
Matt: Getting to effective analytics is a journey. Everyone knows the adage ‘Garbage in Garbage out’, and it becomes particularly true when you compound the issue by aggregating ‘garbage’ across schools. Once again though this is not about boiling the ocean. Do you have to align everything across an MIS? Absolutely not. Should you seek alignment over time, in the things that really matter? Absolutely yes. You can do this in a way that does not undermine the context of different schools – thats vital. It is also not about a doctrine of ‘top down’. In my mind it’s about identifying good practice around data, and making sure all schools understand the importance of that. Good practices with data lead to much less wasted time further down the chain, and not only wasted time but the impact of not really being sure about your data providence. The typical reaction to seeing numbers, metrics, percentages is that we believe them. In too few cases are the underpinning assumptions challenged – “how was this data derived”, “what moderation is in place across schools to ensure that an apple in one is an apple in another”.
Yes, technically, the MIS databases were aligned in so much as NPAT standardised on naming conventions for Aspects. Yes we put in place data extraction technology and we warehouse that data and layer education modelling on top (the calculations that do your %GLD, Combined Y1/2 Phonics etc). That’s business as usual really. The fun starts again with the people and process elements. As soon as you visualise data in a more effective way, and don’t forget we’re not inventing new data here – we are just taking data you already have available – you instantly see gaps. You instantly notice things that aren’t right. And when I say ‘you’ I mean from CEO down. That can be a scary place for some because we lift up all the rocks. I think that’s great, because this is absolutely not about blame for a legacy of data whose quality can be improved, it’s about finally having access to the tools to quickly spot variations and to scaffold the people and the processes to ensure data is reliable.
There is no one I have worked with that has not done something different in their schools after joining and visualising data, and that’s a great thing.
Me: We had several U-turns and changes throughout the process as we switched our position on the types of assessment data and teacher assessment descriptors. How were you able to manage these changing demands from a technical perspective?
Matt: You’d started with fairly granular objectives in teacher assessment if I remember rightly: levels and sub-levels, or steps and stages, or milestones and smaller objectives. The change specifically did not provide too much of a technical challenge around how we got the data, but I think we found it a particularly challenging time to understand the way in which you wanted to visualise it – how leaders and staff would need to see that in the most digestible form. We use both a flexible visualisation approach with the Trust with Microsoft Power BI, as well as our own Apps built for Office 365. PowerBI is naturally easier and quicker for us to adapt than the code in our Apps, but by the same stretch our Apps can provide a more effective interface at times for staff.
The biggest issue though is both from a technical and a data perspective. We lose consistency, and history. For me a major incentive for a mature approach to data and analytics is having access to this history so we can build trend analysis and forecasting. Every time we decide to do something different it makes that more difficult. In this case with NPAT those decisions and changes were actually dealt with fairly early on and we’ve collectively been consistent since.
Me : After a year or so of being to analyse pupil information, we then started the conversation around how we could use technology to start to predict future attainment. You introduced me to the concept of Machine Learning. Can you explain (to a non-data specialist) how Machine Learning works?
Matt: There is a lot of hype around Artificial Intelligence (AI) and Machine Learning (ML) right now. Three years ago everything was Big Data, in much the same way. In many respects there is absolutely nothing new about ML, its been an active research area since the 1950’s and arguably in different forms well before that. Today, ML is a subset of the domain of AI and deals with the ability of computers to learn from data. It is technology that is now prevalent in just about every other aspect of our lives – from blocking spam in emails, to recommending products on Amazon, films on Netflix, and of course most recently in developing self-driving cars.
ML is itself subdivided into:
- ways that we set out specific parameters for the computer and where we know what we are looking for (supervised learning),
- where we want the computer to look at unstructured data and classify it itself (unsupervised learning), or
- where we set up a computer to learn through its own exploration – most famously used with Google’s AlphaGo team beating the world champion (reinforcement learning).
These developments have brought ML more recently into the mainstream. Tools are widely available to utilise ML models both with open source approaches, with Microsoft, or in combination. In fact, Microsoft have recently announced the fact that they are integrating ML/AI approaches with PowerBI, which is really exciting. EdTech companies like CenturyTech integrate ML in their adaptive learning routines.
In these ways, the technology already exists to make what we do in education faster, better, and deliver more impact. We spend 95% of our time looking backwards – what has just happened – whether that is last year or the last ‘data drop’. This is ‘descriptive analytics’ – we are simply describing the things that have happened. Other industries have already moved into ‘predictive analytics’ – using data to predict what is likely to happen in the future. Where we can get to beyond that is with ‘prescriptive analytics’ – if we know what is likely to happen in the future what should we be doing now either to mitigate risk or extend opportunity? The potential for a learning system that provides effective and efficient decision support for the human in an education context is vast.
This isn’t about ML being able to take peoples jobs; that in five years we’ll have robot teachers. This is about us leveraging what computers do far better than us, in order that we focus our human intelligence on the things that the machines will never be able to do (there is some ongoing debate on when the singularity might occur, but I think that’s beyond this blog…).
Me: There came a point where I asked you if you thought you would be able to predict our SATs results last year and you went off and did something clever on your laptop and came back with some results. What did you do?
Yes, we got to that point where we knew that we had good consistent data, and conceivably enough to do something meaningful from a predictive point of view. We tend to like lots of data for reliability, and when you boil it down a Year 6 cohort is not a lot of kids, even over 7 schools. However, we were at the point 18 months in where we had one years of historical data on the same basis as the current Y6 cohort. The same ‘schema’ if you like, of the things that we thought might matter in predicting outcomes.
Truth be told this was new to me. We’d been active in development at Coscole around the AI stack being released by Microsoft and also at that time had been engaged in another predictive proof of concept with Microsoft and one of the largest MATs around Progress 8. Looping back to my mantra of not boiling the ocean, I thought why not try something – it’ll either work or not work. If it works, that’d be pretty cool. If it doesn’t work, we’d be interested in why not.
The easy bit was actually the ‘data wrangling’. This is normally the bit of any data scientists life that consumes 60-80% of their time – finding data, cleaning data, putting it in a form that can actually be consumed by something and then do something useful… The joy for me is that we’ve done most of that: we have all your data warehoused, clean, ready to go.
I set out then to run a very simple experiment. This type of work is not new, lots of providers do it with their own data and I’m sure in more advanced ways, FFT, Rising Stars, CenturyTech etc – but for me it was a validation of others results and I was personally interested in the correlations with the MAT data. The question I was interested in was “Can we use school owned, in year data, from different data sources, for reliable prediction?” . If so, the follow up would be more important – “How should this impact current data collection practices to save time for staff, and to highlight interventions early?”. I also had an academic interest in a methods comparison study in the context of my PhD.
The experiment was a straightforward linear regression model trained on your prior year Year 6 MAT data including a selection of pupil characteristics and principally their standardised scores in Reading and Maths. I completed this in both Python as an open source approach as well as in Microsoft AzureML. I then used this model to run against your (then) current Year 6 cohort to predict their Reading and Maths test outcomes.
The results were interesting – in one way or the other I’d described earlier. Or both – pretty cool.
In Part 2, I’ll be asking Matt to explain how accurate his predictions were when we opened the envelopes* on results day in July 2018 and what the implications of this are for adopting predictive analytics for outcomes or identifying those at risk in the future.
*We didn’t really open any envelopes on results day. It was a downloadable csv. at the much more civil time of 8am this year rather than waiting till midnight.