Open the dashboard

For teachers · Worksheet 5 of 5

How the Atlas Forecast learns and how accurate it is

Our own forecast learns from the mistakes the models have made at each station. And we publish its track record, even when it gets it wrong. How do you measure that honestly?

Aim

To calculate an average error, and understand why a forecast is compared with a benchmark and why it has to be tested with data it did not see while learning.

  1. ForecastThe system combines six models for every hour.
  2. MeasureThe stations measure what really happened.
  3. Learn from the errorFrom months of differences it learns how much, and when, each model gets it wrong.

Activity

  1. 10 min
    Calculating an average error. With this sample data: we said 18, 20 and 15 °C and the measurements were 16.5, 21 and 15 °C. Work out each day's difference without its sign, and their average. If you have the class challenge running, repeat it with your own data.
  2. 10 min
    The published track record. In “How accurate is the Atlas Forecast”, read the average temperature error for today (0–24 h) of the Atlas Forecast and of the median of the models. What percentage of the error is saved?
    See the track record
  3. 10 min
    Why it says what it says. In the simple view for a place where it is active (for example, Girona), open the Atlas Forecast explanation: which models weigh most and which nearby stations it uses.
    Open Girona Open the school's location
  4. 10 min
    Week by week. In the weekly chart on the track record page, look for weeks when the Atlas did worse than the models. What does that tell us?

Questions

  1. What is the average error of the sample data? Why do we remove the sign before taking the average?
  2. What is the Atlas Forecast compared with? Why is it hard to beat?
  3. What is cross-validation? Why would it not be valid to measure accuracy with the same data it learned from?
  4. Why is the Atlas Forecast active only in some areas?
  5. Does a good average track record guarantee that it will be right tomorrow?

Answers for teachers
  1. The differences are 1.5, 1 and 0 °C; the average is 2.5 / 3 ≈ 0.83 °C. Without removing the sign, an error of +2 and another of −2 would give an average of 0, as if we had not been wrong at all.
  2. With the median of the six models it combines (ECMWF IFS and AIFS, GFS, ICON, Météo-France and Met Office), which is already more accurate than almost any single model. The percentage improvement is (error of the median − error of the Atlas) / error of the median.
  3. Forecasting each station with a system that learned without it, as happens at any point on the map where there is no station. Using the same data would be like sitting an exam with the questions already marked: the system can “memorise” and look better than it is (overfitting). In addition, every morning we check what the system in use said the day before.
  4. It needs stations to learn from, and it switches on in an area once it has at least 2,000 verified hours and beats the models. Until then, it keeps learning.
  5. No. It is an average: there are days when the models are more accurate, and the page shows it. That is why it is published week by week.