2025 Tornado Warning Performance Assessment
In Spring 2022, the Northern Tornadoes Project (NTP) undertook the first independent assessment of tornado warning performance in Canada, with the goal of the initiative to help improve both tornado warning performance in Canada and the understanding of tornado warning issues in the Canadian context. That first assessment covered the years 2019 to 2021, and a second assessment covering the 2022 season was completed in Spring 2023. Blog posts presenting a 'report card' and details of the performance assessments were shared ( and here, respectively) and results were published in the journal Atmosphere-Ocean ().
Below is our 2025 report card and detailed performance assessment covering the 2023 and 2024 seasons (there weren't enough significant tornadoes in 2023 for a report card based just on that season).
We’re happy to report that the overall score continues to improve - from 33.3% for 2019-2021, to 56.6% for 2022, to 60.7% for 2023-2024.
The following is the NTP's tornado warning 'report card' for 2023-2024:
Report Card Background Information and Interpretation
Environment and Climate Change Canada (ECCC) has the federal mandate for issuing severe weather watches and warnings across Canada. Their national alerting performance targets are as follows:
* "Probability Of detection for tornado warnings equal or better than 0.50",
* "Tornado warnings issued at least 10 minutes before the events 60% of the time",
* "Convective watches issued at least 6 hours before the events 80% of the time", and
* "Probability Of detection for convective watches equal or better than 0.65".
Note that 'convective' here means related to thunderstorms - so 'convective watches' refers to both severe thunderstorm watches and tornado watches. The NTP assessment however focuses mainly on tornado warnings.
The following is interpretation information for each of the NTP's three Report Card criteria:
- The Probability of Detection (POD) answers the question, “What fraction of all tornadoes documented over the period had a tornado warning?” The ideal score here is 100%, but ECCC has chosen a target of 50%, or 0.5. Based on the 2023-2024 data, the calculated POD for all tornadoes is 0.36 (or 36%) - less than the 50% target. The POD is slightly better when considering only tornadoes that occurred within the Doppler range of radars (0.38) and notably higher when considering only tornadoes rated EF2 or higher on the EF scale (0.53, though the sample size is small at only 19 tornado events). The best POD value (0.57) was found for hybrid-type tornadoes, followed closely by supercell-type tornadoes (0.56). Note that nearly all violent (EF4-EF5) tornadoes are supercell tornadoes, and nearly all tornado-related fatalities are caused by tornadoes rated EF2 or higher. Because the POD deals with whether a tornado was warned or not, this criterion was given the highest weighting by the NTP: 60% of the total score.
- Ideally, for 100% of tornadoes, tornado warnings should give those affected enough time to take protective action. Here, ECCC has chosen to use a target of 10 minutes of warning lead time for 60% of tornadoes. Using the 2023-2024 data, NTP found that this target was met 20.7% of the time - far less than 60%. The percentage of tornado events meeting the 10-minute target is slightly higher when considering only tornadoes that occurred within the Doppler range of radars (21.5%). NTP has given this criterion a weighting of 30% because while lead time is secondary to whether a warning was issued it is still important that those affected have enough time to take safety precautions.
- A tornado watch is a type of ‘convective watch’ that is issued by ECCC. For events where a tornado was confirmed, NTP considers the optimal alerting progression to be a tornado watch issued for area followed by a tornado warning. The ECCC target of a tornado watch being issued six hours prior to the event 80% of the time was relaxed by NTP because the 6-hour threshold was met by so few events, and having a tornado watch out for even an hour before a tornado warning aids with the alerting process. Using the 2023-2024 data, NTP found that this relaxed target was met 56.9% of the time. The focus of our assessment is on tornado warnings rather than tornado watches, so this criterion was assigned only a 10% weighting.
Adding up the weighted scores for the three criteria gives the total score out of 100: 60.7, or a grade of ‘C’.
Details related to the calculation of POD and lead time are discussed in the detailed assessment that follows.
NTP National Tornado Warning Performance Assessment
In the spirit of improving tornado alerting effectiveness for all Canadians, the NTP conducts independent assessments of tornado warning performance in Canada. We used NTP tornado data and ECCC tornado warnings to assess warning accuracy and timeliness. There are a number of different ways to improve tornado warning performance, and the NTP provides recommendations near the end of the assessment.
Assessment Data
For tornado data, we used NTP-confirmed tornadoes over land from 2023-2024 (N=179). Start times were investigated and obtained for each tornado event. These are the same tornado events that are in ECCC's records since NTP confirmations are discussed with ECCC each season to ensure agreement.
For tornado warnings, we used the official ECCC tornado warnings that were the initial warning for that region (N=338), obtained via NTP partner Instant Weather. All related ECCC convective watches, warnings and advisories were also obtained.
ECCC does issue other warnings and advisories that mention the possibility of a tornado. Severe thunderstorm warnings can contain the phrase, “Remember, severe thunderstorms can produce tornadoes”. Also, weather advisories for funnel clouds indicate the possibility of “a weak landspout tornado”. However, in both cases, these ‘tornado tags’ are used far more often than tornadoes actually occur. For example, in 2022, 1660 severe thunderstorm warnings used the tornado tag, which is 43% of all severe thunderstorm warnings issued.
Methods
To keep the assessment simple, we take an 'all of Canada' approach, combining the data from all regions of the country. We then employ the widely used contingency table model to calculate the Probability of Detection (POD), the False Alarm Ratio (FAR), the Critical Success Index (CSI) and the Bias. These parameters tell us the following:
POD – What fraction of tornadoes were correctly tornado-warned? (0 is worst, 1 is best)
FAR – What fraction of tornado-warned events were false alarms? (1 is worst, 0 is best)
CSI – How well did the tornado warnings correspond to confirmed tornadoes? (0 is worst, 1 is best)
Bias – Over-warning (>1) versus under-warning (<1) (1 is usually considered optimal)
We use a somewhat relaxed but realistic definition of a tornado warning success or 'hit': the tornado warning has to be issued before, during, or within 10 minutes from the start of the tornado event. This covers a large number of scenarios, including those in which a spotter reports a tornado to ECCC but the weak, brief tornado has dissipated by the time the warning is broadcast (we believe 10 minutes is fair here). This extra time allowance should not be a factor for stronger, longer-lived tornadoes that often have a lifetime beyond 10 minutes.
When dealing with multiple tornado events, we consider one tornado warning for a region with three tornadoes in it as three separate hits.
And here is how each parameter is calculated:
POD = H / (H+M) FAR = FA / (H+FA) CSI = H / (H+M+FA) Bias = (H+FA) / (H+M)
where H is hits, M is misses and FA is false alarms.
Assessment Results
The following image shows the contingency tables and parameter values using the 2023-2024 data, again considering all tornadoes.
We can look at the POD in a number of ways. For example, is the POD better when the area of interest is confined to the Doppler domain of the radar coverage, since that is where storm rotation signatures can be detected? For tornadoes occurring within Doppler range the POD is 0.38 – so, not much higher than for all tornadoes (0.36).
Below are the POD values by EF-scale rating. The table shows that the POD increases going from EF0 to EF2+ (from 0.18 to 0.53), with the second highest POD (0.48) for the EF0-Default ranking (i.e., no damage indicators were impacted). The majority of the successfully-warned EF0-Default tornadoes were either from supercells in Doppler range, or were landspouts that were documented in real-time by witnesses on social media (and likely directly reported to ECCC offices as they were imminent or occurring).
We can also compare the POD values for different tornado types. The NTP defines three types of tornadoes: supercell, quasi-linear convective system (QLCS) and landspout. Supercell storms are capable of producing the strongest tornadoes, up to the top of the EF scale at EF5. Landspout tornadoes form with loosely organized storms and therefore typically produce weak damage, EF0 to EF1 on the EF scale. QLCS tornadoes occur along the leading edge of a line of storms and are typically stronger and longer-lived than landspout tornadoes, though not quite up to the level of supercell tornadoes.
Landspout tornadoes rarely have an identifiable signature on radar and are often brief, making them the hardest to warn for. Other the other hand, supercell thunderstorms often have prominent, long-lived features that are often easy to detect with radar, and they occur in a dynamic environment that can sometimes be forecast days in advance. That makes it less challenging to anticipate supercell tornadoes.
It is not always so cut-and-dried when it comes to tornado type, however. In some cases, the tornado appears to be a combination of types - what we call 'hybrid' tornadoes.
The tornado type was subjectively determined for each tornado using (where available) radar imagery, lightning data, satellite imagery, surface weather analysis maps, storm environment information, and visual characteristics of the parent storm. The number of tornadoes, percentage of total tornadoes, and POD for each tornado type are as follows:
The POD is notably higher for supercell (0.56) and hybrid (0.57) tornadoes, with supercell tornadoes also making up the largest proportion of tornadoes in the study at nearly 40%. The POD for hybrid tornadoes is also relatively high, due to the fact that many of the hybrid cases involve supercell processes (and therefore radar-detected storm rotation), and there is a small sample size (7). For QLCS tornadoes, the POD is lowest at only 0.17.
The next table shows when tornado warnings were issued relative to the recorded tornado start times. A reminder here that the first two rows would be considered ‘hits’ in our assessment. The table also compares the 2019-2021 and 2022 average percentages with the 2023-2024 percentages. The percentage of tornado warnings issued before the tornado start time increased from 11% to 26% over the years covered in these evaluations. The proportion of tornadoes with no warning has decreased over the years from 74% to 63%, but remains a high percentage.
We can break it down even further by examining the lead time of tornado warnings by EF-scale rating, as seen in the table below. For the 2023-2024 period, there were notably few EF2+ tornadoes (19) compared to 2019-2021 (50) and 2022 (30) evaluations. While the sample size is small, 10 or 53% of the EF2+ tornadoes in 2023-2024 had tornado warnings issued before or during the event, which is a significant improvement over previous years (30% in 2019-2021, 33% in 2022).
NTP Recommendations
Based on our independent assessments of tornado warning performance in Canada since 2019, the NTP has recommended several actions to improve results including:
* Increasing the number of tornado warnings to increase POD and CSI, without significantly increasing FAR,
* Improving lead times by finding ways to get tornado warnings out sooner, mainly by increasing forecaster proficiency and therefore confidence, and
* Ensuring forecasters have cutting-edge tornado detection, nowcasting and forecasting tools, especially those related to Canada's recently upgraded radar network.
All of these are still important, and it is good to see the slight increase in the number of tornadoes with a tornado warning issued before the tornado occurred for the 2023-2024 seasons. It is also good to see that the slight improvement in overall score on the report card did not come at the expense of a higher false alarm ratio, which remained at the 0.81 recorded for the last assessment.
However, there are still many tornadoes occurring without any tornado warning issued, and of particular concern is that nearly half of EF2-rated 'strong' tornadoes did not have a tornado warning. These EF2 tornadoes occurred with supercell or QLCS parent storms, some with visible storm rotation on radar. It would be good to know if these events have any common characteristics from a forecasting and nowcasting perspective.
On the other hand, there are also still many tornado warnings that are false alarms. While reducing false alarms is of less importance than ensuring all EF2+ tornadoes have a tornado warning, it would still be worth knowing if these events had any commonalities.
To that end, we recommend that a study be undertaken to examine more closely the tornado events that did not have a warning, and those where tornado warnings were false alarms, to try to identify any common forecasting and/or nowcasting issues. The EF2 tornado events that went unwarned should be prioritized. It is possible improvements related to radar - more radar-based training, different scanning strategies, and/or better radar-based tools - would help to address gaps in these areas. The NTP may be able to provide assistance here.
Summary
The NTP has completed its third independent assessment of tornado warning performance in Canada, covering the 2023 and 2024 seasons. The main results are:
* Though the Probability of Detection of 0.36 for tornado warnings is still well below the ECCC target of 0.5, it is a significant improvement over 0.23 for 2019-2021 and a slight improvement over 0.35 in 2022,
* Some improvements in tornado warning POD and lead time for 2023-2024, and a notable increase in the number of tornado watches preceding successful tornado warnings, led to the highest score to date (60.7%) on the NTP’s tornado warning performance assessment ‘report card’, and
* While a tornado warning was issued before tornado development 26% of the time in 2023-2024 (up from 11% for 2019-2021 and 24% in 2022), 63% of tornadoes still had no tornado warning - including 9 tornadoes rated EF2.
Acknowledgements
Thanks to Instant Weather for help with tracking down all of the alert messages in 2023-2024, and to Misheyla Iwasiuk for allowing us to use her amazing Scarth tornado photo once again.
