Historical benchmark · low Earth orbit

Borea Benchmark Quantifying the operational value of better space-weather information in low Earth orbit

A historical benchmark: same storm, same object, same propagator; only the geomagnetic information changes.

The re-entry margin is not calibrated and neither registered milestone (M5b, M5c) passed its criterion: read the limits before using any number.

Submit a forecast Reproduce the table

Storm clock · hours from the storm start, not from t0

Illustration, not data: a procedural Earth, a schematic aurora, a symbolic shell of low-orbit objects and a schematic storm curve. The clock moves the picture only; no result on this page changes.

Key numbers

On all storms with the DTM2020 base, knowing the geomagnetic indices and the solar flux instead of freezing them at the forecast epoch lowers the absolute error of the altitude lost at 168 h and of the re-entry epoch (difference between the median absolute errors, 90 % band excluding zero). On the 75 storms since 2000 that have a forecast report, the operational forecast recovers about two thirds of the gain at 168 h (DTM2020); on the re-entry epoch neither path is distinguishable from zero on that subset.

Error difference · altitude lost · 168 h
−610m
Difference between the median absolute errors of the altitude lost at 168 h: observed indices − persistence
90 % band [−745, −495]
Negative: a smaller error than with persistence.
On the 75 storms since 2000 with a report: operational forecast −468 m, observed indices −680 m (DTM2020)
Error difference · re-entry epoch
−8.2h
Difference between the median absolute errors of the re-entry epoch: observed indices − persistence
90 % band [−10.0, −4.9]
Negative: a smaller error than with persistence.
On the 75 storms since 2000 with a report: operational forecast −3.9 h, observed indices −3.9 h; neither band excludes zero
Sample
275
geomagnetic storms with Kp ≥ 7, 1970–2023
1 605 distinct objects, 1 131 real re-entries
Re-entry window
0.70–0.78
Coverage of the predicted re-entry window in storms, across bases and paths
Below the registered minimum of 0.80: the re-entry margin is not calibrated. Read the limits
Storms with a report
75
storms since 2000 where a report exists
On these storms the forecast gain on the re-entry epoch is not distinguishable from zero.

First two tiles: all storms, DTM2020 base, 90 % band. The other bases and the storms since 2000 with a report are in the table.

What this measures

A historical benchmark: same storm, same object, same propagator; only the geomagnetic information changes.

A geomagnetic storm heats the thermosphere and raises the neutral density at the altitudes of low Earth orbit within hours. For an object already low this is a drag problem: it loses altitude faster than a forecast issued before the storm anticipates, its predicted position drifts along track and, if it is close to re-entry, it comes down earlier than planned. The density models used operationally are driven by geomagnetic indices; when the indices are not yet known, which is the situation of every forecast issued before the storm, the error of the input adds to the error of the model.

Illustration, not data: schematic shape of the three information paths after t0.

The three information paths

observed indices: the geomagnetic indices later observed, together with the observed daily solar flux (F10.7); what a perfect forecast of these inputs would have given (hindcast ceiling).
persistence: the 3-hourly ap and Kp of the interval that starts at t0, held for the whole horizon, with the solar flux frozen at t0. The held interval ends 3 h after t0, so this path uses up to 3 h of information not yet available at t0.
operational forecast (RSGA): the daily Ap of the NOAA SWPC Report and Forecast of Solar and Geophysical Activity issued at or before t0, with the solar flux frozen at t0. The report covers the UTC day of t0 and the next two days; from the fourth day on, its last value is held. Before 2000 no report exists and this path falls back to persistence (counted and declared).

The benchmark measures how much of that error comes from not knowing the indices. It takes every geomagnetic storm with Kp ≥ 7 of 1970–2023 (275 storms) and every catalogue object with perigee below 300 km at the forecast epoch t0 = storm start − 24 h (1 605 distinct objects, 1 131 real re-entries). Each object is propagated through the same storm with the same propagator and the same 60-member uncertainty layer; the only thing that changes is the information path fed to the density model: the geomagnetic indices and the solar flux later observed, the indices of the interval that starts at t0 held for the whole horizon with the solar flux frozen at t0 (persistence), or the operational NOAA SWPC forecast issued at or before t0. The result is measured in metres of altitude lost at 72 h, 168 h and 336 h and in hours of re-entry epoch, with 2 000-resample bootstrap bands over storms, and tested on storms the fitted tables never saw (leave-one-storm-out).

The benchmark does not forecast the storm, is not an operational product and contains no learned model: no advantage of an AI method is shown here. It reports what information is worth, not how to obtain it.

The table

Every cell is a value read from a registered result file of the research programme; the file and key are in the data-source attribute of the cell. Bands are 90 % storm-cluster bootstrap bands. The registered range for a calibrated 10–90 band is [0.80, 0.95]: coverages are reported next to it, not read against it, and none of the registered milestones (M5b, M5c) passed its criterion. The research repository is available on request.

Information paths: observed indices persistence operational forecast (RSGA) The three information paths

Altitude lost: coverage of the 10–90 band and median absolute error of the ensemble median

At 72 h only the coverage of the 10–90 band is registered, for persistence and the operational forecast: no error, no difference between paths and no width exist at that horizon, and the observed-indices mode has not been read there.

Density base	Information path	72 h	168 h		336 h		Objects (storms)
Density base	Information path	Coverage 10–90	Coverage 10–90	Median abs. error, m	Coverage 10–90	Median abs. error, m	Objects (storms)
DTM2020 operational	observed indices	—	0.842 [0.827, 0.857]	1 330 [1 230, 1 451]	0.801 [0.784, 0.818]	2 558 [2 285, 2 786]	3 804 (275)
DTM2020 operational	persistence	0.774 [0.747, 0.799]	0.867 [0.851, 0.883]	1 940 [1 797, 2 124]	0.858 [0.841, 0.873]	3 306 [2 997, 3 661]	3 804 (275)
DTM2020 operational	operational forecast (RSGA)	0.701 [0.672, 0.731]	0.786 [0.766, 0.805]	1 651 [1 507, 1 775]	0.757 [0.736, 0.778]	2 972 [2 817, 3 262]	3 804 (275)
NRLMSIS 2.1	observed indices	—	0.916 [0.904, 0.927]	1 236 [1 133, 1 342]	0.903 [0.891, 0.916]	2 279 [2 108, 2 460]	3 804 (275)
NRLMSIS 2.1	persistence	0.880 [0.859, 0.898]	0.925 [0.912, 0.937]	1 547 [1 399, 1 697]	0.907 [0.895, 0.921]	2 779 [2 573, 3 027]	3 804 (275)
NRLMSIS 2.1	operational forecast (RSGA)	0.829 [0.806, 0.852]	0.874 [0.858, 0.890]	1 636 [1 451, 1 788]	0.850 [0.832, 0.868]	3 013 [2 757, 3 273]	3 804 (275)
JB2008	observed indices	—	0.841 [0.816, 0.865]	1 017 [845, 1 173]	0.815 [0.787, 0.841]	2 245 [1 912, 2 493]	1 138 (89)
JB2008	persistence	0.776 [0.731, 0.818]	0.875 [0.846, 0.900]	1 693 [1 452, 1 940]	0.880 [0.853, 0.904]	3 248 [2 881, 3 831]	1 138 (89)

Source: data/derived/benchmark_table.csv → m5c_eval_v1.json (summary), d050_h72_coverage_v1.json (h72.storms).

Re-entry window (truth: TIP message, else decay message, else SATCAT day)

Density base	Information path	Coverage 10–90	Width / observed time	Median abs. error, h	Re-entries (storms)
DTM2020 operational	observed indices	0.717 [0.690, 0.743]	0.59 [0.57, 0.61]	27.0 [25.0, 31.0]	1 131 (258)
DTM2020 operational	persistence	0.777 [0.752, 0.800]	0.83 [0.80, 0.86]	35.1 [32.2, 38.0]	1 131 (258)
DTM2020 operational	operational forecast (RSGA)	0.702 [0.676, 0.728]	0.59 [0.57, 0.61]	32.0 [29.0, 35.0]	1 131 (258)
NRLMSIS 2.1	observed indices	0.761 [0.738, 0.785]	0.78 [0.75, 0.80]	30.0 [27.0, 33.0]	1 132 (258)
NRLMSIS 2.1	persistence	0.782 [0.759, 0.804]	0.89 [0.84, 0.92]	34.0 [31.3, 37.0]	1 132 (258)
NRLMSIS 2.1	operational forecast (RSGA)	0.761 [0.736, 0.782]	0.78 [0.74, 0.82]	35.9 [32.9, 39.9]	1 132 (258)
JB2008	observed indices	0.701 [0.639, 0.756]	0.55 [0.52, 0.58]	27.9 [23.1, 32.1]	274 (78)
JB2008	persistence	0.763 [0.705, 0.810]	0.87 [0.81, 0.98]	33.2 [29.5, 40.7]	274 (78)

Source: data/derived/benchmark_table.csv → m5c_eval_v1.json (summary, reentry).

Value of information: difference between the median absolute errors of two paths, same objects

The value of information is the difference between the median absolute errors of two paths, computed on the same rows (same storm window and object, truth available for both), with a 90 % band from a bootstrap over storms; negative means the first path has the smaller error; bold means the band excludes zero. On all storms with the DTM2020 base, knowing the geomagnetic indices and the solar flux instead of freezing them at the forecast epoch changes the median absolute error at 168 h by −610 m and the re-entry epoch error by −8.2 h. On the 75 storms since 2000 that have a forecast report, the operational forecast recovers about two thirds of the gain at 168 h (DTM2020); on the re-entry epoch neither path is distinguishable from zero on that subset.

Storm set	Density base	Pair	Δ abs. error 168 h, m	Δ abs. error 336 h, m	Δ re-entry epoch error, h	n at 168 h: paired rows (storms)
all storms	DTM2020 operational	observed − persistence	−610 [−745, −495]	−748 [−1 009, −538]	−8.2 [−10.0, −4.9]	1 870 (257)
all storms	DTM2020 operational	forecast − persistence	−289 [−437, −182]	−333 [−586, −47]	−3.2 [−5.1, −0.9]	1 870 (257)
all storms	DTM2020 operational	forecast − observed	321 [198, 428]	415 [230, 684]	5.0 [2.2, 6.8]	1 870 (257)
all storms	NRLMSIS 2.1	observed − persistence	−311 [−426, −196]	−500 [−751, −298]	−4.0 [−7.0, −2.0]	1 870 (257)
all storms	NRLMSIS 2.1	forecast − persistence	89 [−60, 214]	235 [−46, 451]	1.9 [−0.1, 4.3]	1 870 (257)
all storms	NRLMSIS 2.1	forecast − observed	400 [242, 521]	735 [510, 970]	6.1 [3.9, 9.0]	1 870 (257)
all storms	JB2008	observed − persistence	−676 [−909, −481]	−1 004 [−1 410, −773]	−5.4 [−11.6, −2.2]	906 (89)
storms since 2000 with a report	DTM2020 operational	observed − persistence	−680 [−861, −488]	−672 [−1 312, −366]	−3.9 [−12.5, 0.4]	765 (75)
storms since 2000 with a report	DTM2020 operational	forecast − persistence	−468 [−661, −288]	−654 [−1 305, −208]	−3.9 [−10.1, 0.6]	765 (75)
storms since 2000 with a report	DTM2020 operational	forecast − observed	212 [35, 356]	18 [−317, 490]	0.3 [−3.7, 6.0]	765 (75)
storms since 2000 with a report	NRLMSIS 2.1	observed − persistence	−351 [−515, −143]	−579 [−919, −221]	−8.3 [−16.8, −1.2]	765 (75)
storms since 2000 with a report	NRLMSIS 2.1	forecast − persistence	−53 [−210, 206]	98 [−268, 598]	2.2 [−2.6, 9.5]	765 (75)
storms since 2000 with a report	NRLMSIS 2.1	forecast − observed	298 [113, 550]	677 [349, 1 111]	10.2 [2.9, 18.8]	765 (75)
storms since 2000 with a report	JB2008	observed − persistence	−742 [−972, −514]	−1 137 [−1 505, −749]	−6.3 [−12.7, −3.0]	765 (75)

Source: data/derived/benchmark_table.csv → m5c_eval_v1.json (value_of_information). The differences between the p90 absolute errors and the registered verdict tables are in the CSV and in data/derived/benchmark_table.md.

Erratum (2026-10-09): the header of benchmark_table.md says "paired differences"; the values are differences between the median absolute errors of two paths on paired rows (research repository, error E-014). The file is unchanged because its checksum is part of the recorded provenance.

Limits

Read these before using any number on this page.

The re-entry margin is not calibrated. The coverage of the predicted re-entry window in storms is between 0.70 and 0.78 across bases and paths, below the registered minimum of 0.80; the registered milestones M5b and M5c did not pass.
Source: m5c_eval_v1.json summary.*.reentry (post-D-038); registered verdicts m5c_verdict_v1.json, m5b_verdict_v1.json not rewritten (D-038); docs/03 «Correzione D-038 applicata», «Verdetto M5c», «Verdetto M5b»; D-029, D-032
The defect is worse below 200 km of perigee and is already present in quiet windows. In the quiet controls the coverage is between 0.69 and 0.80; below 200 km (DTM2020, observed indices, 37 re-entries) it is 0.62 with 0.38 of the re-entries after the predicted window, against 0.83 and 0.17 in the 260–300 km band: the final descent runs late there.
Source: m5c_eval_v1.json quiet_controls.*.reentry and quiet_late_by_perigee_band; docs/03 «Dove manca la discesa finale»
The coverage at 72 h is reported without a verdict: between 0.70 and 0.88 across bases and paths. No error and no difference between paths are registered at that horizon.
Source: d050_h72_coverage_v1.json; docs/03 «Il numero mancante a 72 h»; D-050
The machine-learning attempts of the research programme did not pass the criteria registered before computing: a residual correction on each density base, a multi-model ensemble with bias correction and non-negative weights, the research version of DTM2020, zero-shot time-series foundation models as index forecasters, and coronagraph CME catalogues as precursors of the storm. None enters this kit and no number from them is shown here.
Source: docs/03 (results R2, R5, R6, R10 and the residual models of the first milestones); D-019, D-021, D-035, D-036, D-039, D-043, D-044
Forecasting the storm itself 1–3 days ahead is out of scope for a physical reason: the orientation of the interplanetary magnetic field that decides whether a coronal mass ejection couples to the magnetosphere is measurable only at L1, about an hour before arrival. The operational forecast verified above does not issue Kp ≥ 7 alarms at that lead time on this sample.
Source: docs/03 «Il previsore operativo non emette allarmi Kp ≥ 7 a 1–3 giorni»; D-043, D-050
The 81-day mean of F10.7 follows the base. DTM2020 and JB2008 read the centred mean of their index tables, which includes observed values after t0, in every path they run: frozen at t0 outside the observed path, read at every time in it. NRLMSIS 2.1 reads the trailing mean of the days before t0 in the persistence and forecast paths, with no value after t0, and in the observed path the centred mean of the pymsis table at every time.
Source: research repository: src/borea/consequences/index_paths.py (module docstring, «Solar drivers»), density.py (BaseDensity.__call__), borea.models.msis_baseline.f107a_trailing, eval/dtm2020.py, eval/jb2008_baseline.py; D-020, D-023
The demonstrator is a hindcast with indices, not a forecast issued at t0. The 81-day mean of F10.7 follows the base. DTM2020 and JB2008 read the centred mean of their index tables, which includes observed values after t0, in every path they run: frozen at t0 outside the observed path, read at every time in it. NRLMSIS 2.1 reads the trailing mean of the days before t0 in the persistence and forecast paths, with no value after t0, and in the observed path the centred mean of the pymsis table at every time.
Source: docs/03 «Dimostratore 2024», declarations; demo_2024_v1.json (bases dtm2020, msis, jb2008; modes_reported); the same code as the benchmark: research repository: src/borea/consequences/index_paths.py (module docstring, «Solar drivers»), density.py (BaseDensity.__call__), borea.models.msis_baseline.f107a_trailing, eval/dtm2020.py, eval/jb2008_baseline.py; D-020, D-023
Two of the three density bases are licensed for non-commercial use only: DTM2020 for academic, non-commercial purposes, NRLMSIS 2.1 for research, academic and non-profit purposes. The benchmark was run under research use. Space-Track element sets are not redistributed; only aggregates with citation.
Source: docs/07; README of the kit

The operational forecast, verified

The operational path was verified directly against persistence on the 75 storms since 2000 that have a report at t0, with a rule registered before the computation (D-033): the forecast must lower the RMSE of the daily Ap at 24 h, 48 h, 72 h and on the 0–168 h mean, with the band excluding zero. It does not: the check at 48 h fails, and the forecast is read as not carrying more information than persistence on these storms. Where it does carry information is on the Kp scale and on the weekly mean level of the indices, not on the peak day.

Horizon	Δ RMSE of daily Ap, forecast − persistence	Beats persistence
t0 + 24 h	−8.0 [−12.3, −3.3]	yes
t0 + 48 h	1.5 [−7.5, 9.3]	no
t0 + 72 h	−7.8 [−11.9, −3.9]	yes
mean 0-168 h	−7.4 [−10.3, −4.5]	yes

On the Kp scale the forecast does beat persistence at every horizon (skill against persistence): t0 + 24 h: 0.38 [0.31, 0.45]; t0 + 48 h: 0.38 [0.13, 0.55]; t0 + 72 h: 0.40 [0.17, 0.54]; mean 0-168 h: 0.68 [0.59, 0.75].

Numbers transcribed from docs/03 of the research repository («Verifica del previsore operativo»); the result file itself is not in the kit. Transcription: site/docs03_forecaster_skill.json.

Skill on strong storms against ordinary days: on all 8 744 daily reports of 2000–2023 the forecast beats persistence with a skill of 0.48, 0.51 and 0.50 on the first, second and third forecast day; on the 75 Kp ≥ 7 storms the same skill is 0.13, 0.17 and 0.19. The skill that an all-days verification measures is mostly skill on quiet days and on recurrence.

2024 demonstrator

Two storms outside the benchmark sample, run with the same layer without any recalibration: the Gannon storm (2024-05-10 to 2024-05-12, Kp max 9.0) and the October storm (2024-10-10 to 2024-10-11, Kp max 8.7). Coverage is reported per horizon and information path without a criterion: two storms are two units, there is no power and no band across storms. The bands shown are object-bootstrap bands, optimistic, for monitoring only. Results are shown at storm level only: no object is named and no per-object row is published.

passive, circular: passive catalogue objects with perigee below 300 km at t0, no manoeuvre in the window, circular orbits
passive, eccentric: passive catalogue objects with perigee below 300 km at t0, eccentric orbits (Gauss propagator)
Starlink: Starlink satellites with perigee below 450 km at t0, no upward jump before t0, stable ballistic coefficient; truth truncated at the first upward jump after t0

Sample at storm level

Storm	Population	Objects in window	Eligible	Run	Perigee at t0, km (p10 / p50 / p90)
Gannon (May)	passive, circular	1 075	37	30	148 / 251 / 274
Gannon (May)	passive, eccentric	1 075	76	75	171 / 240 / 292
Gannon (May)	Starlink	2 695	100	57	300 / 370 / 435
October	passive, circular	1 008	48	36	154 / 250 / 288
October	passive, eccentric	1 008	76	76	171 / 254 / 289
October	Starlink	3 289	596	224	359 / 443 / 446

Source: demo_2024_v1.json (runs.*.windows), demo_2024_extra_v1.json (perigee_t0).

Coverage of the altitude lost, DTM2020 base (monitoring, not criterion)

Storm	Population	Information path	Coverage 10–90 of the altitude lost (rows with truth)
Storm	Population	Information path	24 h	72 h	168 h
Gannon (May)	passive, circular	persistence	1.000 [1.00, 1.00] n 19	0.421 [0.21, 0.63] n 19	0.692 [0.46, 0.92] n 13
Gannon (May)	passive, circular	operational forecast (RSGA)	1.000 [1.00, 1.00] n 19	0.895 [0.79, 1.00] n 19	0.923 [0.77, 1.00] n 13
Gannon (May)	passive, circular	observed indices	1.000 [1.00, 1.00] n 19	0.895 [0.79, 1.00] n 19	0.923 [0.77, 1.00] n 13
Gannon (May)	passive, eccentric	persistence	0.934 [0.87, 0.98] n 61	0.500 [0.40, 0.60] n 60	0.714 [0.61, 0.80] n 56
Gannon (May)	passive, eccentric	operational forecast (RSGA)	0.869 [0.79, 0.93] n 61	0.800 [0.72, 0.88] n 60	0.911 [0.84, 0.96] n 56
Gannon (May)	passive, eccentric	observed indices	0.885 [0.82, 0.95] n 61	0.850 [0.77, 0.92] n 60	0.857 [0.77, 0.93] n 56
Gannon (May)	Starlink	persistence	0.519 [0.40, 0.63] n 52	0.375 [0.25, 0.50] n 48	0.500 [0.38, 0.62] n 42
Gannon (May)	Starlink	operational forecast (RSGA)	0.385 [0.27, 0.50] n 52	0.375 [0.27, 0.48] n 48	0.452 [0.33, 0.57] n 42
Gannon (May)	Starlink	observed indices	0.442 [0.33, 0.56] n 52	0.438 [0.31, 0.56] n 48	0.452 [0.33, 0.57] n 42
October	passive, circular	persistence	0.913 [0.83, 1.00] n 23	1.000 [1.00, 1.00] n 15	1.000 [1.00, 1.00] n 15
October	passive, circular	operational forecast (RSGA)	0.870 [0.74, 0.96] n 23	1.000 [1.00, 1.00] n 15	1.000 [1.00, 1.00] n 15
October	passive, circular	observed indices	0.913 [0.83, 1.00] n 23	0.933 [0.80, 1.00] n 15	0.933 [0.80, 1.00] n 15
October	passive, eccentric	persistence	0.939 [0.89, 0.98] n 66	0.969 [0.94, 1.00] n 65	0.951 [0.90, 0.98] n 61
October	passive, eccentric	operational forecast (RSGA)	0.909 [0.85, 0.95] n 66	0.892 [0.83, 0.95] n 65	0.853 [0.77, 0.92] n 61
October	passive, eccentric	observed indices	0.879 [0.80, 0.94] n 66	0.831 [0.75, 0.91] n 65	0.836 [0.75, 0.92] n 61
October	Starlink	persistence	0.469 [0.41, 0.53] n 175	0.258 [0.20, 0.31] n 163	0.242 [0.19, 0.30] n 153
October	Starlink	operational forecast (RSGA)	0.480 [0.42, 0.54] n 175	0.258 [0.20, 0.31] n 163	0.209 [0.16, 0.26] n 153
October	Starlink	observed indices	0.469 [0.41, 0.53] n 175	0.221 [0.17, 0.28] n 163	0.229 [0.17, 0.28] n 153

Source: demo_2024_v1.json (aggregate.*.horizons). Bands: object bootstrap, optimistic.

The Starlink population reads as effective decay by construction, never as a density model: the objects may be thrusting, and the altitude observed on the October objects falls less than predicted. No comparison with the operator is made.

Re-entries, DTM2020 base

Re-entries counted within the run horizon, with the coverage of the predicted re-entry window (DTM2020 base). Reported, not read.

Storm	Population	Information path	Objects run	Re-entered in the run horizon	Window coverage	Median abs. error, h	Median width, h
Gannon (May)	passive, circular	persistence	30	24	0.833	21.6	92
Gannon (May)	passive, circular	operational forecast (RSGA)	30	24	0.875	11.1	64
Gannon (May)	passive, eccentric	persistence	75	0	—	—	—
Gannon (May)	passive, eccentric	operational forecast (RSGA)	75	0	—	—	—
Gannon (May)	Starlink	persistence	52	12	0.333	159.9	275
Gannon (May)	Starlink	operational forecast (RSGA)	52	12	0.417	99.7	188
October	passive, circular	persistence	36	26	0.962	10.0	55
October	passive, circular	operational forecast (RSGA)	36	26	0.923	12.7	42
October	passive, eccentric	persistence	76	3	0.667	81.4	70
October	passive, eccentric	operational forecast (RSGA)	76	3	0.667	95.9	58
October	Starlink	persistence	177	15	0.667	47.4	192
October	Starlink	operational forecast (RSGA)	177	15	0.533	45.8	152

Source: demo_2024_v1.json (aggregate.*.reentry).

The forecast in force at t0

Daily Ap forecast of the report in force at t0 against the Ap later observed (GFZ). Neither storm was forecast on its peak day.

Storm	Day	Ap forecast (RSGA in force at t0)	Ap observed (GFZ)
Gannon (May)	2024-05-09	12	5
Gannon (May)	2024-05-10	37	105
Gannon (May)	2024-05-11	50	271
October	2024-10-09	10	18
October	2024-10-10	5	97
October	2024-10-11	14	116

Source: demo_2024_extra_v1.json (rsga_in_force_at_t0).

Bring your own forecast

You can evaluate your own path of geomagnetic indices with the same consequence layer as this table: same base, same propagator, same uncertainty layer, same seed, with persistence, the RSGA scenario and the observed indices of the same base on the same objects beside it as references, the difference between the median absolute errors of your path and of each reference with its storm-bootstrap band.

UploadOne JSON file in the format borea-external-path/1.
RunOne run at a time on the project server; a private page counts down to your result.
ResultA private page reached only by its link; listed publicly only if you ask.

What to send

One JSON file in the format borea-external-path/1 (documented in FORECAST_FORMAT.md of the kit): one series of daily Ap or 3-hour ap per storm, issued at t0 = storm start − 24 h, covering at least 168 h after t0, on the GFZ ap scale. The storm list with t0 and the required keys, an example file and the format document are in the kit.

What you get

The evaluation runs on the project server, one job at a time, on the 20 leave-one-storm-out storms of the sample, with DTM2020 operational or NRLMSIS 2.1 as the base; the evaluation took 42 min when the command was timed once on that server (a measure of one machine, not a promise); each run waits for the ones before it, and its private page counts down to the result. The result is a private page reached only by the link given at submission: the same tables as above for your path, the references, the differences between the median absolute errors, the provenance block and the four result files. It appears in the public list only if you ask for it, under the display name you give, if any, and the list is not a ranking.

Limits of the service

3 evaluations per day per network address, 12 per day in total, files under 4 MB; no account, no email; the form sets one short-lived cookie: keep the link to your result. The uploaded file is deleted 7 days after the evaluation or 30 days after submission, whichever comes first; the aggregate result is kept. The declaration of research, academic or non-profit use is required: the density models run under academic research licences on the project server. The benchmark cannot verify that your path uses no information observed after t0: that is self-declared, and no result is read as forecast skill. An external path is never a reference and changes nothing on this page.

Submit a forecast file Public results (opt-in)

Reproduce

The kit (code under MIT, derived tables under CC BY 4.0) and the research repository are available on request from [email protected].

Clone the kit, create a virtual environment and install the pinned dependencies: python -m venv .venv && .venv/bin/pip install -r requirements-lock.txt && .venv/bin/pip install -e .
Regenerate the table from the registered result files copied in data/source/: .venv/bin/python -m borea_benchmark.table. The command stops with FAILED if any file differs from the manifest or any cell differs from its source.
Regenerate this page: .venv/bin/python -m borea_benchmark.site; run the tests: .venv/bin/python -m pytest.

Re-running the propagation itself (not needed for the table) requires accepting the licences of two of the three density bases: DTM2020 (CNES, MCM/DTM licence: academic, non-commercial use; no modification or redistribution of code or coefficients) and NRLMSIS 2.1 (NRL Open Source Academic Research License: research, academic and non-profit use only; pymsis builds the model from source downloaded from NRL). JB2008 (pyatmos, MIT) runs without one. NRLMSISE-00 is not a base of the layer. Geomagnetic indices are GFZ (CC BY 4.0); the forecast archive is NOAA SWPC (a work of the US Government); element sets and re-entry records come from Space-Track with a personal account and are not redistributed.

Models and licences

Borea Benchmark is a free, non-commercial research service. It runs the density models below unmodified on the project's server and returns only aggregate error statistics; the model software is never distributed.

NRLMSIS 2.1. This software incorporates the MSIS® empirical atmospheric model software designed and provided by NRL. Use is governed by the Open Source Academic Research License Agreement contained in the file nrlmsis2.1_license.txt. Licence and source: https://map.nrl.navy.mil/map/pub/nrl/NRLMSIS/NRLMSIS2.1/. MSIS® is a registered trademark of the Government of the United States of America.

DTM2020. DTM2020 (operational) © CNES. Used unmodified under the MCM/DTM licence, for academic, non-commercial purposes. Reference: S. Bruinsma and C. Boniface (2021), The operational and research DTM-2020 thermosphere models, J. Space Weather Space Clim. 11, 47, doi:10.1051/swsc/2021032. Licence: https://github.com/swami-h2020-eu/mcm/blob/master/LICENSE.

Novelty

No equivalent work was found with the systematic search protocol registered on 2026-09-22.