Observation catalog should be filtered against masked forecast for M and S evaluation tests
|Reported by:||liukis||Owned by:||liukis|
|Priority:||major||Milestone:||CSEP Version 12.1.0|
Jeremy pointed out that original implementation of M and S tests had a logical error - email communication from January 4, 2011, "possible logical mistake in the M- and S- tests":
In the M- and S-tests, to normalize we count the number of eqks that fall inside unmasked forecast bins using relm_NumberEventsCatalog(vForecast, cObservedCatalog). The way that I had my scripts set up, this number might not be the same as the total number of eqks in the catalog. In particular, I didn't filter the catalog to contain only the events that fell within unmasked bins. But then the M- and S-tests could use the entire catalog to compute the observed likelihood, so this would bias the results. Perhaps this is more clear w/ an example. Consider an M-test with a model that only has unmasked bins in southern California, and let's say the model provides rates in every magnitude bin. Let's say there were 20 events total, 10 in unmasked bins, the other 10 in northern California. When we count the number of eqks that fall inside unmasked bins, we get the correct number (10) and normalize the magnitude forecast so its total rate is 10. Then we compute the likelihood of the observation given the magnitude forecast, but we end up using all 20 eqks to do so (b/c all of them fall within magnitude bins that are unmasked). The way to avoid this is to filter the catalog against the original lat/lon/depth/mag forecast and then proceed. Perhaps you're already doing this further upstream, but if not, we should correct it. I also note that this will almost never be a problem for the S-test because it's very unlikely (perhaps even not allowed) that a forecast masks part of a spatial cell.