Opened 5 years ago
Closed 5 years ago
#265 closed defect (fixed)
Observation catalog should be filtered against masked forecast for M and S evaluation tests
Reported by: | liukis | Owned by: | liukis |
---|---|---|---|
Priority: | major | Milestone: | CSEP Version 12.1.0 |
Component: | Toolkit | Version: | 10.7.0 |
Keywords: | Cc: |
Description
Jeremy pointed out that original implementation of M and S tests had a logical error - email communication from January 4, 2011, "possible logical mistake in the M- and S- tests":
In the M- and S-tests, to normalize we count the number of eqks that fall inside unmasked forecast bins using relm_NumberEventsCatalog(vForecast, cObservedCatalog). The way that I had my scripts set up, this number might not be the same as the total number of eqks in the catalog. In particular, I didn't filter the catalog to contain only the events that fell within unmasked bins. But then the M- and S-tests could use the entire catalog to compute the observed likelihood, so this would bias the results. Perhaps this is more clear w/ an example. Consider an M-test with a model that only has unmasked bins in southern California, and let's say the model provides rates in every magnitude bin. Let's say there were 20 events total, 10 in unmasked bins, the other 10 in northern California. When we count the number of eqks that fall inside unmasked bins, we get the correct number (10) and normalize the magnitude forecast so its total rate is 10. Then we compute the likelihood of the observation given the magnitude forecast, but we end up using all 20 eqks to do so (b/c all of them fall within magnitude bins that are unmasked). The way to avoid this is to filter the catalog against the original lat/lon/depth/mag forecast and then proceed. Perhaps you're already doing this further upstream, but if not, we should correct it. I also note that this will almost never be a problem for the S-test because it's very unlikely (perhaps even not allowed) that a forecast masks part of a spatial cell.
Change History (2)
comment:1 Changed 5 years ago by liukis
- Owner set to liukis
- Status changed from new to accepted
comment:2 Changed 5 years ago by liukis
- Resolution set to fixed
- Status changed from accepted to closed
Note: See
TracTickets for help on using
tickets.