Opened 5 years ago

Closed 5 years ago

#265 closed defect (fixed)

Observation catalog should be filtered against masked forecast for M and S evaluation tests

Reported by: liukis Owned by: liukis
Priority: major Milestone: CSEP Version 12.1.0
Component: Toolkit Version: 10.7.0
Keywords: Cc:

Description

Jeremy pointed out that original implementation of M and S tests had a logical error - email communication from January 4, 2011, "possible logical mistake in the M- and S- tests":

In the M- and S-tests, to normalize we count the number of eqks that
fall inside unmasked forecast bins using
relm_NumberEventsCatalog(vForecast, cObservedCatalog).  The way that I
had my scripts set up, this number might not be the same as the total
number of eqks in the catalog.  In particular, I didn't filter the
catalog to contain only the events that fell within unmasked bins.
But then the M- and S-tests could use the entire catalog to compute
the observed likelihood, so this would bias the results.  Perhaps this
is more clear w/ an example.  Consider an M-test with a model that
only has unmasked bins in southern California, and let's say the model
provides rates in every magnitude bin.  Let's say there were 20 events
total, 10 in unmasked bins, the other 10 in northern California.  When
we count the number of eqks that fall inside unmasked bins, we get the
correct number (10) and normalize the magnitude forecast so its total
rate is 10.  Then we compute the likelihood of the observation given
the magnitude forecast, but we end up using all 20 eqks to do so (b/c
all of them fall within magnitude bins that are unmasked).  The way to
avoid this is to filter the catalog against the original
lat/lon/depth/mag forecast and then proceed.  Perhaps you're already
doing this further upstream, but if not, we should correct it.  I also
note that this will almost never be a problem for the S-test because
it's very unlikely (perhaps even not allowed) that a forecast masks
part of a spatial cell.

Change History (2)

comment:1 Changed 5 years ago by liukis

  • Owner set to liukis
  • Status changed from new to accepted

comment:2 Changed 5 years ago by liukis

  • Resolution set to fixed
  • Status changed from accepted to closed
Note: See TracTickets for help on using tickets.