CSEP SCEC Testing Center requirements for a model submission
For the CSEP Testing Center at SCEC to support more forecasts groups and to host more models and algorithms, we request that modelers comply with the following set of requirements to enter their models into the Testing Center.
The CSEP Testing Center at SCEC follows quarterly releases of January1, April 1, July1 and October 1 of each year. Source code repository is "frozen" one month priory to the release date meaning that there are no code changes are allowed one month before the release date. For example, code freeze date for January 1st release is December 1st of previous year (March 1 for the April 1st release, June 1st for the July 1st release, and September 1st for the October 1st release). This good software practice guarantees that all code changes targeted for upcoming release, including new models, are intensively tested for the whole month on SCEC certification server before code base of new release is installed on SCEC operational server. We ask modelers to allow enough time for their model submission before the code freeze for targeted CSEP release they would like their model to be part of. Installation of new model within testing center requires work on CSEP development team side after the model codes are submitted for installation. Please consider submitting the model at least two months priory to the targeted CSEP release date. This will allow for one month of interactive installation process between the modeler and the CSEP development team before the code freeze takes place.
CSEPRQ1 We request that the modeler provide source code along with rules to generate executable file if applicable. This includes specification of:
- All dependent packages the model is relying on (including version numbers), and rules for how to build them
- Compiler (including version number) if applicable
- Rules to build executable(s) is applicable
CSEPRQ2 We request that the modeler install the model's source code on CSEP integration server csep-devel.usc.edu (accessible through motion.usc.edu), and builds executable(s) if applicable.
- Please contact John Yu (johnyu@…) to get an account
CSEPRQ3 We request that the modeler provide command(s) required to invoke the model.
Currently, CSEP testing framework supports ANSS and CMT authorized data sources.
- ANSS input catalog is provided in ZMAP format (ASCII or Matlab):
- Please refer to examples of input catalogs
- CMT input catalog is provided as reduced data set of raw CMT catalog in ASCII format
- Please refer to examples of input catalog
CSEPRQ4 We request that the model read input catalog in CSEP provided format. CSEP testing framework "communicates" to the model through input parameter file which contains run-time information about forecast the model is asked to generate. "Key=Value" format of the parameter file is used by most of the installed models to communicate such information.
CSEPRQ5 We request that the model accept a path to the input parameter file as a command-line argument.
CSEPRQ6 We request that the model read the following information from input parameter file provided by the CSEP:
- Forecast start date
- Forecast end date
- Path to the input catalog file
- Path to the output forecast file
- Seed value, or path to the file storing the seed for random number generator (if applicable)
- Any other parameters the model requires at run-time
CSEP development team will work with each modeler to identify a complete set of input parameters for his/her model.
Expected format for the result forecast file is XML-based ForecastML. Forecast templates are defined for each testing region within CSEP testing framework. Please refer to available forecasts templates.
CSEPRQ7 We request that the model generate a forecast in ForecastML format based on XML template the CSEP testing framework provides.
In order for the CSEP development team to verify that the installed model generates the expected result, and to be able to reproduce the forecast, we ask modelers to provide reference data. That reference data will be used by the model's acceptance test within CSEP testing framework.
CSEPRQ8 We request that modeler provides a set of reference data for the model acceptance test. The reference data consists of:
- Input catalog
- Parameters used to generate forecast
- Expected forecast