close Warning: Can't synchronize with repository "(default)" (/work/source/svn/csep does not appear to be a Subversion repository.). Look in the Trac log for more information.

Opened 6 years ago

Closed 5 years ago

#280 closed defect (fixed)

Failure to export raw catalog data from SVN when link to raw archived catalog is detected through existing observation catalog file

Reported by: liukis Owned by: liukis
Priority: major Milestone: CSEP Version 12.7.0
Component: Toolkit Version: 12.4.0
Keywords: Cc:

Description (last modified by liukis)

Retrospective processing of CA three-month-models with CL evaluation test with '--disableExistingDataStaging' flag fails if observation catalog has corresponding raw catalog file as a link to raw catalog in some other runtime directory and such original catalog is archived in SVN. Processing attempts to generate a link to original raw catalog in runtime directory of original processing. Since raw catalog does not exist in runtime directory anymore (it's archived in SVN and only corresponding metadata file exists in runtime directory), soft link to non-existent file is generated causing error message:

Test date: 2008-03-31
Status: FAILED 
Errors: Mon, 2012-05-07 16:02:22 Dispatcher ERROR    Exception occured <type 'exceptions.OSError'> = [Errno 2] No such file or directory: '/home/csep/operations/dispatcher/runs/csep/2012_05/20120507160216/pid_8319/import_raw.dat'

OSError: [Errno 2] No such file or directory: '/home/csep/operations/dispatcher/runs/csep/2012_05/20120507160216/pid_8319/import_raw.dat'



Runtime date: 2008-3-31
Runtime host: csep-op
Runtime directory: /home/csep/operations/dispatcher/runs/csep/2012_05/20120507160216/pid_8319

Processing start time: 2012-05-07 16:02:16.156025
Processing end time: 2012-05-07 16:02:22.446702

Please see /home/csep/operations/dispatcher/logs/batch_2012_5_7_2008-3-31 log file for a detailed report.

Script to invoke reprocessing: csep-op:/home/csep/operations/reprocess/csep_12_4_0/batch_Trac278and279_5.7.2012.init.xml

Change History (4)

comment:1 Changed 6 years ago by liukis

  • Owner set to liukis
  • Status changed from new to accepted

comment:2 Changed 6 years ago by liukis

Special care should be taken in cases when metadata file points to original import_raw.dat file that does not exist in the directory (/home/csep/operations/dispatcher/runs/csep/20080501001001/import_raw.dat). Metadata files in original "/home/csep/operations/dispatcher/runs/csep/20080501001001" directory should be searched for corresponding metadata file to retrieve SVN tag for archived file.

[csep@csep-op pid_28716]$ pwd
/home/csep/operations/dispatcher/runs/csep/2012_04/20120419100818/pid_28716

[csep@csep-op pid_28716]$ cat scec.csep.ANSS.DataSource.1334855318.462932.1.meta
# /home/csep/operations/dispatcher/runs/csep/2012_04/20120419100818/pid_28716/import_raw.dat
DataFilename = /home/csep/operations/dispatcher/runs/csep/2012_04/20120419100818/pid_28716/import_raw.dat = string

DataFileIsLink = /home/csep/operations/dispatcher/runs/csep/20080501001001/import_raw.dat = string

ProducedBy = ['/usr/local/csep/src/generic/Dispatcher.py', '--year=2008', '--month=3', '--day=31', '--enableForecastMap', '--waitingPeriod=0', '--enableForecastXMLTemplate', '--publishServer=csep-usc@shear.usc.edu', '--disableRawDataDownload', '--tests=S M', '--configFile=/home/csep/operations/reprocess/csep_12_4_0/dispatcher_ANSS1932_V8.1.init.xml', '--disableExistingDataStaging', '--logFile=/home/csep/operations/dispatcher/logs/batch_2012_4_18_2008-3-31', '--publishDirectory=/home/scec-03/csep/us/results/data/us/usc/california', 'runtimeDirectory=/home/csep/operations/dispatcher/runs/csep/2012_04/20120419100818/pid_28716', 'runtimeTestDate=2008-03-31'] = string

CreationDateTime = 2012-01-13T10:25:02 = time

FileFormat = ASCII = string

FileDescription = Raw catalog data downloaded from ANSS data source. = string

[csep@csep-op pid_28716]$

comment:3 Changed 6 years ago by liukis

  • Description modified (diff)
  • Summary changed from Failure to export raw catalog data from SVN when multiple links to observation catalog exist to Failure to export raw catalog data from SVN when link to raw archived catalog is detected through existing observation catalog file

comment:4 Changed 5 years ago by liukis

  • Resolution set to fixed
  • Status changed from accepted to closed
Note: See TracTickets for help on using tickets.