close Warning: Can't synchronize with repository "(default)" (/work/source/svn/csep does not appear to be a Subversion repository.). Look in the Trac log for more information.

Opened 6 years ago

Closed 6 years ago

Last modified 6 years ago

#270 closed enhancement (fixed)

Optimize read of large forecasts files for T and W evaluation tests

Reported by: liukis Owned by: liukis
Priority: major Milestone: CSEP Version 12.1.0
Component: Toolkit Version: 10.7.0
Keywords: Cc:

Description

When 4 years worth of one-day forecasts files are read by T and W evaluation tests, numpy.loadtxt() sums up to hours of runtime. Write custom functionality to read large forecasts into numpy.array to reduce runtime.

Change History (3)

comment:1 Changed 6 years ago by liukis

  • Owner set to liukis
  • Status changed from new to accepted

comment:2 Changed 6 years ago by liukis

  • Resolution set to fixed
  • Status changed from accepted to closed

comment:3 Changed 6 years ago by liukis

Added CSEPFile.read() function which is almost 2 times faster than numpy.loadtxt():

liukis@maria-liukis-computer 49>python -m timeit -n 1 -r 1 -s 'import CSEPFile;' 'CSEPFile.read("STEP_2010-08-12.dat")' 1 loops, best of 1: 3.01 sec per loop

vs.

liukis@maria-liukis-computer 50>python -m timeit -n 1 -r 1 -s 'import numpy as np;' 'np.loadtxt("STEP_2010-08-12.dat")' 1 loops, best of 1: 5.47 sec per loop

Note: See TracTickets for help on using tickets.