Here are some notes on tools that presently (as of ealry Sep2018) exist. I've used these things for numerous projects. They are well tested. They give the correct answers, but may not run very quickly, especially on our very clunky mcs.
This file started as: /home/sco/README.PAS_images.txt
Finding images (particularly PAS images)
=========================================
Just to get a list of fits images:
% fitsfind /home/sco/HET_work/gc_probes_0 > Big.List
% fitsfind /home/sco/sco/Red_PhaseI_Data >a
% wc -l a
2022 a # This only took a second or so to run! 2000 PFC images!
To find a list of PAS images of a certain type on a certain date:
% ls
BaseDir Date
% pas_imlistgen l gc2 N
% cat List0
/home/sco/HET_work/gc_probes_0/20180828/gc2/20180828T105750.8_gc2_sci.fits
**** This is the last image in the list for this basedir/date
=============================================================================================
Tools to parse useful information from the image name:
- Get general date/time info
% pas_times.sh /home/sco/HET_work/gc_probes_0/20180828/gc2/20180828T105750.8_gc2_sci.fits N
6812.957031 239.457062 10.96954 (days_since_2000,days_since_jan1,hours_since_UT=0)
- Get the image basename
% just_fitsname.sh /home/sco/HET_work/gc_probes_0/20180828/acm/20180828T110428.2_acm_sci.fits
25 20180828T110428.2_acm_sci
* Get time in seconds and times since an input "last" time
% pasname.sh 20180828T105750.8_gc2_sci 0.0
39470.8 39470.8
% pasname.sh 20180828T110428.2_acm_sci 38000.0
39868.2 1868.2
=============================================================================================
=============================================================================================
I have older tools for finding acm image nearset in time: acam_nearest and acam_nearest_2
But, they are not very good:
1) they both take acm image names (really should take a time to be more general
2) If the time to searched for is less than (even slightly) to closest image
time, bot codes simply hand back the last image in the list.
What I need then is something that takes a PAS time (for instance from pasname.sh)
and finds the CLOSEST in time. NO CONSIDERATION of before or after!
=============================================================================================
My thoughts:
1) make a list with imlistgen to survey the image subdiretory
2) build a script that uses the resulting list.IMAGES to make
a list of image_number, Time (in seconds) , full path name
3) This secondary list, which can be periodically updated, is the
one I search to find time matches.
Based on my thoughts above, I have assembled some new "file search" tools.
pas_imlistgen
finds the images of a given PAS type in a basedir/date
pas_timelist:
take a list of images and make a list of i,time,filename
i = image number
time = times of image in units of seconds (since 0h UT)
filename = fullpath name of the image
pas_timelist_update:
Update the list from pas_timelist in case new images have been added to
the basedir/date.
Here is an example of the using the first two scripts above to find gc1 images:
% cat BaseDir Date
/home/sco/HET_work/gc_probes_0
20180828
% pas_imlistgen l gc1 N
% ls
BaseDir Date List0 list.IMAGES S/ t1/
% cat list.IMAGES
/home/sco/HET_work/gc_probes_0/20180828/gc1/20180828T104213.9_gc1_sci.fits
/home/sco/HET_work/gc_probes_0/20180828/gc1/20180828T104712.9_gc1_sci.fits
/home/sco/HET_work/gc_probes_0/20180828/gc1/20180828T105325.9_gc1_sci.fits
/home/sco/HET_work/gc_probes_0/20180828/gc1/20180828T105747.4_gc1_sci.fits
/home/sco/HET_work/gc_probes_0/20180828/gc1/20180828T110142.8_gc1_sci.fits
/home/sco/HET_work/gc_probes_0/20180828/gc1/20180828T110425.2_gc1_sci.fits
/home/sco/HET_work/gc_probes_0/20180828/gc1/20180828T110617.6_gc1_sci.fits
% cat List0
/home/sco/HET_work/gc_probes_0/20180828/gc1/20180828T110617.6_gc1_sci.fits
*** The last image that I requested using the "l" argument, is written to
the file named List0.
The file "list.IMAGES" contains all of the gc1 images. We can run this through
the pas_timelist routine to get a more usefule file:
% pas_timelist list.IMAGES List.Times N
% cat List.Times
1 38533.9 /home/sco/HET_work/gc_probes_0/20180828/gc1/20180828T104213.9_gc1_sci.fits
2 38832.9 /home/sco/HET_work/gc_probes_0/20180828/gc1/20180828T104712.9_gc1_sci.fits
3 39205.9 /home/sco/HET_work/gc_probes_0/20180828/gc1/20180828T105325.9_gc1_sci.fits
4 39467.4 /home/sco/HET_work/gc_probes_0/20180828/gc1/20180828T105747.4_gc1_sci.fits
5 39702.8 /home/sco/HET_work/gc_probes_0/20180828/gc1/20180828T110142.8_gc1_sci.fits
6 39865.2 /home/sco/HET_work/gc_probes_0/20180828/gc1/20180828T110425.2_gc1_sci.fits
7 39977.6 /home/sco/HET_work/gc_probes_0/20180828/gc1/20180828T110617.6_gc1_sci.fits
The values in "List.Times" can be easily searched and used to find a
particular gc1 image that matches the closest time for some other event.
At this stage it would be useful to make some timing on mcs with real
data directories. I put pas_imlistgen and pas_timelist onto mcs and run
some tests using different Dates. All images are in /hetdata/data
[sco@mcs ~/pas_tests]$ pas_imlistgen l gc1 N
[sco@mcs ~/pas_tests]$ pas_timelist list.IMAGES A N
n1 = number number in list.IMAGES
n2 = number number in List0
T = approximate time in seconds to run pas_timelist
Date gc1 gc2 acm
---------- ---------- ----------
n1 n2 T n1 n2 T n1 n2 T
20180905 2758 1 87 2596 1 81 363 1 11
20180906 1063 1 33 4633 1 147 115 1 4
20180910 7484 1 238 4547 1 145 227 1 7
20180924 2430 1 76 2710 1 85 420 1 15
Next, I pulled over the 20180910 data for gc1 to scohome. Here I found:
Date gc1
----------
n1 n2 T
20180910 7484 1 66
So, while scohome was 3.6 times faster, it still took a full minute to
survey the times.
**** After a rewrite of pasname.sh, where I now allow input of fullpath names.
Now when I re-run pas_timelist it takes 36 sec, not 66sec.
% pas_timelist list.IMAGES A N
a 1 Thu Sep 27 19:09:29 UTC 2018
a 2 Thu Sep 27 19:10:05 UTC 2018
36.000000 (aprroximate run time in seconds)
I added some code to pas_timelist that allowed me to measure the processing time
(T in the table above). As we typically have over 2000 gc1,2 images on a good
night, the long pas_timelist T values (a minute or more) are prohibitive. Clearly
the runtimes for pas_timelist are long, and I should proceed with finishing the
pas_timelist_update script. This script will decrease subsequent runs by nt reprocessing
the full list of images listed in list.IMAGES from pas_imlistgen. I have improved
the performance of pasname.sh, but a significant inrease in efficiency will also
be obtained by developing "pas_timelist_update".
Found a way to significantly improve the making of the time list! I was wondering why I would
hear my disk carnking away for 30 sec when I was processing list.IMAGES with pas_timelist. I
wondered if there was some intrinsic overhead associated with a shell script run. Thanks to
a consult with Jim Fowlerthis morning, this was confirmed. So, I wrote a new version of the
time decoder called pasnames.sh (pasname.sh ---> pasnames.sh) that workks directly on a list
of images. The results are amazing.
% wc -l list.IMAGES
7484 list.IMAGES
*** Running as a script:
% pas_timelist list.IMAGES list.out N
a 1 Fri Sep 28 17:36:42 UTC 2018
a 2 Fri Sep 28 17:37:11 UTC 2018
29.000000 (aprroximate run time in seconds) *** Takes 29 second to run
*** Running as a program that processes a list:
% pasnames.sh list.IMAGES 0.0 *** Takes about 0.3 second to run
*** Running as a script using pasnames.sh:
% pas_timelist list.IMAGES list.out N
a 1 Fri Sep 28 18:15:49 UTC 2018
a 2 Fri Sep 28 18:15:49 UTC 2018
0.000000 (aprroximate run time in seconds) *** In other words, less than a second
Perhaps now I need to reconsider if I need an updated version of
pas_timelist. I'll re-run some of the timing experiments above on mcs.
To get a tool like that used in lmap:
acam_nearest_2 $fsci $acam_nearest_query
I'll kluge up something above, but which uses the new, much faster, pas_timelist.