Erly Mar2020 Notes
Last updated: Mar21,2020

  1. The current reduction steps.
  2. The Problem.
  3. Does image_catalogs work on the "problem" image.
  4. Does image_catalogs fail in AcmRunDate?
  5. Handling image stacks.
  6. The general astrometry code wcsf.


The current reduction steps.

Here are the most current reduction steps. 

  cd /home/sco/ACM_work_Oct2019/red_20191027
  AcmRunDate 20191027                                # Remember BaseDir file   
  AcmRunDate_Clean Y
 

# View summary in the  file = ARD.Summary

For each onsky set: 
  isetwcs 1
  usno_look_wcs_fitslist isetwcs.images             # Optional review of image set  
        usno_look_wcs_run blah.fits N               # For a super quick check
This process is evolving, but these steps get us through the revised WCS stage.

Here are some practical points regarding this reduction process. 

 
  1) For the five Oct nights, after I finished through the WCS point the 
     total size of the reuction directories was 20 GB. When I deletef the 
     ./local_red/HDFIX directories this dropped to 12 GB (40% reduction).
     The HDFIX step can be quickly reproduced, so this is a practical 
     size reduction step following the AcmRunDate procedure.

Return to top of page.


The Problem.

In Feb2020 I developed a suite of codes that collect and utilize calibration material. These routines, as well as code to establish moments-based image catalogs, can be run in batch mode. Early use of these tools revealed some cases of images where things failed to work. In these notes I try to discern where these problems arise. 


# An image that will not process:
/home/sco/tmp/T3/local_red/FIXUP/20191027T013204.9_acm_sci.fits       (on sco2019) 
/home/mcs/sco/tmp/local_red/FIXUP/20191027T013204.9_acm_sci.fits      (on mcs) 

  The basic problem seems to be that the image catalog section 
  of the routine failed to build a cdfp file. 

The new routines I am intergating and testing now are: 

# Process and collect cal material for a night of acm data 
% AcmRunDate  
Usage: AcmRunDate 20191018  
arg1 - date of desired night to reduce (YYYYMMDD)  
Additional options: 
     -v = print verbose comments 
 --help = view a help document 

# Solve/install WCS for acm images in a set (images at a unique sky position)
% isetwcs
Both AcmRunDate and isetwcs are fairly high level routines. Also, the image listed above is a tough case: low S/N with only one bright star.

Return to top of page.


Does image_catalogs work on the "problem" image.

Does the image_catalogs run well on the test image? 



% cat list.0
/home/mcs/sco/tmp/local_red/FIXUP/20191027T013204.9_acm_sci.fits
% image_catalogs list.0 none N N          # I do not clean files, I do not run in debug mode 

[sco@mcs T1]$ cat image_catalogs.summary
 1 86 56.5877 1.3334 20191027T013204.9_acm_sci 

[sco@mcs 20191027T013204.9_acm_sci]$ pwd
/home/mcs/sco/tmp/T1/local_red/IMAGE_CATS/20191027T013204.9_acm_sci

[sco@mcs 20191027T013204.9_acm_sci]$ ls
20191027T013204.9_acm_sci_BGboxes.reg  20191027T013204.9_acm_sci_detsig.fits  20191027T013204.9_acm_sci.parlab
20191027T013204.9_acm_sci_bkg1.fits    20191027T013204.9_acm_sci.fits	      20191027T013204.9_acm_sci.props
20191027T013204.9_acm_sci.cat0	       20191027T013204.9_acm_sci_label0.fits  20191027T013204.9_acm_sci_sigma.fits
20191027T013204.9_acm_sci.cdfp	       20191027T013204.9_acm_sci.params       20191027T013204.9_acm_sci.table

[sco@mcs 20191027T013204.9_acm_sci]$ linecnt 20191027T013204.9_acm_sci.cdfp
 87

*** Yes, this works to a succesful completion. I find 86 sources.

Return to top of page.


Does image_catalogs fail in AcmRunDate?

This is a long run on mcs (about 20min). 

 
[sco@mcs ~/tmp2]$ cat BaseDir
/hetdata/data

[sco@mcs ~/tmp2] AcmRunDate 20191027  

Results: 
It seems I was killing off the cdfp files in the cleaning process. When 
turn that arg to "N" (no clean) I get the full cdfp files: 
[sco@mcs 20191027T025731.9_acm_sci]$ pwd
/home/mcs/sco/tmp2/local_red/IMAGE_CATS/20191027T025731.9_acm_sci
[sco@mcs 20191027T025731.9_acm_sci]$ ls
20191027T025731.9_acm_sci_BGboxes.reg  20191027T025731.9_acm_sci_detsig.fits  20191027T025731.9_acm_sci.parlab
20191027T025731.9_acm_sci_bkg1.fits    20191027T025731.9_acm_sci.fits	      20191027T025731.9_acm_sci.props
20191027T025731.9_acm_sci.cat0	       20191027T025731.9_acm_sci_label0.fits  20191027T025731.9_acm_sci_sigma.fits
20191027T025731.9_acm_sci.cdfp	       20191027T025731.9_acm_sci.params       20191027T025731.9_acm_sci.table
I should clean up wcsf_imgcat and image_catalogs so that they produce the same sort of catalogs. For instance, the wcsf_imgcat code will build a cdfp file using the WCS of an image that is available. However, when I run image_catalogs, not even roughly corrected WCS are available. I should NOT have the wcsf_imgcat code building a cdfp. It is ironic that wcsf_imgcat would not produce Ra,Dec values, but this is an action beyond the scope of initial catalog production. That can be done by an independent script when the appropriate WCS-calibrated image becomes available. The routine that makes the cdfp file will be useful elswhere: 
 
% cdfp_from_imgcat0 $locfits $debug 

Usage: cdfp_from_imgcat0 20170420T101656.1_acm_sci.fits N 
arg1 = name of FITS image run through imgcat0 
arg2 = run in debug mode? (Y/N)  

IMPORTANT:   I must modify it so that it does not use the basename.cat0 file, 
             but rather uses the more general basename.table file.
Finally, here are some notes I compile on Sat Mar7,2020 to prep for upgrading isetwcs. 
 
% cat list.0
/home/sco/AUTO_match_for_wcs/S/20191018T024545.3_acm_sci.fits

To run batch things:
   fitslists list.0
   dssbase_run_markII list.for_mode2 none none none none list N
   dssbase_list Y
   dssbase_review Y
   image_catalogs list.0 Set0 Y N

To review image catalogs:
   ds9_imgcat0 /home/sco/AUTO_match_for_wcs/S/20191018T024545.3_acm_sci.fits N

To build Active.Images
   active_imgcat0 /home/sco/AUTO_match_for_wcs/S/20191018T024545.3_acm_sci.fits -v

Following a run of AcmRunDate we'll have a table file (EX1) that summarizes the processed image database. The images will have assigned "iset" values. To summarize the different iset groups we can used the file named table_onsky_sets.summary: 

 
% cat table_onsky_sets.summary 
Summary of targets sets from table_onsky_sets.sh run
Number of images in input list      =     536
Number of unique targets identified =      14
Number of images with target values =     129
     iset  Nims      RA_2000    DEC_2000
      1       1    14:11:16.21 +50:09:00.7
      2       3    16:53:35.69 +27:47:17.7
      3      24    23:18:40.06 +55:31:21.2
      4       9    23:18:40.69 +55:31:14.7
      5      10    23:18:40.50 +55:31:21.4
      6       9    23:18:40.52 +55:31:12.2
      7       9    23:18:42.67 +55:31:16.7
      8      19    23:18:41.94 +55:31:08.1
      9      10    23:18:43.72 +55:31:14.2
     10      10    23:26:12.50 +15:58:07.8
     11       3    00:30:21.31 +24:16:10.6
     12      14    19:12:43.98 +47:22:36.9
     13       7    23:06:27.18 -05:02:32.0
     14       1    01:24:28.76 +57:50:13.8
Return to top of page.


Handling image stacks.

The image sets for each iset group can be stacked amd processed. We can use the onsky_set routine in an interactive mode to collect the image sets (arg3="Y"), but in the example below I just collect and stack all 9 images in the original set. 

 
 
% onsky_set EX1 4 N N  
Checking EX1.images file 
 9 9 
% ls List.FINAL_ISET  
List.FINAL_ISET

To stack the image set: 
% iset_stacker 4 List.FINAL_ISET mean  

Hence, to get a stack for iset-3 

   onsky_set EX1 3 N N
   iset_stacker 3 List.FINAL_ISET mean
 

The stacked image will be:  ./local_red/ISET_STACKS/iset_3_stack.fits

I put the image stacking and catalog option into AcmRunDate, so....

% cat BaseDir 
/media/sco/DataDisk1/sco/AD/HET_work/acm_nights
% AcmRunDate 20191027 Y 
% image_catalogs_review  
% image_catalogs_review list.STACKS
Return to top of page.


The general astrometry code wcsf.

The routine wcsf is meant to derive and install WCS in a single FITS image. 

 
 
Run wcsf on a stacked image:

In /home/sco/tmp2/local_red
For 20191027 I have (from AcmRunDate) four iset groups, each is stacked.

Summary of targets sets from table_onsky_sets.sh run
Number of images in input list      =     422
Number of unique targets identified =       4
Number of images with target values =      37
     iset  Nims      RA_2000    DEC_2000
      1       9    23:26:07.95 +15:56:54.5
      2      12    20:10:40.69 +40:41:23.2
      3       8    03:53:36.35 -00:06:32.7
      4       8    08:15:12.38 +07:29:23.9

To process iset=2 
wcsf /home/sco/tmp2/local_red/ISET_STACKS/iset_2_stack.fits

I make the script SETUP to prepare a testing directory where I can run this.

 cd /home/sco/tmp2
 SETUP T1 2 
 cd T1
 wcsf /home/sco/tmp2/local_red/ISET_STACKS/iset_2_stack.fits
 
**** This way I can test/modfy the wcsf_ codes until they work. 

As of Thurs Mar12,2020 the rough and viscat codes are working. 

The code to concentrate on now is wcsf_final: 
    cat list.start
        /home/sco/tmp2/local_red/ISET_STACKS/iset_2_stack.fits
    wcsf_final list.start Y Y 

Recall:
Usage: wcsf_final list.0 Y N 
arg1 - Name of file with list of FITS images (can be full path)
arg2 - Interactively assemble a new USNO catalog (Y/N)
arg3 - run in debug mode (Y/N)

Note: I could do this in a fairly empty directory (./s just stores list.start): 
% ls
list.start  local_red/	s/  
% wcsf_final list.start Y Y
Return to top of page.


Back to calling page