Useful Notes from Dec2018

In late Dec2018 I began to solve a shitload of silly matplotlib problems associated with using an interactive cursor. In the end, this also turned into a python refresher. Here are some notes.

 


Thi file was:   /home/sco/README.Dec21_2018    (on scohome) 


Dec2018: Where things stand
===========================

  My image processing codes (ds9_profiles,imstats) are well testsed and workng on 
several machines. Questions remain about how best to subtract sky, extract photometric 
parameters, etc...., but the basic code infrastructure is in place. 

  I have a script (wcsf_imgcat) that will process a list of FITS images to the point 
of creating sky maps and other images as well as a simple catalog of moment-based 
image parameters. The "wcsf" part of the name indicates this was build for finding 
sources with which to astrometrically calibrate the images. 

  After some thought, it seems that I really want a system hat starts with the catalog. 
The first set of parameters could be moments-based, but eventually I want a catalog of 
parameters that can be used for image classification (i.e. star-gal sep). With such a 
catalog I can then proceed to higher quality WCS and ZP calibration.  U now have a working
insteractive cursor and plotting setup. Hence, it seems a good time to start with wcsf_imgcat
and create a concise catalog builder with a goal of image classes and "best-estimate" sky 
maps. 

The state of scohome:
I have two hard disk copies of my system:
/dev/sdb1      1922726704 576614400 1248436736  32% /media/sco/470f7c55-8ab2-4ed4-bc7f-b21445176bce
/dev/sdc1      1922726704 576614420 1248436716  32% /media/sco/fb0d7e99-4ac6-4b31-be53-b63d101a3b6c
For example, I can go to the bash script area and see my codes: 
% cd /media/sco/470f7c55-8ab2-4ed4-bc7f-b21445176bce/home
% ls
sco/  scob/  sco_back/	scoc/
% cd sco/Installs/install_sco_20180522/codes/bash 
*** I see my bash scripts here. 
Note also that I now have working versions of my codes on buckaroo and banzai. 
I have stored all of the data I care about on the buckaroo scodata disk (and 
most everything on banzai also). 

===============================================================================
A simple test using the Test data from ds9_profiles 

% cd Test_ds9_profiles
% mkdir tcat1
% cd tcat1
% imgcat0 ../images/Rsco2039.fits pfc 10.0
**** it does not run anymore! 
/home/sco/Installs/install_sco_20180522/bin/binc/otw/skyhistfit.sh: line 55: poly3.x_for_miny: No such file or directory
*** So, once again I have modified a code and fucked up other codes.

I do make lots of calls to imgcat0 in various scripts, but these are fairly old 
and most of them should be considered obsolete. However, in the spirit of preserving 
old thinsg that (mostly) work, I am going to build imgcat1: the initial change will be 
to simply add a debug flag to the command line. 

*** This was an easy fix. I just had to change one file name in the skyhistfit.sh script.  
    In making this fix I used imgcat1. This really only adds a debug argumnt to the command line. 

Sat Dec 22 09:12:08 CST 2018
============================
  In the course of the abover work, I made a simple plot using the Rsco2039.table file and 
xyplotter_auto. I saw an interesting star-gal sep space, but this immediately raised a desire 
to have a tool that I think I can now cobble together without a lot of hassle now that I have 
a working interactive cursor routine (like the table_point_selector routine used in xyfitter). 
The tool I envision:
   1) Plots a parameter space from a table file and lets user select a point
   2) the point is dentified and the i1,2,j1,j2 coordinates are extracted 
   3) this box area is zoomed to in a specified ds9 frame where the image is displayed.  

Mon Dec 24 09:40:14 CST 2018
============================
I have: 
  imgcat0               = makes a reasonable catalog based on zeroeth and first image moments 
  ds9_imgcat0           = displays the diagnositic images from imgcat0 
  table_point_selector  = view (X,Y) space from a table file and select points

I'll just manually fuck around with things for a while this morning. 
   
I can run table_point_selector (tps) on the imgcat0 table file = Rsco2039.table
  table_point_selector Rsco2039 logEqRad isomag30 N 
At the end of each run (after I enter "E") I have: 
  table_point_selector.X,Y,LineNum,Xstats,Ystats
The file of importance here is:   table_point_selector.LineNum
It gives me the line numbers of the points I selected ("s") 
In my example run I selected 3 big (presumed) galaxies: 
% cat table_point_selector.LineNum
 1 
 2 
 4 

I want to get the table lines for these (i.e. the lines after "# data") 

% lineget.py Rsco2039.table 1
       1     5911     943  1038  1016  1101    13.697   990.81 1057.88    15.6  0.873  -67.68    76.88   1.6373
% lineget.py Rsco2039.table 2
       2     3815     652   738  1146  1222    14.211   695.20 1183.03    15.2  0.577  -50.71    61.77   1.5422
% lineget.py Rsco2039.table 4
       4      877     538   603   914   939    16.661   573.98  925.63    13.0  0.386   82.96    29.61   1.2229
                       i1    i2    j1    j2               Xc     Yc 

To look at the field:
 % ds9_imgcat0 Rsco2039.fits N
*** I verify, using circle regions with tthe Xc,Yc  manually entered that these 
    are my 3 big galaxies. 

To zoom in with xpa: 
% xpaset -p ds9 pan to 573.98 925.63  physical        # centers N3389
% xpaset -p ds9 zoom 4                                # zooms in 4 times 

Then I can pan to another galaxy and retain the same (visually set) zoom factor 
% xpaset -p ds9 pan to 990.81 1057.88 physical        # centers N3379 
% xpaset -p ds9 pan to 695.20 1183.03 physical        # centers N3384 

Using the box size (i1,i2,j1,j2) does not seem to be that critical. I 
just pan to the position and manually set the zoom for each object (which 
will be about the same for most objects). 

I'll kluge this up in one script:    imgcat0_viewer Rsco2039.fits N  
------>  I have a working version of imgcat0_viewer!

Once again, some information on using show() would be so helpful.
I submitted a question to a site (forum?) called:  edureka 
  https://www.edureka.co/community/34032/record-limits-displayed-plot-using-matplotlib-show-module
Probably won't get an answer, but you never know. 


I also found some intersting stuff googling:   "matplotlib forum show()"
See:  https://stackoverflow.com/questions/458209/is-there-a-way-to-detach-matplotlib-plots-so-that-the-computation-can-continue

Another very useful page:   https://realpython.com/python-matplotlib-guide/#why-can-matplotlib-be-confusing
Near the bottom there is some interesting stuff on interactive mode. 

Then I looked at:  https://matplotlib.org/faq/usage_faq.html#what-is-interactive-mode
and tried this (very interesting) 
  import matplotlib.pyplot as plt
  plt.ion()
  plt.plot([1.6, 2.7])

**************  THIS COULD BE WHAT I HAVE BEEN LOOKING FOR! 


I did these steps and interactively built a plot: 
>>> import matplotlib.pyplot as plt
>>> plt.ion()
>>> plt.plot([1.6, 2.7])
[]
>>> plt.title("interactive test")

>>> plt.xlabel("index")

>>> plt.ylabel("boodle")

>>> quit()

****** I now have, in icurs3.py, a code that is much more efficient in 
       identifyng points. 

Another thing that would be extremely useful is being able to learn 
value for (or) lot properties. I keep seeing references to rcParams.
This is importat I think, but as usual, no clear explanations. 

It seems that rcParams is some entity (list?, dictionary?)  that sets properties:
-------------------------------------------------------
def visualize_2D_trip(self, trip):
        plt.figure(figsize=(30,30))
        rcParams.update({'font.size': 22})       # I guess the sets a new font size.

        # Plot cities
        plt.scatter(trip[:,0], trip[:,1], s=200)
-------------------------------------------------------

Maybe I just fuck around in python: 
% python 
Python 2.7.4 (default, Sep 26 2013, 03:20:26) 
>>> import matplotlib.pyplot as plt
>>> type(rcParams)
Traceback (most recent call last):
  File "", line 1, in 
NameError: name 'rcParams' is not defined

>>> type(plt.rcParams)
                  # so RcParams is a class

>>> len(plt.rcParams)
179

>>> print plt.rcParams
{'figure.subplot.right': 0.9, 'animation.frame_format': 'png', 'mathtext.cal': 'cursive', .....
I seem to get this huge list of stuff 

Maybe I just use this to query?    What about figure.subplot.right?
What the fuck is { } ?    
   Ans:  It is a dictionary,   see:  https://www.tutorialspoint.com/python/python_dictionary.htm

========================================================================
Fiarly clear text on lists and tuples:    https://realpython.com/python-lists-tuples/

Define a list: 
 a = ['foo', 'bar', 'baz', 'qux']                            # lists use [ ]

Define a tuple:
 t = ('foo', 'bar', 'baz', 'qux', 'quux', 'corge')           # tuples use ( ) 
========================================================================

How do I use a dictionary? 

To set the values: 
dict = {'Name': 'Zara', 'Age': 7, 'Class': 'First'}
print "dict['Name']: ", dict['Name']
print "dict['Age']: ", dict['Age']
 
Here is how it works in my example above: 
>>> print "rcParams[ Andale Mono ]"
rcParams[ Andale Mono ]


>>> import matplotlib.pyplot as plt
>>> type(plt.rcParams)

>>> str(plt.rcParams)          # big list 
 
>>> f = "rcParams[ Andale Mono ]"
>>> print f
rcParams[ Andale Mono ]

>>> plt.rcParams.keys()    #creates a list of just the keys 

Once I know a valid key name, I can get the value with: 
>>> plt.rcParams.get('axes.titlesize')
'large'

So, to get a listing of all keys in rcParams:
>>> plt.rcParams.keys()
# This creates an enormously long single line. How do I get 
  a list of the key value with one key value per line? 


=====================================================
Now I am just looking at examples 
 To reste valeus: 
    params = {'savefig.dpi'          : dpi,
              'savefig.format'       : 'png',
              'axes.formatter.limits': [-3, 3]}
    rcParams.update(params)
=====================================================


=====================================================
Here are two ways to print the (big!) plt.rcParams dictionary: 

import matplotlib.pyplot as plt

method1: 
for keys,values in plt.rcParams.items():
    print "\n"
    print(keys)
    print(values)

method 2: 
import json
print(json.dumps(plt.rcParams, indent = 4))

method 3:
from pprint import pprint 
pprint(plt.rcParams)

-------------------------------------------------------------------
Practical point:   It is difficult (for me) to enter this 
interactively, so I make a file and then run it:
 
% cat p1.py
#!/usr/bin/env python
import matplotlib.pyplot as plt
ffc = open('File.rcParams', 'w')
for keys,values in plt.rcParams.items():
 ffc.write( "\n %s\t%s \n" % (keys,values) ) 
ffc.close()

% python p1.py
% ls 
File.rcParams  p1.py  S/
% cat File.rcParams               # this s a viewable file! 
-------------------------------------------------------------------
=====================================================

When I measure the figure made by icurs3.py it is not 8x6
figure.figsize  [8.0, 6.0] 
    params = {'savefig.format'       : 'png',
              'axes.formatter.limits': [-3, 3]}
    rcParams.update(params)

I do get a new, larger (but bot 9x9) plot inicurs3.py with: 
# experimental
paramso = {'savefig.format'       : 'png',
          'figure.figsize'       : [9.0, 9.0],
          'axes.formatter.limits': [-3, 3]}
plt.rcParams.update(paramso)
**** So, I am at least able to change soemthing I want changed! 

*************** Fiannly  ********************************************
Here is how I can query the current box plot limits. 
print "Current X1,X2,Y1,Y2 limits: %s %s %s %s \n"%  (plt.axis())
**********************************************************************


***** Finally, here is an example of how to make a nice list of rcParam  *****    
#!/usr/bin/env python
'''Print a readbakle table of the default rcParams dictionary'''
import matplotlib.pyplot as plt
ffc = open('File.rcParams', 'w')
for keys,values in plt.rcParams.items():
 ffc.write( " %-30s       %-50s\n" % (keys,values) ) 
ffc.close()


Fri Dec 28 10:11:03 CST 2018
============================
A note from Greg Z:

Hi Steve,

I don't do interactive mode often, but I found this brief example set that 
may or may not be useful: https://www.datacamp.com/community/blog/bokeh-cheat-sheet-python

This is an alternative to matplotlib, and should be included in anaconda.  
I haven't used it, but there is a cheatsheet: 
https://s3.amazonaws.com/assets.datacamp.com/blog_assets/Python_Bokeh_Cheat_Sheet.pdf

I haven't ever used it, but I'm happy to talk with you (via zoom or skype) 
and I could at least see exactly what you are trying to do and may be of 
more help.  Let me know.


 




Back to calling page