Munging tools
Last Updated: Oct10,2019


The job of extracting lines and/or columns from a file is referred to these days as munging. I have lots of these tools described in my big list of routines, but it is often had to remember their names and uses. I have tried to order the tables of routines below by topic, but there is obvious cross-over in lots of cases. One thing tonote is that I list several routines with the ".sh" appendix. This denotes that these are OTW codes (as opposed to bash scripts). These codes are good examples of things that use table munging scripts to extract textual and numeric quantities, perform (possibly) detailed evaluations, and then produce some output (usually another table file with the same number of rows as the input table).



  1. Running lists of my munging tools.
  2. Example 1: Cutting up the VIRUS temperatures list..
  3. Example 2: Building and using Tables from FITS headers
  4. Example 3: Plotting from a single Table file


A Running list of my munging tools.

Here is a big list of links.


Primitive text file operations

linecnt   Return line count for a file.
getNline   Grab the Nth line from a file.
select_file_lines   Grab certain lines from a file using a list of line numbers.
cut_by_line   Grab a sequential set of lines from a file.
column_fake   Build a column of string values.
data_strip   Strip off the data lines following "# data".
every_nth_line   Extract every Nth line from a file.
lineget.py   Get a specified line number after the "# data" marker.
namelist   Create a file of line numbers (for PointNames).
cat_line_pairs   Combine pairs of lines in a file.

Operate on Table files.

cdfp2table   Convert cdfp to table file.
table_checker   Check properties of a table file.
colget.py   Extract a specified column number from a table file.
fits2table   Build a Table from FITS headers.
table_column_find   Find column number of a column in a table file.
table_column_fake   Make a fake column for a table file.
table_column_pull   Extract a specified column by NAME from a table file.
table_merge   Combine two table files.
table_column_math   Combine 2 columns in a table file using a math operator.
table_mathop   OTW code to apply math operation to 1 table column.
table_xy_boxclean   Graphically isolate (X,Y) data.
table_point_selector   Graphically select from (X,Y) data.
split_table_by_selector   Use point_selector output to divide a table.
point_selector_stats   compute X or Y stats
table_PAS_times.sh   Build a table of PAS image times.
table_stats.sh   Compile stats for table column.
table_ds9_imstats.sh   Compile stats for ds9_imstats_fitslist tables.

Using masks with Table files.

table_mask_builder   Build subset tables based on teaxt and number rules.
get_table_rows   Grab table rows based on a selection mask.
table_text_mask   Build selection mask based on text string matching.
table_fprange_mask   Build selection mask based on floating point range.
masklogic.sh   Combine masks using logical operator.


Example 1: Cutting up the VIRUS temperatures list..

I want to inspect the VIRUS temperature list and select lines that meet a changing array of criteria. You can read about inspecting the VIRUS list here.



Example 2: Building and using Tables from FITS headers

In May2018 began assembling tools to install useful things into the headers of acm images that I have reduced. You can read about how I used some of my munging tools for Building and using Tables from FITS headers.



Example 3: Plotting from a table file.

The case of Example 2 above is a little complicated. Sometimes I want to simply plot different data sets from a single Table file. You can read about how I used some of my munging tools for Plotting from a single Table file.



Back to calling page