Examples using awk and sed

The unix awk and sed commands are very useful, but I can never remember the syntax. Here I collect some simple examples.

  1. A sed example: remove those Mac newline characters!
  2. Grabbing columns from a file.
  3. Combining (column) files with paste.


A sed example: remove those Mac line-skips!.

The unix sed routine is very useful for making lobal substitions in a file: 

 

$ cat a
This is a test.

$ sed 's/test/bunny/g' a >a_new 

$ cat a_new
This is a bunny.
In the above example I made the new file named "a_new" that has my desired changes: I changed "test" to "bunny". A more practical example is the one below where I remove the next-line markers commonly seem in files from Mac OSX systems. Below I show a way to do this manually in vi, then I show a script that uses sed to do it: 
 

Method 1: Just use vi
To edit in a special command sequence like a character string (in vi) just preceed 
the command sequence with a cntrl-v. For example:
 cntrl-v cntrl-M 

Hence, my substitution line might look like:
:1,$s/^M//

Method 2: A script that uses sed
#!/bin/sh
#
# Remove carraige returns (^M) from a file
#
sed 's/^M//g' $1 > $1_new
I don't have to do this a lot, but I usually spend a lot of time trying to remember how to do it when I have to. Just to make this even easier (and easier to remember) I put it in a bash script named "mac_newline_delete". You can read about mac_newline_delete here and see a practical example>/a>.



Grabbing columns from a file.

Most of the times I use awk I just want to grab a few columns from file1 and direct them to file2. 


% cat file.1
23:57:08.488 +61:07:09.06 J2000 -> 1716.855 1582.139
23:57:10.983 +61:07:23.50 J2000 -> 1698.617 1596.066
23:57:13.478 +61:07:37.93 J2000 -> 1680.384 1609.987
23:57:15.974 +61:07:52.36 J2000 -> 1662.148 1623.909
23:57:18.470 +61:08:06.79 J2000 -> 1643.917 1637.835

% awk < file.1 '{ print $5 "  " $6}' > file.2 

% cat file.2 
1716.855  1582.139
1698.617  1596.066
1680.384  1609.987
1662.148  1623.909
1643.917  1637.835


Combining (column) files with paste.

Although paste is not an awk command, I use it a lot with awk. For instance, I might take a few columns from one file, take a few columns from another file, then combine the two extracted column sets to make a new file. Here is how. 

% cat file.1 
     1.00      1.00
     1.00     86.67
     1.00    172.33
     1.00    258.00
     1.00    343.67

% cat file.2 
1716.855  1582.139
1698.617  1596.066
1680.384  1609.987
1662.148  1623.909
1643.917  1637.835

% paste file.1 file.2 >file.3 
% cat file.3 
     1.00      1.00     1716.855  1582.139
     1.00     86.67     1698.617  1596.066
     1.00    172.33     1680.384  1609.987
     1.00    258.00     1662.148  1623.909
     1.00    343.67     1643.917  1637.835


Back to computer basics page