CISC3130: Review of Commands
A review
Spring 2011
Review: File systems
 Absolute pathname, relative pathname
 Commands: cd, pwd, mkdir, rmdir, rm, mv
 File examination: cat, head, tail, more, less, wc, file
 File manipulation: chmod, editors (vi, emacs, pico)
 Directory exploration: ls
 Understand the long listing format
 Different types of file
 Regular file:
 Directory:
 Links: …
 Understand permission setting of file/directory
 Compare text files files: diff, comp
Review: system info & misc
 /etc/passwd: stores user accounts in the system
 Different fields separated by :
 Understand the meaning of the fields
 who, who am i: list the users currently logged on
 ps: list processes running in the system
 date: current date
 sleep: resume executing next command after delaying for
specified amount of time
Shell related commands
 chsh: change default shell to use
 Startup configuration files for bash
 .bash_profile,
 .bash_rc
 To find out the name of the shell that you are using:
 From command line, type
echo $0
 Shell built-in commands
 echo
 set
 alias
 read
Filters commands
 For each command:
 Understand the basic functionalities of the following commands
 Know how to read online manual for more info.
 Common behavior:
 If no files are specified in command line, process standard input
 Output is generally written to standard output
 Benefits of such design?
 sort
 cut, paste
 comp, diff
 uniq
 tr
 grep family
 grep - uses regular expressions for pattern matching
 fgrep - file grep, does not use regular expressions, only
matches fixed strings but can get search strings from a
 egrep - extended grep, uses a more powerful set of
regular expressions but does not support
backreferencing, generally fastest
Hints on bash scripting
 Start with requirement, for example:
“Write a script that reports the number of processes run by
each of the users that are currently online”.
 Identify sub-tasks:
 Find out all users that are currently online
 For a given user, find out the number of processes belonging to
 Compose solution using solution to sub-task
for user in ## all users currently online
## find out the number of processes run by $user
Solve sub-task #1: users online
 Search for candidates …
 Is there a command that fits this job description exactly?
 Well, command who almost fits…
 Does the output of the command directly usable ?
 No
 Try the command out …
 How to process the output of the command so that it can be
used ?
 How to select only the first field of output ?
 Test and confirm your solution:
who | cut –d ‘ ‘ –f 1
Solve subtask #2: number of
 Task #2: to find out # of processed by a user $user
 No command fits this requirement directly…
 But command ps can give us all processes in the system …
 The output of ps also tells us the user each process belongs to
 Command wc can be used to count # of processes, as ps
displays one process per line
 Here, we are collecting info (using ps), and processing the
info in multiple steps using multiple commands
 Now you try to put them together …
Final solution
Another example:
 Requirement:
“ For all regular files under a directory given in command line,
displays what kind of file it is (using command file)”.
 Sub-tasks:
 Find all regular files under current directory
 Psedocode:
for filename in ##list of all regular file under $dir
file $filename
 Psedocode:
for filename in ##the list of all files under $dir
## if the file is regular file
file $filename
Solving subtask
 Find all files under a directory dir
ls $dir
 So we have
for filename in ls $dir
## if the file is regular file
file $filename
 Any problem ?
Sed: Stream-oriented, Non-Interactive,
Text Editor
 Look for patterns one line at a time, like grep
 Change lines of the file
 Non-interactive text editor
 Editing commands come in as script
 There is an interactive editor ed which accepts the same
 A Unix filter
 Superset of previously mentioned tools
Two power filters
sed and awk
Introduction to sed: substitution
 Substitute “command”: s
 changes all occurrences of a regular expression into a new value.
 to change "day" in file old to "night" in the "new" file:
sed s/day/night/ <old >new
 Generally: replace occurrence of a pattern in standard input,
with a given string, and display result in standard output
 sed s/regular expression/replace string/
Introduction to sed: substitution
 If you have meta-characters in the command, quotes are
 sed 's/3.1415[0-9]*/PI/' <old >new
 To mark a matching pattern
 grep –n count mylab1.cpp | sed s/count/<count>/
 One can use any letter to delimit different parts of command s
 If delimiter appears in regular expr or replace str, escape them
 To change /usr/local/bin to /common/bin:
 sed 's/\/usr\/local\/bin/\/common\/bin/' <old >new
 It is easier to read if you use other letter as a delimiter:
 sed 's_/usr/local/bin_/common/bin_' <old >new
 sed 's:/usr/local/bin:/common/bin:' <old >new
 sed 's|/usr/local/bin|/common/bin|' <old >new
How sed works?
 sed, like most Unix utilties, read a line at a time
 By default, sed command applies to the first occurrence of
the pattern in a line.
[zhang@storm ~]$ sed 's/aa*/bb/'
ab ab
bbb ab
 To apply to every occurrence, use option g (global)
 sed 's/aa*/bb/g‚
How sed works: aggressive matching
 sed finds the longest string in the line that matches the
pattern, and substitute it with the replacing string
 Pattern aa* matches with 1 or more a’s
[zhang@storm ~]$ sed 's/aa*/bb/'
Substitution with referencing
 How to mark all numbers (integers or floating points)
using angled brackets?
 E.g., 28 replaced by <28>, 3.1415 replaced by <3.1415>
 Use special character "&“, which refer to string that matches the
pattern (similar to backreference in grep.)
 sed 's/[0-9][0-9]*\.[0-9]*/(&)/g'
 You can have any number of "&" in replacement string.
 You could also double a pattern, e.g. the first number of a line:
$echo "123 abc" | sed 's/[0-9]*/& &/'
123 123 abc
Multiple commands
 To combine multiple commands, use -e before each command:
 sed -e 's/a/A/' -e 's/b/B/' <old >new
 If you have a large number of sed commands, you can put them
into a file, say named as sedscript
# sed comment - This script changes lower case vowels to upper case
each command must be on a separate line.
 Invoke sed with a script:
 sed -f sedscript <file.txt >file_cap.txt
sed interpreter script
 Alternatively, starts the script file (named CapVowel) with
 #!/bin/sed -f
and make the file executable
 Then you can evoke it directly:
 CapVowel <old >new
Restrict operations
 Restrict commands to certain lines
 Specifying a line by its number.
sed '3 s/[0-9][0-9]*//' <file >new
 Specifying a range of lines by number.
sed '1,100 s/A/a/' All lines containing a pattern.
 To delete first number on all lines that start with a
"#," use:
 sed '/^#/ s/[0-9][0-9]*//'
 Many other ways to restrict
Command d
 A useful command deletes every line that matches the
restriction: "d.“
 To look at the first 10 lines of a file, you can use:
 sed '11,$ d' <file
 i.e., delete from line 11 to the end of the file
 If you want to chop off the header of a mail message, which is
everything up to the first blank line, use:
 sed '1,/^$/ d' <file
Command q
 This command is most useful when you wish to abort the editing
after some condition is reached.
 Ex: another way to duplicate the head command is:
 sed '11 q' which quits when the eleventh line is reached.
 To keep first word of a line, and delete the rest of the line, mark first
word with the parenthesis:
 sed 's/\([a-z]*\).*/\1/'
 Recall: regular exprssions are greedy, and try to match as much as
 "[a-z]*" matches zero or more lower case letters, and tries to be as big
as possible.
 ".*" matches zero or more characters after the first match. Since the
first one grabs all of the lower case letters, the second matches anything
 Ex:
$echo abcd123 | sed 's/\([a-z]*\).*/\1/'
Backreference (cont’d)
 If you want to switch two words around, you can remember
two patterns and change the order around:
 sed 's/\([a-z][a-z]*\) \([a-z][a-z]*\)/\2 \1/’
 To eliminate duplicated words:
 sed 's/\([a-z]*\) \1/\1/'
 If you want to detect duplicated words, you can use
 sed -n '/\([a-z][a-z]*\) \1/p’
 Numeric value can have up to nine values: "\1" thru "\9.“
 To reverse first three characters on a line, you can use
 sed 's/^\(.\)\(.\)\(.\)/\3\2\1/'
Sed commands & scripts
 Each sed command consists of up to two addresses and an
action, where the address can be a regular expression or
line number.
 A script is nothing more than a file of commands
sed: a conceptual overview
 All editing commands in a sed script are applied in order
to each input line.
 If a command changes input, subsequent command address
will be applied to current (modified) line in the pattern
space, not original input line.
 Original input file is unchanged (sed is a filter), and the
results are sent to standard output (but can be redirected
to a file).
Why awk?
 It is an excellent filter and report writer.
 Many UNIX utilities generates rows and columns of
 AWK is an excellent tool for processing these rows and
columns, and is easier to use AWK than most conventional
programming languages.
 A. Aho, B. W. Kernighan and P. Weinberger.
Features of awk
 a pseudo-C interpretor,
 it understands same arithmetic operators as C
 string manipulation functions
 search for particular strings and modify output.
 AWK also has associative arrays, which are incredible useful,
and is a feature most computing languages lack. Associative
arrays can make a complex problem a trivial exercise.

File Attributes