Advanced Scripting and Command Line
Usage with tshark and Related Utilities
March 31st, 2008
Sake Blok
Research & Development Engineer @ ion-ip
[email protected] / [email protected]
SHARKFEST '08
Foothill College
March 31 - April 2, 2008
SHARKFEST '08 | Foothill College | March 31 - April 2, 2008
Agenda
Part I : Why and how to use scripting?

GUI versus CLI

Collect, Filter, Transform, Present, Validate

How?
Part II : Useful CLI Tools

Wireshark CLI tools

Other CLI tools
Part III : Examples

CLI command piping

Scripting
Contest !?!
Questions?
SHARKFEST '08 | Foothill College | March 31 - April 2, 2008
GUI versus CLI
graphical USER interface vs COMMAND line interface
GUI -> powerful “fixed” functionality
CLI combined with other tools -> flexibility
CLI -> automate repetitive tasks
CLI not only when GUI is unavailable
SHARKFEST '08 | Foothill College | March 31 - April 2, 2008
5 steps of network analysis
Collect raw data
Filter raw data
Transform data into information
Present Information
Validate information
SHARKFEST '08 | Foothill College | March 31 - April 2, 2008
How?
What information do I need?

visualize your output
What (raw) data sources do I have?

Know the output formats of your data sources
What tools are available?

What can they do, browse through manpages for unknown
options
Practice & Experiment & be creative 
SHARKFEST '08 | Foothill College | March 31 - April 2, 2008
Part II : Tools
Wireshark CLI tools

tshark

dumpcap

editcap

mergecap

capinfos
Other CLI tools

|, >, for … do … done, `<command>`

cut, sort, uniq, tr

sed, awk

scripting (sh/perl/python/…)
SHARKFEST '08 | Foothill College | March 31 - April 2, 2008
tshark (1)
CLI version of wireshark
Similar to tcpdump, but statefull
uses dumpcap as capture engine
standard options: -D, -i, -c, -n, -l, -f, -R, -s, -w, -r
name resolving (-n)
time stamps (-t<format>)
decode as (-d tcp.port==8080,http)
preferences (-o <pref>:<value>)
SHARKFEST '08 | Foothill College | March 31 - April 2, 2008
tshark (2)
output formats (-V -T <format>)

summary, use of column prefs

Verbose (-V), hex dump (-x)

PDML (-T pdml)

fields (-T fields -E <sep> -e <field1> -e <field2> …)
statistics (-z …)

protocol hierarchy (-qz io,phs)

conversations (-qz conv,eth , -qz conv,tcp)

i/o statistics (-qz io,stat,10,ip,icmp,udp,tcp)
SHARKFEST '08 | Foothill College | March 31 - April 2, 2008
dumpcap
used by (wire|t)shark
can be used separately
options similar to tshark
stateless so traces can run forever
fast! only network->disk
ring buffers extremely useful

dumpcap -i 5 -s0 -b filesize:16384 -files:1024 -w ring.cap
SHARKFEST '08 | Foothill College | March 31 - April 2, 2008
capinfos
display summary of a tracefile
all info vs specific info
$ capinfos sharkfest-1.cap
File name: sharkfest-1.cap
File type: Wireshark/tcpdump/... - libpcap
File encapsulation: Ethernet
Number of packets: 3973
File size: 1431813 bytes
Data size: 1368221 bytes
Capture duration: 1299.436650 seconds
Start time: Thu Jan 17 11:37:16 2008
End time: Thu Jan 17 11:58:55 2008
Data rate: 1052.93 bytes/s
Data rate: 8423.47 bits/s
Average packet size: 344.38 bytes
$ capinfos -ae sharkfest-*.cap
File name: sharkfest-1.cap
Start time: Thu Jan 17 11:37:16 2008
End time: Thu Jan 17 11:58:55 2008
File name: sharkfest-2.cap
Start time: Thu Jan 17 11:39:27 2008
End time: Thu Jan 17 12:02:52 2008
SHARKFEST '08 | Foothill College | March 31 - April 2, 2008
editcap
used to manipulate a capture file
use ‘editcap –h’ to see all options
select frame ranges or time ranges


editcap -r sharkfest-1.cap tmp.cap 1-1000 2001-3000
editcap -A "2008-01-17 11:40:00" -B "2008-01-17 11:49:59"
sharkfest-1.cap tmp.cap
split file in chunks

editcap -c 1000 sharkfest-1.cap tmp.cap
change snaplen, time


editcap -s 96 sharkfest-1.cap tmp.cap
editcap -t -3600 sharkfest-1.cap tmp.cap
SHARKFEST '08 | Foothill College | March 31 - April 2, 2008
mergecap
used to merge captures
use ‘mergecap –h’ to see all options
based on timestamps

mergecap -w out.cap in-1.cap in-2.cap
or just append each file

mergecap -a -w out.cap in-1.cap in-2.cap
SHARKFEST '08 | Foothill College | March 31 - April 2, 2008
Part II : Tools
Wireshark CLI tools

tshark

dumpcap

editcap

mergecap

capinfos
Other CLI tools

|, >, for … do … done, `<command>`

cut, sort, uniq, tr

sed, awk

scripting (sh/perl/python/…)
SHARKFEST '08 | Foothill College | March 31 - April 2, 2008
|, >, for … do … done, `<command>`
Command piping with |

use ‘man bash’ for more info
ls -1t | head
Output redirection with >

ls -1t | head > newfiles.txt
Looping with for … do … done

for word in ‘one’ ‘two’ ‘three’; do echo $word; done
Command evaluation with backtics (``)

for file in `ls -1t | head`; do echo $file; head -1 $file;echo "";done > firstlines.txt
Variable assignments

newfile=`echo ${file}.bak`
SHARKFEST '08 | Foothill College | March 31 - April 2, 2008
cut, sort, uniq, tr
cut

-c (by position)
cut -c1-10 /etc/passwd

-f [-d ‘<delimiter>’] (by field)
cut -d ':‘ -f1 /etc/passwd

with no option
sort names.txt

-r (reverse sorting)
sort -r names.txt

-n (numerical sort)
du -ks * | sort -rn | head

with no option
sort names.txt | uniq

-d (only ‘doubles’)
sort names.txt | uniq -d

-c (show count)
sort names.txt | uniq -c

“ ” “_”
echo “one two” | tr “ “ “_”

-d “\015”
cat dosfile.txt | tr –d “\015” > unixfile.txt
sort
uniq
tr
SHARKFEST '08 | Foothill College | March 31 - April 2, 2008
sed, awk
both powerful scripting languages
sed - stream editor for filtering and transforming text



-e 's/<deleteme>//'
-e 's/<replaceme>/<withthis>/'
-e 's/^.*\(<keepme>\).*\(<andme>\).*$/\1 \2/‘
awk - pattern scanning and processing language


netstat -an | awk '$1=="TCP" {print $4}' | sort | uniq –c
… | awk '{printf(“ %s tcp.port==%s",sep,$1);sep="||"}'
SHARKFEST '08 | Foothill College | March 31 - April 2, 2008
scripting
parsing output when command piping is not enough
automate execution of tshark/dumpcap/mergecap etc
anything you can think of 
use your own favorite language (sh/perl/python/etc)
SHARKFEST '08 | Foothill College | March 31 - April 2, 2008
Part III : Examples
Using command piping

Counting http response codes

Top 10 URL's

All sessions with session-cookie XXXX
Using scripting

All sessions for user XXXX (shell script)

Synchonize time between trace files (perl script)
SHARKFEST '08 | Foothill College | March 31 - April 2, 2008
Test setup used to create capture files
HTTP Server
SMTP server
POP3 server
pop3
smtp
Internet
pop3
http
smtp
icmp
PC 1 : WebMail
(sharkfest-1.cap)
PC 2 : POP/SMTP client
(sharkfest-2.cap)
SHARKFEST '08 | Foothill College | March 31 - April 2, 2008
Example 1:
counting http response codes (1)
Problem
I need an overview of http response codes
Output
table with http response codes & counts
10 200
4 302
5 404
Input
Existing capture file with http traffic
Steps
print response codes
make table
SHARKFEST '08 | Foothill College | March 31 - April 2, 2008
Example 1:
counting http response codes (2)
Command:
tshark -o "tcp.desegment_tcp_streams:TRUE"
-r sharkfest-1.cap -R "http.response"
-T fields –e http.response.code |
sort | uniq –c
New tricks learned:

-o <pref>:<value>

-T fields –e <field>

| sort | uniq -c
SHARKFEST '08 | Foothill College | March 31 - April 2, 2008
Example 2:
Top 10 requested URL's (1)
Problem
Output
I need a list of all URL’s that have been visited
Sorted list with requested URL’s and count
13 http://www.shop.com/index.html
10 http://www.shop.com/images/logo.gif
Input
Steps
Existing capture file with http traffic
Print http.host and http.request.uri
Strip everything after “?”
Combine host + uri and format into normal URL
count url’s
make top 10
SHARKFEST '08 | Foothill College | March 31 - April 2, 2008
Example 2:
Top 10 requested URL's (2)
Command:
tshark -r sharkfest-1.cap -R http.request
-T fields -e http.host -e http.request.uri |
sed -e 's/?.*$//' |
sed -e 's#^\(.*\)\t\(.*\)$#http://\1\2#' |
sort | uniq -c | sort -rn | head
New tricks learned:

sed -e 's/?.*$//‘
remove unnecessary info

sed -e 's#^\(.*\)\t\(.*\)$#http://\1\2#‘
 transform

| sort | uniq -c | sort -rn | head
 top10
SHARKFEST '08 | Foothill College | March 31 - April 2, 2008
Example 3:
All sessions with cookie XXXX (1)
Problem
I know in which “session” a problem exists, but
I need all data from that session to work it out
Output
New capture file with whole tcp sessions that
contain cookie PHPSESSID=c0bb9d04cebbc765bc9bc366f663fcaf
Input
Existing capture file with http traffic
Steps
extract port numbers based on cookie
create new filter based on port numbers
filter tcp sessions to a new capture file
SHARKFEST '08 | Foothill College | March 31 - April 2, 2008
Example 3:
All sessions with cookie XXXX (2)
Command:
tshark -r sharkfest-1.cap -w cookie.cap
-R `tshark -r sharkfest-1.cap -T fields -e tcp.srcport -R "http.request and
http.cookie contains \"PHPSESSID=c0bb9d04cebbc765bc9bc366f663fcaf\"" |
awk '{printf(“ %s tcp.port==%s",sep,$1);sep="||"}‘ `
New tricks learned:

tshark -R `<other command that generated filter>`

awk '{printf(“ %s tcp.port==%s",sep,$1);sep="||"}'
SHARKFEST '08 | Foothill College | March 31 - April 2, 2008
Example 4:
All sessions for user XXXX (1)
Problem
A particular user has multiple sessions and
I need to see all sessions from that user
Output
New capture file with all data for user xxxx
Input
Existing capture file with http data
Steps
link session cookies to user
extract session cookies
create new capture file per session cookie
merge files
SHARKFEST '08 | Foothill College | March 31 - April 2, 2008
Example 4:
All sessions for user XXXX (2)
New tricks learned:
Script:
$ cat example6.sh
#!/usr/bin/bash
file=$1
user=$2
for cookie in `tshark -r $file -R "http.request and http
contains $user" -T fields -e http.cookie | tr " " "_"`
do
sid=`echo $cookie | cut -d '_' -f 2 | tr -d "\015"`
tmpfile="tmp_`echo $sid | cut -d '=' -f 2`.cap"
echo "Processing session cookie $sid to $tmpfile"

for … do … done

<var>=`echo …| …`

cut -d <FS> -f <x>

tr –d “\015”


mergecap -w <outfile> <infile1>
<infile2> …
`ls -1 tmp_*.cap`
tshark -r $file -w $tmpfile -R `tshark -r $file "http.request
and http.cookie contains \"$sid\"" -T fields -e
tcp.srcport | awk
'{printf("%stcp.port==%s",sep,$1);sep="||"}'`
done
mergecap -w $user.cap `ls -1 tmp_*.cap`
rm `ls -1 tmp_*.cap`
SHARKFEST '08 | Foothill College | March 31 - April 2, 2008
Example 5:
Synchronize time between trace files (1)
Problem
In order to analyse a problem, two traces at
different locations need to be compared, but
the timestamps are different
Output
New file with “synchronized” timestamps
Input
Two capture files (with icmp packets)
Steps
Make sure to ping between the capture host
Match icmp packets in both files
Calculate min and max difference in time
Create new file with corrected timestamps
SHARKFEST '08 | Foothill College | March 31 - April 2, 2008
Example 5:
Synchronize time between trace files (2)
Script:
foreach $seq (sort keys %seq) {
next unless defined $times1{'req'}{$seq};
next unless defined $times1{'resp'}{$seq};
next unless defined $times2{'req'}{$seq};
next unless defined $times2{'resp'}{$seq};
$ cat example7.pl
#!/usr/bin/perl
$file1 = shift || die;
$file2 = shift || die;
$file3 = shift;
$reqdiff= $times2{'req'}{$seq} - $times1{'req'}{$seq};
$respdiff= $times2{'resp'}{$seq} - $times1{'resp'}{$seq};
$tshark = '/cygdrive/c/Program\ Files/Wireshark/tshark.exe';
$editcap = '/cygdrive/c/Program\ Files/Wireshark/editcap.exe';
printf "%s : %9.3f %9.3f %9.3f %9.3f -> %15.9f %15.9f\n", $seq,
$times1{'req'}{$seq},
$times2{'req'}{$seq},
$times2{'resp'}{$seq},
$times1{'resp'}{$seq},
$reqdiff, $respdiff;
$options = "-T fields -e frame.time -e icmp.type -e icmp.ident -e icmp.seq -R icmp";
$cmd1 = $tshark . " -r $file1 " . $options . " 2>/dev/null |";
$cmd2 = $tshark . " -r $file2 " . $options . " 2>/dev/null |";
$t1 = \%times1;
$t2 = \%times2;
if( !defined $mindiff ) {
$mindiff = $reqdiff;
$maxdiff = $respdiff;
} else {
$mindiff = $reqdiff unless $mindiff > $reqdiff;
$maxdiff = $respdiff unless $maxdiff < $respdiff;
}
sub readline {
my ($p,$line) = @_;
chomp $line;
$line =~ s/\015//;
($time,$type,$ident,$seq) = split '\t',$line;
$id = $ident . "-" . sprintf "%05d",$seq;
$seq{$id} = 1;
if( $type == 8 ) {
$p->{'req'}{$id} = &conv_time($time);
} else {
$p->{'resp'}{$id} = &conv_time($time);
}
}
}
$diff = -int( 500000*($mindiff + $maxdiff) ) / 1000000;
printf "\n";
printf "Minimum difference found : %15.9f\n", $mindiff;
printf "Maximum difference found : %15.9f\n", $maxdiff;
printf "\n";
sub conv_time {
my ($str) = @_;
($dummy,$h,$m,$s) = ($str =~ /^(.*) (\d\d):(\d\d):(\d\d\.\d+)$/);
return 3600 * $h + 60 * $m + $s;
}
if( abs($diff)<0.000001 ) {
printf "File \"%s\" and \"%s\" are already time synchronized.\n",$file1, $file2;
} else {
printf "File \"%s\" needs to be adjusted %f seconds to match file \"%s\"\n",
$file2, $diff, $file1;
if( $file3 eq "" ) {
printf "\nExecute \"editcap -t %f %s <outfile>\" to create a synchronized\nversion of file \"%s\"\n", $diff, $file2,
$file2;
} else {
$cmd = $editcap . " -t $diff " . $file2 . " $file3";
system $cmd;
printf "\nExecuted \"editcap -t %f %s %s\" to create a synchronized\nversion of file \"%s\"\n", $diff, $file2, $file3,
$file2;
open(F, $cmd1);
while(<F>) {
&readline($t1, $_);
}
close(F);
open(F, $cmd2);
while(<F>) {
&readline($t2, $_);
}
close(F);
}
}
SHARKFEST '08 | Foothill College | March 31 - April 2, 2008
Summary

tshark, dumpcap, capinfos, editcap, mergecap

tshark+scripting can complement GUI

use little building blocks and combine them

Hopefully you are triggered to experiment 
See http://www.euronet.nl/~sake/sharkfest08/ for the presentation,
used trace files, example-scripts and notes of this session.
SHARKFEST '08 | Foothill College | March 31 - April 2, 2008
Contest
Create a one-liner that creates one capture file for each
pop3-user with the name <username>.cap
These files should contain all pop3 traffic for the specific
user and nothing else.
Use file sharkfest-2.cap (available at
http://www.euronet.nl/~sake/sharkfest08/)
Send your one-liner to me at [email protected] today before
11:59 PM. I will draw 2 winners from the one-liners that
work as stated above. I will e-mail back my solution and
the list of winners on Tuesday.
Prices can be collected by the winners on Wednesday,
please look me up at lunch 
SHARKFEST '08 | Foothill College | March 31 - April 2, 2008
Questions?
?
SHARKFEST '08 | Foothill College | March 31 - April 2, 2008
Descargar

Slide 1