Text processing: grep, sed & awk

Three tools do most real-world log hunting and text wrangling on Linux, and they're built to chain together with pipes rather than work alone.

# grep — find lines matching a pattern
grep "ERROR" app.log
grep -i "error" app.log         # case-insensitive
grep -r "TODO" ./src            # recursive, through a whole directory
grep -v "DEBUG" app.log         # invert — show lines that DON'T match
grep -c "ERROR" app.log         # just count the matches

# sed — find and replace, stream-edit a file
sed 's/staging/production/' config.txt          # first match per line
sed 's/staging/production/g' config.txt         # every match per line
sed -i 's/staging/production/g' config.txt      # edit the file in place

# awk — pull out and work with specific columns
awk '{ print $1, $4 }' access.log          # print the 1st and 4th space-separated fields
awk -F: '{ print $1 }' /etc/passwd     # custom field separator — usernames from /etc/passwd
ps aux | awk '{ print $2, $11 }'          # PID and command, from ps output

Where they really earn their keep is chained together — filter with grep, extract a column with awk, count or dedupe with sort | uniq:

# the top 5 IP addresses hitting your server, from an access log
awk '{ print $1 }' access.log | sort | uniq -c | sort -rn | head -5