103.2 Process text streams using filters

Introduction

This lesson is about small text tools.

The main question is:

How do you take text input, change it, shorten it, sort it, or inspect it without opening a big editor?

What you should be able to do after this lesson:

• Use common text filters.
• Read only the beginning or end of a stream.
• Sort and deduplicate lines.
• Transform characters.
• Check file checksums.

Big Idea: Small Tools, Simple Jobs

UNIX-style work is often:

one tool does one small text job -> pipe it to the next tool

That is why filters matter so much.

Basic Viewers

Print a whole file

cat file.txt

Read page by page

less file.txt

First lines

head file.txt
head -n 20 file.txt

Last lines

tail file.txt
tail -n 20 file.txt

Counting and Numbering

Count lines, words, bytes

wc file.txt
wc -l file.txt

Add line numbers

nl file.txt

Selecting and Rearranging Text

Select fields

cut -d: -f1 /etc/passwd

Example output:

root
bin
daemon
mail
ftp
http
nobody
dbus

What this shows:

• cut -d: -f1 extracts the first colon-separated field
• on /etc/passwd, that first field is the username

Sort lines

sort file.txt

Show unique lines

uniq file.txt
sort file.txt | uniq

Important beginner rule:

• uniq only removes adjacent duplicates
• that is why sort | uniq is so common

Join lines side by side

paste file1.txt file2.txt

Character Changes

Translate characters

echo "hello" | tr 'a-z' 'A-Z'

Example output:

HELLO

What this shows:

• tr can translate one set of characters into another
• uppercase conversion is one of the most common first examples

Simple text editing with sed

echo "color" | sed 's/color/colour/'

Splitting Files

split -l 100 bigfile.txt chunk_

This creates smaller files from a large input.

Inspecting Data

Show bytes in another form

od -c file.bin | head

This is helpful when a file is not plain text.

Checksums

Checksums help confirm file integrity.

md5sum file.iso
sha256sum file.iso
sha512sum file.iso

Reading Compressed Text Without Fully Extracting

zcat file.gz
bzcat file.bz2
xzcat file.xz

This is useful when you want to inspect content quickly.

Practice Step by Step

1) Count and sort usernames from `/etc/passwd`

1. Extract usernames: cut -d: -f1 /etc/passwd
2. Sort them: cut -d: -f1 /etc/passwd | sort
3. Count them: cut -d: -f1 /etc/passwd | wc -l

2) Convert text to uppercase

1. Run: echo "linux" | tr 'a-z' 'A-Z'
2. Result: LINUX

Cheat Sheet

• cat = print file content
• less = view long text page by page
• head = first lines
• tail = last lines
• wc = count lines, words, bytes
• nl = number lines
• cut = select fields
• sort = sort lines
• uniq = collapse adjacent duplicates
• paste = join lines side by side
• tr = translate characters
• sed = simple stream editing
• split = split large files
• od = inspect raw data
• md5sum, sha256sum, sha512sum = checksums
• zcat, bzcat, xzcat = read compressed text