Pipes, Redirection & Filters

💤0
Lv 10 XP
← 🧱 Foundations · Command Line Skills

Pipes, Redirection & Filters

Intermediate ⭐ 80 XP ⏱ 18 min #cli#pipes#text-processing

Compose small tools into powerful one-liners with pipes, redirection, and filters.

📖Theory

The Unix philosophy is small tools that do one thing, combined. Three mechanisms make that possible:

  • Pipe | — send one command’s output into the next command’s input
  • Redirection > >> < — connect output/input to files
  • Streams — stdout (1) for results, stderr (2) for errors

The classic filters transform text streams: grep (match lines), sort, uniq, cut, wc, sed (substitute), awk (field processing). Chaining them forms a pipeline that generates → filters → summarizes.

🌍Real-World Example
# Top 5 IPs hitting a web log
awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -5

# Count error lines, ignore the rest
grep -i error app.log | wc -l

# Extract the 2nd column, dedupe, save to a file
cut -d, -f2 data.csv | sort -u > names.txt

# Replace text on the fly
sed 's/localhost/127.0.0.1/g' config.txt
✍️Hands-On Exercise
  1. Build a pipeline that counts how many unique words appear in a file.
  2. Use grep, sort, and uniq -c to find the most common log message.
  3. Redirect a command’s errors to a file while showing normal output on screen.
  4. Use cut to extract one column from a CSV and sort -u to dedupe it.
🧾Cheat Sheet▾
TaskCommand
Pipecmd1 | cmd2
Redirect out (overwrite)cmd > file
Appendcmd >> file
Redirect errorscmd 2> file
Merge stderr→stdoutcmd 2>&1
Discard outputcmd > /dev/null
Count lineswc -l
Field extractcut -d, -f2 / awk '{print $2}'
Substitutesed 's/a/b/g'
💬Common Interview Questions▾
What does a pipe do?

It connects the stdout of one command to the stdin of the next, letting you chain small tools into a processing pipeline without temporary files.

What's the difference between stdout and stderr?

stdout (fd 1) carries normal output; stderr (fd 2) carries error/diagnostic messages. Keeping them separate lets you redirect or filter each independently.

How do you discard a command's error output?

Redirect stderr to the null device: cmd 2>/dev/null.

📚Official Documentation

📝 My notes on this topic

Auto-saves as you type