Chapter 8: Redirection and Pipes in Linux

In Linux, redirection and pipes are powerful tools for controlling the flow of data between commands, files, and devices. Redirection allows you to send a command’s output to a file or use a file as input, while pipes let you chain commands by passing the output of one command as input to another. This chapter covers the essentials with practical examples to help you master these concepts.

Redirection Basics

Redirection uses special operators (>, >>, <, 2>, etc.) to control where input comes from or where output goes.

1. > (Output Redirection)

Redirects a command’s standard output (stdout) to a file, overwriting the file if it exists.

What it does: Sends the output of a command to a specified file, replacing its contents.

Examples:

Save the list of files in the current directory to files.txt

 ls > files.txt

 ls > files.txt

This creates or overwrites files.txt with the output of ls.

Save the system’s memory usage to memory.txt

free -h > memory.txt

free -h > memory.txt

The free -h command’s output is written to memory.txt.

2. >> (Append Output)

Appends a command’s output to a file without overwriting it.

What it does: Adds the output to the end of a file, preserving existing content.

Examples:

Append the current date to log.txt

date >> log.txt

date >> log.txt

This adds the output of date to log.txt without erasing its contents.

Append a list of running processes to process_log.txt

ps aux >> process_log.txt

ps aux >> process_log.txt

Each time you run this, new process information is added to the file.

3. < (Input Redirection)

Uses a file as the standard input (stdin) for a command.

What it does: Feeds the contents of a file to a command as its input.

Examples:

Sort the contents of names.txt

sort < names.txt

sort < names.txt

The sort command reads names.txt and outputs the sorted lines.

Send the contents of email_list.txt to the mail command

mail -s "Test Email" user@example.com < email_list.txt

mail -s "Test Email" [email protected] < email_list.txt

The mail command uses email_list.txt as the email body.

4. 2> (Error Redirection)

Redirects standard error (stderr) to a file.

What it does: Captures error messages (not regular output) and sends them to a file.

Examples:

Redirect errors from a find command to errors.txt

find / -name "missingfile" 2> errors.txt

find / -name "missingfile" 2> errors.txt

Permission-denied errors or other issues are saved to errors.txt.

Redirect both output and errors to separate files

ls /etc /nosuchdir > output.txt 2> errors.txt

ls /etc /nosuchdir > output.txt 2> errors.txt

The ls output goes to output.txt, and errors go toerrors.txt.

5. 2>&1 (Combine Output and Error)

Redirects standard error to the same destination as standard output.

What it does: Combines stderr with stdout, often used with > or >>.

Examples:

Save both output and errors from ls to combined.txt

ls /etc /nosuchdir > combined.txt 2>&1

ls /etc /nosuchdir > combined.txt 2>&1

Both the successful output and any errors are written to combined.txt.

Append both output and errors to log.txt

grep -r "error" /var/log >> log.txt 2>&1

grep -r "error" /var/log >> log.txt 2>&1

All output and errors from the grep command are appended to log.txt.

Pipes

Pipes (|) allow you to send the output of one command directly as input to another, enabling powerful command chaining.

What it does: Connects the stdout of one command to the stdin of another.

1. Basic Piping with |

Examples:

Search for a string in a file and count the matches

cat file.txt | grep "error" | wc -l

cat file.txt | grep "error" | wc -l

cat file.txt outputs the file, grep "error" filters for lines containing “error,” and wc -l counts those lines.

List all .txt files and sort them

ls | grep ".txt$" | sort

ls | grep ".txt$" | sort

ls lists files, grep ".txt$" filters for .txt files, and sort orders them alphabetically.

2. Combining Pipes with Multiple Commands

Examples:

Display the top 5 largest files in a directory

ls -l | sort -k 5 -nr | head -n 5

ls -l | sort -k 5 -nr | head -n 5

ls -l lists files in long format, sort -k 5 -nr sorts by the fifth field (size) in reverse numerical order, and head -n 5 shows the top 5.

Find processes containing “python” and format the output

ps aux | grep "python" | awk '{print $2, $11}'

ps aux | grep "python" | awk '{print $2, $11}'

ps aux lists processes, grep "python" filters for “python,” and awk prints the process ID and command name.

3. Using tee with Pipes

The tee command reads from stdin and writes to both a file and stdout.

What it does: Saves piped output to a file while passing it to the next command.

Examples:

Save a sorted list to a file and display it

ls | sort | tee sorted_files.txt

ls | sort | tee sorted_files.txt

The sorted list is written to sorted_files.txt and displayed on the screen.

Save and display filtered log entries

cat /var/log/syslog | grep "error" | tee error_log.txt

cat /var/log/syslog | grep "error" | tee error_log.txt

Lines containing “error” are saved to error_log.txt and shown in the terminal.

More Practical Examples

The redirection and pipe operators work identically in RHEL and Debian, as they are part of the Bash shell. Here are some practical scenarios combining multiple concepts:

Log Analysis: Monitor a log file for errors and save critical ones

cat /var/log/syslog | grep "CRITICAL" > critical_errors.txt

cat /var/log/syslog | grep "CRITICAL" > critical_errors.txt

This saves only “CRITICAL” log entries to critical_errors.txt.

System Monitoring: Check disk usage and save the top 3 directories:

du -sh /var/* | sort -hr | head -n 3 > disk_usage.txt

du -sh /var/* | sort -hr | head -n 3 > disk_usage.txt

du -sh shows directory sizes, sort -hr sorts in descending order, and head -n 3 saves the top 3 to disk_usage.txt.

Error Debugging: Run a script and log both output and errors:

./myscript.sh > script_output.txt 2>&1

./myscript.sh > script_output.txt 2>&1

All output and errors from myscript.sh are saved to script_output.txt.

Filtering and Formatting: List all running services and save those matching “http”:

systemctl list-units | grep "http" | tee http_services.txt

systemctl list-units | grep "http" | tee http_services.txt

This filters for HTTP-related services, displays them, and saves them to http_services.txt.

Tips for Redirection and Pipes

• Overwriting vs. Appending: Use > with caution, as it overwrites files. Use >> to safely append.
• Error Handling: Always consider redirecting stderr (2>) when debugging scripts or commands.
• Combining Tools: Pipes shine when combining commands like grep, awk, sort, and sed for data processing.
• Performance: For large datasets, grep -r or find might be faster than piping multiple commands.

This chapter equips you with the tools to manipulate command input and output efficiently. Practice these examples in your terminal to see how they work in real-time. In the next chapter, we’ll explore archives and compression to manage files effectively.

Summary Cheat Sheet