Bash / Command Line Pipes and Redirection

Tags: #bash #commandline #pipes #linux #redirection

Last Reviewed: 16/08/2024

Part of Linux

File Descriptors:

• Definition: A file descriptor is a handle to a resource like a file, opened by a program via a request to the Linux kernel. The kernel assigns the next available file descriptor, typically starting at 3, since the first three (0, 1, 2) are reserved.
• Standard File Descriptors:
  • 0 (Standard Input, stdin): Receives input, typically from the keyboard, another program, or a redirected file.
  • 1 (Standard Output, stdout): Outputs data, usually to the terminal.
  • 2 (Standard Error, stderr): Outputs error messages, also to the terminal by default.

Pipes:

• Anonymous Pipes: Used to connect the output (stdout) of one program to the input (stdin) of another. For example, using the pipe (|) symbol, you can chain commands together, such as filtering output from ls -l through grep.
• Named Pipes: Created using the mkfifo command. These require both a reader and a writer to be active simultaneously, effectively connecting the output of one program to the input of another through a named file.

Redirection:

• Output Redirection: Redirects stdout or stderr to a file using the > symbol. For instance, program > file.log writes the program's output to file.log. To redirect both stdout and stderr to the same file, use 2>&1.
• Input Redirection: Uses the < symbol to supply input to a program from a file. For example, program < input.txt reads input from input.txt.

Combining Pipes and Redirection:

• You can combine pipes and redirection to handle complex data processing scenarios. For example, using program1 | program2 > output.txt redirects the output of program2 to output.txt.

Special Cases:

• Discarding Output: By redirecting output to /dev/null, you effectively discard it, as this is a special file that discards all data written to it.
• Custom File Descriptors: Beyond 0, 1, and 2, you can create custom file descriptors, e.g., 4>log.txt, which could be used for special cases in scripts or programs.

Appending vs. Truncating:

• Truncating: Using a single > truncates the target file before writing.
• Appending: Using » appends the output to the existing content of the file without truncating it.