发布于

mpstat: Displaying CPU Statistics for Each Available CPU

Functionality:
The mpstat command is used to display statistics for each available CPU.

Syntax:
mpstat [ options ]

Overview:
mpstat (Multi-Processor Statistics) is a tool for displaying statistics on the performance of individual CPUs, functioning as a real-time monitoring utility. While similar to vmstat, mpstat focuses solely on CPU performance statistics. To install mpstat on Ubuntu or CentOS, you can use the following commands:

For Ubuntu:

sudo apt install sysstat

For CentOS:

sudo yum install sysstat

Options:

  • -P: Specify a CPU number or use ALL to display statistics for all CPUs.

Examples:

Running mpstat without any options will display overall performance statistics for all CPUs:

$ mpstat
Linux 6.8.0-45-generic (Ubuntu22-VirtualBox)   2024年10月15日   _x86_64_   (2 CPU)

13:48:03     CPU   %usr   %nice   %sys   %iowait   %irq   %soft   %steal   %guest   %gnice   %idle
13:48:03     all   0.11   0.02    0.10     0.01    0.00    0.17     0.00     0.00     0.00   99.60

Using the -P ALL option provides both overall CPU performance statistics as well as detailed statistics for each individual CPU:

$ mpstat -P ALL
Linux 6.8.0-45-generic (Ubuntu22-VirtualBox)   2024年10月15日   _x86_64_   (2 CPU)

13:59:40     CPU   %usr   %nice   %sys   %iowait   %irq   %soft   %steal   %guest   %gnice   %idle
13:59:40     all   0.11   0.02    0.10     0.01    0.00    0.17     0.00     0.00     0.00   99.60
13:59:40       0   0.11   0.02    0.10     0.01    0.00    0.31     0.00     0.00     0.00   99.45
13:59:40       1   0.10   0.02    0.10     0.01    0.00    0.03     0.00     0.00     0.00   99.75

You can also specify a particular CPU by using the -P n option, where n represents the CPU number starting from 0:

$ mpstat -P 0
Linux 6.8.0-45-generic (Ubuntu22-VirtualBox)   2024年10月15日   _x86_64_   (2 CPU)

14:18:14     CPU   %usr   %nice   %sys   %iowait   %irq   %soft   %steal   %guest   %gnice   %idle
14:18:14       0   0.11   0.02    0.10     0.01    0.00    0.31     0.00     0.00     0.00   99.45

$ mpstat -P 1
Linux 6.8.0-45-generic (Ubuntu22-VirtualBox)   2024年10月15日   _x86_64_   (2 CPU)

14:18:17     CPU   %usr   %nice   %sys   %iowait   %irq   %soft   %steal   %guest   %gnice   %idle
14:18:17       1   0.10   0.02    0.10     0.01    0.00    0.03     0.00     0.00     0.00   99.75

Field Descriptions:

  • %usr: Percentage of CPU time spent in user mode (excluding processes with a negative nice value). Calculation: (usr/total)*100
  • %nice: Percentage of CPU time spent on processes with a negative nice value. Calculation: (nice/total)*100
  • %sys: Percentage of CPU time spent in kernel mode. Calculation: (system/total)*100
  • %iowait: Percentage of time spent waiting for I/O operations. Calculation: (iowait/total)*100
  • %irq: Percentage of CPU time spent handling hardware interrupts. Calculation: (irq/total)*100
  • %soft: Percentage of CPU time spent handling software interrupts. Calculation: (softirq/total)*100
  • %idle: Percentage of CPU time spent idle, excluding time waiting for I/O operations. Calculation: (idle/total)*100

发布于

AWK: A Powerful Text Processing Tool and Programming Language

Overview:
AWK is a robust text processing tool and programming language, primarily used for formatting, analyzing, and processing text on Unix and Linux systems. It excels at handling structured text like tables, CSV files, and logs.

Syntax:

awk -f 'scripts' -v var=value filename
awk 'BEGIN{ print "start" } pattern{ commands } END{ print "end" }' filename

Explanation:
AWK reads files or input streams (including stdin) line by line, processing text data based on user-specified patterns and actions. It is particularly useful for structured text. While AWK can be used directly from the command line, it is more often employed through scripts. As a programming language, AWK shares many features with C, such as arrays and functions.

Options:

  • -F: Specifies the field separator (can be a string or regular expression).
  • -f 'scripts': Reads AWK commands from the script file 'scripts'.
  • -v var=value: Assigns a value to a variable, passing external variables to AWK.

AWK Script Structure:

  • pattern: Matches specific lines.
  • { commands }: Executes actions on matching lines.
  • filename: The file to be processed by AWK.

An AWK script typically consists of three optional parts: a BEGIN block, pattern matching, and an END block. The workflow proceeds as follows:

  1. Execute the BEGIN statement.
  2. Process each line from the file or standard input, executing the pattern matching.
  3. Execute the END statement.

AWK Built-in Variables:

  • $0: The current record (line).
  • $n: The nth field (column) of the current record, where $1 is the first column and $n is the nth column.
  • FS: The field separator (default is a space or tab), can be customized with the -F option.
  • OFS: Output field separator (used for formatted output).
  • RS: Record separator (default is newline).
  • ORS: Output record separator (default is newline).
  • NR: Current line number (starting from 1).
  • NF: Number of fields (columns) in the current line.

AWK Operators:

  • Arithmetic Operators:
    +, -, *, /, %, ^
    Increment and decrement operators (++, --) can be used as prefixes or suffixes.
  • Assignment Operators:
    =, +=, -=, *=, /=, %=, ^=
  • Regular Expression Operators:
    ~: Matches regular expression.
    !~: Does not match regular expression.
  • Logical Operators:
    ||: Logical OR
    &&: Logical AND
  • Relational Operators:
    <, <=, >, >=, !=, ==
  • Other Operators:
    $: Refers to a field by its number.
    Space: Concatenates strings.
    ?:: Ternary operator.
    in: Checks if a key exists in an array.

AWK Regular Expression Syntax:

  • ^: Matches the start of a line.
  • $: Matches the end of a line.
  • .: Matches any single character.
  • *: Matches zero or more occurrences of the preceding character.
  • +: Matches one or more occurrences of the preceding character.
  • ?: Matches zero or one occurrence of the preceding character.
  • []: Matches any character in the specified range.
  • [^]: Matches any character not in the specified range.
  • () and |: Subexpressions and alternations.
  • \: Escape character.
  • {m}: Matches exactly m occurrences of a character.
  • {m,}: Matches at least m occurrences.
  • {m,n}: Matches between m and n occurrences.

AWK Built-in Functions:

  • toupper(): Converts all lowercase letters to uppercase.
  • length(): Returns the length of a string.

Custom Functions in AWK: AWK scripts can include user-defined functions. For example:

function square(x) {
  return x * x;
}

To use the function:

awk '{ print square($1) }' file.txt

AWK Control Flow Statements:

  • if-else: Conditional statements.
  • while and do-while: Loops.
  • for: Standard loops, including array traversal with for-in.
  • break and continue: Loop control.
  • exit: Terminates the script execution.
  • next: Skips the remaining commands for the current line.
  • return: Returns a value from a function.
  • ?:: Ternary operator for conditional expressions.

AWK Arrays: AWK supports associative arrays, meaning array indexes can be strings as well as numbers. Arrays in AWK don’t need to be declared or sized; they are created as soon as you assign a value to an index.

Examples:

1. Basic Example:

$ echo "hello" | awk 'BEGIN{ print "start" } END{ print "end" }'
start
end

2. Using Built-in Variables: To print the first and third columns of a file:

awk '{ print $1, $3 }' test.txt

3. Using External Variables:

$ a=100
$ b=100
$ echo | awk '{ print v1 * v2 }' v1=$a v2=$b
10000

4. Using Regular Expressions: To print the second column of lines starting with “a”:

awk '/^a/ { print $2 }' test.txt

5. Using Built-in Functions: Convert all lowercase letters to uppercase:

awk '{ print toupper($0) }' test.txt

6. Handling Different Delimiters: For CSV files with comma-separated values:

awk -F ',' '{ print $1, $2 }' test.csv

7. Writing and Running AWK Scripts: Save an AWK script to a file (e.g., script.awk):

BEGIN { FS=","; OFS=" - " }
{ print $1, $3 }

Run the script:

awk -f script.awk test.csv

Conclusion:

AWK is a versatile and powerful tool for text processing, offering rich features like pattern matching, regular expressions, and scripting capabilities. From simple one-liners to complex data analysis scripts, AWK excels at processing structured text efficiently and flexibly.

发布于

A Guide to the sed Stream Editor

Function Overview:
sed is a stream editor that reads text from files or input streams line by line, edits the text according to user-specified patterns or commands, and then outputs the result to the screen or a file. When used in conjunction with regular expressions, it is incredibly powerful.

Syntax:

sed [options] 'command' file(s)
sed [options] -f scriptfile file(s)

Explanation:
sed first stores each line of the text in a temporary buffer called the “pattern space.” It then processes the content of this buffer according to the given sed commands. Once the processing is complete, the result is output to the terminal, and sed moves on to the next line. The content of the file itself is not altered unless the -i option is used. sed is mainly used to edit one or more text files, simplify repeated text file operations, or create text transformation scripts. Its functionality is similar to awk, but sed is simpler and less capable of handling column-specific operations, while awk is more powerful in that regard.

Options:

  • -e: Use the specified commands to process the input text file.
  • -n: Suppress automatic output (only prints lines modified when used with the p command).
  • -h: Display help information.
  • -V: Display version information.

Parameters:

  • command: The command to be executed.
  • file(s): One or more text files to be processed.
  • scriptfile: A file containing a list of commands to execute.

Common Actions:

  • a: Append text after the current line.
  • i: Insert text before the current line.
  • c: Replace the selected lines with new text.
  • d: Delete the selected lines.
  • D: Delete the first line of the pattern block.
  • s: Replace specified characters.
  • h: Copy the pattern block’s content to an internal buffer.
  • H: Append the pattern block’s content to the internal buffer.
  • g: Retrieve content from the internal buffer and replace the text in the current pattern block.
  • G: Retrieve content from the internal buffer and append it to the current pattern block.
  • l: List non-printable characters in the text.
  • L: Similar to l, but specifically for handling non-ASCII characters.
  • n: Read the next input line and apply the next command to it instead of reapplying the first command.
  • N: Append the next input line to the current pattern block and insert a new line between them, changing the current line number.
  • p: Print the matching lines.
  • P: Print the first line of the pattern block.
  • q: Quit sed.
  • b label: Branch to the location marked by label in the script; if the label doesn’t exist, the branch goes to the end of the script.
  • r file: Read lines from a file.
  • t label: Conditional branch to a marked location, starting from the last line. If the condition is met, or a T/t command is used, the branch jumps to the specified label or the end of the script.
  • T label: Error branch. If an error occurs, this branches to the labeled command or the end of the script.
  • w file: Write the processed block of the pattern space to the end of a file.
  • W file: Write the first line of the pattern space to the end of a file.
  • !: Execute the following commands on all lines not selected by the current pattern.
  • =: Print the current line number.
  • #: Extend comments to the next newline character.

Replacement Commands:

  • g: Global replacement within a line (used with the s command).
  • p: Print the line.
  • w: Write the line to a file.
  • x: Exchange the text in the pattern block with the text in the internal buffer.
  • y: Translate one character to another (not used with regular expressions).
  • &: Reference to the matched string.

Basic Regular Expression (BRE) Syntax in sed:

  • ^: Match the beginning of a line.
  • $: Match the end of a line.
  • .: Match any single character except a newline.
  • *: Match zero or more of the preceding characters.
  • []: Match a single character from a specified range.
  • [^]: Match a single character not in the specified range.
  • (..): Capture a substring.
  • &: Save the matched text for later use in replacements.
  • <: Match the start of a word.
  • >: Match the end of a word.
  • x{m}: Match exactly m occurrences of x.
  • x{m,}: Match at least m occurrences of x.
  • x{m,n}: Match between m and n occurrences of x.

To match the start of a word, use \<. To match the end of a word, use \>.

Extended Regular Expression (ERE) Syntax in sed:

  • \b: Match a word boundary (not supported by default in sed regular expressions).
  • +: Match one or more occurrences of the preceding character.

Practical Examples:

1 Print specific lines:
To print only lines 1 and the last line:

sed -n '1p;$p' test.txt

2 Delete lines:
To delete the second line:

sed '2d' filename

3 Basic match and replace:
Replace spaces with hyphens:

echo "hello world" | sed 's/ /-/g'

4 Advanced match and replace:
Reverse words in a string:

echo "abc def ghi" | sed 's/\([a-zA-Z]*\) \([a-zA-Z]*\) \([a-zA-Z]*\)/\3 \2 \1/'

5 Multiple edits:
Replace “Hello” with “Hi” and “Goodbye” with “Farewell” in one command:

sed 's/Hello/Hi/; s/Goodbye/Farewell/' example.txt

6 Read a file:
Insert content from an external file after lines matching a pattern:

sed '/Line 2/r extra.txt' data.txt

7 Write to a file:
Save processed content into a new file:

sed 's/World/Everyone/' input.txt > output.txt

In summary, sed is a versatile and efficient tool for editing text in a stream, offering powerful pattern matching and text transformation capabilities when combined with regular expressions. From basic line printing to advanced text manipulation, sed serves a wide range of text processing needs.

发布于

What is the od Command and How to Use It?

The od (octal dump) command is a versatile tool that outputs the contents of a specified file in various formats such as octal, decimal, hexadecimal, floating-point numbers, or ASCII characters. It displays the content to the standard output (usually the terminal), with the leftmost column showing the byte offset, starting from 0.

Function:

The od command outputs file content in various formats like octal, decimal, hexadecimal, floating-point, or ASCII, with the byte offset displayed in the leftmost column. It can handle both text and binary files and is typically used to view file data that cannot be directly displayed in the terminal, such as binary data. The command can interpret the file content and output its values in various formats, whether they are IEEE754 floating-point numbers or ASCII codes. You might also want to check out the hexdump command, which by default outputs data in hexadecimal format but isn’t as powerful as od.

Syntax:

od [OPTION…] [FILE…]

Key Options:

  • -A RADIX or --address-radix=RADIX: Specifies the radix (base) for the byte offset. By default, the offset is displayed in octal.
  • -j BYTES or --skip-bytes=BYTES: Skips the specified number of bytes before displaying the file content.
  • -N BYTES or --read-bytes=BYTES: Outputs only the specified number of bytes.
  • -S [BYTES] or --strings[=BYTES]: Outputs strings at least BYTES bytes long (default is 3).
  • -v or --output-duplicates: Ensures that duplicate data is not omitted in the output.
  • -w [BYTES] or --width[=BYTES]: Sets the number of bytes to display per line (default is 32 bytes).
  • -t TYPE or --format=TYPE: Specifies the format of the output. Options include:
    • a: Named characters (e.g., newline is shown as “nl”).
    • c: Printable characters or escaped sequences (e.g., newline is shown as “\n”).
    • d[SIZE]: Signed decimal integers of SIZE bytes (default is sizeof(int)).
    • f[SIZE]: Floating-point numbers of SIZE bytes (default is sizeof(double)).
    • o[SIZE]: Octal integers of SIZE bytes (default is sizeof(int)).
    • u[SIZE]: Unsigned decimal integers of SIZE bytes (default is sizeof(int)).
    • x[SIZE]: Hexadecimal integers of SIZE bytes (default is sizeof(int)).
    The SIZE can be specified as 1 (byte), or as uppercase letters like C (char), S (short), I (int), and L (long). For floating-point numbers, SIZE can be F (float), D (double), or L (long double).
  • --help: Displays help information.
  • --version: Displays version information.

Parameters:

  • FILE…: One or more files whose content will be displayed.

Examples:

Example 1: Basic Output

$ cat test.txt
abcd 12345
$ od test.txt 
0000000 061141 062143 030440 031462 032464 000012
0000013

In this output, the first column shows the byte offset (default in octal).

Example 2: Show Byte Offset in Decimal

$ od -Ad test.txt 
0000000 061141 062143 030440 031462 032464 000012
0000011

Example 3: Hide Byte Offset

$ od -An test.txt 
 061141 062143 030440 031462 032464 000012

Example 4: Output in Hexadecimal (4 Bytes per Group)

$ od -tx test.txt 
0000000 64636261 33323120 000a3534
0000013

Example 5: Output in Hexadecimal (1 Byte per Group)

$ od -tx1 test.txt
0000000 61 62 63 64 20 31 32 33 34 35 0a
0000013

Example 6: Display Named ASCII Characters

$ od -ta test.txt
0000000   a   b   c   d  sp   1   2   3   4   5  nl
0000013

Or display printable characters and escape sequences:

$ od -tc test.txt
0000000   a   b   c   d       1   2   3   4   5  \n
0000013

Example 7: Hexadecimal with Original Characters

$ od -tcx1 test.txt
0000000   a   b   c   d       1   2   3   4   5  \n
         61  62  63  64  20  31  32  33  34  35  0a
0000013

Example 8: Specify Bytes per Line

$ od -w8 -tc test.txt
0000000   a   b   c   d       1   2   3
0000010   4   5  \n
0000013

Example 9: Remove Spaces Between Columns

To remove spaces between columns during od output:

  1. Use -An to hide the offset.
  2. Use -v to avoid omitting duplicate data.
  3. Use -tx1 to output one byte per group in hexadecimal format, and -w1 to display one byte per line.
  4. Finally, pipe the output to awk to concatenate it into a single line.
$ od -An -w1 -tx1 test.txt | awk '{for(i=1;i<=NF;++i){printf "%s",$i}}'
616263642031323334350a

发布于

Linux xargs Command Passes Arguments to Other Commands

Description: xargs is used to pass arguments to other commands and is an essential component for building one-liner commands.

Syntax:
xargs [OPTIONS] [COMMAND]

Overview:
xargs takes input from stdin, separated by spaces or newline characters, and passes it as space-separated arguments to other commands. However, be careful when filenames or strings contain spaces, as xargs may misinterpret them.

Options:

  • -0, --null: Default option. If stdin contains special characters like backticks (), backslashes (\), or spaces, xargs` restores them to regular characters.
  • -a, --arg-file=FILE: Reads input from the specified file instead of stdin.
  • -d, --delimiter=DEL: Specifies the delimiter to separate input. By default, xargs uses spaces and newlines, outputting arguments separated by spaces.
  • -E EOF_STR: Sets an end-of-input string. If none is specified, input has no terminator. EOF_STR must be a separate field (i.e., space or newline separated).
  • -e, --eof[=EOF_STR]: Same as -E, but non-POSIX compliant. Use -E if available.
  • -I REPLACE_STR: Assigns each argument to the specified placeholder (e.g., {}, $, @). Useful for positioning arguments when there are multiple parameters. For example:find . -name "*.txt" | xargs -I {} cp {} /tmp/{}.bak
  • -i, --replace[=REPLACE_STR]: Same as -I, but REPLACE_STR is optional and defaults to {}. Use -I for POSIX compliance.
  • -L MAX_LINES: Limits the number of input lines per execution, implying the -x option.
  • -l, --max-lines[=MAX_LINES]: Same as -L. Defaults to 1 line. Use -L for POSIX compliance.
  • -n, --max-args=MAX_ARGS: Specifies the maximum number of arguments to pass to the command at once.
  • -o, --open-tty: Reopens stdin to /dev/TTY before running the command in a subprocess, useful for interactive applications.
  • -P, --max-procs=MAX_PROCS: Sets the maximum number of parallel processes. Default is 1. Use with -n or -L for batch processing.
  • -p, --interactive: Prompts the user for confirmation before executing each command.
  • --process-slot-var=NAME: Sets an environment variable with a unique value for each running subprocess. Once a process finishes, the value is reused.
  • -r, --no-run-if-empty: Stops xargs from running if there is no input. This is the default behavior.
  • -s, --max-chars=MAX_CHARS: Limits the maximum number of characters (including command, spaces, and newlines) in the command.
  • --show-limits: Displays the system’s command-line length limitations.
  • -t, --verbose: Prints the command to stderr before executing it.
  • -x, --exit: Exits if the command line exceeds the specified character limit (-s).
  • --help: Displays help information.
  • --version: Displays version information.

Parameters:

  • COMMAND: The command string to execute.

Examples:

Example 1
Some commands don’t accept piped arguments directly. Use xargs to pass them:

# Incorrect: `ls` cannot accept piped input directly
find /sbin -perm +700 | ls -l

# Correct: use `xargs` to pass arguments to `ls`
find /sbin -perm +700 | xargs ls -l

Example 2
Show system command-line length limitations:

$ xargs --show-limits

Example 3
Restore shell special characters like backticks:

$ echo '`0123`4 56789' | xargs -t echo

Example 4
Set the delimiter for reading input as a comma:

$ echo 01234 , 56789 | xargs -E ","

Example 5
Solve “argument list too long” errors when working with many files:

# Add a suffix to all files in the current directory
ls | xargs -t -i mv {} {}.bak

Example 6
Set how many lines to pass as arguments at a time:

$ echo -e "01234\n56789\n01234" | xargs -t -L 2 echo

Example 7
Merge multi-line input into a single line:

$ cat test.txt | xargs

Example 8
Kill processes in combination with ps, grep, awk, and kill:

$ ps -ef | grep spp | awk '{printf "%s ",$2}' | xargs kill -9