Finding the right information on time can be a challenge when working with large files in the terminal. Especially log files, which contain lots of details about the activity of a system.
Luckily, there are also lots of file searching tools for cutting through unnecessary info and getting only the data that interests you.
We’ll make a quick dive into some of the most basic file reading tools that we use on a daily basis when handling log files.
Note: the commands have a fragment from their man description added besides them. If you don’t know what man is, try the man command in the terminal to find out.
The most basic command for reading a file from the terminal. It reads files sequentially, writing them to the standard output.
As most built-in Linux commands, it comes with several flags that can enhance its functionality. However, it’s frequently used without any of them to display the entire content of a file. Here’s an example:
It only displays the contents of sample.txt to the standard output. By using it in combination with the pipe “|” operator, it can deliver this output to other terminal commands instead of the standard output.
“The pipe(8) daemon processes requests from the Postfix queue manager to deliver messages to external commands.”
The pipe operator is used between two commands. As an operator, its function is to pass the output of the first command to the input of the second command. Here’s an example:
The pipe operator already comes in handy and lets us filter the sample1.txt. In the first example, it’s used in combination with head to display the first line from sample1.txt. The second example it’s self-explanatory, but you should try it out anyway.
“The tail utility displays the contents of file or, by default, its standard input, to the standard output.”
As you saw in the example above, tail can be used to display lines of text from a file, starting from the end of the file upwards.
In combination with the -f flag, tail can constantly display in the terminal the lines of text that are appended to a file. This is the normal behavior of a log file; information is added constantly as new events happen in the system. Here’s an example of tail -f :
So tail displayed the last 10 lines from sample.log and now waits to display the next lines of text that will be appended.
“The grep utility searches any given input files, selecting lines that match one or more patterns.”
Grep is a powerful and easy to use tool for finding the right information inside a given context. The context can be one or more files, and the correct information can be one or more patterns that interest you. Here’s an example:
In this example, I used grep to search for all the contexts in sample.log that contain the ‘EVENT‘ pattern. The response was very fast in this case, where the file had about 360 lines of text.
You can check how many lines a file has by using cat -n. See how fast a commands’ response is by typing time in front of it and then executing it.
You can also check this simple explanation of the log entry levels (e.g., EVENT) and what they mean here.
The most useful command combination when working with log files is tail and grep.
Let’s say you monitor the actions related to a specific ID. The logs can have dense information, so you only want those actions displayed and the rest ignored. The following command will do precisely that: tail -1f file_name | grep ’42’
It will display the last line from file_name if ’42’ is present and any other lines (when appended) where ’42’ is found. Tail will run until you interrupt its execution.
To search for a pattern in multiple files, you can simply type: grep ’42’ file1 file2
To search a file for multiple patterns, type: grep -e ’42’ -e ’43’ file1
This option is most useful when multiple -e options are used to specify multiple patterns.
-A,-B, -C flags
Other useful flags for grep are -A and -B. They specify a context to be printed (B) before or (A) after the line that contains the pattern is displayed. You can also use the -C flag, which results in a combination of both -A and -B flags.
Many command combinations can be done only with the commands mentioned above, especially with the help of pipe.
Feel free to try them and see how it goes.
Extra: If you’re on MAC OS, try the following command – say hello world
More about file searching in the terminal and our services here: