Log Level and Logrotate

Log

An application log is a file that contains information about events that have occurred within a software application. These events are logged out by the application and written to the file. It contains errors, informational events and warnings. The format and content of an application log are determined by the developer of the software program.

1. Log Level

Most common log levels are as follows:

Debug: The DEBUG log level should be used for information that may be needed for diagnosing issues and troubleshooting or when running application in the test environment for the purpose of making sure everything is running correctly
Info: The standard log level indicates that something happened, the application entered a certain state, etc. For example, a controller of your authorization API may include an INFO log level with information on which user requested authorization if the authorization was successful or not. The information logged using the INFO log level should be purely informative and not looking into them on a regular basis shouldn’t result in missing any important information.
Warn: This log level indicates that something unexpected happened in the application, a problem, or a situation that might disturb one of the processes. But that doesn’t mean that the application failed. The WARN level should be used in situations that are unexpected, but the code can continue the work. For example, a parsing error occurred that resulted in a certain document not being processed.
Error: This log level should be used when the application hits an issue preventing one or more functionalities from properly functioning. Eg: the ERROR log level can be used when one of the payment systems is not available, but there is still the option to check out the basket in the e-commerce application or when your social media logging option is not working for some reason.
Fatel: This log level tells that the application encountered an event or entered a state in which one of the crucial business functionality is no longer working. A FATAL log level may be used when the application is not able to connect to a crucial data store like a database or all the payment systems are not available and users can’t checkout their baskets in your e-commerce.

2. Log rotate

Many applications and system services will store log files. These log files give insight into how their system is performing, and are invaluable when troubleshooting issues. However, log files can get unwieldy very quickly. For example, if your web server software logs every visit to your website, and you get thousands of viewers per day, there will be way too much information to feasibly squeeze into one text file.

Logrotate will periodically take the current log files, rename them, optionally compress them, and generate a fresh file to which an application can continue sending its logs. The logrotate command is invoked automatically from cron, and most services have their own log rotation configuration that is implemented when they’re installed. This configuration tells logrotate what it should do with the old log files. For example, how many of them should it keep around before deleting, should it compress the files, etc.

Eg: default config for apt history.log

/var/log/apt/history.log {
  rotate 12
  monthly
  compress
  missingok
  notifempty
}

Here, this config file will run monthly, create a maximum of 12 archives, all compressed logs and will skip rotation if the log file is empty. Here are some selected logrotate configuration keywords.

Name	Descriptions
daily	Log files are rotated every day.
weekly	Log files are rotated if the current weekday is less than the weekday of the last rotation or if more than a week has passed since the last rotation. This is normally the same as rotating logs on the first day of the week, but if logrotate is not being run every night a log rotation will happen at the first valid opportunity.
monthly	Log files are rotated the first time logrotate is run in a month (this is normally on the first day of the month).
notifempty	Do not rotate the log if it is empty (this overrides the ifempty option).
nocompress	Old versions of log files are not compressed.
compress	Old versions of log files are compressed with gzip by default.
missingok	If the log file is missing, go on to the next one without issuing an error message.
delaycompress	Postpone compression of the previous log file to the next rotation cycle. This only has an effect when used in combination with compress.
copytruncate	Truncate the original log file to zero size in place after creating a copy, instead of moving the old log file and optionally creating a new one. It can be used when some program cannot be told to close its logfile and thus might continue writing (Truncate the original log file to zero size in place after creating a copy, instead of moving the old log file and optionally creating a new one.

Let's take an example for 7 logs, we can make the following configurations.

/opt/var/app/logfile.log {
  rotate 7
  daily
  missingok
  delaycompress
  compress
  notifempty
  copytruncate
}

Output:

Log(Ruby)

1. Log levels(Ruby)

In Ruby, there are five common log levels: FATAL, ERROR, WARN, INFO, and DEBUG. The meaning of each of these log levels is provided as follows:

Debug: The debug level should be used for detailed messages that assist in debugging. Typically, these logs are used by developers or system admins who need as much detail as possible about what might be going on in the system. In Ruby, this is the most verbose level.
Info: In the case of info-level logs, these provide information regarding normal application processing. This could be log information about services starting and stopping, or application metrics.
Warn: Warnings indicate something’s wrong. It might not be an error since the application may have recovered. Or, the warning may inform us of a rare business scenario we want to track.
Error: When we see errors, we know something failed. Perhaps a transaction was unable to update the database. Or maybe a dependency couldn’t be reached. Overall the application still functions, but things aren’t working properly.
Fatal: Fatal logs should only be used when something happens that causes the application to crash. This type of error may need support or intervention from the development team. It shouldn’t be used for general business errors.
Unknown: The unknown logging level will catch anything that doesn’t have a known level. It’s a catch-all for anything that might sneak through.

2. Limit Log Size (Ruby)

Logs can become really big files unless you keep track of their size. Log files will cover up all the disk space and there will be many problems occurring due to that. So it is important to limit log size. Reference here: Ruby guides

Problems of growing logs

Large files are harder to manipulate.
File system runs out of space.

Example:

MEGABYTE = 1024 ** 2

ONE_HUNDRED_MEGABYTES = MEGABYTE * 100

logger = Logger.new("my_log.txt", 1, ONE_HUNDRED_MEGABYTES)

The 1st parameter is where to save log messages. The 2nd parameter is how many log files you want to keep (log rotation), and the last parameter is the maximum file size.

Logrotate for docker container

As there are log files for applications and systems, there is a log file for every docker container. We can rotate docker logs the same as we manage application and system logs.

Default logging driver

We can configure different logging drivers for containers. By default, the stdout and stderr of the container are written in a JSON file located in /var/lib/docker/containers/[container-id]/[container-id]-json.log

1. Docker

Log rotate using docker run command:

We can specify the logging driver and options in the docker run command.

For example:

$ docker run \
    --log-driver json-file \
    --log-opt max-size=10M \
    --log-opt max-file=5 \
    alpine echo hello world

Here,

--log-driver json-file is a json-file which is default.

--log-opt max-size=10M is to rotate log if file size is more than 10M

--log-opt max-file=5 is to retain a total of 5 backup log files.

2. Docker compose

Log rotate using docker compose:

The logging driver and options can also be configured using docker compose.

For example:

version: '3.2'
services:
  nginx:
    image: 'nginx:latest'
    ports:
      - '80:80'
    logging:
      driver: "json-file"
      options:
        max-size: "2K"
        max-file: "4"

Here,

max-size option is used to limit the log size.

max-file option is used to limit the number of logs.

Output

This is the output of the above docker-compose file. Here, a new log file is created if size exceeds ‘2k’. The latest log file is without number i.e. container-id-json.log and the old log file is the greatest number i.e. container-id-json.log.3. So every time a new log file is created, the oldest log gets deleted.

Contributors: Ujan Man Shakya ↩