Python Logging Examples That Log to a File and Rotate It

Python's logging module comes down to three decisions: which level to keep, where the records go, and what each line looks like. The smallest working setup is one basicConfig() call in your entry point plus logging.getLogger(__name__) at the top of every module. Everything else here is a variation on that: writing to a file, rotating the file, giving each module its own logger, and moving the setup into a dictionary or config file.

All examples use the standard library only and each one is complete, so copy them as they are. The Logging HOWTO and the Logging Cookbook are the official references when you want the long version.

The smallest Python logging example#

This prints every level to stdout with a timestamp, the level name and the logger name. It also shows how to log variables and a traceback.

import logging
import sys

logging.basicConfig(
    stream=sys.stdout,
    level=logging.DEBUG,
    format="%(asctime)s %(levelname)-8s %(name)s: %(message)s",
)
logger = logging.getLogger(__name__)

user = "indra"
attempts = 3
logger.debug("Loaded settings for %s", user)
logger.info("Login ok for %s after %d attempt(s)", user, attempts)
logger.warning("Only %d MB free on %s", 512, "/var")

try:
    1 / 0
except ZeroDivisionError:
    logger.error("Division failed for %s", user, exc_info=True)

Three things to notice. The message uses %s and %d placeholders with the values passed as extra positional arguments, not an f-string. Logging merges them only when a record is actually going to be written, so a filtered debug() call costs almost nothing. The exc_info=True keyword attaches the current traceback to the record. And the logger name is __name__, which is __main__ when you run the file directly and the dotted module path when it's imported.

Without any configuration at all, logging prints WARNING and above to stderr in the default format severity:logger name:message. That's why a bare logging.info("...") in a fresh script shows nothing while logging.warning("...") does.

If you prefer brace formatting for the output layout, pass style="{" to the Formatter. The message itself still uses % placeholders; the two are independent.

formatter = logging.Formatter("{asctime} {name} {levelname:8s} {message}", style="{")

Logging levels and which messages get through#

The level names map to numbers, and a record passes when its level is at least the threshold it's checked against.

Two thresholds are in play. The logger's level decides whether a record is created at all. Each handler can then have its own level, which filters again before writing. A common arrangement is INFO on the console handler and DEBUG on the file handler, with the logger itself at DEBUG so both receive what they need.

To set the level from a command-line flag, convert the name to the constant:

import argparse
import logging

parser = argparse.ArgumentParser()
parser.add_argument("--log", default="INFO")
args = parser.parse_args()

level = getattr(logging, args.log.upper(), None)
if not isinstance(level, int):
    raise ValueError(f"Invalid log level: {args.log}")
logging.basicConfig(level=level)

Custom levels are possible but the HOWTO advises against them, and it calls them a very bad idea inside a library. Stick with the six built in.

Python logging to a file#

The quickest file logger is basicConfig() with filename. The encoding argument has been accepted since Python 3.9, and you want it set so non-ASCII text doesn't depend on the platform default.

import logging

logging.basicConfig(
    filename="app.log",
    encoding="utf-8",
    level=logging.INFO,
    format="%(asctime)s %(levelname)s %(name)s: %(message)s",
)
logger = logging.getLogger(__name__)
logger.info("Service started")
logger.error("Non-ASCII is fine: Øresund, Malmö")

Run it in a fresh interpreter. The file appears in the current working directory, which is the first place to look when you think logging "isn't writing anything".

Most real programs want both console and file, with different thresholds. That means building the handlers yourself:

import logging
import sys

fmt = logging.Formatter("%(asctime)s %(levelname)s %(name)s: %(message)s")

console = logging.StreamHandler(sys.stderr)
console.setLevel(logging.INFO)
console.setFormatter(fmt)

file_handler = logging.FileHandler("app.log", encoding="utf-8")
file_handler.setLevel(logging.DEBUG)
file_handler.setFormatter(fmt)

root = logging.getLogger()
root.setLevel(logging.DEBUG)
root.addHandler(console)
root.addHandler(file_handler)

Do this once, in the script that starts the program. Modules never create handlers; they only call getLogger(__name__) and log.

File rotation with RotatingFileHandler#

A plain FileHandler grows forever. The rotating handlers in logging.handlers cap that. RotatingFileHandler rolls the file over when it reaches a size and keeps a fixed number of old copies, renamed with .1, .2 and so on.

import logging
from logging.handlers import RotatingFileHandler

handler = RotatingFileHandler(
    "app.log",
    maxBytes=5_000_000,
    backupCount=5,
    encoding="utf-8",
)
handler.setFormatter(logging.Formatter("%(asctime)s %(levelname)s %(name)s: %(message)s"))

root = logging.getLogger()
root.setLevel(logging.INFO)
root.addHandler(handler)

TimedRotatingFileHandler rotates on a schedule instead. Rolling at midnight and keeping two weeks of files looks like this:

from logging.handlers import TimedRotatingFileHandler

handler = TimedRotatingFileHandler(
    "app.log", when="midnight", backupCount=14, encoding="utf-8"
)

If an external tool such as logrotate manages the files, use WatchedFileHandler instead. It notices when the file it's writing to has been moved or removed and reopens it. The full list of handlers and their arguments is in the logging.handlers reference.

Size-based rotation suits services with unpredictable volume, and time-based rotation suits cases where people will be grepping "yesterday's log".

Python logging example with multiple modules#

Logger names form a dot-separated tree. A logger named app is the parent of app.db and app.http, and the root logger sits above everything. Records travel up that tree: a record logged on app.db is handled by app.db's handlers, then app's, then root's, unless some logger on the way has propagate set to false.

That's why the convention is one line per module:

logger = logging.getLogger(__name__)

In a package laid out as app/__init__.py, app/db.py and app/http.py, those loggers are automatically named app.db and app.http. The entry script configures root once, and every module's output flows to it with its origin in the %(name)s field. The example below simulates that layout in one file so it can run on its own.

import logging
import sys

handler = logging.StreamHandler(sys.stdout)
handler.setFormatter(logging.Formatter("{levelname:8s} {name}: {message}", style="{"))
root = logging.getLogger()
root.addHandler(handler)
root.setLevel(logging.INFO)

# In a real project these are logging.getLogger(__name__) in app/db.py and app/http.py
db = logging.getLogger("app.db")
http = logging.getLogger("app.http")

logging.getLogger("app.db").setLevel(logging.DEBUG)  # turn one module up

db.debug("SELECT took %d ms", 12)        # shown: app.db is at DEBUG
http.debug("GET /health")                # hidden: app.http inherits INFO from root
http.info("GET /orders -> %d", 200)

logging.getLogger("app").setLevel(logging.WARNING)  # quiet the whole package
http.info("dropped: app.http now inherits WARNING from app")
db.debug("still shown: app.db keeps its own DEBUG level")
db.warning("slow query on orders")

This is the mechanism behind "show me debug output from my database module only". A module like the one in the SQLite in Python examples gets its own logger for free, and you dial it up or down from the entry point without touching its code. The same applies to coroutines; logging from async code uses the identical calls.

If you're writing a library rather than an application, the rules flip. Don't configure anything, don't log to the root logger, and don't add handlers other than NullHandler. The HOWTO recommends exactly one line on the library's top-level logger so nothing prints when the host application hasn't set up logging:

import logging
logging.getLogger("foo").addHandler(logging.NullHandler())

Python logging config with dictConfig and fileConfig#

Once you have more than a couple of handlers, code-based setup gets noisy. dictConfig() takes a dictionary describing formatters, handlers, loggers and root, and it's the method the HOWTO recommends for new applications.

import logging
import logging.config

CONFIG = {
    "version": 1,
    "disable_existing_loggers": False,
    "formatters": {
        "simple": {"format": "%(asctime)s - %(name)s - %(levelname)s - %(message)s"},
    },
    "handlers": {
        "console": {
            "class": "logging.StreamHandler",
            "level": "DEBUG",
            "formatter": "simple",
            "stream": "ext://sys.stdout",
        },
    },
    "loggers": {
        "app.db": {"level": "DEBUG"},
    },
    "root": {"level": "INFO", "handlers": ["console"]},
}

logging.config.dictConfig(CONFIG)
logging.getLogger("app.db").debug("debug from app.db is on")
logging.getLogger("app.http").debug("debug from app.http is off")
logging.getLogger("app.http").info("info reaches the root console handler")

The ext://sys.stdout value tells the config system to resolve an external Python object. Class names can be relative to the logging package, such as logging.StreamHandler, or any importable dotted path.

A rotating file handler fits the same shape. Add it to handlers and list it under root:

"file": {
    "class": "logging.handlers.RotatingFileHandler",
    "level": "DEBUG",
    "formatter": "simple",
    "filename": "app.log",
    "maxBytes": 5000000,
    "backupCount": 5,
    "encoding": "utf-8",
},

Because the input is just a dictionary, you can keep it in a file. JSON needs nothing beyond the standard library:

import json
import logging.config

with open("logging.json", encoding="utf-8") as f:
    logging.config.dictConfig(json.load(f))

YAML works the same way once you have a YAML parser, which the standard library doesn't ship. The HOWTO shows the YAML form of the config above with version: 1 at the top, stream: ext://sys.stdout on the handler and propagate: no on the named logger.

The older fileConfig() reads an INI-style file. It's still fully supported and the HOWTO's sample is short enough to keep around:

[loggers]
keys=root,simpleExample

[handlers]
keys=consoleHandler

[formatters]
keys=simpleFormatter

[logger_root]
level=DEBUG
handlers=consoleHandler

[logger_simpleExample]
level=DEBUG
handlers=consoleHandler
qualname=simpleExample
propagate=0

[handler_consoleHandler]
class=StreamHandler
level=DEBUG
formatter=simpleFormatter
args=(sys.stdout,)

[formatter_simpleFormatter]
format=%(asctime)s - %(name)s - %(levelname)s - %(message)s

Load it with logging.config.fileConfig("logging.conf", disable_existing_loggers=False). That last argument matters, and it's the subject of the next section. The logging.config reference documents every key both functions accept.

Python logging to file not working and other gotchas#

These cover most of the "nothing shows up" and "everything shows up twice" reports.

• basicConfig ran too late. If anything logged a WARNING or higher before your basicConfig(filename=...) call, the module-level functions already attached a stderr handler to root, and basicConfig() does nothing once root has handlers. Imports that log at import time are the usual cause. Configure logging first thing in your entry point, or pass force=True to replace whatever is there.
• Level set in the wrong place. A FileHandler at DEBUG attached to a root logger still at the default WARNING writes only warnings and above. Set the logger level too.
• Module loggers silently disabled. Both fileConfig() and dictConfig() default disable_existing_loggers to true. Any logger created before the call, which includes every getLogger(__name__) in an already imported module, is switched off unless it or an ancestor is named in the config. Set it to false unless you have a reason not to.
• Duplicate lines. A handler on app.db plus a handler on root means each record is written twice, because records propagate upward. Either keep handlers on root only, or set propagate to false on the child. Adding a handler inside a function that runs more than once produces the same symptom.
• f-strings in log calls. logger.debug(f"user={user}") formats the string every time, even when DEBUG is off. Use logger.debug("user=%s", user) and let logging defer the work.
• Mixing up output style and message style. style="{" on a Formatter changes the layout string only. Messages still take % placeholders; the cookbook explains why that can't change without breaking existing code.
• Handlers inside a library. If your package adds a StreamHandler or FileHandler on import, the application using it can't control the output. Add a NullHandler and nothing else.
• Relative file paths. filename="app.log" is relative to the process's working directory, not to the script. Under a service manager or cron that directory is often not what you expect, so pass an absolute path in production. The same care applies when you log output from child processes, as in the subprocess timeout examples.

Quick reference#

A checklist for a new project:

1. Put logger = logging.getLogger(__name__) at the top of every module. Never configure logging inside a module.
2. Configure once, in the entry point, before any other import can log. Use basicConfig() for scripts and dictConfig() for anything bigger.
3. Set disable_existing_loggers to False in every dictConfig() or fileConfig() call.
4. Pass values as arguments, logger.info("x=%s", x), not as f-strings.
5. Use exc_info=True in an except block to capture the traceback.
6. Put handlers on root only. Tune per-module verbosity with setLevel() on the named loggers, not with extra handlers.
7. For files, set encoding="utf-8", use an absolute path in production, and choose RotatingFileHandler or TimedRotatingFileHandler so the file can't grow unbounded.
8. For libraries, add a NullHandler to the top-level logger and stop there.

The format fields you'll use most are %(asctime)s, %(levelname)s, %(name)s and %(message)s. The Formatter Objects section of the logging reference lists the rest.

Quick answers