Creating GCP-compatible structured log messages in Python

The Google Cloud Platform supports structured logging. All it takes is writing JSON objects to stdout or stderr. With structlog, it’s straightforward to set up a processor chain that produces compatible log messages.

There are two special fields that are essential for filtering and searching logs:

• The severity field takes one of nine different values from DEBUG to EMERGENCY.

• The message field is shown in the log entry line in the GCP Logs Explorer.

Thus, we first map the ten structlog methods to the nine LogSeverity values:

from structlog import EventDict

STRUCTLOG_TO_GCP = {
    "critical": "CRITICAL",
    "debug": "DEBUG",
    "err": "ERROR",
    "error": "ERROR",
    "failure": "ALERT",
    "fatal": "EMERGENCY",
    "info": "INFO",
    "log": "DEFAULT",
    "msg": "DEFAULT",
    "warning": "WARNING",
    # Unused GCP severity: "NOTICE"
}


def add_gcp_severity(
    _, method_name: str, event_dict: EventDict
) -> EventDict:
    event_dict["severity"] = STRUCTLOG_TO_GCP.get(method_name, 
                                                  "DEFAULT")
    return event_dict

Then, we use two built-in structlog processors to rename the event field to message and output everything as JSON:

import structlog

structlog.configure(
    processors=[
        add_gcp_severity,
        structlog.processors.EventRenamer("message"),
        structlog.processors.JSONRenderer(),
    ]
)

This produces messages in the right format:

log = structlog.get_logger()
log.info("Hello World!", key="value")
# {"key": "value", "severity": "INFO", "message": "Hello World!"}