Logging, Streamhandler and Standard Streams

Logging, StreamHandler and standard streams

The following script, log1.py:

import logging, sys

class SingleLevelFilter(logging.Filter):
    def __init__(self, passlevel, reject):
        self.passlevel = passlevel
        self.reject = reject

    def filter(self, record):
        if self.reject:
            return (record.levelno != self.passlevel)
        else:
            return (record.levelno == self.passlevel)

h1 = logging.StreamHandler(sys.stdout)
f1 = SingleLevelFilter(logging.INFO, False)
h1.addFilter(f1)
rootLogger = logging.getLogger()
rootLogger.addHandler(h1)
h2 = logging.StreamHandler(sys.stderr)
f2 = SingleLevelFilter(logging.INFO, True)
h2.addFilter(f2)
rootLogger.addHandler(h2)
logger = logging.getLogger("my.logger")
logger.setLevel(logging.DEBUG)
logger.debug("A DEBUG message")
logger.info("An INFO message")
logger.warning("A WARNING message")
logger.error("An ERROR message")
logger.critical("A CRITICAL message")

when run, produces the following results.

C:\temp>log1.py
A DEBUG message
An INFO message
A WARNING message
An ERROR message
A CRITICAL message

As you'd expect, since on a terminal sys.stdout and sys.stderr are the same. Now, let's redirect stdout to a file, tmp:

C:\temp>log1.py >tmp
A DEBUG message
A WARNING message
An ERROR message
A CRITICAL message

So the INFO message has not been printed to the terminal - but the messages directed to sys.stderr have been printed. Let's look at what's in tmp:

C:\temp>type tmp
An INFO message

So that approach appears to do what you want.

Python Logger : StreamHandler not controlling my terminal stdout?

It seems it is related to the fact that I have multiple modules - each module can have a logger to console, but the main script must remain in control and have its own logger set (which I did not have).
This page solved my problem: https://docs.python.org/3/howto/logging-cookbook.html#logging-to-multiple-destinations

Making Python loggers output all messages to stdout in addition to log file

All logging output is handled by the handlers; just add a logging.StreamHandler() to the root logger.

Here's an example configuring a stream handler (using stdout instead of the default stderr) and adding it to the root logger:

import logging
import sys

root = logging.getLogger()
root.setLevel(logging.DEBUG)

handler = logging.StreamHandler(sys.stdout)
handler.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
handler.setFormatter(formatter)
root.addHandler(handler)

logger configuration to log to file and print to stdout

Just get a handle to the root logger and add the StreamHandler. The StreamHandler writes to stderr. Not sure if you really need stdout over stderr, but this is what I use when I setup the Python logger and I also add the FileHandler as well. Then all my logs go to both places (which is what it sounds like you want).

import logging
logging.getLogger().addHandler(logging.StreamHandler())

If you want to output to stdout instead of stderr, you just need to specify it to the StreamHandler constructor.

import sys
# ...
logging.getLogger().addHandler(logging.StreamHandler(sys.stdout))

You could also add a Formatter to it so all your log lines have a common header.

ie:

import logging
logFormatter = logging.Formatter("%(asctime)s [%(threadName)-12.12s] [%(levelname)-5.5s]  %(message)s")
rootLogger = logging.getLogger()

fileHandler = logging.FileHandler("{0}/{1}.log".format(logPath, fileName))
fileHandler.setFormatter(logFormatter)
rootLogger.addHandler(fileHandler)

consoleHandler = logging.StreamHandler()
consoleHandler.setFormatter(logFormatter)
rootLogger.addHandler(consoleHandler)

Prints to the format of:

2012-12-05 16:58:26,618 [MainThread  ] [INFO ]  my message

Does Python logging write to stdout or stderr by default?

If no filename argument is passed to logging.basicconfig it will configure a StreamHandler. If a stream argument is passed to logging.basicconfig it will pass this on to StreamHandler otherwise StreamHandler defaults to using sys.stderr as can be seen from the StreamHandler docs

class logging.StreamHandler(stream=None)

Returns a new instance of the StreamHandler class. If stream is specified, the instance will use it for logging output; otherwise, sys.stderr will be used.

and the source code:

class StreamHandler(Handler):
    """
    A handler class which writes logging records, appropriately formatted,
    to a stream. Note that this class does not close the stream, as
    sys.stdout or sys.stderr may be used.
    """

    def __init__(self, stream=None):
        """
        Initialize the handler.
        If stream is not specified, sys.stderr is used.
        """
        Handler.__init__(self)
        if stream is None:
            stream = sys.stderr
        self.stream = stream

python logs to both stdout and stderr

You can use logging.StreamHandler()

like that:

logging.basicConfig(handlers=[logging.StreamHandler()], level=logging.INFO, format='%(asctime)s - %(name)s - %(levelname)s - %(message)s')
log = logging.getLogger()

If you also want to that certain level would go stderr (only Error or Error and Warning) and other stdout. you can do something like suggested here by @crosswired:

logger = logging.getLogger("__name__")
logger.setLevel(logging.DEBUG)
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
formatter = logging.Formatter(format)

h1 = logging.StreamHandler(sys.stdout)
h1.setLevel(logging.DEBUG)
h1.addFilter(lambda record: record.levelno <= logging.INFO)
h1.setFormatter(formatter)

h2 = logging.StreamHandler()
h2.setLevel(logging.WARNING)
h2.setFormatter(formatter)
logger.addHandler(h1)
logger.addHandler(h2)
logger.info("info")
logger.error("error")

Python logging split between stdout and stderr

Logging messages using the stream handler appear red in console

The PyDev console highlights messages to stderr in red by default. Python's logging.DEBUG will send messages to stderr. If you wish to change this behavior, see this post: Logging, StreamHandler and standard streams

To change the colors in PyDev, see here: http://pydev.org/manual_adv_interactive_console.html

How to specify levels in python logging module?

You could create your own Handler:

logger = logging.getLogger('DBMQ')
logger.setLevel(logging.DEBUG)

class MyStreamHandler(logging.StreamHandler):
    def emit(self, record):
        if record.levelno == self.level:
            super().emit(record)

stream_handler = MyStreamHandler()
stream_handler.setLevel(logging.INFO)

logger.addHandler(stream_handler)

file_handler = logging.FileHandler('./data/file.log')
file_handler.setLevel(logging.WARNING)

logger.addHandler(file_handler)

---

Related Topics