How to stop INFO messages displaying on spark console?
Thanks @AkhlD and @Sachin Janani for suggesting changes in .conf
file.
Following code solved my issue:
1) Added import org.apache.log4j.{Level, Logger}
in import section
2) Added following line after creation of spark context object i.e. after val sc = new SparkContext(conf)
:
val rootLogger = Logger.getRootLogger()
rootLogger.setLevel(Level.ERROR)
How to turn off INFO logging in Spark?
Just execute this command in the spark directory:
cp conf/log4j.properties.template conf/log4j.properties
Edit log4j.properties:
# Set everything to be logged to the console
log4j.rootCategory=INFO, console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n
# Settings to quiet third party logs that are too verbose
log4j.logger.org.eclipse.jetty=WARN
log4j.logger.org.eclipse.jetty.util.component.AbstractLifeCycle=ERROR
log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO
log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO
Replace at the first line:
log4j.rootCategory=INFO, console
by:
log4j.rootCategory=WARN, console
Save and restart your shell. It works for me for Spark 1.1.0 and Spark 1.5.1 on OS X.
How to disable all logging info to the spark console from .net application
You need to rename/copy the file log4j.properties.template
to log4j.properties
.
If you want to see fewer logs, you can set the logging level to ERROR
instead of WARN
.
You can also put these lines to avoid logging the other errors you got.
log4j.logger.org.apache.spark.util.ShutdownHookManager=OFF
log4j.logger.org.apache.spark.SparkEnv=ERROR
Source
How to turn off INFO from logs in PySpark with no changes to log4j.properties?
I used sc.setLogLevel("ERROR")
because I didn't have write access to our cluster's log4j.properties file. From the docs:
Control our logLevel. This overrides any user-defined log settings. Valid log levels include: ALL, DEBUG, ERROR, FATAL, INFO, OFF, TRACE, WARN
Turn off pyspark logging through python sript
To remove (or modify) logging from a python script:
conf = SparkConf()
conf.set('spark.logConf', 'true') # necessary in order to be able to change log level
... # other stuff and configuration
# create the session
spark = SparkSession.builder\
.config(conf=conf) \
.appName(app_name) \
.getOrCreate()
# set the log level to one of ALL, DEBUG, ERROR, FATAL, INFO, OFF, TRACE, WARN
spark.sparkContext.setLogLevel("OFF")
docs configuration
docs setLogLevel
Hope this helps, good luck!
Edit: For earlier versions, e.g. 1.6, you can try something like the following, taken from here
logger = sc._jvm.org.apache.log4j
logger.LogManager.getLogger("org"). setLevel(logger.Level.OFF)
# or
logger.LogManager.getRootLogger().setLevel(logger.Level.OFF)
I haven't tested it unfortunately, please, let me know if it works.
How can set the default spark logging level?
http://spark.apache.org/docs/latest/configuration.html#configuring-logging
Configuring Logging
Spark uses log4j for logging. You can configure it by adding a log4j.properties file in the conf directory. One way to start is to copy the existing log4j.properties.template located there.
The following blog about "How to log in spark" https://www.mapr.com/blog/how-log-apache-spark suggest a way to configure log4j, and provide suggestion which includes directing INFO level logs into a file.
Disable INFO messages in Spark for an specific application
You must do this:
import org.apache.log4j.Logger
import org.apache.log4j.Level
Logger.getLogger("org").setLevel(Level.OFF)
Logger.getLogger("akka").setLevel(Level.OFF)
before creating a SparkContext.
Do something like this:
Logger.getLogger("org").setLevel(Level.OFF)
Logger.getLogger("akka").setLevel(Level.OFF)
val sc = new SparkContext(conf)
Also, you can use getRootLogger
method:
Logger.getRootLogger().setLevel(Level.ERROR)
Related Topics
Why Can a Python Dict Have Multiple Keys with the Same Hash
Python - Typeerror: 'Int' Object Is Not Iterable
Matplotlib Log Scale Tick Label Number Formatting
How to Pick "X" Number of Unique Numbers from a List in Python
How to Check If Directory Exists in Python
How to Remove Gaps Between Subplots in Matplotlib
Parsing HTML in Python - Lxml or Beautifulsoup? Which of These Is Better for What Kinds of Purposes
What Is the Pythonic Way to Unpack Tuples
How to Overwrite/Print Over the Current Line in Windows Command Line
How to Get a Complete List of Object's Methods and Attributes
What Is the Syntax to Insert One List into Another List in Python
Wrapping a C Library in Python: C, Cython or Ctypes
How to Efficiently Calculate a Running Standard Deviation
How to Force Python to Be 32-Bit on Snow Leopard and Other 32-Bit/64-Bit Questions