adhocshare

Knowledge is Power

Configure log files on HDP platform

by · Published · Updated

 

  1. Kafka
  2. Storm
  3. Ranger
  4. HDFS
  5. Zookeeper
  6. Oozie
  7. Knox
  8. Hive & Hive metastore

 

1. Kafka

Kafka currently uses org.apache.log4j.DailyRollingFileAppender, which doesn’t allow us to specify the max backup index, max file size. And by default, rolls every hour creating 24 files a day.

Appender: org.apache.log4j. RollingFileAppender
Roll criteria: Every 100MB.
Retention: 9 backup logs.

Steps:

  1. Login to Ambari
  2. Goto the config tab of the Kafka broker.
  3. Expand Advanced kafka-log4j
  4. Change the appender from org.apache.log4j.DailyRollingFileAppender to org.apache.log4j.RollingFileAppender.
  5. Comment out the appender’s date pattern: 
    #log4j.appender.kafkaAppender.DatePattern='.'yyyy-MM-dd-HH
#log4j.appender.kafkaAppender=org.apache.log4j.DailyRollingFileAppender
  6. Add the following lines for each appender (change the appender name): 
    log4j.appender.kafkaAppender=org.apache.log4j.RollingFileAppender
log4j.appender.kafkaAppender.MaxFileSize=100MB
log4j.appender.kafkaAppender.MaxBackupIndex=9
  7. Repeat the steps for all the appenders in the section.
  8. Rolling restart Kafka brokers for the changes to take effect.

Issues/Further Investigation:

The kafka.out doesn’t roll and grows huge. There is a open ticket with HortonWorks on this issue.

 

2. Storm

We need to set the log viewer settings to update the clean up frequency

logviewer.cleanup.age.mins – Value in mins for the worker logs to age for cleaning.

logviewer.cleanup.interval.secs – Value in secs for frequency for the logviewer cleanup process to trigger.

Steps:

  1. Login to Ambari
  2. Goto the config tab of the Kafka broker.
  3. Expand Custom storm-site
  4. Add the following two properties: 
    logviewer.cleanup.age.mins : XXX (Value in mins for the worker logs to age for cleaning)
logviewer.cleanup.interval.secs : XXX (Value in secs for frequency for the logviewer cleanup process to trigger)
  5. Restart Storm components. (esp.. Supervisors – rolling restart)

 

3. Ranger

Ranger logs are divided into 2 parts: admin and usersync

admin – Ranger admin logs – access logs, xa log and xa sql log

usersync – User sync logs

SSH to the server where Ranger Admin is installed and make the following changes:

  1. Admin logs : /usr/hdp/current/ranger-admin/ews/webapp/WEB-INF/log4j.xml
    Change that to RollingFileAppender, see example below:

    <appender name="xa_log_appender">
<param name="file" value="${catalina.base}/logs/xa_portal.log" />
<param name="datePattern" value="'.'yyyy-MM-dd" />
<param name="append" value="true" />
<layout>
<param name="ConversionPattern" value="%d [%t] %-5p %C{6} (%F:%L) - %m%n" />
</layout>
</appender>

    Change the above to:

    <appender name="xa_log_appender">
<param name="file" value="${catalina.base}/logs/xa_portal.log" />
<param name="maxFileSize" value="100MB" />
<param name="maxBackupIndex" value="9" />
<layout>
<param name="ConversionPattern" value="%d [%t] %-5p %C{6} (%F:%L) - %m%n" />
</layout>
</appender>

    Repeat the process for sql_appender as well.

  2. UserSync logs (usr/hdp/current/ranger-usersync/conf/log4j.xml):
    Currently it uses DailyRollingFileAppender which doesn’t clean up the old files.

    xm<appender name="logFile">
<param name="file" value="${logdir}/usersync.log" />
<param name="DatePattern" value="'.'yyyy-MM-dd" />
<layout>
<param name="ConversionPattern" value="%d{dd MMM yyyy HH:mm:ss} %5p %c{1} [%t] - %m%n"/>
</layout>
</appender>

    Change the above to:

    <appender name="logFile">
<param name="file" value="${logdir}/usersync.log" />
<param name="maxFileSize" value="100MB" />
<param name="maxBackupIndex" value="9" />
<layout>
<param name="ConversionPattern" value="%d{dd MMM yyyy HH:mm:ss} %5p %c{1} [%t] - %m%n"/>
</layout>
</appender>
  3. Restart Ranger.

 

4. HDFS

Audit Logs:

This log is currently configured with DailyRollingFileAppender, have to change this to RollingFileAppender to enforce size based rolling and cleanup.

Steps:

  1. Login to Ambari
  2. Goto the config tab of HDFS component.
  3. Filter for Advanced hdfs-log4j
  4. Update the following log appenders: 
    #log4j.appender.DRFA=org.apache.log4j.DailyRollingFileAppender log4j.appender.DRFA=org.apache.log4j.RollingFileAppender
...
# Rollver at midnight
#log4j.appender.DRFA.DatePattern=.yyyy-MM-dd
...
#log4j.appender.DRFA.MaxBackupIndex=30
log4j.appender.DRFA.MaxFileSize=256MB
...
#log4j.appender.DRFAS=org.apache.log4j.DailyRollingFileAppender
log4j.appender.DRFAS=org.apache.log4j.RollingFileAppender log4j.appender.DRFAS.File=${hadoop.log.dir}/${hadoop.security.log.file} log4j.appender.DRFAS.layout=org.apache.log4j.PatternLayout
log4j.appender.DRFAS.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n
#log4j.appender.DRFAS.DatePattern=.yyyy-MM-dd
log4j.appender.DRFAS.MaxFileSize=100MB
log4j.appender.DRFAS.MaxBackupIndex=9
hdfs.audit.logger=INFO,console
log4j.logger.org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit=${hdfs.audit.logger}
log4j.additivity.org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit=false
#log4j.appender.DRFAAUDIT=org.apache.log4j.DailyRollingFileAppender
log4j.appender.DRFAAUDIT=org.apache.log4j.RollingFileAppender
log4j.appender.DRFAAUDIT.File=${hadoop.log.dir}/hdfs-audit.log
log4j.appender.DRFAAUDIT.layout=org.apache.log4j.PatternLayout
log4j.appender.DRFAAUDIT.layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n
#log4j.appender.DRFAAUDIT.DatePattern=.yyyy-MM-dd
log4j.appender.DRFAAUDIT.MaxFileSize=100MB
log4j.appender.DRFAAUDIT.MaxBackupIndex=9
mapred.audit.logger=INFO,console
log4j.logger.org.apache.hadoop.mapred.AuditLogger=${mapred.audit.logger}
log4j.additivity.org.apache.hadoop.mapred.AuditLogger=false
#log4j.appender.MRAUDIT=org.apache.log4j.DailyRollingFileAppender
log4j.appender.MRAUDIT=org.apache.log4j.RollingFileAppender
log4j.appender.MRAUDIT.File=${hadoop.log.dir}/mapred-audit.log
log4j.appender.MRAUDIT.layout=org.apache.log4j.PatternLayout
log4j.appender.MRAUDIT.layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n
#log4j.appender.MRAUDIT.DatePattern=.yyyy-MM-dd
log4j.appender.MRAUDIT.MaxFileSize=100MB
log4j.appender.MRAUDIT.MaxBackupIndex=9

 

5. ZooKeeper

# Max log file size of 10MB
log4j.appender.ROLLINGFILE.MaxFileSize=256MB
# uncomment the next line to limit number of backup files
#log4j.appender.ROLLINGFILE.MaxBackupIndex=10

 

6. Oozie

Steps:

  1. Login to Ambari
  2. Goto the config tab of Oozie component.
  3. Filter for oozie.instance.id}] %m%n 
    #Commented this to add RollingFileAppender
#log4j.appender.oozieops=org.apache.log4j.DailyRollingFileAppender
#log4j.appender.oozieops.DatePattern='.'yyyy-MM-dd
log4j.appender.oozieops=org.apache.log4j.RollingFileAppender
log4j.appender.oozieops.MaxFileSize=256MB
log4j.appender.oozieops.MaxBackupIndex=20
log4j.appender.oozieops.File=${oozie.log.dir}/oozie-ops.log
log4j.appender.oozieops.Append=true
log4j.appender.oozieops.layout=org.apache.log4j.PatternLayout log4j.appender.oozieops.layout.ConversionPattern=%d{ISO8601} %5p %c{1}:%L - %m%n

#Commented this to add RollingFileAppender
#log4j.appender.oozieinstrumentation=org.apache.log4j.DailyRollingFileAppender #log4j.appender.oozieinstrumentation.DatePattern='.'yyyy-MM-dd log4j.appender.oozieinstrumentation=org.apache.log4j.RollingFileAppender log4j.appender.oozieinstrumentation.MaxFileSize=256MB
log4j.appender.oozieinstrumentation.MaxBackupIndex=20
log4j.appender.oozieinstrumentation.File=${oozie.log.dir}/oozie-instrumentation.log log4j.appender.oozieinstrumentation.Append=true
log4j.appender.oozieinstrumentation.layout=org.apache.log4j.PatternLayout log4j.appender.oozieinstrumentation.layout.ConversionPattern=%d{ISO8601} %5p %c{1}:%L - %m%n

#Commented this to add RollingFileAppender
#log4j.appender.oozieaudit=org.apache.log4j.DailyRollingFileAppender
#log4j.appender.oozieaudit.DatePattern='.'yyyy-MM-dd
log4j.appender.oozieaudit =org.apache.log4j.RollingFileAppender
log4j.appender.oozieaudit.MaxFileSize=256MB
log4j.appender.oozieaudit.MaxBackupIndex=20
log4j.appender.oozieaudit.File=${oozie.log.dir}/oozie-audit.log
log4j.appender.oozieaudit.Append=true
log4j.appender.oozieaudit.layout=org.apache.log4j.PatternLayout log4j.appender.oozieaudit.layout.ConversionPattern=%d{ISO8601} %5p %c{1}:%L - %m%n

#Commented this to add RollingFileAppender
#log4j.appender.openjpa=org.apache.log4j.DailyRollingFileAppender
#log4j.appender.openjpa.DatePattern='.'yyyy-MM-dd
log4j.appender.openjpa=org.apache.log4j.RollingFileAppender
log4j.appender.openjpa.MaxFileSize=256MB
log4j.appender.openjpa.MaxBackupIndex=20
log4j.appender.openjpa.File=${oozie.log.dir}/oozie-jpa.log
log4j.appender.openjpa.Append=true
log4j.appender.openjpa.layout=org.apache.log4j.PatternLayout
log4j.appender.openjpa.layout.ConversionPattern=%d{ISO8601} %5p %c{1}:%L - %m%n

 

7. Knox

  1. Login to Ambari
  2. Goto the config tab of Oozie component.
  3. Filter for gateway-log4j
  4. Update the following log appenders: 
    #Commented this to add RollingFileAppender
#log4j.rootLogger=ERROR, drfa
log4j.rootLogger=ERROR, rfa
log4j.logger.org.apache.hadoop.gateway=INFO#log4j.logger.org.apache.hadoop.gateway=DEBUG
#log4j.logger.org.eclipse.jetty=DEBUG
#log4j.logger.org.apache.shiro=DEBUG
#log4j.logger.org.apache.http=DEBUG
#log4j.logger.org.apache.http.client=DEBUG
#log4j.logger.org.apache.http.headers=DEBUG
#log4j.logger.org.apache.http.wire=DEBUG
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{2}: %m%n

#Commented this to add RollingFileAppender
#log4j.appender.drfa=org.apache.log4j.DailyRollingFileAppender
#log4j.appender.drfa.File=${app.log.dir}/${app.log.file}
#log4j.appender.drfa.DatePattern=.yyyy-MM-dd
#log4j.appender.drfa.layout=org.apache.log4j.PatternLayout
#log4j.appender.drfa.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} (%F:%M(%L)) - %m%n
log4j.appender.rfa=org.apache.log4j.RollingFileAppender
log4j.appender.rfa.File=${app.log.dir}/${app.log.file}
log4j.appender.rfa.MaxFileSize=256MB
log4j.appender.rfa.MaxBackupIndex=10
log4j.appender.rfa.layout=org.apache.log4j.PatternLayout
log4j.appender.rfa.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} (%F:%M(%L)) - %m%n

log4j.logger.audit=INFO, auditfile

#Commented this to add RollingFileAppender
#log4j.appender.auditfile=org.apache.log4j.DailyRollingFileAppender log4j.appender.auditfile=org.apache.log4j.RollingFileAppender
log4j.appender.auditfile.MaxFileSize=256MB
log4j.appender.auditfile.MaxBackupIndex=10
log4j.appender.auditfile.File=${app.log.dir}/${app.audit.file}
log4j.appender.auditfile.Append = true
#log4j.appender.auditfile.DatePattern = '.'yyyy-MM-dd
log4j.appender.auditfile.layout = org.apache.hadoop.gateway.audit.log4j.layout.AuditLayout
  5. Filter for ldap-log4jUpdate the following log appenders: 
    #Commented this to add RollingFileAppender
#log4j.rootLogger=ERROR, drfa
log4j.rootLogger=ERROR, rfa
log4j.logger.org.apache.directory.server.ldap.LdapServer=INFO log4j.logger.org.apache.directory=WARN
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{2}: %m%n

#Commented this to add RollingFileAppender
#log4j.appender.drfa=org.apache.log4j.DailyRollingFileAppender
#log4j.appender.drfa.File=${app.log.dir}/${app.log.file}
#log4j.appender.drfa.DatePattern=.yyyy-MM-dd
#log4j.appender.drfa.layout=org.apache.log4j.PatternLayout
#log4j.appender.drfa.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} (%F:%M(%L)) - %m%n

log4j.appender.rfa=org.apache.log4j.RollingFileAppender
log4j.appender.rfa.File=${app.log.dir}/${app.log.file}
log4j.appender.rfa.MaxFileSize=256MB
log4j.appender.rfa.MaxBackupIndex=10
log4j.appender.rfa.layout=org.apache.log4j.PatternLayout log4j.appender.rfa.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} (%F:%M(%L)) - %m%n

     

8. Hive & Hive metastore

  1. Login to Ambari
  2. Goto the config tab of Hive component.
  3. Filter for Advanced hive-log4j
  4. Change,
    • DRFA -> RFA in all places
    • log4j.appender.RFA=org.apache.log4j.DailyRollingFileAppender -> log4j.appender.RFA=org.apache.log4j.RollingFileAppender
  5. Add
    1. log4j.appender.RFA.MaxFileSize=100MB
    2. log4j.appender.RFA.MaxBackupIndex=10
  6. Sample below, 
    # Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# Define some default values that can be overridden by system properties
hive.log.threshold=ALL
hive.root.logger=INFO,RFA
hive.log.dir=${java.io.tmpdir}/${user.name}
hive.log.file=hive.log

# Define the root logger to the system property "hadoop.root.logger".
log4j.rootLogger=${hive.root.logger}, EventCounter

# Logging Threshold
log4j.threshold=${hive.log.threshold}

#
# Daily Rolling File Appender changed to Rolling File Appender (DRFA -> RFA)
#
# Use the PidDailyerRollingFileAppend class instead if you want to use separate log files
# for different CLI session.
#
# log4j.appender.RFA=org.apache.hadoop.hive.ql.log.PidDailyRollingFileAppender

log4j.appender.RFA=org.apache.log4j.RollingFileAppender

log4j.appender.RFA.File=${hive.log.dir}/${hive.log.file}

# Rollver at midnight
#log4j.appender.RFA.DatePattern=.yyyy-MM-dd
log4j.appender.RFA.MaxFileSize=100MB

# 30-day backup
log4j.appender.RFA.MaxBackupIndex=10
log4j.appender.RFA.layout=org.apache.log4j.PatternLayout

# Pattern format: Date LogLevel LoggerName LogMessage
#log4j.appender.RFA.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n
# Debugging Pattern format
log4j.appender.RFA.layout.ConversionPattern=%d{ISO8601} %-5p [%t]: %c{2} (%F:%M(%L)) - %m%n


#
# console
# Add "console" to rootlogger above if you want to use this
#

log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} [%t]: %p %c{2}: %m%n
log4j.appender.console.encoding=UTF-8

#custom logging levels
#log4j.logger.xxx=DEBUG

#
# Event Counter Appender
# Sends counts of logging messages at different severity levels to Hadoop Metrics.
#
log4j.appender.EventCounter=org.apache.hadoop.hive.shims.HiveEventCounter


log4j.category.DataNucleus=ERROR,RFA
log4j.category.Datastore=ERROR,RFA
log4j.category.Datastore.Schema=ERROR,RFA
log4j.category.JPOX.Datastore=ERROR,RFA
log4j.category.JPOX.Plugin=ERROR,RFA
log4j.category.JPOX.MetaData=ERROR,RFA
log4j.category.JPOX.Query=ERROR,RFA
log4j.category.JPOX.General=ERROR,RFA
log4j.category.JPOX.Enhancer=ERROR,RFA


# Silence useless ZK logs
log4j.logger.org.apache.zookeeper.server.NIOServerCnxn=WARN,RFA
log4j.logger.org.apache.zookeeper.ClientCnxnSocketNIO=WARN,RFA

     

 

References

  1. https://community.hortonworks.com/articles/8882/how-to-control-size-of-log-files-for-various-hdp-c.html
  2. https://discuss.pivotal.io/hc/en-us/articles/202296718-How-to-Change-Hadoop-Daemon-log4j-properties

 

Related posts:

  1. Adding compression codec to Hortonworks data platform
  2. Best practices for Namenode and Datanode restarts
  3. Permanently add jars to hadoop
  4. Connecting to Apache Phoenix

Tags:

You may also like...

Leave a Reply

Your email address will not be published. Required fields are marked *