But what’s ASL? Apparently, it’s the Apple System Log. There’s a nice summary on Domain of the Bored. He also gives the key hint: you can use syslog(1) to read the binary ASL files.

I didn’t delve too deeply into the flags. It appeared that just running syslog spat out all the information I required. But, it comes out encoded like cat -v. But you can pipe it through unvis(1) to clean that up.

$ syslog | unvis | less

Normally, Console.app would take care of all this transparently. But when you’re ssh’d into a mac, that’s not an option. So it’s good to know about syslog(1).

Looking closer at the flags, you can say syslog -E safe in place of piping through unvis(1).

Why use logging?

Well, you could use System.out.println(). But as alluded to above, this is a very coarse tool. There’s no way to turn it off without altering the code. With a logging system, you get much greater control, including the ability to control what gets logged and where it ends up. And you get to control it at runtime without altering your code.

Generally, proper logging gives you better visibility into your project and its behaviour.

Perhaps most importantly you should do it in order to not piss off your sysadmins by generating a very large non-rotateable file.

Which logging system to use?

There are quite a few out there.

• log4j
• JDK logging

In addition, there are “abstraction libraries,” which allow you to choose different logging systems to actually use. For example:

• commons-logging
• slf4j

My advice is that unless you’re writing a library that you expect 3rd parties to use (over whom you have no control), then you should stick to log4j. It’s extremely widely used and well known. It works very well. It’s also wonderfully configurable.

If you do have to write a library for 3rd parties, then commons-logging is the more popular choice. Whilst it certainly has it’s flaws, it does appear to work for me. Caveat emptor.

But what about JDK Logging?

JDK Logging was patterned after log4j. Only they made the configuration a lot less flexible. The main flaw (as I see it) is that you can only configure it via a system property, which means it’s inherently global. When you’re running several webapps in the same JVM (via tomcat), this is a fatal flaw.

Actually, I’ve since learned that tomcat works around this by providing it’s own replacement implementation of LogManager. Regardless, the configuration is still very anaemic when compared with log4j.

What should I log?

• At DEBUG, log whatever the hell you like so you know you’ve been down a particular code path, or what a certain piece of data looks like.
• At INFO, support should be able to trace through transactions and figure out what’s going on.
• At WARN, note things that could constitute a problem.
• At ERROR, note things are are a problem.
• At FATAL, note why you cannot continue (“my CPU is on fire”).

All messages above DEBUG must be readable by non-developers — plain English, please!

Remember to include as much context as possible. Don’t just note that something failed. Say what the conditions were that caused that failure.

Beginning log4j

First, create a very simple log4j.properties (you can use log4j.xml but it’s more verbose and the extra features it makes available are rarely needed).

  log4j.rootLogger=WARN, stdout
  log4j.appender.stdout=org.apache.log4j.ConsoleAppender
  log4j.appender.stdout.layout=org.apache.log4j.SimpleLayout
  log4j.logger.com.me=INFO

NB: I set the default logging level to WARN and left our code logging at INFO. This is in order to reduce junk output from 3rd party libraries like Spring and Hibernate. Of course, you can always enable that if needed.

Place this somewhere it will end up at the root of your classpath.

• For maven, put it in src/main/resources/log4j.properties.
• For an ant project, put it in src/log4j.properties.

Now, create a logger and use it.

  package com.me;
  public class Foo {
    // Non-static allows use of getClass().
    private Logger LOG = Logger.getLogger(getClass());
 
    public void logSomething() {
      LOG.info("something");
    }
  }

That should end up on your console. Hurrah!

NB: Always using a consistent field name for your logger makes it much easier to grep for log messages later on. It’s a good candidate for an Eclipse template.

How do I enable DEBUG for a single class/package?

Add this to log4j.properties.

 log4j.logger.org.springframework=DEBUG

Everything else will continue to be logged at the level of the rootLogger, but Spring stuff will come out more verbosely.

To understand how this works, you have to understand the idea of “logger hierarchies.” If you have configured the rootLogger as WARN and the “com.me.proj2″ logger as INFO, this is how the logger hierarchy could end up looking.

How do I customize the output format?

Configure a different Layout in log4j.properties. e.g. PatternLayout.

 log4j.appender.stdout=org.apache.log4j.ConsoleAppender
 log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
 log4j.appender.stdout.layout.conversionPattern=%d{ISO8601} %c{2} %p %m%n

That configures each line with the following fields:

• The date & time in standard ISO 8601 format.
• The last two components of the classname which made the log message.
• The priority level.
• The message itself.
• A newline.

How does log4j get configured?

The configuration process is fairly simple. The first time you call log4j, it looks on the classpath for log4j.xml and then log4j.properties. That’s all there is to it, unless you change it.

There are several ways to impose your own order on things.

• You can specify a log4j.configuration system property pointing a file. But as we discussed earlier, system properties are generally best avoided.
• You can use the log4j API (PropertyConfigurator or DOMConfigurator) to find the config file in the location you want. You have to execute this code before the rest of your app starts up.

If it still doesn’t seem to be doing what you expect, you can (as a one-off) specify the log4j.debug system property on the command line. That will show you where log4j is looking for its configuration on stderr.

How do I configure logging in my tests?

Generally you don’t want logging in tests. They’re verbose enough already, and JUnit will let you know when something’s wrong. You can enforce this by creating a second log4j.properties that lives in the classpath of your tests, which sets the rootLogger to WARN (and nothing else more detailed).

• For maven, put it in src/test/resources/log4j.properties.
• For an ant project, put it in test/log4j.properties.

How should I configure log4j in a webapp?

As early as possible. This lets you see debug output from the rest of your app starting up.

That is, in a special ServletContextListener. Trying to configure log4j from within a Spring applicationContext is far, far too late (this appears to be a common mistake).

Spring has a special purpose Log4jConfigListener for this purpose. You can write your own, however. Here’s an example of the web.xml configuration.

  <context-param>
    <param-name>log4jConfigLocation</param-name>
    <param-value>/WEB-INF/log4j.properties</param-value>
  </context-param>
  <listener>
    <listener-class>org.springframework.web.util.Log4jConfigListener</listener-class>
  </listener>

How should log4j be setup for development / production in a webapp?

Don’t have a default /log4j.properties on the classpath. Instead, have the Log4jConfigListener set up in the web.xml, pointing at /WEB-INF/log4j.properties (as shown above). That file should contain your development logging setup (INFO level output to the console) as well as commented out examples of production logging setup (WARN level output to a rotated file).

Create a file META-INF/context.xml (for tomcat), which contains a commented out override for the log4jConfigLocation context-param. e.g.

 <context>
   <!--
   <parameter override="false"
              name="log4jConfigLocation"
              value="file:///etc/myapp/log4j.properties"
              />
   -->
 </context>

When packaged as a war, the two files are easily extracted and can be used to override the logging behaviour.

 % unzip -p ~/foo.war META-INF/context.xml > $sitecode/conf/Catalina/localhost/foo.xml
 % vi $CATALINA_BASE/conf/Catalina/localhost/foo.xml
 % unzip -p ~/foo.war WEB-INF/log4j.properties > /etc/myapp/log4j.properties
 % vi /etc/myapp/log4j.properties

This is probably the most flexible solution. It means that you can deploy the same war file anywhere, and just have a small amount of one-time setup to get it going.

How do I get my log files rotated?

Log4j comes with a builtin DailyRollingFileAppender which will give you a new log file every day. However, I’ve tended to prefer the DatedFileAppender addon, which works in a very similar manner to cronolog. That is, each log file is named with the current date in it.

This is how it’s configured in log4j.properties.

log4j.rootLogger = WARN, logfile
log4j.appender.logfile=biz.minaret.log4j.DatedFileAppender
log4j.appender.logfile.Directory=${catalina.base}/logs
log4j.appender.logfile.Prefix=myapp.
log4j.appender.logfile.Suffix=.txt
log4j.appender.logfile.layout=org.apache.log4j.PatternLayout
log4j.appender.logfile.layout.ConversionPattern=%d{ISO8601} %p %c{2} - %m%n

What about exceptions?

Lots of people tend to log that an exception occurred and then move on as if nothing happened.

  // FAIL
  try {
    someOperation();
  } catch (SomeNastyException e) {
    LOG.error(e.getMessage());
  }

This is bad because it loses all the context associated with the exception, as well as failing to stop the course of the program. As I noted before, you’re much better off rethrowing it for something higher up to deal with.

  try {
    someOperation();
  } catch (SomeNastyException e) {
    throw new RuntimeException(e);
  }

Whilst this doesn’t work in all cases, it’s a far better default stance to take.

You might also want to look at Best Practises For Exception Handling.

How do I get a unique per-request ID in my logs?

Tools like postfix and sendmail include the message-id in the log files. This is invaluable for tracing a message over several lines of a log file. I needed to do this in a webapp and found log4j’s Nested Diagnostic Context. This is how I implemented it.

  public class AddIdListener implements ServletRequestListener {
    private static long requestNum = 0;
    private static synchronized long getNextRequestNum() {
      return requestNum++;
    }
    public void requestInitialized(ServletRequestEvent sre) {
      NDC.push(String.valueOf(getNextRequestNum()));
    }
    public void requestDestroyed(ServletRequestEvent sre) {
      NDC.pop();
    }
  }

With this configured in the web.xml, you need to use the PatternLayout to get it into the logs.

 log4j.appender.stdout.layout.conversionPattern=%d{ISO8601} %x %c{2} %p %m%n

Now, all requests with the same value in the %x field belong to the same request.

This may not be a perfect solution, but it works quite well for me.

A performance tip

Sometimes, a call to a logger can be expensive. Particularly in the middle of a tight loop. For this, you can ask log4j if it’s even going to bother using the output, which is quicker.

  for (BigHairyObject bho : bigHairyObjects) {
    if (LOG.isDebugEnabled()) {
      LOG.debug("here, bho=" + bho);
    }
  }

Normally, logging in Java is quite simple: you add log4j to your classpath, then create a log4j.properties to say what gets logged. If you’re running as part of a webapp, you use something like Spring’s Log4jConfigListener to ensure that the configuration gets applied as soon as the webapp is active.

Cocoon is different. By default (in development) you run it using a combination of the cocoon-maven-plugin and mvn jetty:run. This is quite cunning (as Grzegorz explained in a comment a while back), it lets you edit all sorts of stuff and have it dynamically reloaded. In order to make the cocoon “block” work with jetty, the maven plugin creates things like web.xml for you automatically, so there’s no chance to edit things. Drat.

Now, if you follow the documentation for logging in Cocoon, it advises:

The usual Cocoon web application sets up Log4j through the Cocoon Spring Configurator.

Lovely advice, except that by the time Spring has started up, read your logging configuration and applied it, a great deal of interesting events have already occurred. You really need to enable logging as early as possible using a ServletContextListener.

Thankfully, it’s possible to do so, even when using mvn jetty:run.

First, you need to create a log4j configuration that’s suitable. I think it has to be XML, which is a shame as it’s more complicated than the properties file. I wanted to change the defaults to get my FlowScript calls to log to the console. This is what I ended up with in etc/log4j.xml.

  <!DOCTYPE log4j:configuration SYSTEM "log4j.dtd">
  <log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/">
    <appender name="stdout" class="org.apache.log4j.ConsoleAppender">
      <param name="target" value="System.err"/>
      <layout class="org.apache.log4j.PatternLayout">
        <param name="ConversionPattern" value="%d{ISO8601} %c{2} %p - %m%n"/>
      </layout>
    </appender>
    <logger name="cocoon">
      <level value="INFO" />
    </logger>
    <logger name="org.apache.cocoon.components.flow.javascript.fom">
      <level value="INFO" />
    </logger>
    <root>
      <priority value="WARN"/>
      <appender-ref ref="stdout"/>
    </root>
  </log4j:configuration>

Note that we shut everything up (WARN) by default, and then explicitly enable messages for things we want to see (INFO). I’ve found that even in development, this helps to tell the wood from the trees.

With that in place, you have to edit pom.xml in order to tell the cocoon-maven-plugin to use that instead if its default. The default pom should have a build/plugins/plugin section, to which this stanza needs adding.

  <configuration>
    <!-- Gets copied to target/.../WEB-INF/log4j.xml -->
    <customLog4jXconf>etc/log4j.xml</customLog4jXconf>
  </configuration>

Finally, you need to arrange for the auto-generated web.xml to be patched with a reference to Log4jConfigListener. This is done through Cocoon’s slightly arcane mechanism, xpatch. Create a file src/main/resources/META-INF/cocoon/xpatch/log4j.xweb which looks like this.

  <xweb xpath="/web-app"
        unless="comment()[contains(., 'Log4j Configuration')]"
        insert-after="node()[1]">
    <!--Log4j Configuration-->
    <context-param>
      <param-name>log4jConfigLocation</param-name>
      <param-value>/WEB-INF/log4j.xml</param-value>
    </context-param>
    <listener>
      <listener-class>org.springframework.web.util.Log4jConfigListener</listener-class>
    </listener>
  </xweb>

Now if you run mvn jetty:run in your block and inspect the generated web.xml, you should see the above patched in to place. Also, you should be able to generate messages on the console from within FlowScript by doing:

  cocoon.log.info("hello world");

The procedure above is a hassle. But the benefit of being able to see logging messages coming out on the console in front of you is significant.

One final point to note. When you do run mvn jetty:run, you’ll see a few log4j errors, i.e.

log4j:WARN No appenders could be found for logger (org.apache.commons.configuration.ConfigurationUtils).
log4j:WARN Please initialize the log4j system properly.

log4j:WARN No appenders could be found for logger (org.apache.commons.jci.stores.MemoryResourceStore).
log4j:WARN Please initialize the log4j system properly.

As far as I can tell these are completely ignorable, just very annoying. They appear to happen before jetty itself starts up, and are irrelevant to the web app (as far as I can see).

Except I tried that in Eclipse (using WTP) and it didn’t work.

I tried copying it to the $CATALINA_BASE/conf directory instead¹.

Still no joy.

I’ve just found the answer, after looking in the tomcat source. It turns out that tomcat’s alternative logging implementation (JULI) is enabled via a system property. This happens inside catalina.sh:

# Set juli LogManager if it is present
if [ -r "$CATALINA_BASE"/conf/logging.properties ]; then
  JAVA_OPTS="$JAVA_OPTS -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager"
  LOGGING_CONFIG="-Djava.util.logging.config.file=$CATALINA_BASE/conf/logging.properties"
fi

But Eclipse runs the Java code directly, without using catalina.sh. So the properties never gets set. You have to set them by hand in the “Run Configurations” dialog. Like this:

Of note there is that I’ve imported the logging.properties file into the Servers project of my workspace. It seemed liked a useful place to put it.

Of course, after realising that, I soon find the same information in the WTP FAQ: How do I enable the JULI logging in a Tomcat 5.5 Server instance?.

Anyway, now I might be able to debug that ClassLoader issue…

¹ Inside Eclipse with WTP, that will be something like $WORKSPACE/.metadata/.plugins/org.eclipse.wst.server.core/tmp0.

So, I’ve switched my Apache over to using cronolog in order to get date based logfiles automatically. I’ve used it on a project at work recently and it really works a treat.

I also noticed that PostgreSQL has grown the ability to take care of its own log files in recent versions. So I’ve switched that to doing date based logging with automatic rotation as well. Lovely.

All is not perfect of course. Oh no, that would be far too simple. There’s still the issue of removing old logfiles and/or compressing them. But that seems to be a smaller integration problem.

Of course the trigger for all this activity was finding a 220Mb access_log lying around. Doh!

About half way through writing it, I remembered the CPAN, for all your feedy needs. Almost. At the moment, it’s still a bit too easy to create Atom feeds that the feed validator doesn’t like. I need to have a bit of a think about the best way to get things like “id” in there for each log message. But it should still be usable for now.