Tag Archives: Java

Understanding Java Generics’ super and extends

Do you know the difference in Java generics between <? super E> and <? extends E>? If you have any doubt, read on.

In Java, classes and interfaces can have type parameters, like a List of Numbers instead of just a List:

List<Number> list = new ArrayList<>();

With the diamond notation <>, the Java 7 compiler uses type inference to deduce the argument type so you can abbreviate instantiation to new ArrayList<>().

You can also type the variable that holds the reference to the ArrayList instance as a List, a Collection, or a Serializable.

Let’s now say you have a pair of methods that take a List of Numbers. The first method printNumbers() gets Numbers out of the List:

public void printNumbers(List<Number> list) {
    for (Number number : list) {
        System.out.print(number);
        System.out.print(", ");
    }
}

The second method, fillNumbers(), adds Numbers to the List:

public void fillNumbers(List<Number> list) {
    Number n = Integer.MAX_VALUE;
    numbers.add(n);
    numbers.add(3);
}

The method getting Numbers out of the List could just as easily work with a List of Integers or a List of Floats:

List<Number> floats = new ArrayList<>();
floats.add(Float.valueOf(1.2f));
printNumbers(floats);

What we really want though is a List, not a List:

List<Float> floats = new ArrayList<>();
floats.add(Float.valueOf(1.2f));
printNumbers(floats);

But that causes the compiler to fail with:

The method printNumbers(List) in the type Test is not applicable for the arguments (List)

To use a List, the method must be typed with List<? extends Number>.

public void printNumbers(List<? extends Number> list)

The ? extends tells the compiler that we want to use some unknown (the ?) subtype of Number. We explicitly constrain the wildcard (the ?) to represent the unknown subtype of Number. We say the method is a covariant use of the List of Numbers; it works with any List of any subtype of Number.

On the other hand, you should be able to give the method adding Numbers to the List any old List of Objects:

List<Object> objects = new ArrayList<>();
fillNumbers(objects);

That does not compile:

The method fillNumbers(List<Number>) in the type GenericsSuperExtendsDemo is not applicable for the arguments (List<Object>)

You get this to work with List<? super Number>:

public void fillNumbers(List<? super Number> list)

The ? super Number is an explicitly constrained wildcard that represents some unknown supertype of Number. We call that one a contravariant use of the List of Numbers and it works for any List of any supertype of Number.

Here is the complete example:

package test;

import java.util.ArrayList;
import java.util.List;

public class GenericsSuperExtendsDemo {

    public static void main(String[] args) {
        new GenericsSuperExtendsDemo().go();
    }

    public void go() {
        System.out.println("Hello World!");

        List<Number> numbers = new ArrayList<>();
        fillNumbers(numbers);
        printNumbers(numbers);

        {
            List<Number> floats = new ArrayList<>();
            floats.add(Float.valueOf(1.2f));
            printNumbers(floats);
        }
        {
            List<Float> floats = new ArrayList<>();
            floats.add(Float.valueOf(1.2f));
            printNumbers(floats);
        }
        {
            List<Integer> integers = new ArrayList<>();
            printNumbers(integers);
        }
        {
            List<Object> objects = new ArrayList<>();
            fillNumbers(objects);
        }
    }

    public void printNumbers(List<? extends Number> list) {
        for (Number number : list) {
            System.out.print(number);
            System.out.print(", ");
        }
    }

    public void fillNumbers(List<? super Number> list) {
        Number n = Integer.MAX_VALUE;
        list.add(n);
        list.add(3);
    }

}

The inspiration for this article is a paper presented at OOPSLA 2016 titled “Java and Scala’s Type Systems are Unsound.”

Happy Coding,
Gary Gregory

Loading a Log4j Configuration for a specific EJB

You can load and unload a specific Log4j 2 configuration file for a given EJB. How? Use @PreDestroy and @PostConstruct. This gives you separately deployable EJBs with separate Log4j configurations. Ka-Pow!

For example:

public class MySessionEJB implements SessionBean {

    private static final String LOGGER_CONFIG = "/path/to/log4j2.xml";
    private static final String LOGGER_CONTEXT_NAME = "MySessionEJB";
    private static LoggerContext logctx;
    private static Logger logger;

    @PostConstruct
    @TransactionAttribute(value=TransactionAttributeType.NOT_SUPPORTED)
    private void postConstruct() {
        logctx = Configurator.initialize(LOGGER_CONTEXT_NAME, LOGGER_CONFIG);
        logger = logctx.getLogger("com.whatever.myejb");
    }

    @PreDestroy
    @TransactionAttribute(value=TransactionAttributeType.NOT_SUPPORTED)
    private void preDestroy() {
        Configurator.shutdown(logctx);
    }
}

Happy Coding,
Gary Gregory

635862519545313516-70822833_finding-nemo-now-what

Of the demise of FindBugs and Monty Python

umdfindbugsIt turns out that FindBugs, the Java bug hunting tool used by legions of Java developers, after being proclaimed dead, has issued a Monty Python-like “I’m not dead yetrejoinder on Hacker News.

What is going on here?

FindBugs is a development-time tool that finds bugs in Java programs using static analysis. It’s free and distributed under the Lesser GNU Public License out of The University of Maryland. Most developers use FindBugs through Maven and the Maven Findbugs Plugin. David Hovemeyer who did his Ph.D. thesis on FindBugs founded the project. Since then Bill Pugh has been the project lead and primary developer. The current developers listed for FindBugs are Bill Pugh and Andrey Loskutov.

Reviewing Andrey’s November 2nd post (edited in quotes for minor typos and clarity), here what seems to be the Good, the Bad and the Ugly, no not that one, but the film is a classic and the images below are all in good fun.

The Good

thegoodThe Good is thin: FindBugs has two committers with push karma: Andrey Loskutov and Tagir Valeev. That’s not much for such a complex project but better than none. Tagir’s last commit (Google Code Archive) was on 2015-04-09 and Andrey does not consider him active. Andrey has stated that he himself “has no free time to work on the project”. (There is code on the Google Code Archive and then on GitHub.)

The Bad

thebadBill Pugh, the project lead, has not been active and is the bottleneck for access rights to critical parts of the project. According to Andrey:

“Only the project leader Bill Pugh has admin rights for the project web page and the GitHub project group and page. We cannot deploy any website update, we can’t add new project members, we can’t manage code access rights, we can’t publish releases to the well-known update sites without his help. Without him, we have no admin rights to anything; we can only push to the repository.”

This seems to be par for the course in many FOSS projects, the benevolent dictator acts as a gatekeeper to resources and once a bus or vacation enters the pictures, this locks out contributors. Bill Pugh, the project lead, has not been active or responsive it seems (until now, more on this later).

The Ugly

theuglyFindBugs has been around for a long time now (ten years) and has accrued a fair amount of technical debt.

Andrey states that the code is very complex, has “organically grown” over a decade. Not surprisingly and like many other FOSS projects, and authors have not done much in terms of documentation. Poor public interfaces apparently compound the hurt.

“Most of the code consists of the very low level bytecode related stuff, tightly coupled with the ancient BCEL library, which doesn’t scale and is not multi-thread safe.”

I hear you there. I help out over at Apache Commons, the current home of the Apache Commons BCEL component, where we released in July a long overdue version 6.0. Apache published the previous version 5.2; wait for it… in June 2006! Ouch, FindBugs, I feel your pain. It might be too late for FindBugs and Apache Commons BCEL developers to work together to salvage this dependency:

 “No one enjoys maintaining this code, at least not me. I see no future for FindBugs with the BCEL approach, and see no way to get rid of it without investing lot of effort, and without breaking every detector and possibly many third party tools. This is the biggest issue we have with FindBugs today, and most likely the root cause for all the evil. This code can’t be fixed, it must be rewritten.”

What are the alternatives?

Luckily, there are other byte code fiddling tools out there. We have (quoted with minor edits from their respective sites):

  • ASM (INRIA, France Telecom license) from the OW2 consortium. ASM is an all-purpose Java bytecode manipulation and analysis framework. You can use ASM to modify existing classes or dynamically generate classes, directly in binary form. ASM provides common transformations and analysis algorithms allow you to assemble easily custom complex transformations and code analysis tools.
  • Apache Commons Weaver (Apache 2.0 license) manipulates existing class files. It looks like FINDBUGS has started to work on some integration with ASM based on looking at a few commits.
  • Javassist (Mozilla Public License 1.1, LGPL 2.1, Apache 2.0 licenses) is a class library for editing bytecodes in Java; it enables Java programs to define a new class at runtime and to modify a class file when the JVM loads it. Unlike other similar bytecode editors, Javassist provides two levels of API: source level and bytecode level. If the users use the source-level API, they can edit a class file without knowledge of the specifications of the Java bytecode. Javassist designed the whole API with only the vocabulary of the Java language. You can even specify inserted bytecode in the form of source text; Javassist compiles it on the fly. On the other hand, the bytecode-level API allows the users to edit directly a class file as other editors.
  • Byte Buddy (Apache 2.0 license) is a code generation and manipulation library for creating and modifying Java classes during the runtime of a Java application and without the help of a compiler. Other than the code generation utilities that ship with the Java Class Library, Byte Buddy allows the creation of arbitrary classes and is not limited to implementing interfaces for the creation of runtime proxies. Furthermore, Byte Buddy offers a convenient API for changing classes either manually, using a Java agent or during a build.

From FindBugs to HuntBugs

Andrey thinks that because the code is as it is, there are not so many people willing to contribute. GitHub sees some pull requests, but most of them are smaller fixes or enhancements which are not reviewed or tested.

These conditions has led Tagir to start his own project called HuntBugs, a new Java bytecode static analyzer tool based on Procyon Compiler Tools aimed to supersede FindBugs. HuntBugs is currently in early development stage and published under the Apache 2.0 license. It also sports seven contributors on GitHub, an almost good sign, but all of these contributors only have a handful of commits each. Almost all commits are from Tagir, so the word “community” seems generous. The Procyon Compiler Tools are published on BitBucket also under the Apache 2.0 license.

The lack of community around FindBugs is what killed it’s momentum. This happens over and over in the FOSS world.

Like many FOSS projects and again to no one’s surprise, there is little support from any companies or organizations. The FindBugs sponsors page states that the most recent funding for FindBugs comes from a Google Faculty Research Awards. No code patches, no testing, no paid developers. Of course, Andrey remarks, some companies do use FindBugs in commercial products like SonarSource and Coverity, not to mention countless companies using FindBugs in their build processes.

As it is today, even without further development, FindBugs is still useful. Sadly it cannot find bugs that are specific to Java 8 or Java 9’s early releases.

What’s on the horizon?

There are two paths here: FindBugs will raise from its ashes or a fork will take over. There are a number of steps Andrey outlines in his post to allow the current contributors (himself really) to continue the project. These are mostly project governance and access rights issues. However, having stated that his time is limited, does this seem realistic? This assumes that Bill follows the steps to further open up FindBugs and that this will lead to a community forming around a more open FindBugs . Summarizing from Andrey’s post, the following needs to happen:

  • Update the site to point to GitHub instead of SourceForge.
  • Shut down the old SourceForge bug tracker and forums and point to GitHub instead.
  • Allow to grant access rights to the GitHub project.
  • Grant right to publish the new releases all known download sites.
  • Configure automated build and test (hello Travis CI and Coveralls)
  • Attract contributors

None of these seems difficult or insurmountable; it just needs Bill’s blessing and a new project governance model that allows other contributors to come in have proper access rights. Perhaps FindBugs could adopt some of Apache’s nomenclature: You can be a committer with push rights and above that a Project Management Committee (PMC) member, with full access to the project. Only votes from PMC members vote are binding on release votes. This would allow many committers to come in under the mentorship of a PMC.

Hello Bill!

fireeating

Bill eating a bug.

Then after all this, Bill pops up on Hacker News:

“FindBugs isn’t dead (although my participation had been in hibernation for a while).

I’ve been juggling far too many projects, but I’m now working to move FindBugs back into the active rotation.

I also want announce I’ll be working with GrammaTech as part of the Swamp Project, and they will be helping with rebooting the FindBugs project. This has been in the works for a long time (almost a year), and although I’ve known that GrammaTech was likely to win an award, this hasn’t been official and something I could talk about until recently. Was hoping to have something a little more concrete to talk about as far as that goes; but I don’t yet have the information I wanted to share.

Thanks to all the FindBugs fans and supporters who lobbied for me to return to active maintenance of FindBugs. Give me a week to get up to speed on current project needs.”

So that sounds good…

Now what?

This is all fine and good and I hope that Bill can work with the current contributors, or the one contributor to rejuvenate FindBugs. What Andrey asks sounds reasonable, but that is not enough. I believe that better project governance would really help FindBugs move forward again.

635862519545313516-70822833_finding-nemo-now-what

What about bringing FindBugs under the auspices of an Apache top-level project? Bringing a project under the Apache umbrella is done through the Apache Incubator under a well-defined process. This would really deal with all project governance issues and infrastructure in one go. The project sources could still live in GitHub, but the Apache-way of doing things would address many if not all of Andey’s concerns.

What do you think?

Cheers and Happy Coding,
Gary

Apache Log4j 2.7 is out!

log4j-logo-2-7Apache Log4j 2.7 is heading out to Maven Central. Here’s are the highlights of what’s new since 2.6.2.

 

  • The RoutingAppender can be configured with scripts.
  • A new Appender, the ScriptAppenderSelector can create another Appender as specified by a Script.
  • Users can now inject context data from other sources than ThreadContext. Values can be any Object, not just Strings.
  • The SocketAppender now supports IO buffering.
  • Added the ability to generate Log4j 2 XML configuration file from a ConfigurationBuilder.
  • Hello Scala! We’ve added Logging APIs for Scala 2.10 and 2.11.
  • Added options to exclude stack trace from JSON, XML and YAML layouts.
  • Added Core API Configurator.shutdown(LoggerContext, long, TimeUnit).
  • FileAppender and RollingFileAppender can create files on-demand.
  • PatternLayout added a color ANSI option to %message and %xThrowable.
  • org.apache.logging.log4j.core.LoggerContext now implements Closeable.
  • Add ThreadContextMap2 interface supporting method putAll(Map<String, String>).
  • Add JUnit Rule implementations to manage the thread context.
  • The Core AbstractConfiguration and AbstractManager now track its LoggerContext; add Configuration.getLoggerContext().

Continuing the Asynchronous Epic

  • We’ve added support for java.util.concurrent.LinkedTransferQueue to the AsyncAppender.
  • Added optional support for the Conversant DisruptorBlockingQueue in AsyncAppender.
  • Added optional support for JCTools MPSC bounded lock-free queue in AsyncAppender.

Continuing the GC-free Epic

  • Continuing the GC-free epic, we’ve added support for garbage-free ThreadContext map. This is disabled by default, and users need to enable this explicitly.
  • Also in GC-free-land, we changed LogEvent‘s internal data structure for context data to be garbage-free. We added the method LogEvent#getContextData() and deprecated getContextMap().

Continuing the Builder Epic

  • Added Builders for the ConsoleAppender, FileAppenderRoutingAppenderSocketAppender, and ServletAppender (and deprecated factory methods).
  • Builders can be generic.
  • Builder can subclass another Builder.

And bug fixes as well, for which you can just see the Log4j site.

Happy Log4j Logging!
Gary

 

Changing log levels in Log4j2

How do you change log levels in Log4j 2 when the Logger API does not have a setLevel() method?

The Configurator class provides several methods to change levels in Log4j. An important feature to understand before using these APIs is that logger names are hierarchical, see Refining Logging with Hierarchical Loggers.

Configurator.setAllLevels(String, Level)

The Configurator.setAllLevels(String, Level) method sets the level of the given named logger and all child loggers. Recall that in Log4j, loggers are hierachical. If you have the loggers com.foo.acom.foo.b, and com.foo.c, and you call setAllLevels("com.foo", Level.DEBUG) then all the loggers I listed will be set to DEBUG as will "com.foo". A logger "com.bar" will remain as is.

Configurator.setLevel(Map<String, Level>)

The method Configurator.setLevel(Map<String, Level>) will set each named logger to its Level from the map. No child loggers will be affected unlike setAllLevels(String, Level).

For example:

Map&amp;lt;String, Level&amp;gt; map = new HashMap&amp;lt;&amp;gt;();
map.put(&quot;com.foo&quot;, Level.DEBUG);
map.put(&quot;com.bar&quot;, Level.DEBUG);
Configurator.setLevel(map);

This will set the loggers "com.foo" and "com.bar" to DEBUG but no existing child loggers of either will be affected. If you create a new child logger though, that child logger will inherit its level from its parent.

Configurator.setLevel(String, Level)

The method Configurator.setLevel(String, Level) sets the named logger to the given level without affecting child loggers (unlike setAllLevels(String, Level)).

Configurator.setRootLevel(Level)

Finally, Configurator.setRootLevel(Level) sets the level of the root logger to the given level without affecting child loggers. The root logger is the topmost logger with a name of "" (the empty string).

Happy Coding,
Gary Gregory

All about Java 9

These videos are all about Java 9 from the Devoxx conference:

Thanks to Arnaud Héritier for gathering these links for the Maven dev mailing list.

The Java lowercase conversion surprise in Turkey

You can convert Strings to lowercase in Java with String.toLowerCase().

The following code will work anywhere in the world except Turkey:

import java.util.Locale;

public class LocaleTest {

  @Test
  public void test() {
    Assert.assertEquals("title", "TITLE".toLowerCase());
  }

}

In Turkey, this will give you:

org.junit.ComparisonFailure: expected:<t[i]tle> but was:<t[?]tle>
	at org.junit.Assert.assertEquals(Assert.java:115)
	at org.junit.Assert.assertEquals(Assert.java:144)
	at com.garygregory.util.LocaleTest.testBug2(LocaleTest.java:23)
	... 24 more

Why is that?

Under the covers, the call to toLowerCase() becomes toLowerCase(Locale.getDefault()), and of course getDefault() changes depending on your country. So what really happens in Turkey is toLowerCase(Locale.forLanguageTag("TR"))).

In the Turkish locale, the Unicode LATIN CAPITAL LETTER I becomes a LATIN SMALL LETTER DOTLESS I. That’s not a lowercase “i”.

If you index maps with String keys that include the letter I and you normalize keys to lowercase, your program will have a bug when running on the Turkish locale.

How are you supposed to deal with that?! Well, luckily, you can use a Locale that does not perform such conversions, namely Locale.ENGLISH and more generically, Locale.ROOT.

You defensively written code becomes one of:

Assert.assertEquals("title", "TITLE".toLowerCase(Locale.ENGLISH));
Assert.assertEquals("title", "TITLE".toLowerCase(Locale.ROOT));

If you write your code “in English”, the Locale.ENGLISH is fine, but consider Locale.ROOT for more general support.

Java defines Locale.ROOT as:

/**
 * Useful constant for the root locale. The root locale is the locale whose
 * language, country, and variant are empty ("") strings. This is regarded
 * as the base locale of all locales, and is used as the language/country
 * neutral locale for the locale sensitive operations.
 *
 * @since 1.6
 */
static public final Locale ROOT = createConstant("", "");

As a reminder, Java provides localization support with the java.util.Locale class, from the Javadoc:

A Locale object represents a specific geographical, political, or cultural region. An operation that requires a Locale to perform its task is called locale-sensitive and uses the Locale to tailor information for the user. For example, displaying a number is a locale-sensitive operation— the number should be formatted according to the customs and conventions of the user’s native country, region, or culture.

The Locale class implements IETF BCP 47 which is composed of RFC 4647 “Matching of Language Tags” and RFC 5646 “Tags for Identifying Languages” with support for the LDML (UTS#35, “Unicode Locale Data Markup Language”) BCP 47-compatible extensions for locale data exchange.

For further experimentation, the following Java system properties affect how Java initializes the default locale:

  • user.language
  • user.region
  • user.script
  • user.country
  • user.variant

Here is the full source code code for the test:

package com.garygregory.util;

import java.util.Locale;

import org.junit.Assert;
import org.junit.Assume;
import org.junit.ComparisonFailure;
import org.junit.Test;

public class LocaleTest {

  private static final Locale TURKISH = Locale.forLanguageTag("tr");

  @Test
  public void testBug1() {
    Assume.assumeTrue(Locale.getDefault() != TURKISH);
    Assert.assertEquals("title", "TITLE".toLowerCase());
    Assert.assertEquals("title", "TITLE".toLowerCase(Locale.getDefault()));
  }

  @Test(expected = ComparisonFailure.class)
  public void testBug2() {
    Assert.assertEquals("title", "TITLE".toLowerCase(TURKISH));
  }

  @Test
  public void testFix() {
    Assert.assertEquals("title", "TITLE".toLowerCase(Locale.ENGLISH));
    Assert.assertEquals("title", "TITLE".toLowerCase(Locale.ROOT));
  }

}

In summary, use toLowerCase(Locale.ROOT) instead of toLowerCase() for internal maps.

Happy Coding,
Gary Gregory