Java on Desktop Goes Native

Introduction

As more developers are catching the Jave desktop train this article should fill some gaps. In this article I describe a 100% Pure Java cross platform a solution for both development and deployment of Java desktop applications that look and behave like native applications. For added bonus the solution is free in every sence of the word [0]. The solution is part of my free (GPL) desktop application framework jApp which is freely available from my web site at http://www.sparetimelabs.com .

Background

jDraft is a 100% Pure Java 2D CAD freeware application that builds on my Free application framework, jApp.

At work we do most of the development with Borland JBuilder which can generate native executables for the tree main platforms: MacOS, Linux and Windows. On my spare time I prefer Eclipse [17]. Not only because I think it is an excellent tool but also because it is free, which is a thing to consider when one is developing an application with zero income model.

Going native

For the past five years Java Web Start [18] has been my only deployment mode for jDraft. Recently it has shown up on my radar that not everybody shares my enthusiasm for Web Start. And indeed the user experience is less than optimal in some key areas, especially in the integration with the Desktop regardless of weather your desktop manager is Finder, Explorer, Gnome or KDE.

My philosophy is that for a good user experience the user should not be aware that the application is written in Java. Rather the application should blend in with the crowd and look like any first class citezent. The Java Swing toolkit gives a passable, if not perfect, imitation of the native look and feel, and it should be even better with the up coming Java 6 release [21]. Reference [3] gives a good introduction to the subject. But on the desktop integration side the Java API is conspicious by absence.

The most fundamental aspect of a point and click user interface is just that, the user needs to be able to launch applications and open documents by just clicking at the icons on the desktop.

The icons should represent the functionality and content of the application and documents respectively.In addition to file associations I also wanted a solution that does not require installers. Although installers are common, it is a fact that an installer adds nothing positive to the user experience and mostly serves to patch shortcoming in the desktop's application deployment system.

Building the executables

I was also a litle disappointed that the Windows solution involved some commercial JNI code and a native DLL to do the necessary registry magic for file associations. This ment either sacrifying my Pure Java concept or no-installer goal.

A few companies make commercial turnkey solution , a Javalobby article [7] provides a good list of these.

Luckily I did not have to resort to them because I stumbled on 'jstub' [8], a free Ant task for packaging Java applications into native executables for Windows and Linux.

An interesting technical detail worth mentioning is how 'jstub' creates the native executables. This is based on the observation that the Java Virtual Machine apparently uses 'zlib' [22] compression library to read the compiled Java classes. Because 'zlib' does not mind that a file has a preamble before the actual compressed file starts it is possible to add most anything to the beginning of any executable '.jar' file. Just create a stub or bootstrap code that passes the file itself to Java and stick the '.jar' file at the end. For Windows the stub is written in C and for Linux it is actually a shell script.

A delightfull detail is that the even the code inside 'jstub' that inserts the icon resource into the .EXE is pure Java, making it possible to use it on any Java / Ant supported platform . Well done, OrangeVolt . The comple Ant task definition for building the executables is too long to include here but an outline the task defs is in Example 1.

<!-- build the mac os executable -->
<jarbundler name="MyApplication-MacOSX" 
	dir="./bin" 
	shortname="MyApp" 
	signature="myap" 
	icon="MyApplication.icns" 
	jar="MyApplication.jar" 
	mainclass="mypackage.MyAppMainClass" 
	infostring="MyApplication - Copyright (c) MyCompany">	
	</jarbundler>

<!-- build the windows executable -->
<jstub output="MyApplication-Win.exe"
	archive="MyApplication.jar" 
	execute="javaw -jar "${app:absolutepath}""  
	mode="win32" 
	icon="MyApplication.ico" />

<!-- build the linux executable -->
<delete file="MyApplication-Linux" quiet="true"/>
<jstub output="MyApplication-Linux"
	archive="MyApplication.jar" 
	execute="java -jar ${app:absolutepath}"  
	mode="unix" />
<chmod file="MyApplication-Linux" perm="a+rx-w"/> 	

Implementing the file associations

To do the necessary Windows registry hacking for the file associations JNI code still seemed necessary.

Fortunatelly all modern (NT based) Windows come with a command line utility REG.EXE which can be used to manipulate the registy. And with the Java API 'Runtime.execute()' method it is possible to execute command line tools of the underlying operating system. A pure Java solution, albeit a borderline case!

By making my jDraft application to do the association registration by itself on the first launch a separate installer becames unnecessary.Listing 2 is a Java class that implements a clean interface to the shell command line including the possibility to 'sudo' commands. With this 'Shell' class setting a Windows registry key is as eysy as obtaining a Shell instance and giving it a command line to execute as in example 2.

Shell sh=new Shell();
String cmd="REG ADD HKCR\\.myext /f /v \"\" /t REG_SZ /d MyApplicationKey", null);
sh.execute(cmd);

Listing 1 ( full listing here ), or browse javadoc for class Shell

public class Shell {
   public String execute(String cmd, final String input, int timeout) {
      m_Exception=null;
      OutputStream output = new ByteArrayOutputStream();
      m_Command = cmd;
      m_EditCode = -1;
      try {
         if (m_TraceCommands)
            System.out.println("execute: " + cmd);
         final Process cmdproc = Runtime.getRuntime().exec(cmd);
         final PrintStream stdin = new PrintStream(cmdproc.getOutputStream());
         InputStream stdout = cmdproc.getInputStream();
         createForkedPipe(stdout, m_TraceCommands ? System.out : null, output);
         InputStream errout = cmdproc.getErrorStream();
         createForkedPipe(errout, m_TraceCommands ? System.err : null, output);
         if (input != null) {
            stdin.println(input);
            stdin.close();
         }

         java.util.Timer timer = new java.util.Timer();
         if (timeout > 0)
            timer.schedule(new TimerTask() {
               public void run() {
                  cmdproc.destroy();
               }
            }, timeout);
         int res = cmdproc.waitFor();
         timer.cancel();
         m_EditCode = res;
      } catch (IOException e) {
         m_Exception=e;
         System.err.println(e.getMessage());
      } catch (Exception e) {
         m_Exception=e;
         e.printStackTrace();
      }
      if (m_TraceCommands)
         System.out.println("exit code: " + m_EditCode);
      return output.toString();
   }
}

Handling file associations on Windows is pretty simple and on Mac OS totally trivial but Linux with its multitude of distros, desktop managers and vague and shifting standards is a different story. I guess thas what freedom is fundametally about.

Be that as it may, it boils down to the fact that icon and file associations are handled in the popular Gnome and KDE desktop manager by writing some description files into specific system directories and then calling a system script or two to inform the desktop manager of the newcomers.

Although there is an ongoing effort to standardize the desktop manager interfaces [24] not every distro is up to speed and the standard leaves some important points open, like the location of some key scripts and config files. To overcome this I attempted to collect a list of all likely place an and work from there. This turned out to be more than I had bargained for and I was about to give up, when someone pointed out that all this has been worked out and the knowledge embedded into a set of scripts called XDG-Utils [23] .

Unfortunately, in the best tradition of Linux there is no guarantee that these scripts exist in the system and if they do, where they might be. To solve this chicken and egg problem I embedded the scripts as resources into my Java class XDGUtils, listing 2, which writes them out to the disk for execution. With this strategy I can execute these scripts like any other shell command using the the Shell class.

Listing 2

public class XDGUtils {
   static public String execute(Shell shell, String sudo,String command, String input) throws Exception {
      String output ="";
      output= shell.execute(sudo+"sh "+command, input);
      if (shell.getExitCode() != 0) { 
         String script = command;
         int i = script.indexOf(" ");
         if (i < 0)
            i = script.length();
         String args = script.substring(i);
         script = script.substring(0, i);

         File scriptFile = new File(System.getProperty("user.home") + File.separator + script);
         boolean useTemp = !scriptFile.exists();
         if (useTemp) 
            AssociationService.copyResourceToFile(XDGUtils.class, script,scriptFile);
         output = shell.execute(sudo + "sh "+scriptFile.getAbsolutePath() + args, input);
         if (useTemp)
            scriptFile.delete();
      }
      return output;
   }
}

A further complication with Linux desktop integration model and a strange concept for anyone coming from Windows is that in Linux one cannot just write to the system directories and execute commands as one sees fit. In Linux you need to run with root rights to do certain things. That is good from security point of view but it means that your Java application is likely to run with limited user rights making for example some of the installation related directories inaccessible.

Fortunately most distros include the command 'sudo' [12] which allows you to execute commands with root rights, if you know the password.

So by asking the user for the password and executing commands with 'sudo' most anything can be executed. Granted, you still cannot write to the system directories but you can write the necessary files temporarily to the user's home directory or some such place where the user has write access and then copy the files into the right place with root rights using 'sudo' and 'cp', the Linux copy command.

Using the AssociationService

To use the AssociationService you need to come up with various things like application name and icon, file extension, mime type and icon for each document type the application handles. Table 1 gives a list of all the things you need.

Icons need to conform to a naming convention and stored eiher in your project sources or as resource of some class as described in the table. An application ID is used as a part of icon file names and keys to Windows and Linux registries, so you should take care to pick up a unique name that is unlikely to conflict with other keys.

Mac OS signatures and file types can be registered to ensure uniquenes.

Mime types can be registerd and are usually (for application spefic file types) of the form "application/vnd.company.type".

Listing 3 (read full listing ) or browse javadoc for class Association

public class Association {       /**
        * Creates an imutable file association that relates file name extension (".myext"), 
        * mime type ("application/vnd.mycompany.myext"), icon ("myAppDocIcon.png") and
        * description ("A MyApp application Document")
        * @param extension the extension including the preceding dot
        * @param mimeType the mime type in the form type/subtype
        * @param magic the magic xml fragment, can be null
        * @param icon the icon name, including image type extension, excluding path
        * @param description end user description of this association
        */
       public Association(String extension,String mimeType,String magic,String icon,String description) {
          if (!extension.startsWith("."))
             throw new IllegalArgumentException("File type extension ('"+extension+"')should start with a '.'");
          if (mimeType==null)
             throw new IllegalArgumentException("Argument 'mimeType' must not be null");
          if (description==null)
             throw new IllegalArgumentException("Argument 'description' must not be null");
          if (icon==null)
             throw new IllegalArgumentException("Argument 'icon' must not be null");
          m_FileExtension=extension;
          m_MimeType=mimeType;
          m_Description=description;
          m_MagicPattern=magic;
          m_Icon=icon;
       } 
   }

Listing 4 (read full listing) or browse javadoc for class AssociationService

public class AssociationService {
   /**
    * Registers/unregister the application and file associations with the platforms Desktop.
    * Registers/unregisters the associations with the platforms Desktop Manager (Gnome/KDE/Finder/Explorer) so that
    * both the application and the documents have the correct appearance (icon) and can be launched/opened
    * by double clicking at the icon on the Desktop.
    * the application lauched b
    * @param install weather to install or uninstall
    * @param systemInstall install into system or for the user (Linux only)
    * @param password the root (admin) password for 'sudo'ing the regisration (Linux only)
    * @param mainClass the mainClass of the application
    * @param resourceClass the class that contains the resources (icons)
    * @param applicationID the internal name for the application (used as key in regisration)
    * @param applicationName the user name for the application
    * @param description the user description of the application
    * @param applicationDir the directory used to store registration related files, needs to be writable
    * @param associations list of associations to register
    * @throws Exception if something goes wrong
    */

   public void register(//
         boolean install,//
         boolean systemInstall,//
         String password,//
         Class mainClass,//
         Class resourceClass,//
         String applicationID,//
         String applicationName,//
         String description, //
         String applicationDir,//
         Iterable associations//
   ) throws Exception {
      assertNotNull("mainClass", mainClass);
      assertNotNull("resourceClass", resourceClass);
      assertNotNull("applicationID", applicationID);
      assertNotNull("applicationName", applicationName);
      assertNotNull("description", description);
      assertNotNull("associations", associations);
      assertNotNull("applicationDir", applicationDir);
      if (!(new File(applicationDir)).exists())
         throw new IllegalArgumentException("Argument applicationDir='" + applicationDir + "' which doens not exist.");
      if (associations.iterator().hasNext() == false)
         throw new IllegalArgumentException("Argument 'associations' cannot be empty.");

      m_Install = install;
      m_SystemInstall = systemInstall;
      m_RootPassword = password;
      m_MainClass = mainClass;
      m_ResourceClass = resourceClass;
      m_ApplicationID = applicationID;
      m_ApplicationName = applicationName;
      m_Description = description;
      m_Associations = associations;
      m_ApplicationDir = applicationDir;

      String osname = System.getProperty("os.name").toLowerCase();
      if (osname.indexOf("window") >= 0)
         windowsInstall();
      if (osname.indexOf("linux") >= 0)
         linuxInstall();
   }
}

In Linux the icon for the actual application executable file cannot be changed easily. The convention is that the executable is hidden in somewhere in the system and an 'Application.desktop' file takes it place on the users desktop. To get an idea on how to use AssociationService see example 3.

LinkedList associations=new LinkedList();
associations.add(new Association(
	".myext",
	"application/vnd.mycompany.myext",
	null,"MyApplicationDocumentIcon",
	"My Application Document");
assocService.register(
	true, // install it
	false, // into the system,
	null,// not password
	MyAppMainClass.class, // this is the main class
	MyAppMainClass.class, // and resource class,
	"MyApp", // applicationID
	"MyApplication", // real name
	"an Application for Doing MyStuff",
	System.getProperty("user.home")+".MyApplication",
	associations);

Packaging for distribution

To get the user's browser to download your nicely packaged application it needs to have a specific type.

For Windows the established format is '.zip' file and for Linux its '.gz'. The premier browsers for each platform, Explorer and Firefox, are configured, out of the box, to download files of these types. To automatically create both as part of your build process using Ant script is trivial with the 'zip' and 'gzip' tasks. (For those working on a budget there is the added advatage that even the most basic internet service providers usually provide download statistics by file type, if not by file, so you can track the number of downloads easily).

The above formats also work for Macintosh Finder but most applications seem to be delivered in the form of a mountable disk image '.dmg'. The advantage is that Safari, the Mac browser, automatically downloads, mounts and opens these images which then appear as ordinary folders. The user can then just drag the application to where he/she wants.

Creating a .dmg disk image is not trivial.

Fortunately Apple has had the foresight to create command line tools in addition to the showy graphical tools and some bright fellow has worked out the details [13], even if the instructions are slightly out dated by now. With command line tools and Ant's ability to execute them the build process of creating disk images can be totally automated.

Formatting the disk image to put a file system on it requires root priviledges which means either embedding the password into the build script or doing some interactive input during build process, neither of which is a good practice.

Instead I chose to pre-create a large and empty but formated disk image manually, which process needs admin right, and then use the build process to clone that, copy my Mac OS exececutable onto it and then compress the clone creating an optimally sized image, all of which can be done with normal user rights.

Example 4 outlines the Ant script that packages all three executables.

<!-- package mac executable into a .dmg disk image -->
<copy file="EmptyMyApplication-MacOSX.dmg" tofile="temp.dmg"/>
<exec executable="hdid" dir="." >   
<exec executable="cp" dir="." > 
	 
<exec executable="hdiutil" dir="." >   
<delete file="MyApplication-MacOSX.dmg"/>
<exec executable="hdiutil" dir="." >  
	 
<delete file="temp.dmg"/>
		
<!-- package the windows executable into a zip file-->
<zip destfile="MyApplication-Win.zip" basedir="." includes="MyApplication-Win.exe"/>
		
<!-- package the linux executable into a gzip file -->
<gzip  src="MyApplication-Linux" destfile="MyApplication-Linux.gz"/>

It presuppose the existence of an empty Mac OS disk image file. Correct naming is essential as the script refers to the volume both by filename and volume name. Example 5 gives the shell commands necessary to create that image.

hdiutil create -megabytes 10 EmptyMyApplication-MacOSX.dmg -layout NONE 
hdid -nomount EmptyMyApplication-MacOSX.dmg

sudo newfs_hfs -v MyApplication-MacOSX /dev/disk[number]
hdiutil eject /dev/disk[number]

In this article I've tried to explain the process using cleaned up and simplified examples. They should be usable "as is" out of the box, but of course there is the chance that I've messed things up in creating the example. Therefore, as always, your best bet is to go and look at the real source code. The complete and debugged build script and source code for the for the classes mentioned can be found from my website.

Nobody is perfect

Unfortunately that is not quite so. The 'chmod' Ant task only works on MacOS and Linux. chmod is necessary to make the Linux executable, well, executable. Of course this is not absolutely necessary, you can leave this task to the end user, but for best user experience the executable bit should really be factory set. One way around this problem would be to use the 'tarfileset' Ant task that allows setting the mode bits for files packed into '.tar' files which could then be packaged as'.gz' files, which is common practice.

The 'cp' command to copy files is not available in Windows and while Ant support the 'copy' Ant task it does not appear to produce a package that MacOS would regard as a proper functional '.app' package. I suspect that this is because 'copy' does not preserve file permissions which I believe is an inherent limitation of the Java file IO API which cannot not support this concept because Windows, the lowest common denominator, doesn't support it.

One way to get around the afore mentioned problems on Windows would be to install and configure Cygwin [19] which brings the power of Unix to Windows. It certainly support the 'cp' command and I believe it can simulate file permissions with the 'ntea' option [20] .

Also there is no way, as far as I know, to create .dmg images on any other platform than Mac OS X. Of course this is not absolutely required if you are happy to deliver your application in '.gz','.zip' or '.tar' archive, which should be almost as good. In summary, if you wan to develop native looking pure Java applications with the described methodology for all platforms then the best platform for development is Mac OS X. In Linux and Windows you need to sacrify packaging the Mac executables in '.dmg' files. Developing in Windows also means finding a way to set the file permission as outlined above.

A word of caution, I've not tested the alternative methods outlined above, after all, I've moved all my development to Mac OS X so I have no need to ...

Also worth mentioning is that at present 'jstub' does not support transparency on appication icons, which is a pitty. As it is an Open Source project you should be able fix this though [9]. While you are at it, please add support for the up coming Windows Vista icons too!

References