Monday, December 18, 2006

JRake, Part 3: Running

So far, we've managed to compile and test our Java code using 100% JRuby. Now we just need a way to run it.

Assuming our application is web-based, the obvious solution is to wrap it up in a war file and deploy it into a container. But as anyone who has worked on a large project knows, creating and deploying war files can be a painfully slow process. Our original mission was to cut as much time as possible off our edit-compile-run cycle, so this clearly isn't the right solution. (We will need to generate war files at some point, of course, for deploying into production environments, but we'll save that work for another day).

Some app servers have an "autodeploy" feature that lets you deploy your app into an already running server, which is a huge step in the right direction - though things can still take a fair bit of time to deploy. Even when autodeploy works, it requires an extra alt-tab (command-tab for you mac users) over to the shell to kick off the build, followed by an alt-tab over to the browser to see the results, which wastes another second or two. There's also timing issue: if you hit "refresh" in the browser before the build and deploy have finished, you won't see your changes (or worse, you'll see some of your changes). So while autodeploy is definitely a big improvement, it's still not an ideal solution.

What we really want is for Java code to work the same way that dynamically typed languages do: you change your code, hit "save", and view the results in the browser.

JRake Server

I think I've managed to achieve something close to this, by setting up a Jetty server to act as a kind of proxy for my main application. Whenever a request comes in, it performs the following steps:

  1. Calls out to the JRake script to compile any out-of-date code.
  2. Creates a new classloader, and uses it to reload the application's main servlet class.
  3. Creates an instance of the servlet and forwards the original HTTP request on to it.

The upshot is that I can now make changes in my IDE, and see those changes (almost) immediately in my browser. No new virtual machines are started up along the way, no extra alt-tabs are needed, and I don't have to worry about hitting the web page before the deploy is complete. These may seem like minor points, but when you're doing this stuff eight hours a day, those little annoyances can add up.

Here's the code for the main program, which starts up the Jetty server:

require 'java'
require 'reload_handler'

server = org.mortbay.jetty.Server.new(3030)
handler = ReloadHandler.new('lib/rakefile.rb', :main_compile, ['tmp/main/'], 'com.example.HelloWorldServlet')
server.set_handler(handler)

server.start
puts "\nHit stop server\n\n"
$stdin.readline
server.stop

And the code for the reload handler:

require 'java'
require 'rake'
require 'jrake'

class ReloadHandler < org.mortbay.jetty.Handler

# todo: Should extend AbstractHandler
# (need JRuby to support extending abstract classes)

def initialize(rakefile, target, classpath, classname)
super()

@rakefile = rakefile
@target = target
@classpath = classpath
@classname = classname
end

def handle(target,request,response,dispatch)
begin
# Compile any out of date files
Rake.application.clear
load @rakefile
target = Rake.application.lookup(@target)
raise "Target not found: #{@target}" if target.nil?
target.invoke

# Load the servlet class, create an instance, and delegate to it
servlet_class = load_class(@classpath, @classname)
servlet = servlet_class.new_instance
param_types = to_java_array(java.lang.Class, [
javax.servlet.ServletRequest,
javax.servlet.ServletResponse
])
method = servlet_class.get_method("service", param_types)
args = to_java_array(java.lang.Object, [request,response])
method.invoke(servlet, args)

rescue Exception => e
trace = "#{e.class}: #{e.message}<br/>#{e.backtrace.join('<br/>')}"
response.writer.write("
<html>
<body>
<pre>{trace}</pre>
</body>
</html>
")
end

request.handled = true
end

def start
end

def stop
end

def setServer(server)
@server = server
end

def getServer
@server
end
end

Again, I've set up a sample project that includes all the pieces - JRuby, Rake, Jetty, and associated scripts. You can get it from subversion here:

svn://svn.foemmel.com/blog/jrake/running

If you're interested in the scripts but don't want to download all the third-party stuff, just check out the lib directory:

svn://svn.foemmel.com/blog/jrake/running/lib

Monday, December 11, 2006

JRake, Part 2: Testing

In my previous post I showed how to integrate JRuby/Rake with javac, in a way that eliminated the need to start up a new virtual machine for each compilation. This was fairly straightforward - I used JRuby's high level java integration to load the compiler class from the system classloader, then invoked its compile method.

Next on my list was integration with JUnit, which turned out to be a bit more complicated. I was able to load the main JUnit classes (e.g. org.junit.runner.JUnitCore) using the same high level mechanism as before, but for the actual test classes, I had to use a separate, non-system classloader. That way I could specify the location of my test classes from within the rakefile (as opposed to having to include those directories on the system classpath) plus I could recompile and rerun my tests without having to start up a new virtual machine.

In the end I got it working without too much trouble. The junit method below takes two arguments: the location of the directory containing the test classes, and the classpath to use when running the tests. It then creates a new classloader for the specified classpath, uses it to load the test classes, and passes those test classes into JUnitCore for execution:

def junit(test_class_dir, class_path)

# Append the test_class_dir to the class_path, if necessary
class_path += [test_class_dir] unless class_path.member?(test_class_dir)

# Make sure test_class_dir has trailing slash
unless test_class_dir[-1,1] == '/'
test_class_dir = test_class_dir + '/'
end

# Scan test_class_dir for test class files
class_names = []
FileList["#{test_class_dir}/**/*Test.class"].each do |class_file|
class_names << class_file[test_class_dir.length,
class_file.length - test_class_dir.length - '.class'.length].gsub('/', '.')
end
fail "No test classes found" if class_names.empty?

# Load the test classes via a new classloader
classes = load_classes(class_path, class_names)

# Run the tests
runner = org.junit.runner.JUnitCore.new
runner.add_listener(org.junit.internal.runners.TextListener.new)
result = runner.run(to_java_array(java.lang.Class, classes))
fail("Unit tests failed") unless result.was_successful
end

def load_classes(class_path, class_names)

# Make sure directories have a trailing slash, otherwise URLClassLoader ignores them
class_path.each { |element| element << '/' if FileTest.directory?(element) && element[-1,1] != '/' }

# Convert classpath elements to URLs
urls = to_java_array(java.net.URL, class_path.map { |element| java.net.URL.new('file:' + element) })

# Create a class loader for the specified class path
loader = java.net.URLClassLoader.new(urls)
return class_names.map { |class_name| loader.loadClass(class_name) }

end

Unfortunately, I had to use the internal JUnit TextListener to display the results of the tests, since JRuby doesn't currently support extending existing Java classes in such a way that I could provide my own subclass of RunListener. This will hopefully be fixed in a future version of JRuby.

Interactive JRake

Now, while all these classloading tricks are nice, they don't really help if all we do is start up the virtual machine, run the tests, shutdown, and repeat, since the original goal was minimize those startups. I've therefore created an "interactive" version of JRake, to keep the virtual machine warmed up between tests. All it does is sit in a loop, prompting the user for targets to build, and building them. A sample session looks like this:

jrake shell v0.0

Valid targets:
clean - deletes all generated files
main_compile - compiles the main code
unit_compile - compiles the unit test code
unit_test - runs the units tests
help - displays this help text
exit - quits the application

A blank line repeats the previous command.

jrake> main_compile
compiling 1 java file(s)...done
jrake> unit_test
compiling 1 java file(s)...done
.
Time: 0.009

OK (1 test)

jrake> clean unit_test
deleting 10 file(s)...done
compiling 1 java file(s)...done
compiling 1 java file(s)...done
.
Time: 0.002

OK (1 test)

jrake> exit

You can download the entire sample project via subversion here:

svn://svn.foemmel.com/blog/jrake/testing

Just run the "build.bat" or "build.sh" scripts with no arguments to bring up the JRake Shell. If any arguments are passed in, the behavior is the same as before i.e. the targets will be built then the script will terminate.

Next Steps

Now we just need to get the app up and running...

Wednesday, December 6, 2006

JRake, Part 1: Compiling

For as long as I've been working with Java, I've been in search of a build tool that didn't drive me, and those around me, bat-shit crazy. I've come to realize I have two somewhat conflicting requirements for such a tool:

  1. Must be based on a scripting language

    Builds get complicated, and I need a tool that will let me do arbitrarily complex things, preferably in a well-known scripting language with lots of supporting libraries.

  2. Must run on the Java platform

    The Java virtual machine takes time to start up. Since most tools that deal with Java code (e.g. javac) are themselves written in Java, this means there is a rather annoying interval between when you invoke them, and when they start doing actual work. This delay may seem insignificant, but on a large project where a build consists of many calls to many tools and gets run many times a day, all that wasted time can really harsh your flow.

    The solution is for the build tool to treat all of the supporting tools as libraries, not command line applications. For example, instead of calling javac directly, just invoke the method com.sun.tools.javac.Main.main(String[]). That way you only incur the VM startup time once.

The tools I've used in the past don't meet these requirements, of course. Specifically:

  • make fails on both counts (along with most other tools).

  • ant fails on the first count, as it's not based an existing scripting language. While it's true that you can drop down to Java and do complicated things by writing your own tasks, the context switch is pretty jarring. One problem is that you have the extra step of compiling your tasks before you compile your code - and therefore your build script now needs a build script. But the real issue is that Ant tasks only let you put custom logic at the task level, not at the level that actually manages the tasks. For example, if you want to define some dependencies between projects in one place, and use that to drive a bunch of build steps for each project, you're in for a tough time. It can be done (we used XSLT to generate ant scripts on one project) but it ain't pretty.

  • rake, running on top of standard C-Ruby, fails to meet the second requirement. However, Rake itself is a damn nice tool. See Martin Fowler's article for a good summary.

Rake and JRuby

The solution, obviously, is to run Rake on top of JRuby (I will henceforth call this two-headed monster "JRake"). That way we have the power of Ruby at our disposal, with all the snappy goodness of a JVM based tool. The trick is to get JRake to invoke the main javac class directly, without starting a new VM. This actually isn't much of a trick, since integrating with Java is exactly what JRuby was designed for.

And so, after a bit of trial and error, here is a rakefile that does just that. It compiles any out-of-date java files in the "src" directory and puts the resulting class files in the "tmp" directory:

require 'java'

task :default => :compile

task :compile do
src_dir = 'src'
dest_dir = 'tmp'

Dir::mkdir(dest_dir) unless File::exist?(dest_dir)

javac(src_dir, dest_dir)
end

def javac(src_dir, dest_dir)
java_files = get_out_of_date_files(src_dir, dest_dir)

unless java_files.empty?
print "compiling #{java_files.size} java file(s)..."

args = ['-d', dest_dir, *java_files]

buf = java.io.StringWriter.new
if com.sun.tools.javac.Main.compile(
to_java_array(java.lang.String, args), java.io.PrintWriter.new(buf)) != 0
print "FAILED\n\n"
print buf.to_s
print "\n"
fail 'Compile failed'
end
print "done\n"
end
end

def get_out_of_date_files(src_dir, dest_dir)
java_files = []
FileList["#{src_dir}/**/*.java"].each do |java_file|
class_file = dest_dir + java_file[src_dir.length,
java_file.length - src_dir.length - '.java'.length] + '.class'

# todo: figure out why File.ctime doesn't work
unless File.exist?(class_file)
&& java.io.File.new(class_file).lastModified > java.io.File.new(java_file).lastModified
java_files << java_file
end
end
return java_files
end

def to_java_array(element_type, ruby_array)
java_array = java.lang.reflect.Array.newInstance(element_type, ruby_array.size)
ruby_array.each_index { |i| java_array[i] = ruby_array[i] }
return java_array
end

If you want to play around around with this script but can't be bothered to install JRuby, Rake, and the other bits, I've created a complete "Hello World" project that includes everything you need. You can get it from subversion here:

svn://svn.foemmel.com/blog/jrake/compiling

Just checkout the project, make sure JAVA_HOME points to your JDK, and run the "build" script to see it go.

Next Steps

I've gotten JUnit and a few other cool things working with JRake, which I'll write about in my next post.