I maintain the Grails GWT plugin -> http://grails.org/plugin/gwt that was inherited from Peter Ledbrook
This is a nice little plugin, that does quite a lot of funky things in grails scripts, including managing its own dependencies via Ivy.
Being a GWT integration, one of its jobs is to compile the GWT Java into javascript for running in a production setting.
There was already and implementation from the first days of the plugin, of course, that handled this. It had an issue, in common with all GWT based frameworks. Compiling is slow and serial. So, if you have 30 modules (like I do), they will compile one after the other.
There are some tweaks you can add of course, like the GWT worker threads option on the compiler. This purports to add parallelisation to the compilation, and indeed it does. My profiling, however, shows that the threads only get fired up in the last stage of the compilation, converting the AST into javascript variants. This leaves the first bit of the build, (which takes most of the time!), serial.
If you watch your cpu (assuming you have more than one core), you’ll see one core bearing most of the load, and the others firing up occasionally, normally at the end of the build.
This was taking ages (getting to 40-50 mins on a fast dev machine), which isn’t really sustainable.
So, as a method of speeding things up, I investigated a method of using all the CPU resources available.
I implemented a Groovy class that duplicated the current compiler invocation and added a Java 6 Executor pool that let me call the compiler once for each module and have as many in parallel as there were cores available (it auto detects this). This works well, cutting the compile time down by at least a third, sometimes more.
It did leave me with duplicated code though, some in the Grails script and then a copy in the Groovy class. This made me unhappy, its a maintenance headache.
Really, I want to have the logic (an invocation of AntBuilder with a bunch of classpath and environment setup) in the script, as its used for more than just the compilation, its also used for running the GWT client and other tasks.
So, I moved all this logic into the script and passed it in via a Closure to the class. All works fine when using the serial mode (that you know and love).
When in parallel mode, however, it got ugly. I kept hitting wierd ArrayIndexOutOfBoundsExceptions whenever I ran it, and quickly gave up trying to fix it (I have other work to do too…). I revisited it again today, as part of the prep for the next release of the plugin and realised I’d been cavalier with my thread management.
You see, Grails scripts expect to be single threaded, none of their environment is thread safe in any way, especially the global AntBuilder instance.
Whenever I ran in parallel compilation mode, up half a dozen threads at once would hit the global AntBuilder instance via my shared code in the script.
The solution is suprisingly simple and easily understood if you have some threading background. If you have threads, don’t share state unless you are very, very careful
My code was originally :-
gwtJava = { Map options, body ->
..some setup...
ant.java(options, body) <--------- ArrayIndexOutOfBoundsException
}
Here, we are passing a body to be evaluated by the common java run (set up in a particular way).
This is broken when run multi threaded, instead we make our resources local and ensure we liberally clone and Closures we are passed to make sure shared state is minimised. I also altered ‘body’ to accept a param that it calls ‘ant’, so overriding the globally accessed ant instance. It would be possible to do via changing the Closure delegate as well.
gwtJava = { Map options, Closure body ->
def localAnt = new AntBuilder()
body = body.clone().curry(localAnt)
localAnt.java(options, body)
}
The block is now safe to be accessed by as many threads as you like!
