Calling a Clojure Function With :keys Destructuring From Java

I had the occasion to do some hardcore Java-on-Clojure interop at my day job today.

Like this: we have a Clojure function with a signature like

(defn clojure-function [& {:keys [host port]
                           :or {host "localhost", port 8080}}]
  (println "host:" host)
  (println "port:" port))

So parameters to clojure-function are expected to take the form (:host "myhost" :port 1234). Via the magic of the :keys directive, the values are bound to local variables named "host" and "port", corresponding to the keywords :host and :port from the input. The :or bit provides default values if one or the other (or both) of the expected keys is not specified.

This is a very useful idiom in Clojure, one which permits named or "keyword" parameters, which I think are easier to deal with than regular positional parameters. But the need to invoke this method from Java poses some ... shall we say "fun" challenges.

Challenge the First: There is nothing in Java that corresponds to the Clojure keyword. Try to invoke a method like LonoCloudClass.clojure-function(:host, "myhost", :port, myport) and the compiler will immediately complain that :host is invalid syntax. Great. So we'll probably want to pass those things in as strings.

Challenge the Second: The :keys directive and behavior implies a key-value mapping. So it would seem reasonable to pass in an associative data structure like, say, an instance of java.util.Map. Or even a Clojure map literal. That's easy enough to do from the Java side, but, perversely, the Clojure side then complains with a message like "java.lang.IllegalArgumentException: No value supplied for key: {:host "somehost", :port 1234}".

Notice what the exception message claims is the key with no value: it's the entire map! So the map is not what gets destructured by the :keys directive; what gets destructured is a sequence of things that can be grouped in pairs. The first item of each pair needs to be a Clojure keyword, which gets a local variable named after it. The second item of each pair becomes the value of that variable.

Armed with this knowledge, we can take a shot at meeting both of these challenges.

On the Java side:

- Build a java.util.Map with java.util.String instances as keys and some Object type as values.

- Invoke -javaFunction (see definition below) with the Map as its parameter

On the Clojure side:

- Write a function (-javaFunction) that takes a java.util.Map as its only parameter

- Convert the map into a Clojure sequence in which every other item is a keyword

- Pass the sequence as the parameters to the actual Clojure function

(defn- keywordize [coll]
  (apply concat (for [[k v] (into {} coll)] [(keyword k) v])))

(defn -javaFunction [args]
  (apply clojure-function (keywordize args)))

Note that in order to complete the cycle you will need to AOT (ahead-of-time compile) the Clojure namespace that defines -javaFunction and clojure-function. You will also need to add a :gen-class directive to your namespace declaring javaFunction as a static method in the generated Java class. Something like this:

(:gen-class
 :methods [#^{:static true} [javaFunction [java.util.Map] void]])

Also note that the strings that serve as the keys/keywords need to not be prefaced by a colon on the Java side. The Clojure function (keyword "thing") will return a result of :thing, which is exactly what we want in this case.

Building Modern Clojure-now-with-reducers! with JDK 6

The Clojure times, they are a-changin'. Time was, an enterprising young hacker could just clone the Clojure repo from GitHub, run ant to build it, then hack h[er|is] way to [fame|glory]. Now, with the advent of reducers, those days are sadly gone. Or at least slightly delayed.

You see, reducers require some ultra-modern Java-isms; namely, java.util.concurrent.ForkJoin[Pool|Task]. If you have JDK 7 installed as your default JDK (or can arrange to so install it), you have everything you need and your Clojure build should Just Work™.

If you're stuck with JDK 6, you may have a bit more work to do. Your main symptom will be that your Clojure build does *not* Just Work™, but rather fails with an exception of the form java.lang.ClassNotFoundException: jsr166y.ForkJoinPool.

The Clojure Maven build now declares a dependency on an artifact called jsr166y, which provides an implementation of the needed ForkJoin... classes. Running mvn package will detect the new dependency, download the correct jar to your ~/.m2/ directory, and include that jar on the build classpath.

If you prefer to use ant instead of Maven (as I do), you have yet more work to do.

One thing you could do is download the jsr166y jar, stash it somewhere, and manually add it to your classpath. That seems kind of tedious to me; but I guess your tolerance level for tedium could be higher than your tolerance level for Maven.

Luckily, the Clojure core team has anticipated your preference for ant, and they want to alleviate your tedium. Unluckily, their solution actually uses Maven behind the scenes; you ant purists may want to stop reading at this point and just go download the jar and add it to your classpath.

Right before they committed the changes to implement reducers, the Clojure core team added the Maven dependency on jsr166y. Also, for medium-core ant fans, they added a one-time script at the top level of the project: antsetup.sh. Run that script once to create a new file: maven-classpath.properties (and download the jsr166y jar from some Maven repo if you don't already have it). The ant build now includes that properties file and adds the classpath it specifies to the ant build classpath and you're good to go.

Oh. Except for the whole having to sit on the sidewalk ringing a bell and chanting "Unclean!" at regular intervals. But that's a small price to pay for Clojure [fame|glory], right?

Starting a Java Process in the Background from Clojure

clojure.java.shell/sh is a beautiful thing. With it, you can execute a shell process from Clojure, get its output, and generally work your will.

Unless, that is, the shell process you want to execute is a long-lived one, and you want to kick it off from Clojure and let it run in the background while you accomplish other great things with Clojure.

You can observe this behavior with a simple experiment. In a Clojure REPL:

user=> (require 'clojure.java.shell)
user=> (clojure.java.shell/sh "bash" "-c" "sleep 10 &")

Ten seconds should give you time to switch to another terminal window and verify that the process is indeed running in the background. But you will not get your REPL back until that ten seconds is up.

The thing is, sh will hang around and wait until the process finishes, whether or not it's in the background. I'm not sure just why this is. sh uses Java's Runtime.exec() to do its magic; I may update it to use ProcessBuilder.start() instead and see if that helps. I'm not hopeful.

Until then, I've found a workaround for the specific case that's been bugging me. All I have to do is redirect stdout and stderr in the command that I send to the shell. Something like:

clojure.java.shell=> (sh "bash" "-c" "sleep 10 1>/tmp/mylog.log 2>&1 &")

There's a downside to this -- normally you would be able to inspect the return value from sh and see what the stdout of the process looks like. For short-running processes, this is probably exactly what you'd want to do. For long-running processes, you probably don't want sh hanging indefinitely just so you can eventually inspect its output. So you can always open the log file and deal with the output there.

For my specific problem, though, I want to kick off the java process in the background *and* record its pid. To do that, I simply append a second command to my command string:

(sh "bash" "-c" "sleep 10 1>/tmp/mylog.log 2>&1 & \n echo $!")

Now when the shell exits, the value of the :out key of the result has the pid of the new background process. The \n is necessary because neither of the other usual ways of combining multiple commands on one line (&& and ;) seems happy when following a single ampersand.

Installing Xcode 4.3 Confuses Homebrew (Briefly)

I have a lot of catching up to do, iOS-development-wise. Don't get me wrong, I absolutely love the work I get to do at my day job, which is All Clojure All The Time. But that doesn't leave a lot of time for iOS development.

In an effort to not fall too far behind, I installed Xcode 4.3, which promptly trashed my /Developer directory (not a bad thing, really). Homebrew was supposedly updated to cope, but when I tried to brew install graphviz today, it complained about not finding the developer directory there.

I was able to convince graphviz to install without resorting to symlink-ing, using the following incantation:

sudo xcode-select -switch /Applications/Xcode.app/Contents/Developer/

clj-json breaks when running 'lein javac'

I'm working on a potential patch to Mark McGranaghan's excellent clj-json library for converting Clojure data to JSON. Unfortunately, I can't test my changes right now, because of an incompatibility between Leiningen version 1.6.2 and clj-json version 0.4.3, which causes lein javac to throw an exception.

It looks like clj-json has a dev dependency on an old version of a lein-javac plugin for Leiningen, which itself depends on ant-launcher version 1.6.5. It also looks like modern lein is smart enough to use its built-in javac task in preference to the old plugin, but it gets tripped up on the old version of ant-launcher.jar. The specific exception is:

Exception in thread "main" java.lang.NoSuchMethodError: \
org.apache.tools.ant.launch.Locator.fromJarURI(Ljava/lang/String;)\
Ljava/lang/String; (NO_SOURCE_FILE:0)

I've submitted an issue on the clj-json project, but for now a simple workaround is to delete the dev dependency from clj-json's project.clj file, remove everything from its lib/ directory, and run lein deps to refresh dependencies.

Pro Tip: Fullscreen-ness of Apps Seems to be Sticky in Mac OS Lion

I've had a lot of fun the past couple days working with the Terminal app in fullscreen mode. Having entered fullscreen, I can work without distraction, but I can also switch freely back to my desktop or other running apps. Best of all: when I quit Terminal and restart it, it automatically restarts in fullscreen mode. Good job, Apple!

Installing Java on AWS EC2 Instances with Pallet

I have recently been spinning up a lot of t1.micro instances on Amazon's EC2 service. I'm deliberately using minimal / default images, and I need to install Java on them from scratch. There have been reports (and I have experienced myself) that the Java install process hangs indefinitely when installing either Oracle's JRE/JDK or OpenJDK on a t1.micro running Ubuntu. I have been able to install Java successfully by explicitly requiring a 64-bit version of the OS.

Here's how you can do that in your Pallet node-spec:

Starting an EC2 Instance with an Admin User Using Pallet

At my new day job today, I learned the hard way that if you're using Pallet to create an EC2 instance, and you want to use its automated-admin-user crate (which authorizes the user running the crate to login on the new instance via SSH using your RSA key), you really *really* want to run that crate during the :bootstrap phase. Running during the :configure phase does not seem happy using my current configuration of Pallet version 0.6.4 and jclouds version 1.0.0. Here's the relevant Clojure snippet:

(require 'pallet.core)
(require 'pallet.phase)
(require 'pallet.crate.automated-admin-user)

(def server-spec
  (pallet.core/server-spec
   :node-spec node-spec
   :phases {:bootstrap (pallet.phase/phase-fn (pallet.crate.automated-admin-user/automated-admin-user))}))

P.S.: LonoCloud is hiring Clojure developers! Join us!

Implementing Continuations in iOS Using Blocks

Came up with a neat use for blocks in iOS at my day job today: continuations!

Like this: I'm working on a native iOS app that occasionally shows data from a Rails website in a UIWebView. When it does so, it has to make sure the user of the iOS app is authenticated to the web app. This way we should never have to see the web app's login page in the native app, which has its own login screen.

Of course, all of this reeks of asynchronicity. Which usually means callbacks. Which usually means implementing one or twelve protocols. We've tried to encapsulate all of the authentication logic in a custom class that's used by the controller that presents the UIWebViews, but we've also recently come up with a need for multiple controllers, controlling multiple web views. And the code is already (ahem) not as attractive as it could be.

Our original workflow looks something like this:

* Render a UIWebView and invoke its loadRequest: method with some URL.
* Have the controller implement the UIWebViewDelegate protocol.
** In particular, implement webView:shouldStartLoadWithRequest:navigationType:.
** Ask the authenticator object if we need to refresh authentication.
*** If not, just load the request (happy path).
*** If so, tell the authenticator object to refresh authentication and return NO.
*** This may require collecting new authentication credentials from the user, which means another controller and view (and asynchronous request cycle) entirely.
*** It certainly requires a separate asynchronous call to the web app, to verify authentication and set the timeout for this session.
*** Oh yeah -- we need to have remembered the original request and restart it when the dust settles.
* Insanity ensues, in the form of delegates with delegates and very hard-to-follow callback trails and state management.

Enter: blocks.

With blocks we can streamline, thusly (starting with webView:shouldStartLoadWithRequest:...):

* Tell the authenticator object to make sure we're authenticated, passing in a block to execute once we are.
** This block closes over all the state we need to "do the next thing". It can refer to methods in the current controller and access the current controller's instance variables, even though it'll be executed by the authenticator object.
* If the authenticator decides we're already authenticated, it simply executes the block (the happy path).
* If we're not authenticated, but we already know our credentials, open a NSURLConnection to the web app, using the usual callbacks. When that connection finishes successfully, execute the block we were given from the original controller.
* If we're not authenticated and we need new credentials, push the login screen controller on the stack, passing *it* a block that does the NSURLConnection stuff, followed by executing the block from the original controller.
* The login controller presents a modal dialog, accepts the user's input for authentication credentials, then calls its block (the one passed from the previous step, which encapsulates "what to do next" at two levels).

When we can implement this strategy pervasively, every entity will simply ask its collaborators to do whatever they need to do, however they need to do it (including asynchronously), and know that once the task of e.g., login, is accomplished, whatever is supposed to happen next actually does.

This idea was inspired by Continuation Passing Style in compiler theory, in which every function is called with the data it needs to perform its own business, *plus* information about what to do next. When programs are expressed in this style, there is no recursion, no nesting of routines; there is only "do what you're supposed to do, then jump to the next thing". It's not a perfect analogy, but it has been surprisingly effective in making some pretty complex and ugly code easier to understand.

You're Going Down, Emacs!

Time to get serious, yo:

[david@davids-MacBook-Air-3 ~]$ cat .aliases
alias vi='emacs'
alias vim='emacs'
alias nano='emacs'
alias mate='emacs'

iOS Pro Tip: Know Thine API!

I was reading through some code at my current project (P.S.: ThoughtWorks is hiring! Contact me if you want to submit a résumé!), when a good old-fashioned C-style for loop caught my eye. You know the kind:

for (int i = 0; i < someArray.length; i++) {
  // do something with someArray[i]
}

It turns out that if someArray is an NSArray, there is API defined to help with this:

[someArray enumerateObjectsUsingBlock:^(id obj, NSUInteger idx, BOOL *stop) {
  // do something with obj
  // set stop to true if you want to skip the remaining objects for some reason
}];

Note: Yes, I'm aware that there is also syntax for this in Objective-C 2.0, in the form of fast enumeration. The code I was converting was specifically interested in returning the index of a particular object in the array, which is why I went with this form.

Anyway, further reading in the NSArray API led me to another fun method: indexOfObjectPassingTest. It also takes a block, which returns a boolean value indicating whether or not the current object passes the test. If it does, you get the index back and move on with your life. This turned out to be exactly what I needed to port the original bit of code to full modernity.

I also came across an example of code that was using NSString +stringWithFormat. It would create the original format string, conditionally modify it based on the presence of some optional data, then pass all available data in to format the final string. Something like this:

NSString *formatString = @"required: %@";
if (optionalData) {
  formatString = [formatString stringByAppendingString:@", optional: %@"];
}
NSString *result = [NSString stringWithFormat:formatString, requiredData, optionalData];

This seemed potentially error-prone to me; I didn't like the idea of always passing in the (potentially nil) optional data, even if I was convinced there would be no placeholder for it in the format string.

A quick read through the NSString class reference led to the instance method stringByAppendingFormat. This allowed for a workflow more like the following:

NSString *result = [NSString stringWithFormat:@"required: %@", requiredData];
if (optionalData) {
  result = [result stringByAppendingFormat:@", and optional:%@", optionalData];
}

This way I did not have to dance around getting the format string just right before populating it; I could tack on the extra bits on the fly, which just seems cleaner.

iOS Pro Tip: textFieldShouldClear

If you're using UITextFieldDelegate to manage the state of a save button (e.g., to disable saving if a field is empty), you're probably implementing the textField:shouldChangeCharactersInRange:replacementString: delegate method and checking for the field being empty. This will handle the case where the user causes the text field to become empty by deleting all characters, or selecting and cutting them. In this case, you should probably also implement textFieldShouldClear:, to handle the case where the user clicks on the text field clear widget (if it exists). It turns out that action does not call the ...shouldChangeCharacters... callback.

Versioning Core Data Models and Unit Testing with Xcode 4

Today at my day job I got to create a new version of a Core Data model. The Good News: automatic migration really does Just Work in the simple case of adding a new attribute (column) to an entity (table). The Bad News: Unit Tests do not necessarily automatically get the new version of the model.

For reference, I'm using Xcode 4.0.2, build 4A2002a. Also using Google Toolbox for Mac and OCMock for unit testing.

For some reason, my unit test target (we're using Google Toolbox for Mac and OCMock for unit testing) was originally set up with the following files targeted for various build phases:

• Compile Sources: MyModel.xcdatamodel (the original version, underneath the .xcdatamodeld directory)
• Copy Bundle Resources: no model

After creating the new version of the data model, we really want the .xcdatamodeld (the whole directory structure that knows about all versions) to be what gets compiled / bundled.

• Compile Sources: MyModel.xcdatamodeld
• Copy Bundle Resources: MyModel.xcdatamodeld

So I had to delete MyModel.xcdatamodel from the Compile Sources build phase and add MyModel.xcdatamodeld. The really weird thing was I could not get Xcode to add the file using the add dialog in the "Choose items to add" dialog that the plus sign brings up.

Note: to see the "Choose items to add" dialog I'm referring to in Xcode 4, do this:

• Click on the Project name in the Project navigator
• Click on a Target
• Click on "Build Phases" at the top of the project info section
• Expand the "Compile Sources" phase
• Scroll to the bottom of "Compile Sources" and click on the "plus" sign (+)

Using this dialog, Xcode will allow you to select any of the .xcdatamodel files, but not the .xcdatamodeld. The only way I could get the .xcdatamodeld file in "Compile Sources" (and "Copy Bundle Resources") was by clicking on the file in the Project navigator and dragging it into the build phase. Then it worked like a charm.

One other bit of weirdness: A lot of blogs that talk about versioning Core Data models don't mention how to set the selected version in Xcode 4. To do this:

• Select the .xcdatamodeld file in the Project navigator
• Open the Utilities pane (in the upper right corner of Xcode; see image)

• Select the current version from the Versioned Data Model section of the File Inspector (see image)

Final note: The app pictured in the images, TripCents, is not my current client's app; it's one of mine.

Mac Pro Tip: macerror

Ran across an obscure error code working on an iOS app at my day job. It turns out there's a utility on the Mac that will give you a little more to go on than some random negative integer: simply run

macerror [error code]

to get a (slightly) more descriptive version. For instance, I was getting error code -25299 trying to add an entry to my app's keychain; macerror reports that as errKCDuplicateItem.

[app next]: TripCents is now in the iTunes store!

My latest app, TripCents, is now available. It lets you store a collection of named trips and calculates the approximate cost of gas, given the distance for each trip, your car's estimated gas mileage, and the current local cost of gas. I wrote it to help me gauge how much to budget for gas each month, and also to educate my kids as to how much money we actually spend shuttling them around town.

I'm particularly happy about the chance this gave me to flex some new iOS developer muscles. Where TreadCalc was essentially a single-screen app with no persistence, TripCents uses CoreData for persistence, has actual multi-page navigation and editable forms, and it renders its views using custom UITableViewCells. It also offers a lot of opportunity for incremental improvements, such as adding the ability to use Google's APIs for geocoding and directions, and MapKit to render geocoded destinations nicely.

You can read more about TripCents on my RuppWorks, LLC website, or if you just can't wait you can go straight to the store.

Mac Pro Tip: Preferences .plist files

I love my new MacBook Air 11". Used it for most of my flight from Atlanta to Chicago today (about 1.5 hours flying time), and for once didn't have to worry about the guy in front of me leaning his seat back all the way into my lap. Also had a reported ~5.5 hours of battery time left when I had to shut down. The only thing I would do differently next time is get maximum RAM (currently sporting 2GB versus a potential max of 4GB).

When I'm at home, I run the MB Air in clamshell mode connected to a 20" cinema display (old school, baby; that's how I roll). Works great, everything displays where it should in either mode.

Except ...

There's this one app, RSA SecurID, that I occasionally need to connect to ThoughtWorks' sites. And it is decidedly not smart about repositioning itself. I usually have it dragged down out of the way, to the bottom right corner of the cinema display, but today while traveling I found out it always stays off the screen, even without the cinema display connected. Exposé will show it to me when I switch to the app and Show Application Windows, but when I click on it, it zips back off to the (now non-existent) bottom right corner of the cinema display. Restarting does not help, nor does gathering windows nor detecting displays in the Displays settings panel.

Bummer.

So I had to resort to opening the .plist file for the app (located for me at ~/Library/Preferences/com.rsa.Software Token\Desktop.plist) and manually updating its two properties: LastXPosition and LastYPosition. Supposedly, .plist files can be edited with any text editor (e.g., TextEdit), but this one seems to be in some binary format; I ended up have to use Xcode's plist editor to be able to make my changes.

Further Pro Tip: Make sure to tab or click out each field that you've edited. If you just type your change in place and save, it will not take.

First App: TreadCalc!

My first app for iPhone -- TreadCalc -- is live in the iTunes store! It's a little utility I developed for when I have to run on a treadmill and want to know what speed to set to run a given distance in a given time. There are other apps in the store that do this, of course, but I think mine is the only one that does it all on one screen -- no settings, no info; nothing but you, a few buttons (for switching between miles, kilometers, and time entry), and a couple of dials. As a bonus, you can use it as miles-to-kilometers conversion calculator, at least for distances up to 26.9 miles (just over the length of a marathon).

Check it out, yo!

Mac Tech Tip: chflags

I'm in the throes of my very-nearly-annual clean install of Mac OS X to my MacBook Pro (just to generally de-gunk; no other pressing reason). Of course this means I have to reinstall a lot of software (note to self: Hey self! You still need to restore World of Warcraft from your backup, stat!). Dropbox is helping me a lot this year; this time I don't have to recreate e.g., all of my password manager entries. That would be bad.

One of the more obscure things that I take for granted, always lose when I reinstall, and immediately miss is my symlink to /usr, which I like to create so I can use Finder to browse it. This year I decided to see if there was some better, native way to expose /usr to the Finder, and it turns out there is: chflags. One quick call to sudo chflags nohidden /usr, and I'm happily browsing. No muss, no fuss, no stretchmarks.

First Clojure Patch (ding)!

In celebration of the awesomeness that was ClojureConj The First, I just submitted my first patch to the Clojure core codebase! Still waiting for the Lumbering Wheels of Bureaucracy to turn the Massive Gears of Pending Assembla Membership to open the Gates to the Kingdom of Those Who Have Had a Patch Accepted. But the hardest part is done!

I'm just giddy with excitement. Or sleep deprivation. Or both. Probably both.

Using Haml with Rails 3.0

One of the things I [heart] about being a Ruby/Rails developer for ThoughtWorks is that I occasionally get asked to do something really cool on the side, like being a technical reviewer for an upcoming book on Rails 3.0.

In this particular book, the author uses Haml instead of Erb for templating views in the sample code. When I'm reviewing a book, I like to actually exercise as much of the sample code as I reasonably can, as it tends to be among the hardest stuff to get right, and among the easiest ways to discourage a reader when it's not right. But Rails defaults to Erb for templating, and I'm not exactly Haml-enabled.

What's a geek to do?

Well, my friend Google had a thing or two to suggest, but the process is slightly non-trivial so I thought I'd collate it here, thusly:

• NB: All commands are entered from your Rails 3.0 application's root directory.


• Edit [your_app]/Gemfile, and add the line gem 'haml' somewhere convenient.


• Edit [your_app]/config/application.rb, and add the line g.template_engine :haml to the config.generators block (uncommenting the block if necessary).


• gem install haml


• bundle install


• haml --rails .


• git clone git://github.com/psynix/rails3_haml_scaffold_generator.git lib/generators/haml

That last bit is particularly important for allowing Rails' scaffold generators to Do The Haml Thang^(TM) when generating view templates.

Now back to my regularly scheduled tech review ...

P.S.: The book is going to be awesome. Reserve your copy now!