Monday, January 28, 2013

Remote code execution in Java made trivial

SSH offers a very convenient way to execute shell scripts remotely. Code like

ssh myserver.acme.com << EOC
cd $APP_HOME
./start_my_stuff.sh
EOC

are fairly easy to write and read.

But while remote execution itself is easy, writing actual distributed code is total mess.

I’m a Java guy. I wish, I could run Java code remotely as easy as I can do it with shell.
And now, finally, I can.

@Test public void remote_hello_world() throws InterruptedException { ViManager cloud = CloudFactory.createSimpleSshCloud(); cloud.node("myserver.acme.com"); cloud.node("**").exec(new Callable<Void>() { @Override public Void call() throws Exception { String localHost = InetAddress.getLocalHost().toString(); System.out.println("Hi! I'm running on " + localHost); return null; } }); // Console output is transfered asynchronously, // so it is better to wait few seconds for it to arrive Thread.sleep(1000); }

And that snippet will work for you too, only two requirements are:

  • You should be able to SSH to target host without password (you have SSH key-pairs set up).
  • Remote host should have JDK installed, and java command on PATH.
  • That is it, there is no need to install anything special to target servers.

    You can find more details in tutorial at GridKit site.

    How it work?

    A lot of black magic is happening behind the scene. In particular:

  • Classpath of current Java processes is replicated and cached at remote host using SFTP.
  • SSH is used to start remote Java process.
  • All communications between master and slave are tunneled in stdIn and stdOut (thus you can care less about NAT and firewall).
  • Modified version of RMI/serialization is used to allow anonymous classes to be run remotely.
  • By despite all that internal complexity, it just works. I’m using it to run agents on Linux and Solaris. I was using that stuff on Amazon EC2 and I cloud have master process running on my Windows desktop while slave scattered across Unixes.
    It was long road. A lot of issues such as firewalls, bugs in JSch (java SSH client), subtle SSH limitation, etc has been solved along that path.
    But now, I believe, it will “just work” for you too.

    Java vs shell

    But what is the need for such library in a first place? Below are just few of my reasons:

  • I’m a Java guy, I do not want to invest much in shell scripting skills.
  • Java is "really" cross platform, heck I can even debug stuff on Windows then run them on Linux.
  • I have to do a lot to distributed orchestration (starting/stopping processes in right order etc), it is so much easier to do in java.
  • It is hard to write reusable shell scripts, but easy for java (heck, I can even unit test pieces of deployment logic).
  • Remotting is just a remotting

    Ability to effortlessly run remote java code is not much by itself. But it enables you to reach new levels of day-to-day automation (which was prohibitively expensive before).

    But that would be topic for another post.