Jenkins CI Server is great

March 16, 2012

Finally got a Jenkins server installed. Had a host of system issues, like communicating to our source code repo.

Jenkins is a joy to use. Well, it is not perfect, what is? Like, I need to pass the user’s name that invoked a build via Jenkins to the target DOS script (yea, Windows) that eventually invokes the legacy Ant scripts. A quick Google search shows that this is asked in various ways, but no answers. For example, here or here. Hmmmm.

Anyway, now comes a trial use, to see if it is what we really need and can we manage it to do what we will want. With 400 plugins, I don’t see how it could lack. Plus, I’m sure I can use the Groovy plugin to cobble something up. Jenkins even includes a Groovy Console. Finally, there is a road map for possible migration of legacy Ant scripts to Gradle using the Gradle Plugin.

I take back my past snarky comment. Jenkins is not just a pretty face on Cron.

BTW, is there some Wiki law that says a wiki shall never ever have a link to the parent project? If you get tossed into a wiki by following a link, invariably you will click in agony at links that should go to the real home. Instead, you have to edit the URL in the address bar. Since I never curse, I can’t write “wtf”.

Off Topic
Was watching the Easyb intro video. BDD is interesting. Definitely “should” is a better then “test”. With so many great tools why are products still bug ridden?

More stuff

  1. Jenkins home page
  2. Continuous integration Not a very good Wikipedia article
  3. Continuous Integration Much better
  4. Continuous Integration in Agile Software Development
  5. Hooking into the Jenkins(Hudson) API
  6. Five Cool Things You Can Do With Groovy Scripts
  7. Parameterized Builds in Jenkins – choosing subversion folders
  8. Groovy Console
  9. Groovy plugin
  10. Switching to Jenkins–Download and Install Artifact Script for Tester
  11. Gradle Plugin

Using Powershell instead of cygwin for scripting

February 27, 2012

In this blog entry, Windows Tricks For Whacking Lingering Java Processes …, the author resorts to the cygwin linux shell to automate an admin need.  Now cygwin is awesome, bringing some of the best tools from *nix.  However, with Powershell, Microsoft finally transformed the Windows command line into a powerful and usable administration resource.

Below a line from the bash script the author used to find each process given a process name and then loop thru each process ID and terminate it.   Note how the script requires a good understanding of bash scripting and in addition multiple Linux utilities  ps, grep, and sed.   The most important line in the script is:

found=` ps -aW | grep $procText |  sed -e's/^\s*\([0-9][0-9]*\).*/\1/' `
Here is how this works.
ps -aW:   show all processes and windows too.
grep $procText:   only use lines that contain the command line string
sed -e:   run stream editor inline
s///:   substitute
^\s*:  all beginning white space
\([0-9][0-9]*\):  find two or more digits, remember these.
.*:  any characters.
/\1/:  replace with the found digits

Very complex!!!! But, to *nix users this is nothing. Its even fun to do. With a Linux command line you can rule the world. Note that in the above script line, it’s just text processing.

Powershell has a different approach. In Powershell one works with objects (in the OOP sense). Thus, instead of transforming everything to text, one manipulates properties of objects. One pipes object instead of text.

To do:
Show how to do this in Powershell. Unfortunately, wrote the above a long time ago and lost what little Powershell mojo I was developing. Any help?

3/18/12: The powershell approach will probably be expanded from something like this, which just lists the processes by name and process ID:

$strComputer = "."

$colItems = get-wmiobject -class "Win32_Process" -namespace "root\CIMV2" `
-computername $strComputer

foreach ($objItem in $colItems) {
      write-host "Name: " $objItem.Name
      write-host "Process ID: " $objItem.ProcessId

Java’s HTTP Server for browser-based Groovy app

April 10, 2011

Code illustrating use of the HTTP server included in Java JDK 1.6. via Groovy to present a browser-based UI to an app.


The code allows this usage:

main.SimpleServer.serve(0){ s, t, p ->
    // handle the request, response here...
    // now shut down,

Listing 1, How it’s used.

This will create a web server using the host “localhost” at an unused port p, and then automatically open the default browser at “http://localhost:p/”. The closure will be the application. Neat. Of course, you would only use this behind a firewall, etc.

The above code, as a I later discovered, is very similar to other frameworks. Here is a new one I just learned about:

vertx.createHttpServer().requestHandler { req ->
    def file = req.uri == "/" ? "index.html" : req.uri
    req.response.sendFile "webroot/$file"
See also: “Effortless asynchronous application development for the modern web and enterprise”

See also “Java development 2.0: Ultra-lightweight Java web services with Gretty” link, for an approach using Gretty.

Hmmm, just noticed that this is the ‘look’ of a simple node.js example. No way to dupe node.js of course, it is a low level thing, but can streaming event based programming be done with Groovy? Perhaps with GPars. See this discussion node.groovy?.

5/9/2011: Check out blog post on concurrent websockets. Instead of the Java 1.6 embedded server, this is based on Gretty. See also, Gretty/GridGain/REST/Websockets.

A sample session running the example code and using telnet is:

telnet localhost 21224
Connected to localhost.
Escape character is '^]'.
GET /?reply=24 HTTP/1.1

HTTP/1.1 200 OK
Content-length: 15</pre>
<pre>Connection closed by foreign host.

The Eclipse console contains:

+++++++ Simple Server ++++++++++++++++++
uri: [/?reply=24]
protocol: [HTTP/1.1]
query: [reply=24]
path: [/]
params: [[reply:[24]]]

Stopping server ... stopped!

The browser will show:

The app as seen on browser


You have to supply a GUI for running a local Java application. One example could be the setup of a product build. Builds, though usually executed with Ant, Maven, or Gradle, may still require the user to select some target parameters or build type. Why not just use the browser? It can present a more modern interface and with good design, allow more fault tolerant use then that with command line or multiple prompt boxes.

This approach also allows future remote build server use, since the designed browser UI can be reused. The browser is ubiquitous and creating web pages is relatively easy (though can be a hair pulling nightmare sometimes). And, with the use of AJAX and high-level frameworks, like Dojo or JQuery, browser clients can more easily duplicate the usability of dedicated thick client apps. HTML5 (see HTML5 Rocks) is also helping to obliterate any remaining reasons for using a thick-client app.

An embedded server as used here, is great when the task is ad hoc, short lived, or single user. In the provided example, once the input is received the server shuts down. For more complex or ubiquitous use a standard server or more powerful embedded server should be used.

Embedded Server

For a local application that accesses system resources and uses the browser as the UI, using an embedded server is the simplest approach. Pure Javascript, Applets, and other means are very complex and may run against configuration issues with Browser security settings and so forth. In the Java world, that would mean using the more popular Tomcat or Jetty servers.

However, Java 1.6 now includes a light-weight HTTP Server. So, the configuration and requirements are more tractable. Nothing more to download and programmatic configuration concerns are minor. Note that the Java HTTP server does not offer many features, one augments this with custom code. For example, parsing of the request query is not present.

That the package of this server is com.sun… is problematic. Will it stay around, become part of the javax packages, etc? Should the JDK even have this built in? According to this post by M. MacMahone

… is that the API and implementation are a fully supported, publicly accessible component of Sun’s implementation of Java SE 6. It does mean however, that the packages are not formally part of the Java SE platform, and are therefore not guaranteed to be available on all other (non Sun) implementations of Java SE 6.

Incidentally, the Groovy language has an import system, Grape, that can also make use of Tomcat or Jetty as transparently as using the JDK embedded server. See the Further Reading below for an example using Groovlets.

This code illustrates

(more for my future reference)
The list below was some of the things the code used and the final code listed here may no longer have them.

  • API use.
  • Using JQuery in external js files.
  • How to stop the server.
    • With AJAX post
    • Timeout
    • HTTP context
  • Using ScheduledExecutorService to limit runtime.
  • A console ASCII spinner.
  • Groovy GString use.
  • Launching the default browser.
  • Selecting an unused port.
  • Simplistic state machine configuration.
  • Detecting Java version.
  • AJAX using JQuery.
  • Groovy object construction from script.
  • Use of Closure.
  • Basic authentication
  • Quasi Anonymous class use in Groovy
  • access to resources

Of course, not great example of the above, but … Warning, code is not production ready, etc. There is no real exception handling!

How it works.

TODO: Give some idea what all that code does.

The index.html creates a simple form with three buttons (submit, ping, and end), an input field, and a ‘console’ output area.

- submit: send the answer to the server which then gives feedback, correct or wrong. The server then deliberately shuts down.
ping: sends an AJAX request to the ping context which just sends back the time, the response is appended to the console.
end: sends an AJAX request to the ‘stop’ context. The server responds with ‘stopping server …’, then shuts down. All buttons are disabled.

Why Groovy

Groovy is a dynamic JVM based language whose most prominent feature is that it extends the Java syntax to be more usable, i.e., less wordy,. From the example, a method that prints the contents of a map can be defined as:

def showInfo(info){info.each{ k,v -> println "$k = $v" }}

Then invoked as:


The Groovy In Action book is pretty thorough. Chapter 1 makes a good case, and may still be available here.

Why not Groovy? Well, being dynamic can be rough, especially if it impacts the ability of a IDE to offer the features that come from using Java, like completions, etc. The Eclipse plug-in is getting much better, and I read the IntelliJ IDEA groovy support is top-notch. But, the worlds most popular language, JavaScript, is dynamic, and it hasn’t bothered too many people (maybe end users?).


When I first wrote this, I created a complicated “app” with a bunch of class files and so forth. Then I simplified it. Later I said, yuck. Finally I decided that this should just be a simple one method invocation as shown at the beginning of this post. Consequently, a bunch of stuff in this sample code is guarded by conditionals and it works but sure can be simplified.

While coding I ran into a strange classloader issue see this post. There are a few files in this demo

The source code can be downloaded at: here

Listing 4. TestServe.groovy (not a unit test, btw)

//import static main.SimpleServer as SS;   // still won't work!!! GROOVY-4386
import main.SimpleServer

	ss, t, params ->

		def path = t.getRequestURI().getRawPath()

		}else if(path =="/") {
		def answer = params.get("reply")
		reply = (!answer || answer[0] != '"42"') ? "Wrong!" : "Correct!"


Listing 5 SimpleServer.groovy

 * File: SimpleServer.groovy
 * Date: 20110314T2125-05:00
 * Author: jbetancourt

package main

import java.nio.channels.*
import java.nio.*
import java.text.SimpleDateFormat
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit as TU;
import java.util.concurrent.ScheduledExecutorService;
import org.codehaus.groovy.reflection.ReflectionUtils

import edu.stanford.ejalbert.BrowserLauncher;

 * Example of using the Java 1.6 HTTP server
 * @see "
 * spec/com/sun/net/httpserver/package-summary.html"
 * Version 0.02
 * @author jbetancourt
class SimpleServer implements HttpHandler{
	static final STOP_WAIT = 2
	static final HOST = "localhost"
	static final HTTP_PROTOCOL = "http://"
	static final splashText = " Embedded Java 1.6 HTTP server example (ver $version) "
	static final company = " 20110314T2125-05:00 jbetancourt"
	static final UTF_8 = "UTF-8"
	static final version = "0.3"
	static final realmName = "my realm"

	static port =0 // if zero, get unused port
	static url
	static int  counter = 0 // for spinner
	static String basedir   // docbase
	static String scriptDir

	HttpServer server
	static SimpleServer simpleServer
	def SRC_INDEX_HTML = "/src/index.html"
	def serverPropFilePath = ""
	def props = new Properties()
	def browserLauncher
	def ScheduledExecutorService cancelExec
	def authenticate = false
	def Closure handleParams

	 * The only thing you really need.
	 * @param port   if 0, unused port will be used
	 * @param closure  handles the request,response
	 * @return
	static serve(port, Closure closure){
		def ss = new SimpleServer()
		ss.port = port
		ss.handleParams = closure
		ss.serverPropFilePath = ""
		ss.exec(new String[0])
		return ss

	 * Entry point into the demo.
         * @param args
	static main(args) {

		simpleServer = new SimpleServer()

	} // end main

	/**  */
	def exec(args){
		def basedir = new".").getAbsolutePath()
		println("server started. url=$url,basedir=$basedir,scriptDir=$scriptDir,")

		def spb = props.getProperty("progressBar")
		cancelExec = Executors.newScheduledThreadPool(1)

	 * Handle the given request/response.
	 * The first response is the input form.  The subsequent
	 * request evaluates the answer if any, and then the server
	 * is stopped.  Any exception will also stop the server.
	public void handle(HttpExchange t) throws IOException {
			def uri = t.getRequestURI()
			def query = uri.getRawQuery()
			def path = uri.getRawPath()
			def params = parseQuery(query)

				handleParams(this, t, params)
				if(query == null){
					}else if(path =="/") {

		}catch(Exception ex){
			throw ex;

	/**  */
	def showInfo(info){
		println "+++++++ Simple Server ++++++++++++++++++"
		info.each{ k,v ->
			println k + (v ? ': [' + v +']' : ": []")
		println "-------------------------"	


	/**  */
	static splash(){

	/**  */
	def configure(args){
			basedir = new".").getAbsolutePath()
			scriptDir = "$basedir/src/"
			def propFileName = (args.length) >0 ? args[0] :serverPropFilePath
			port  = unusedPort(HOST)
			url = "$HTTP_PROTOCOL$HOST:$port/"

				def getResource = {def resource->

				InputStream context = getResource(propFileName)

		}catch(Throwable ex){
		}finally {

	/**  */
	def createHTTPServer(){
		server = HttpServer.create(new InetSocketAddress(port),0);

		props.propertyNames().iterator().each{key ->
				def conf = props.get(key).split(",")
				def className = conf[0]
				def contextPath = conf[1]? conf[1].trim(): "/"

				def obj = createObjectFromScript(className, this)
				def context = server.createContext(contextPath, obj)

					def authClassName = conf[2]
					def ac = createObjectFromScript(authClassName,
							conf.length > 3? conf[3] : realmName)


				def iProp = key + ".initialState"
					obj.currentState = props.getProperty(iProp)

				iProp = key + ".transitions"

		def context = server.createContext("/", this)
			context.setAuthenticator(new MyAuthenticator("my realm"))

		server.createContext("/stop", [
						println("in handler for stop .. t[" + it + "]")
						sendString(it,"stopping server ....")
				] as HttpHandler);

		server.createContext("/ping", [
					handle:{ ct ->
						println("pinging ...")
				] as HttpHandler);

	} // end createHTTPServer

	 * @return
	static boolean validJavaVersion(){
		def flag = true
		def ver = System.getProperty("java.version");
		if(!ver.contains("1.6")  && !ver.contains("1.7")){
			println("ERROR *** Requires Java 1.6 or above. Detected: $ver");
			flag = false;

		return flag;

	 * Send the initial query page.
	 * @param t the request handler
	 * @param filePath the html file
	 * @return nothing
	static sendPage(HttpExchange t, String filePath) throws IOException {
		OutputStream os = null;
			def fPath = new File(basedir + filePath)
			def uri = fPath.toURI()
			Map>map = t.getResponseHeaders()

			def binary = false

			if(filePath.endsWith( ".js")){
				map.set("Content-Type", "text/javascript; charset=UTF-8")
			}else if(filePath.endsWith(".gif")){
				map.set("Content-Type", "image/gif;")
				binary = true
			}else if (filePath.endsWith(".jpg")) {
				map.set("Content-Type", "image/jpeg;")
				binary = true

			println("sending .... " + fPath)
				byte[] bytes = readFile(fPath.getPath())
				t.getResponseHeaders().set("Content-Encoding", "gzip")
				GZIPOutputStream gos = new GZIPOutputStream(t.getResponseBody())
				def response = fPath.getText()
				t.sendResponseHeaders(HttpURLConnection.HTTP_ACCEPTED, response.length());
				os = t.getResponseBody();
		}catch(FileNotFoundException ex){
		}catch(Exception ex){
				println("close output stream ...")

	 *  Read file into buffer.
	 * @param path
	 * @return
	static byte[] readFile(String path){
		File file = new File(path)

		FileInputStream inStream = new FileInputStream(file)
		FileChannel inChannel = inStream.getChannel();
		def bb = ByteBuffer.allocate(1024*1024)

			int bytesRead = bb
			if(bytesRead == -1){

		return bb.array()

	 * Send the resulting score based on response content.
	 * @param t
	 * @param answer
	 * @return
	static sendString(t, answer ) throws IOException {
		t.sendResponseHeaders(HttpURLConnection.HTTP_OK, answer.length());
		OutputStream os = t.getResponseBody();

	 * Just followed example at:
	 * @see
	 * /concurrent/ScheduledExecutorService.html
	def keepAlive(showProgressBar, Long maxTime){
		def handleBeep = cancelExec.scheduleAtFixedRate(new Runnable(){
					public void run(){

				}, 1, 4, TU.SECONDS);

		cancelExec.schedule(new Runnable(){
					public void run(){
						println("\ncancel beeping")

				},4, TU.MINUTES);

	 *  In shell console, ASCII spinner gives visual feedback of running server.
	 *  Got idea for approach at
	 *  @see
	 *  But, then took out the use of a switch.  As Charles Moore would say,
	 *  never use conditionals when it can be calculated.
	static def progressBar(){
		print("\b${["/","-","\\","-"][counter++ % 4]}")

	/**  */
	Object createObjectFromScript( String className, Object... args ) throws Exception {
		println "Creating $className"
		def gcl = new GroovyClassLoader(this.class.classLoader)
		def path = "$scriptDir${className.replace('.','/')}.groovy"
		def cl = gcl.parseClass( new File(path))
		def ni = cl.newInstance(args)
		return ni;

	 * Get an unused port for server and browser url.
	 * If port is non-zero.
	 * BTW, at shell:
	 *   On windows:  netstat -an
	 *   On linux: netstat -an | grep -i listen
	 * @see
	 * /how-can-i-detect-a-free-port-on-the-server-by-code-from-client-side
	 * I tried simpler ways, but they didn't work. Like using 0 as port.
	 * @param hostname
	 * @return port number
	static int unusedPort(String hostname) throws IOException {
			return port

		int minPort = 8000
		int range = 0xFFFF - 8000
		while (true) {
			int port = minPort + (int) (range * Math.random());
			try {
				Socket s = new Socket(hostname, port);
				s.close(); // is this wise?
			} catch (ConnectException e) {
				return port;
			} catch (IOException e) {
				if (e.getMessage().contains("refused")){
					return port;
				throw e;

	/**	 	 */
	def ping(t){
		def now = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss").format(new Date())
		sendString(t, "$now")

	/**  */
	String getServerProperty(key){
		return props.getProperty(key)

	/**  */
	def start(){

	/**  */
	def launchBrowser(url){
		new BrowserLauncher().openURLinBrowser(url)

	/**  */
	def stopServer(){
		print("\nStopping server ... ")

	/**  */
	static handleException(Exception ex){
		println("ERROR: ${ex.getMessage()}")
		throw ex

	 * Parse query into list of values array.
	 * @see
	 * @param query
	 * @return
	static Map parseQuery(final String query){
		Map params = new HashMap();

		if(!query || query.length() == 0){
			return params

		def key,val

		for (String param : query.split("&")) {
			String[] pair = param.split("=");

			if(pair.length > 0){
				key = URLDecoder.decode(pair[0], UTF_8);

			if(pair.length > 1){
				val = URLDecoder.decode(pair[1], UTF_8);

			List values = params.get(key);
			if (values == null) {
				values = new ArrayList();
				params.put(key, values);
			values.add(!val ? "":val );

		return params;

} // end SimpleServer

// the authenticator class.  Should have been just a simple inner class.
class MyAuthenticator extends BasicAuthenticator {
	/**  */
	public MyAuthenticator(String realm){

	public Authenticator.Result authenticate(HttpExchange t){
		return super.authenticate(t)

	public boolean checkCredentials(String username, String password){
		//printf("user=%s, pass=%s%n", username, password)
		return true
} // end MyAuthenticator class

Listing 6 If there is a setting for a property file it will be loaded and used. This is an example:

# SimpleServer configuration
# context = FQCN,path[,authenticator FQCN, domain name]*
controllerContext = main.AppContext,/question
controllerContext.initialState = running
# simple state transitions = state:next_state[,state:next_state]*
controllerContext.transitions = init\:running,running\:end
# Misc
stop_wait = 2

Listing 7 index.html The ‘app’ UI:

<script type="text/javascript" src=""></script>
	<!--  <script src="/src/jquery.blockUI.js" type="text/javascript"></script> --><script type="text/javascript" src="/src/scripts.js"></script><script type="text/javascript">// <![CDATA[
 		$(document).ready(function(){ 				//$(document).ajaxStart($.blockUI).ajaxStop($.unblockUI); 				pingServer();	 				stopServer(); 		});
// ]]></script></pre>
<div class="box middle lightGrey">
<div class="middle">Embedded HttpServer Demo <span class="tiny">(</span></div>

<hr />

<form id="form1" class="box internal grey" action="/question" method="get" name="form1">
<input id="mode" type="hidden" name="mode" value="mathInput" />
<table width="100%">
<tr style="margin: 12px;">
<td><span id="lblReply" class="heavy"> What is "4"+"2"? </span></td>
<td><input id="reply" type="text" name="reply" /></td>
<td><input id="submitButton" title="submit" type="submit" value="submit" /></td>
<td><input id="pingButton" type="button" value="ping" /></td>
<td><input id="stopButton" type="button" value="end" /></td>

Listing 8 scripts.js JQuery stuff:

// file: scrips.js
// author: jbetancourt
// External JQuery use.
// technique reference:
/* <![CDATA[ */ var pingServer; var stopServer; (function($){ 	pingServer = function(){ 		$('#pingButton').click(function(){ 			$.get('/ping', {mode:"ping"}, 				function(data){ 				    $('#target').append(" "+data) 				},"html") 			    .error(function(response,status,xhr){ 					var msg = "Server does not responsd: "; 					$("#target").html(msg + " " + status + 						(xhr.status ? " xhr.status: [" + xhr.status + "] " 							 + "] xhr.statusText: [" + xhr.statusText + "]" 							 : "") 					); 				}); 		}); 	}; })(jQuery); (function($){ 	stopServer = function(){ 		$('#stopButton').click(function(){ 			$("#submitButton").attr('disabled','disabled'); 			$("#pingButton").attr('disabled','disabled'); 			$("#stopButton").attr('disabled','disabled'); 			$('#target').load('/stop', function(response,status,xhr){ 				if(status == "error"){ 					var msg = "Server does not responsd; "; 					$("#target").html(msg + xhr.status + " " + xhr.statusText); 				} 			}); 		}); 	}; })(jQuery); /* ]]> */

Listing 9 stylesheet.css

.heavy{ font-weight:bolder;}
.box{ border:2px solid black;}
.middle{ margin-left:auto;margin-right:auto;width:60%;}
.internal{ margin:2em;background:#F8F8F8 ;}
.horzCenter { margin-left:auto;margin-right:auto;width:50%;}
.grey { background:#F8F8F8;} .lightGrey { background:#F0F0F0;}
.large{ font-size:xx-large;padding-top:4px; padding-bottom:4px}
body{ width:960px }  .tiny{font-size:small;}

Listing 10 example app AppContext.groovy

 * File: AppContext.groovy
 * Date: 20110320T1952-05:00
 * Author: jbetancourt

package main

import java.text.DateFormat;
import java.text.SimpleDateFormat


 *  The context that hosts the application.
 * @author jbetancourt
class AppContext  implements HttpHandler{
	def SimpleServer server;
	def static RUNNING = 'running'
	def currentState
	def transitions = [:]

	/**  */
	AppContext(SimpleServer server){
		this.server = server
		currentState = server.getServerProperty("initialState")
		def statesProperty = server.getServerProperty("transitions")

	 * Handle the given request/response.
	 * The first response is the input form.  The subsequent
	 * request evaluates the answer if any, and then the server
	 * is stopped.  Any exception will also stop the server.
	public void handle(HttpExchange t) throws IOException {
			def uri = t.getRequestURI()
			def final query = uri.getRawQuery()
			def path = uri.getRawPath()
			def params = server.parseQuery(query)

			def mode = params.get("mode")
					if(mode[0] == "mathInput"){
						def reply = params.get("reply")
					}else if (mode[0]=="ping") {

		}catch(Exception ex){
			throw ex;

	* And, send response.
   def evaluateAnswer(t,answer){
	   def reply
		   reply = (answer[0] != '"23"') ? "Wrong!" : "Correct!"
	   }catch(Exception ex){
		   reply = "wrong"


   /**  */
	def showInfo(info){
		println "++++++++ AppContext +++++++++++++++++"
		info.each{ k,v ->
			println k + (v ? ': [' + v +']' : ": []")
		println "-------------------------"

	/**	 	 */
	def ping(t){
		def now = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss").format(new Date())
		server.sendString(t, "$now")

	/**  */
	def setTransitions(s){
		s.split(",").each{ tran ->
			def kv = tran.split(":")

	/**   */
	def atState(s){
		return currentState == s

	/**   */
	def transitionNextState(){
		def ns = transitions[currentState]
		currentState = (ns ? ns : "end")
		println("  next state=[$currentState] ")

} // end AppContext class


Shown was a simple example of using the JDK 1.6 embedded HTTP server using the Groovy language.


  • April 15, 2011: Added some beginnings of code to handle image resources.
  • August 4, 2011: Just saw an old post on same subject. “Groovy++ in action: DSL for embedding HttpServer”. Added it to references. That one creates a DSL to use the embedded HTTP server. Nice.
  • Dec 4, 2011: This post has an example of using Gretty via Groovy: Five Cool Things You Can Do With Groovy Scripts
  • Feb 11, 2012: The internal JDK Http server is being used here: “How ION uses Virgo“.

Required Software

• Oracle Java JDK
• Groovy 1.8-rc3
• BrowserLauncher2

Dev Software

• Eclipse: Helios Service Release 2
• Windows 7 Pro, 64bit
• Mercurial 1.8.1
• TortiseHG 2.0.2
• MercurialEclipse 1.0.0
• Groovy-Eclipse 2.1.3

Further Reading

Duplicate blog post: Java’s HTTP Server for browser-based Groovy app

Other Servers
“Java development 2.0: Ultra-lightweight Java web services with Gretty”



HTTP Implementations in other languages

* Groovy++ in action: DSL for embedding HttpServer

* Java non-blocking servers, and what I expect node.js to do if it is to become mature

* Sun’s secret web server

* Using

* node.groovy?

* Ratpack

* AsynchronousChannel

* Mp3d project (which uses com.sun.httpserver)

* Groovy Goodness: Groovlets as Lightweight Servlets

* Making a simple web server in Python.

* Embedded HTTP server

* Comparison of web server software

* Groovlets

* Practically Groovy: MVC programming with Groovy templates

* HTTP server API in Sun’s Java SE 6

* Using Sun Java 6 HttpServer to write a functional HTTP test

* Package

* Example for Java HTTP Server API and Ruby WEBrick HTTP Server

* Loop back connection leak


* JDK modules source

* Java Documentation 6.0 JDK Modules

* ServerImpl

* D. Ferrin resource access solution

* Simple Java HttpServer / Handler

* post by hiro345

* Using for comet/cometd

* Separating jQuery Functions into External Files (without selectors!)

Java Dev Using Embedded Groovy Console in Eclipse

December 12, 2010

In development, simple but powerful tools to get at the essence of a code source in order to understand, test, and extend it is essential. This is even more important in Test Driven Development (TDD). Eclipse’s Java Debugger is for most situations, powerful enough. Eclipse has an Expressions View available in the debug perspective to execute snippets of code. The Java Development Toolkit (JDT) also has a Scrapbook facility that allows the creation, storage, and running of experimental code. And, of course, all the other features of the debugger are excellent.

However, when you need it, it’s possible to embed a script engine and have new ways of analyzing and developing code. In listing 1, an app shows the use of the ConsoleWaiter class. When the code executes the at line 43, it opens the Groovy Console which allows the use Groovy shell scripting in a GUI frame, see figure 1. When the console is closed the app continues executing.

Listing 1
* File:
* @author jbetancourt
* Date: 20101213T1718-5
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;

* @author jbetancourt
public class ExampleApp2 {
   static public String greeting = "Hello world!";
   private static final List<String> nodes;

   static {
        nodes = new

   private String title = "Example 2";

   public String getTitle(){
      return title;

   * @param args command line args
   public static void main(String[] args) {
      ExampleApp2 app = new ExampleApp2();

      ConsoleWaiter waiter = new ConsoleWaiter(app);

      waiter.setVar("greet", greeting);
      waiter.setVar("nodes", nodes);
      waiter.setVar("title", app.getTitle());;

screen capture of Groovy console

Console screen capture, click to view

Another screen capture, click to view

This is awesome. In one project I had to examine the contents of a Properties object. Did it have an “email” value? I was stumped when using the Eclipse debugger, it did not show all entries in the Map, at the end was “…”. Sure, I could use the Expressions window, but with the Console I could not only do a get(key), but iterate using Groovy style closures and much more.

The magic that enables this is the ConsoleWaiter.groovy class shown below in listing 2 that was written by John Green. Since a Groovy script is a Java class underneath, in Eclipse you can call Groovy from Java easily (some compiler magic).

Listing 2
 * File:  ConsoleWaiter.groovy

import groovy.lang.Binding;
import groovy.ui.Console;

 * Provides a wrapper for the console.
 * Based on source by John Green
 * Adapted from:
 * Released under the Eclipse Public License
 * I added methods to allow use from Java.
 * The run() method launches the console and causes this thread
 * to sleep until the console's window is closed.
 * Allows easy interaction with the objects alive at a given
 * point in an application's execution, like in a debugger
 * session.
 * Example 1:
<pre> * new ConsoleWaiter().run()

 * Example 2:
<pre> * def waiter = new ConsoleWaiter()
 * waiter.console.setVariable("node", node)
class ConsoleWaiter {
 Console console
 Object source
 boolean done = false;

 /** */
 public ConsoleWaiter(Console inConsole){
    this.console = inConsole

 /** */
 public ConsoleWaiter(Object source){
    console =
    new Console(getClass().classLoader,
    new Binding())
    this.source = source
    console.setVariable("source", source)

 /** */
 public void setVar(String key, Object value){
    console.setVariable(key, value)

 /** 	 */
 public void setVar(String key, List values){
    console.setVariable(key, values)

 /** 	 */
 public void setVar(String key, Object[] values){
    console.setVariable(key, values)

 /** 	 */
 public void run() {
    // I'm a little surprised that this exit() can be private.
    console.frame.windowClosing = this.&exit
    console.frame.windowClosed = this.&exit
    while (!done) {
       sleep 1000

 /** 	 */
 public boolean isDone(){
    return done;

 /** 	 */
 public void exit(EventObject evt = null) {
    done = true

 /** 	 */
 public Console getConsole(){
    return console;

Eclipse Integration

The disadvantage of this approach is that you have to put extraneous code inside the tests or target class. Not only is this tedious and slow, what if code is accidentally deployed like this? A better approach is to just set a breakpoint in the code, and then have the ability to open a script console at that breakpoint, in the Java Stack Frame, that has access to the execution context and classpath. Is there an Eclipse add-in that does this? If not, there should be.


Shown was a simple example of embedding a Groovy console in Java code to allow scripting. Of course, this is not a new idea. It was even mentioned in an older JVM scripting language, Beanshell. Note that it is possible to instead of using a GUI console, to use the Groovy shell, InteractiveShell class. In the reading list below this approach is taken to allow remote scripting of a server hosted application.


  • Oct 10, 2011:
    Interesting tool that could be relevant: YouDebug.
  • March 20, 2012: Not exactly same scenario, but the concept of an embedded script console is found in many products. Jenkins CI Server has one and it uses Groovy. Jenkins Script Console

Further reading

Groovy Object Notation (GrON) for Data Interchange

May 17, 2010


Foregoing the use of JSON as a data interchange when Groovy language applications must interact internally or with other Groovy applications would be, well, groovy.


JavaScript Object Notation (JSON) is considered a language-independent data interchange format. However, since it is based on a subset of the JavaScript (ECMA-262 3rd Edition) language, this is not fully correct. In fact, many languages must marshal to and from JSON using external libraries or extensions. This complicates applications since they must rely on more subsystems and there may be a performance penalty to parse or generate an external object notation.

If an application must only interact within a specific language or environment, such as the Java Virtual Machine (JVM), perhaps using the host language’s data structures and syntax will be a simpler approach. Since Groovy (a compiled dynamic language) has built-in script evaluation capabilities, high-level builders (for Domain Specific Language (DSL) creation) , and meta-programming capabilities, it should be possible to parse, create, transmit, or store data structures using the native Groovy data interchange format (GDIF), i.e., based on the native Groovy data structures.

Syntax example

Below is an example JSON data payload.

JSON (JavaScript) syntax:

{"menu": {
  "id": "file",
  "value": "File",
  "popup": {
    "menuitem": [
      {"value": "New", "onclick": "CreateNewDoc()"},
      {"value": "Open", "onclick": "OpenDoc()"},
      {"value": "Close", "onclick": "CloseDoc()"}

Below is the same data payload; this time using Groovy syntax. Note that there are not too many differences, the most striking is that maps are created using brackets instead of braces. It looks simpler too.

Groovy syntax:

[menu: [
	id: "file",
	value: "File",
	popup: [
	menuitem : [
	 [ value: "New", onclick: "CreateNewDoc()" ],
	 [ value: "Open", onclick: "OpenDoc()" ],
	 [ value: "Close", onclick: "CloseDoc()" ]

Code Example

 * File: GrON.groovy
 * Example class to show use of Groovy data interchange format.
 * This is just to show use of Groovy data structure.
 * Actual use of "evaluate()" can introduce a security risk.
 * @sample
 * @author Josef Betancourt
 * @run    groovy GrON.groovy
 * Code below is sample only and is on an "AS IS" BASIS,
 * or implied.
 * =================================================
class GrON {
    static def message =
    '''[menu:[id:"file", value:"File",
     popup:[menuitem:[[value:"New", onclick:"CreateNewDoc()"],
     [value:"Open", onclick:"OpenDoc()"], [value:"Close",

    /** script entry point   */
    static main(args) {
       def gron = new GrON()
       // dynamically create object using a String.
       def payload = gron.slurp(this, message)

        // manually create the same POGO.
        def obj = [menu:
	    [  id: "file",
	       value: "File",
                 popup: [
                   menuitem : [
                   [ value: "New", onclick: "CreateNewDoc()" ],
                   [ value: "Open", onclick: "OpenDoc()" ],
                   [ value: "Close", onclick: "CloseDoc()" ]

         // they should have the same String representation.
         assert(gron.burp(payload) == obj.toString())

 * @param object context
 * @param data payload
 * @return data object
def slurp(object, data){
	def code = "{->${data}}"  // a closure
	def received = new GroovyShell().evaluate(code)
	return received()

 * @param data the payload
 * @return data object
def slurp(data){
     def code = "{->${data}}"
     def received = new GroovyShell().evaluate(code)
     return received()

 * @param an object
 * @return it's string rep
def burp(data){
     return data ? data.toString() : ""

} // end class GrON

Possible IANA Considerations

MIME media type: application/gron.

Type name: application

Subtype name: gron

Encoding considerations: 8bit if UTF-8; binary if UTF-16 or UTF-32

Additional information:

Magic number(s): n/a

File extension: gron.

Macintosh file type code(s): TEXT


To be determined.


Would GrON be a security hole? Yes, if it is implemented using a simple evaluation of the payload as if it were a script. The example shown above used evaluate() as an example of ingestion of a data object. For real world use, some kind of parser and generator for object graphs would be needed. The advantage would accrue if the underlying language parser could be reused for this.

Now this begs the question, if Groovy must now support a data parser, why not just use JSON with the existing libraries, like JSON-lib?

Is using the Java security system an alternative as one commenter mentioned?


The idea for GrON was formulated about a year ago. Delayed posting it since I wanted to create direct support for it. However, the task required more time and expertise then I have available at this time.

I was debating what to call it, if anything. Another name I considered was Groovy Data Interchange Format (GDIF), but I decided to follow the JSON name format by just changing the “J” to “G” and the “S” to “r” (emphasizing that Groovy is more then a Scripting language, its an extension of Java).


10Sept2011: See also this post: “JSON Configuration file format“.

9Feb2011: Looks like Groovy will get built in support for JSON: GEP 7 – JSON Support

I found (May 18, 2010, 11:53 PM) that I’m not the first to suggest this approach. See Groovy Interchange Format? by DeniseH.

Recently (Oct 3, 2010) found this blog post:
Groovy Object Notation ? GrON?

Mar 22, 2011: Groovy 1.8 will have JSON support built in.

Further Reading

Creative Commons License
Groovy Object Notation (GrON) for Data Interchange
by Josef Betancourt is licensed under a Creative Commons Attribution-Noncommercial-No Derivative Works 3.0 Unported License.
Based on a work at

Switching between versions of tools

April 25, 2010

How does one manage the use of multiple versions of a software tool, such as a programming language or framework?


In a blog post the author, Jeff, shows a way to switch between versions of his development tools, using the Grails framework as an example.   In Linux he creates symlink to the specific version.  He further uses aliases in the ~/.profile to allow quick changes to these symlinks.  Nice post.

Windows 7

Many, I’m sure, have been using that approach, symlinks to the current version.  For example, on Windows 7, I use directory junctions.  Below is a snippet of my java folder.  Notice that all the Groovy versions are in the GroovyVersions subfolder and the file “groovy” will point to one of these.

Directory of c:\java

04/07/2010  05:55 PM    <JUNCTION>     groovy [c:\java\GroovyVersions\groovy-1.7.2]
04/07/2010  05:52 PM    <DIR>          GroovyVersions

The built-in command to create a junction in Windows 7 is:

c:\java>mklink /?
Creates a symbolic link.

MKLINK [[/D] | [/H] | [/J]] Link Target

/D      Creates a directory symbolic link.  Default is a file
symbolic link.
/H      Creates a hard link instead of a symbolic link.
/J      Creates a Directory Junction.
Link    specifies the new symbolic link name.
Target  specifies the path (relative or absolute) that the new link
refers to.

So, to create the shown link one would execute:
mklink /J groovy c:\java\GroovyVersions\groovy-1.7.2
or should that be:
mklink /D groovy c:\java\GroovyVersions\groovy-1.7.2
(todo: review the differences between a junction and a softlink)

Changing versions

Changing to a particular would now require delete of the current junction and then creation of a new one to point to the desired version.  However, this is clunky.  Jeff used shell aliases which could just as easily be done in Windows using batch files or PowerShell commands.


As the original blog post mentioned, this allows easier control over the environment.  Thus, GROOVY_HOME environment variable would not have to change.  It would remain, GROOVY_HOME=C:\java\groovy


NTFS Hard Links, Directory Junctions, and Windows Shortcuts

NFTF symbolic link
NTFS junction point
Symbolic Links on MSDN
Using a Virtual Drive in a Project with Multiple Branches

Ad Hoc Version Control With Git

April 1, 2010

In my blog post “How to do Ad Hoc Version Control” I suggested Mercurial as the DVCS to use.  However, I read that Git is a very popular and powerful alternative to Mercurial.  Over at the VIM Tips Blog, this post, shows how easy it is to create a local folder repository using Git.  Very similar to Mercurial’s steps.

Linked on the Git site we also have this:

Create your own repository anywhere

If you want to get some version control on a simple local project (i.e., it doesn’t have a big remote repository or anything), then you can simply use git init to create your own standalone local repository. For example, if you’re working on some design concepts for a new application, then you could do something like the following:

mkdir design_concepts
git init

Now you can add files, commit, branch, and so on, just like a “real” remote Git repository. If you want to push and pull, you’ll need to set up a remote repository.

git remote add
git pull master

Which is better Git or Mercurial?  
For ad hoc use, it probably doesn’t matter.  What matters is how easy it is to install, learn, and create repositories on demand.   For large systems with many version control requirements, one must evaluate each alternative carefully.  I’m sure advance use will be ‘difficult’ in any VCS system.

A nice recent blog post by C. Beust, “Git for the nervous developer”, provides a nice perspective on using Git, though perhaps not entirely relevant to the stated Ad Hoc usage scenarios.  That post has a nice list of Git references.

What is an optimal ad hoc workflow?
In the original article I gave context for an ad hoc situation. For example, fixing some server configuration files. In that use-case, Git-Flow would be overkill. Perhaps even GitHub Flow too.

A simpler approach is having two “standard” branches: master and dev. The ‘master’ branch is the live or solution branch. The ‘dev’ branch is where you will make any changes or experiments. Rarely would you create a new branch off of dev. If the ad hoc situation is that complicated, you have other issues! In a “lone developer” workflow however, branching off of dev would be more common. Another ingredient is a separate local Git repo to serve as a backup.

The GitHub Flow is still very applicable, of course, and may be better then the above. The GitHub Flow reminds me of “The Flying Fish Approach” that was documented in the cvsbook.

Once one of these attempts on the dev branch (or github flow feature branch) produces a positive result (the dev branch is live while it is current branch), you merge it into the master branch and then tag the master branch. Each success is tagged, that way when the success turns out to be bogus or has unintended consequences, you go back to the last ‘good’ tag. If you don’t use tags, you can just use the commit SHA1 id. Remember the master is always the final live environment of the ad hoc situation.

Now how to use Git to do the above workflow? I don’t know. Still learning Git. :)

Further Reading

  1. Using git for Backup is Asking for Pain
  2. Git
  3. Git Cheatsheet. This one is interactive.
  4. Git Ready
  5. A successful Git branching model
  6. Using Git for Document/Software Version Control
  7. Version control on a 2GB USB drive
  8. Git for designers
  9. Git for the nervous developer
  10. GitHub Flow
  11. The Flying Fish Approach – A Simpler Way To Do It
  12. Using Git as a “Poor Man’s” Time Machine
  13. Using Git as a “Poor Man’s” Time Machine – Part Two
  14. What tools exist for simple one-off sharing of a git repo?
  16. For home projects, can Mercurial or Git (or other DVCS) provide more advantages over Subversion?

How to do Ad Hoc Version Control

March 28, 2010

By Josef Betancourt,  Created 12/27/09    Posted 28 Mar 2010


You will modify a group of files to accomplish something.  For example, you’re trying to get a server or network operational again after some security requirements.   During this process you also need to keep track of changes so that you can back out of dead ends or combine different approaches.  Or you just simply need to temporarily keep revisions of a group of files for a task based project.


Version Control, Revision Control (RCS), Configuration Management (CM), Version Control System (VCS), Distributed Version Control System (DVCS), Mercurial, Git, Agile, Subversion

Lightweight Solution

An lightweight Ad-Hoc Version Control (AHVC) approach may be desirable.  Note that even when there are other solutions in place, a lightweight approach may still be desirable.  What are the requirements of a lightweight and workable solution?

  • Automated:  Thru human error a file or setting may not get versioned or even lost.  Thus, all changes must be tracked.
  • Small:  A large sprawling system that could not even fit on a thumb drive is too big.
  • Multiplatform:  It should be able to run on the major operating systems.
  • Non-intrusive:   Use of this system should not change the target systems in any way.  Ideally should run from a thumb drive or CD.  And, if there is a change, backing it out should be foolproof.
  • Simple:  Anything that requires training or complexity will not be used or adopted.  This reduces collaborative adoption and improvements in tools and process.
  • Fast:   Should be fast and optimized for local use.
  • Distributed:   Since issues can span “boxes”, it should be able to work with network resources.  This reduces the applicability of GUI based solution.
  • Scripting:  Should be easy to optimize patterns of use by creating high-level scripts.
  • Small load:  Of course, we don’t want to grab excessive CPU and memory resources from the target system.
  • Non-Admin:  Even in support situations, full admin access may not be available.
  • Transactional:  Especially in server configuration, changes should be consistent.  Failure to save or revert even one file could be disastrous.
  • Agile:  Not about tools but results.


At home when I create a folder to work on some files, like documents or programming projects, I will usually create a version control system right in the folder.  This has saved me a few times.  I also tried to do this at a prior professional assignment and was partially successful (will be discussed later).

I used a Distributed Version Control System (DVCS).  Since it does not require a centralized server or complicated setup to use, a DVCS meets most of the lightweight requirements.  Though, a VCS is usually used for collaborative management of changing source files it may be ideal here.  One popular use case is managing one’s /etc folder in Linux with a VCS.


A good example of a DVCS is:


“(n) a fast, lightweight Source Control Management system designed for efficient handling of very large distributed projects.” –

Note:  I use Mercurial as the suggested system simply because I started with it.  Git and others are just as applicable.

To create a repository in a folder, one simply executes three commands init, add, and commit.  The “init” creates a subfolder that serves as the folder history or true repository.  The “add” is recursive, adding all the files to version control, and the “commit”, makes these changes permanent.  Of course, one can ‘add’ a subset of files and create directives for files to skip and so forth.



In a command shell

hg init
hg add
hg commit -m “initial commit of project”



The terminology may be a little confusing.  What happened is that now the GIZMO folder has a Mercurial repository which consists of a new .hg folder, and the other existing files and folders comprise the working directory (see Mercurial docs for a more accurate description).  There are no other changes!

That’s all it takes to create a repository.  No puzzling about storage, unique names, hierarchy, and all the details that goes with central servers.  The Mercurial docs show how to do the other required tasks, like go back to a previous changeset or retrieve file versions and so forth.  Here is how to view the list of files in a particular changeset:

c:\Users\jbetancourt\…cts\adhocVersioning>hg -v log -r 0

   changeset:   0:f29a0b0ad03c    user:        Josef Betancourt <josef.betancourt>l    date:        Sat Jun 21 10:53:11 2008 -0400    files:       AdhocVersioning.doc   description: first commit

And, here is a log output using the optional graph log extension (

c:\Users\jbetancourt\...adhocVersioning>hg glog -l 2
@  changeset:   9:25f4c55e4860
|  tag:         tip
|  user:        Josef <josef.betancourt>
|  date:        Fri Mar 26 22:43:56 2010 -0400
|  summary:     removed repo.bat
o  changeset:   8:43a33533c992
|  user:        Josef <josef.betancourt>
|  date:        Thu Mar 25 22:08:35 2010 -0400
|  summary:     removed old files

For the lone individual using ad hoc versioning a sample workflow is give at Learning Mercurial in Workflows.

Ad Hoc Sharing

A DVCS, true to its name, shines in how it allows Distributed versioning sharing of these local repositories.  Thus, when a team is working on a technical issue (ad hoc) it is very easy to share each others work. Mercurial includes an embedded web server that can be used for this.

Mercurial’s hg serve command is wonderfully suited to small, tight-knit, and fast-paced group environments.  It also provides a great way to get a feel for using Mercurial commands over a network.

This is illustrated with the coffee shop scenario, see manual.

A sprint or a hacking session in a coffee shop are the perfect places to use the hg serve command, since hg serve does not require any fancy server infrastructure … Then simply tell the person next to you that you’re running a server, send the URL to them in an instant message, and you immediately have a quick-turnaround way to work together. They can type your URL into their web browser and quickly review your changes; or they can pull a bugfix from you and verify it; or they can clone a branch containing a new feature and try it out.

Of course, this would not scale and is for “on-site” use between task focused group members.

A great workflow image by Leon Bambridge for team sharing.

Another simple scenario is taking a few file documents from one location to another with a flash drive (in lieu of using a Cloud storage service). Instead of doing a copy or cp one can simply create a DVCS repository at the work directory, then clone it on the flash drive. Then at home one pulls to the DVCS repository at home. When finished editing the files, one then pushes to the flash repo, and does the reverse process at the work site. Not only are you not missing any files, you are also keeping track of prior versions. Note, for security reasons, not everyone has unfettered web access or should they.

Revisiting the flash drive scenario above; if you plan to use a flash drive for transport multiple times and the group of files are large, using the “bundle/unbundle” hg commands are a good tool, see Communicating Changes on the Mercurial site.

Every connection must be secure and every file must be encrypted, especially if on flash drives. The security policies of the employer come first. Even if only for your own personal ad-hoc use, you should be careful with exposing your data.


  • Easy to use.The commands needed to perform normal tracking of changes are few and simple.  The conceptual model is also simple, especially if one is not fixated on use of centralized Version Control System.
  • Some file changes may be dependent on or result in other file changes.In a DVCS, commits or check-ins create a “changeset” in the local repository.  This naturally keeps track of related changes.
  • You may need to work on different operating systems.Mercurial runs on many systems including Windows.
  • You don’t want to change the existing system, low intrusion.  Mercurial can be deployed to a single folder, and the repositories it creates do not pollute the target folders.  For example, in the Subversion VCS, “.svn” folders are created in each subfolder in the target.  Not a drawback but complicates things down the line, such as when using file utilities and filters.


Unfortunately, the use of a DVCS is not perfect and has its own complexities.  For Mercurial, in the context of this post, these are handling binary files, versioning non-nested folders, and probably for any VCS is the semantic gap between the project task based view and the versioning mindset.

1. Binary Files

Mercurial is really for tracking non-binary files.  That way the advantages of versioning are realized.  Diffs and merges are not normally applied to Binary files. Further the size of binary files impact performance and storage when they reach a certain size.  Yet, for ad hoc use, binary files will have to be easily tracked.  Binary files could be images, libraries, jars, zips, documents, or data.

Large binaries are a problem with all VCS systems.  One author discussed a technique to allow Git to handle them in lieu of his continued use of Unison.  He said use Git’s “–shared” option:  git clone –shared /mnt/fileserver/stuff.git stuff

Note that Mercurial extensions exist to handle binary files.  One of these is the BigFiles extension.  In essence, BigFiles and other similar approaches, handle large binaries using versioned references to the actual binaries which are stored elsewhere.

Update Oct 29, 2011: Looks like Mercurial 2.0 will have a built-in extension for handling binary files, LargeFiles extension.

Another issue is that since binary files may not be diffed within the dvcs tool set.  In a DVCS one can set an external merge agent.   If one is not available, using the app that created the binary diff and merge is cumbersome.    For example, a Word doc is binary (even though internally it could be all XML) in effect.   Thus, a diff would not reveal a usable view.  One must ‘checkout’ particular revisions and then use Word to do a diff or just manually eyeballing it.  Same thing with a zip, jar, image, etc.

Update 02-02-2012: Some tools allow direct use of external tools to diff “binary” files. I think TortoiseSVN does this, allowing Microsoft Word, for example, to diff.

2. Non-nested target folders.

A scenario may involve the manipulation of folders that are not nested. For example, a business system employs two servers and changes must be made to both for a certain service to work, further, physically moving these folders or creating links is not possible or allowed. Mercurial, at this time, works on a single folder tree, and AFAIK there is no way to use symlinks or junctions to create a network folder graph, at least with my testing.  The ForestExtension or subrepositories experimental feature in Mercurial 1.3 do not qualify since they only enable the handling of a folder tree as multiple repositories.

Sure each folder tree in the graph can be managed, but if a particular change effects files in each tree, there is no easy way to transactionally version them into one changeset, though there are ways to share history between repositories (as in the ShareExtension).

A possible solution is to allow the use of indirect folders.  In Mercurial, work files and the actual repository, the .hg folder, are colocated.  Instead the repository can point to the target folders (containing the work files) to be versioned.  In this way multiple non-nested folders can be managed.  Note that this is not a retreat to the centralized VCS since the repository is still local and distributed using DVCS operations.   Below, the user has created a new Mercurial repository in folder “project”.  This creates the actual repo subdirectory “.hg”, and the indirect actual folders to be versioned are pointed to in a “repos” directive file or using actual symlinks.



repos ——> src_folder1

\—–> src_folder2

\—–> src_folderN

Whether this is useful, possible, or already planned for is unknown.

I mentioned this “limitation” on the Mercurial mailing list and was told that this is not a use case for a DVCS. There are many good reasons why all (?) VCS are focused on the single folder tree.

Update, 2011-08-31 T 11:37
Just learned that Git does have an interesting capability?

It is also possible to have a working tree where .git is a plain ASCII file containing gitdir: , i.e. the path to the real git repository

Though this doesn’t fulfill the non-nested project folders scenario, it does help Git be more applicable in an ad-hoc solution. For example, the repo could be located in a different storage location when the target folder is in a constrained unit.

3. Non-admin install

Updated 25 Aug 2010: In the requirements, non-admin install of the VCS was mentioned. This is where Mercurial fails, somewhat. The default install using the binary, at least on Windows, requires admin privileges. I got around this by first installing on another Windows system, then copying the install target folder to the PC I need to work on. This worked even when I installed on a Windows 7 Pro, and then copied to a Windows XP Pro. No problems yet. The Fossil DVCS does not have this problem.

4. Ignore Files

This is, perhaps, a minor issue. Mercurial, as most VCS do, allow one to filter the files that are versioned in the repo.
In Mercurial one creates an .hgignore file and within it, one can use glob or regular expression syntax to specify the ignore files. Well, this can be tricky. See newsgroup discussion that was started by this post. IMHO, having another syntax declaration that allows specification of directories and files explicitly is the way to go. How do other systems do this? Ant patternsets seem to be pretty usable.

5. Semantic Gap

There is a semantic gap when working on a maintenance problem and switching to the versioning viewpoint.   When versioning systems are routinely used, as in Software Development, this is not an issue, just part of the Software Development Lifecycle or best practice (amazing that some shops don’t use version control).   But, when one uses VC only occasionally as a last resort it’s another story.  QA, Support, and Project Managers, may not be comfortable with repositories, branches, tags, labels, pull, push, and so forth.

When I first tried to use Mercurial for Ad hoc service professionally it quickly lost some of its advantages as the task (fixing a system) reached panic levels (usually the case with customer support and deployment schedules) and simply creating and looking at commit messages failed to follow the workflow.  Manually tracking which tag or branch related to which event of system testing was cumbersome.  Further use would have eventually revealed the patterns of use that would have worked better, but that was a onetime experiment.

A partial solution, other than just getting more expert with the DVCS and better work patterns, is to implement a higher level Domain Specific Language (DSL) that hides the low level DVCS command line and repository centric view.  This could even have a GUI counterpart.  This is not the same as a GUI interface to the DVCS such as TortoiseHg or the Eclipse HG plugin.  What should that DSL  be and is it even possible or useful?

work flow Updates

June 26, 2011: git-flow, is an example of providing high-level operations to enable a specific work flow or model. Perhaps such an approach would be applicable in this AHVC requirements.

Sept 17, 2011: Mercurial Flow-Extension
implements the git-flow branching model.


Naive Approach

The usual approach is to just make copies of the effected folder or files that you will be changing.  You can use suffixes to distinguish them, such as gizmo.1.conf.   It’s very common to see (even in production!) files or folders with people’s initials, gizmo.jb.19.conf.

This gets out of hand very quickly, especially if you are multitasking or working as part of a team and may forget after a good lunch what file “gizmo.24.conf” solved.   This problem is compounded when you need to change multiple files, so for example, gizmo.jb.19.conf may depend on changes to “widget.22.conf”.   This also gets very chaotic when the files to change and track are in different folder trees or even storage system.  Most importantly this will not withstand the throw clay at the wall and see what sticks school of real world maintenance.

One method I’ve seen and used myself is to just clone each folder tree to be modified.  This gives an easy way to back out any changes.  This, alas, is also error prone, resource intensive, and may not be possible on large file sets or very constrained systems.

Client-Server VCS

A Traditional client-server VCS like Subversion can, of course, be used for Ad Hoc Versioning.   With Subversion one can run svnserve, its lightweight server, in daemon mode.  Then create a task based repository:

svnadmin create /var/svn/adHoc

And, import your tree of files:

svn import c:\Users\jbetancourt\Documents\projects\adhocVersioning file:///var/svn/adHoc/project

Plus, Subversion supports offline use.  I think.  Have not used Subversion in a while.

Another effective Subversion feature is the use of local repositories using the “file:// protocol”.

Management consoles

Many systems are managed by various forms of management consoles, graphical user interfaces.  These are client or web based and may be part of the system or a third-party solution.  This is a big advantage from a hands-on administrative point of view.  However, from an automation and scripting viewpoint this is not optimal.  Thus, there is hopefully a API or tool based method of accessing the actual configuration files or databases of the system.  So in this sense, these systems are within the scope of this discussion.

This is not always the case.  One application server comes to mind that was (is?) so complex that there was no way to script it.  Thus, no way to automate the build and release process and versioning of the system.  Consequently, there was also no way to automate the various QA tests that were always panic driven and manually applied.

Managed Approach

The correct or more tractable method is to use a managed approach.  This is a software configuration and distribution system that is usually under the control of the IT staff, for example Microsoft’s System management Server (SMS) or System Center Essentials (SCE) for SMB.  Non-Microsoft solutions are of course available, such as those from IBM Tivoli’s product lineup.

Why is this not always the best approach?  There may be situations where a subset of a managed resource must be modified.  For example, you are a field service engineer and must travel or remotely connect to a client’s system to handle a service call.   This process may also entail making changes to other hosted apps and system configurations, such as network configurations.  Trying to get the IT department to collaborate or change the configuration or schedule of the managed solution may not be possible or timely.  In fact, this would be discouraged (rightly so) since it can wreak havoc on a production system.  Thus, even changing some resource may entail admin of multiple systems, not just a push of a few files and run of some batch files.  It could require interactive set and test.  Picture the Mars Rovers and how the OS reset problem was finally solved.

Closely related to the managed approach is to use a centralized version control system (VCS) or backup support.  Fortunately many operating systems have versioning capabilities built in or readily available.  For example, in the Windows platform one can make System Restore points or use the supported backup subsystems (as found in Windows 7 Professional).  Many *nix’s also have built-in versioning support in the form of installable Version Control Systems or differential backup.  In high-end virtualized systems there are facilities for backup or making snapshots and even transport of live systems.

While these work, there is a certain amount of complexity involved. Also there are issues using the same approach on multiple operating systems.  Another important drawback is that one cannot always modify the target system and, for example, install a VCS, even temporarily.  The common factor in these approaches is that there is a central “server” and associated repository for revision storage.  This is fine when available but not very conducive to ad hoc use.

Versioning File System

A VFS could be of some use.  As far as I know there are no popular file systems that support versioning (as used here).  Digital Equipment’s VAX system had single file versioning and now openVMS.  Microsoft’s Windows was supposed to have this in the future winfiles, but is no longer in the plan(?), though Windows 7 and current servers can allow access to previous file versions as a feature of its   system protection feature.  Plan 9 has a snapshot feature. ZFS has advanced stuff too and I would not be surprised if one can set a folder to be ‘versioned’.

However, a VFS would not help in task based versioning since as discussed previously, there may be a need to change multiple subsystems and track these changes as “change sets”.  Thus, a VFS is not a Revision Control System.

Of course, using a scripted solution (discussed next) in conjunction with a file change notification system (inotify), one could cobble together a folder based VCS.  However, this is outside of our lightweight requirements.

Scripted Solution

Of course, it should be possible, especially in *nix systems, to use the utilities available and construct a tool chain for a lightweight versioning support.  The rich scripting and excellent tools like rsync make this doable.  Some languages such as Perl or Python are ideal for gluing this together.

Yet, this is not optimal since these same tools will not work on all operating systems or require duplication.  For various reasons, for example, it is not always possible to install cygwin on Windows and make use of the excellent *nix utilities and scripting approach.  Likewise, it is not possible to use the outstanding Windows PowerShell in Linux.  This is only a problem of course if we are referring to empowering staff to work seamlessly on different OS or resources.  Having the same tools and workflow are valuable.

Another thing about this alternative is that a custom solution will eventually become or have functions of a version control system like Git, so why not just start with one?


One approach possible by the use of the aforementioned scripted solution is to create a snapshot system.  The DVCS gives us fine grained control of file revisions.  But, do we really need to diff and find out that a command in one batch file used the ‘-R’ option or just get the file with the desired option.  We would know which file we want using task based snapshots.  Before a task is begun, we initiate a snapshot.  This is analogous to the OS type of restore points, except we do this for a specific target set.

NoSQL Database

Finally, there have been alternatives to the Relational Database Management System (DBMS) for many years.  Most recently, this is the NoSQL group of projects such as CouchDB.    CouchDB claims that it is: “Distributed, featuring robust, incremental replication with bi-directional conflict detection and management.”   Those features sound like something an ad hoc version control system should have.  Yet, CouchDB, all?, are document centric.  Still, worth pondering.


Presented were a few thoughts on an approach to ad hoc versioning.  A DVCS was proposed as a lightweight solution and some issues were mentioned.  Alternatives were looked at.  More research is required to evaluate proposal and determine best practices for the discussed scenarios.


7/15/10:  Changed “maintain” to “accomplish” in Scenario as per feedback from K. Grover.

7/23/10:  Forgot that I visited Ben Tsai’s blog where he discusses using Mercurial within an existing VCS such as Subversion, which I’ve also done, but not really the topic I discussed.

Further Reading

“HgInit: Ground up Mercurial”,

Setting up for a Team

“Easy Automated Snapshot-Style Backups with Linux and Rsync”

Using Mercurial as ad-hoc local version control

“Intro to Distributed Version Control (Illustrated)”

Version Control,

“The Risks of Distributed Version Control”,

Subversion Re-education

“Subverting your homedir, or keeping your life in svn” (He now uses Git)

Home directory version control notification

“Managing your web site with Mercurial”, Tim Post,


Mercurial by example

Mercurial (hg) with Dropbox

Mercurial for Git users

Versioning File System

Agile Operations in the Enterprise
Michael Nygard,



Microsoft System Center Essentials

A utility that keeps track of changes to the etc configuration folder:

Version Control for Multiple Agile Teams


DVCS vs Subversion smackdown, round 3

“Using Mercurial as ad-hoc local version control”; Tsai, Ben;

Tracking /etc etc


For a more detailed exposition, see the mecurial tutorial:

The Hg manpage is available at:

There’s also a very useful FAQ that explains the terminology:

There’s also a good README:

HG behind the scenes:


Mercurial Basic workflows

Mercurial BigFiles Extension

Mercurial LargeFiles Extension

Mercurial Subrepos: A past example revisited with a new technique

Mercurial(hg) Cheatsheet for Xen

A Guide to Branching in Mercurial


Nested Repositories


Tracking 3rd-party sources



Git as an alternative to unison


Get every new post delivered to your Inbox.