Jenkins CI Server is great

March 16, 2012

Finally got a Jenkins server installed. Had a host of system issues, like communicating to our source code repo.

Jenkins is a joy to use. Well, it is not perfect, what is? Like, I need to pass the user’s name that invoked a build via Jenkins to the target DOS script (yea, Windows) that eventually invokes the legacy Ant scripts. A quick Google search shows that this is asked in various ways, but no answers. For example, here or here. Hmmmm.

Anyway, now comes a trial use, to see if it is what we really need and can we manage it to do what we will want. With 400 plugins, I don’t see how it could lack. Plus, I’m sure I can use the Groovy plugin to cobble something up. Jenkins even includes a Groovy Console. Finally, there is a road map for possible migration of legacy Ant scripts to Gradle using the Gradle Plugin.

I take back my past snarky comment. Jenkins is not just a pretty face on Cron.

BTW, is there some Wiki law that says a wiki shall never ever have a link to the parent project? If you get tossed into a wiki by following a link, invariably you will click in agony at links that should go to the real home. Instead, you have to edit the URL in the address bar. Since I never curse, I can’t write “wtf”.

Off Topic
Was watching the Easyb intro video. BDD is interesting. Definitely “should” is a better then “test”. With so many great tools why are products still bug ridden?

More stuff

  1. Jenkins home page
  2. Continuous integration Not a very good Wikipedia article
  3. Continuous Integration Much better
  4. Continuous Integration in Agile Software Development
  5. Hooking into the Jenkins(Hudson) API
  6. Five Cool Things You Can Do With Groovy Scripts
  7. Parameterized Builds in Jenkins – choosing subversion folders
  8. Groovy Console
  9. Groovy plugin
  10. Switching to Jenkins–Download and Install Artifact Script for Tester
  11. Gradle Plugin

Virtual Machine Applicance for development environment

January 1, 2012

Configuration of a development environment can be very time consuming, error prone, or difficult. This is especially true when investigating or getting up to speed on a new technology or framework. In a corporate environment this is a also a drain on resources and existing developer staff who must take the time to prep a new developer.

One approach to mitigate this is to use a Virtual Appliance.

Virtual appliances are a subset of the broader class of software appliances. Installation of a software appliance on a virtual machine creates a virtual appliance. Like software appliances, virtual appliances are intended to eliminate the installation, configuration and maintenance costs associated with running complex stacks of software.

A virtual appliance is not a complete virtual machine platform, but rather a software image containing a software stack designed to run on a virtual machine platform which may be a Type 1 or Type 2 hypervisor. Like a physical computer, a hypervisor is merely a platform for running an operating system environment and does not provide application software itself. — Virtual Appliance

Creating a Virtual Machine Applicance
The available VM software such as Oracle VirtualBox and the VMware VM have facilities to generate appliances. Thus, when a functioning development environment is created by a lead tech or group, an appliance can be generated for the rest of the team. This appliance can even be provided using a Virtual Desktop Infrastructure (VDI).

Open Virtualization Format
While a VM system can be used to create individual VM instances that can be reused, a more recent technology (supported by some vendors) is the use of OVF:

… is an open standard for packaging and distributing virtual appliances or more generally software to be run in virtual machines.

The standard describes an “open, secure, portable, efficient and extensible format for the packaging and distribution of software to be run in virtual machines”. The OVF standard is not tied to any particular hypervisor or processor architecture. The unit of packaging and distribution is a so called OVF Package which may contain one or more virtual systems each of which can be deployed to a virtual machine.

An OVF package consists of several files, placed in one directory. A one-file alternative is the OVA package, which is a TAR file with the OVF directory inside. — http://en.wikipedia.org/wiki/Open_Virtualization_Format

Using ready made appliances
Each VM vendor can/does make available an appliance marketplace. Thus, one can find ready-made LAMP based environments with a development software stack, for example.

Alternative 1, an installable virtual disk
Where resources are constrained, such as places where developers are still on 3GB of ram and ancient PCs, a Virtual Machine is just not going to cut it.

One easy alternative is to create a dev environment on an installable soft hard drive. TrueCrypt can be used for this purpose. One simply create a true crypt volume, which is just a single file. Then creates the desired dev env in that volume, and that file can now be copied to load into other dev’s workstations as a new hard drive.

TrueCrypt is really for security and privacy concerns, it encrypts data, so may not be ideal for this application. Since TrueCrypt is so useful as a virtual disk, it would be great if it had the option of not encrypting content. But, that would perhaps be outside of its feature space. For that the next alternative is available.

Alternative 2, use VHD files
An alternative is using something directly targeted at virtual disks such as the VHD file format. However, this does not seem to have easily useful public gui or command support (for the end user: developer).

On Windows following the instructions here and using these Send To scripts will allow one to seamlessly use vhd files as mountable hard disk volumes.

Note that Windows 8 will support native mounting of ISO and VHD files.

Further Reading


KARSH KALE plays “MILAN” LIVE


Hudson/Jenkins CI Server, can’t edit a job?

August 3, 2011

I was looking at a possible use of a Continuous Integration Server to quickly set up a build process.

Downloaded the Jenkins war file, put into Tomcat and defined a simple Job that invoked an Ant file to echo “Hello World!”. Cool, that was easy. But, then I wanted to expand that job to do more. Could not find a modify or edit capability. Huh? What’s up with that?

I searched and found very little. There was even some mention of using SED to edit the Job configuration XML, yeeech! Edit using a text tool for a tree-based data structure?

Anyway, not impressed. Of course, this was a quick tryout. Or maybe Linux people are so perfect they never have to edit their work. :)

Is Jenkins/Hudson just a pretty face on *nix utilities?

I looked at a few other CI Servers. So far Pulse and Team City look interesting, but they are not free.

Updates
Mar 2, 2012: Used one of the latest Jenkins version. Much much better! Though I’m having issues getting Active Directory authentication going. Can log in ok, but then it uses the wrong user “name” that our PCs must use. You know how LDAP has all this distinguished this or that.

Links

  1. Jenkins
  2. Active Directory Plugin

JSON configuration file format

May 8, 2011

JSON is a data interchange format. Should it also be used as a configuration file format, a JSON-CF?

Overview

Had to write yet another properties file for configuration info. Started to think that maybe there are better alternatives. Wondered about JSON for this.

Requirements

What are requirements of a configuration file format?

  • Simple
  • Human readable
  • Cross platform
  • Multi-language support
  • Unicode support

Looks like JSON has all the right qualities.

If all you want to pass around are atomic values or lists or hashes of atomic values, JSON has many of the advantages of XML: it’s straightforwardly usable over the Internet, supports a wide variety of applications, it’s easy to write programs to process JSON, it has few optional features, it’s human-legible and reasonably clear, its design is formal and concise, JSON documents are easy to create, and it uses Unicode.
— Norman Walsh, Deprecating XML

JSON-CF Limitations

  • Instead of angle brackets as in XML, we have quotation marks everywhere.

What does it need?

  • Inline comments, see for example, json-comments
  • Interpolation (property expansion)
  • Namespaces
  • Inheritance
  • Includes
  • Date value
  • Schema
  • Cascading

Example

{
    "_HEADER":{
            "modified":"1 April 2001",
            "dc:author": "John Doe"
    },
    "logger_root":{
            "level":"NOTSET",
            "handlers":"hand01"
     },
    "logger_parser":{
            "level":"DEBUG",
            "handlers":"hand01",
            "propagate":"1",
            "qualname":"compiler.parser"
    },
    "owner":{
             "name":"John Doe",
             "organization":"Acme Widgets Inc."
     },
     "database":{
             "server":"192.0.2.62",     
             "_comment_server":"use IP address in case network name resolution is not working",
             "port":"143",
             "file":"payroll.dat"
      }
}

Programmatic Access using Groovy

Now we can easily read this file in Java. Using Groovy is much easier, of course. Groovy version 1.8 has built-in JSON support, great blog post on this here.

import groovy.json.*;

def result = new JsonSlurper().
                          parseText(new File("config.json").text)

result.each{ section ->
	println "$section\n"
}

>groovy readConfig.groovy
Resulting in Groovy style data structure, GRON, (look ma, no quotation marks):

logger_parser={qualname=compiler.parser, level=DEBUG, propagate=1, handlers=hand01}

owner={organization=Acme Widgets Inc., name=John Doe}

_HEADER={dc:author=John Doe, modified=1 April 2001}

database={port=143, file=payroll.dat, server=192.0.2.62, _comment_server=use IP address in case network name resolution is not working}

logger_root={level=NOTSET, handlers=hand01}

In Groovy you can access the data with GPath expressions:
println “server: ” + result.database.server

You can also pretty print JSON, for example:
println JsonOutput.prettyPrint(new File(“config.json”).text)

Summary

Raised the question of the use of JSON as a configuration file format.

What I don’t like is the excess quotation marks. YAML is more attractive in this sense. But, the indentation as structure in YAML, similar to Python, may not be wise in configuration files.

Well, what is the answer, should there be a JSON-CF? I don’t know. A very perceptive caution is given by Dare Obasanjo commenting on use of new techniques in general:

So next time you’re evaluating a technology that is being much hyped by the web development blogosphere, take a look to see whether the fundamental assumptions that led to the creation of the technology actually generalize to your use case.

Updates

  • After writing and posting this I searched the topic and, of course, this is not a new question. Search. I updated the reading list below with some useful links.
  • JSON Activity Streams are an example of how JSON is used in new ways.
  • schema.org types and properties as RDFS in the JSON format: schema.rdfs.org/all.json
  • Just learned that node.js uses the NPM package manager which uses a JSON config file format.
  • Jan 7, 2012: Java JSR 353: Java API for JSON Processing

Further Reading
A very simple data file metaformat
JSON configuration file format
GRON
Cascading Configuration Pattern
NPM configuration file format
XML or YAML for configuration files
Using JSON for Language-independent Configuration Files
INI file
JSON+Comments
Comparison of data serialization formats
Data File Formats, in Art of Unix Programming, Eric Steven Raymond.
ConfigParser – Work with configuration files
JSON
java.util.Properties
RFC 4627
XML-SW,a skunkworks project by Tim Bray. Brings a bunch of XML complex together into one spec.
YAML
ISO 8601
Learning from our Mistakes: The Failure of OpenID, AtomPub and XML on the Web
Groovy 1.8 Introduces Groovy to JSON
JSON-LD
JSON Schema

 


” Sacred Place ” , R.Towner / P. Fresu, live in Innsbruck , Part 4


Easy stream parsing using Groovy, CVS example

March 22, 2011

You use every combination of options but that dam command won’t give you what you want?

I faced this last week at work. I had to get a list of my commits to CVS. I tried a bunch of stuff and also searched for a solution. None really worked well. An example of an approach is shown here: “how to search cvs comment history“.

The root problem is that the output of many tools are not always easily reusable. In this situation (and I’m sure in more modern tools like Subversion, Git, or Mercurial) the output resembles (I took out work related info):

=============================================================================
RCS file: /cvs/A...
Working file: Java So..
head: 1.1
branch:
locks: strict
access list:
keyword substitution: kv
total revisions: 4;     selected revisions: 3
description:
----------------------------
revision 1.1
date: 2011/03/  
filename: Produc...tsA
branches:  1....;
file Produ...
----------------------------
revision 1.1.4.1
date: 201....
filename: ProductsA....;
AS.....
----------------------------
revision 1.1.2.1
date: 2011/0
filename: ProductsA....;
ExampleNightMare - ....
=============================================================================

RCS file: /cvs/Am...
Working file: Java S..
head: 1.1
branch:
locks: strict
access list:
keyword substitution: kv
total revisions: 4;     selected revisions: 3
description:
----------------------------
revision 1.1
date: 2011/03/  
filename: Pro...

This output goes on for thousands of lines! Sure if you use a tool often and dug into its idioms or have a guru near by, you could probably get what you want, but …. (of topic, but why don’t Man pages and other docs give examples for every option?).

There is no need to take out the dragon book and start writing a parser (is ‘parser’ the correct term in this context?), or even create a DSL. If your very familiar with real scripting languages like Python, Perl, or even pure shell utilities, this is easy. If your not, on Windows (and don’t use Powershell), or just as another approach, Groovy is easy to use.

The usual pattern I would imagine is to just just read the input and trigger on a start phrase to indicate a block of interest, then the data is captured when the including line is subsequently detected in the input stream. However, in my situation depicted above, I did the opposite, I got the data I needed, but only printed it out when I got a subsequent trigger phrase, the commit comment.

Sure you could generalize or find some tool that does this, but you’d probably spend more time learning the tool or creating a reusable system that only you need or understand.

// file: ParseCvsLog_1.groovy
// Author: jbetancourt

def inside = false
def workingFile

new BufferedReader(new InputStreamReader(System.in)).eachLine(){ s ->
	
	if(s.startsWith("Working file:")){
		inside = true
		workingFile = s.split("Working file:")[1] // got what I want?
	}
  
	// this indicates that it is.
	def found = s ==~ /.*ExampleNightMare.*/
	if(found){
		println(workingFile)   // send to next pipe
		inside = false
	}  
	
}

Probably not a good example of idiomatic Groovy code, but easy to follow. A Groovy expert could probably do it on one line (I don’t like those smarty one-liners; one week later, you don’t know what you did.).

This is used as (all one line):

cvs inscrutable bunch of gibberish that doesn't answer question | groovy ParseCvsLog_1.groovy > myChanges.txt

Nothing new in this post, of course. The value of any scripting approach is that it is infinitely adaptable. And, when the scripting language is easy to use, the results could even be reusable. Perl, Python, and Ruby, for example, have great facilities for sharing of snippets and modular code solutions. Groovy and other JVM based languages like Scala are beginning to add this capability to Java environments.

Updates

  • 20110323T1906-5: Cleaned up the sample code a little; don’t want to give the wrong impression.
  • 20110402T1702-5: While looking thru the book “Groovy In Action” noticed that section 13.5.3 Inspecting version control, deals with this subject.

Further Reading


Cascading Configuration Pattern

October 24, 2010

Synopsis

Many existing systems load configuration in cascade to reduce the use of duplicate properties and allow fine grained customization. This post expresses this common usage into a design pattern.

Keywords:   SCM, CM, Properties, Groovy, Config, CSS

Content:  Context, Forces, Solution, Consequences, Implementation, Code Example, Related Patterns, Related Intellectual Property, Further Reading

Context

A business system uses property (configuration) files to configure particular environments or subsystems.   Many environments share the same properties and values, however, some are different and crucial.  To avoid missing any properties, all the properties are duplicated in each file, and required differences are changed appropriately. 

For example, if there are 100 properties required and there are 23 possible environments, that is 2300 lines of source to manage.  If there are any duplicated properties that do not vary between environments, then there is an opportunity to simplify the configuration system.  In this case, if we make only a “root” file have the full set of properties,  the total size is given by: 

T = L+npL; 
where 
    T is total size, 
    n is number of files, 
    p is percent of each file that is overridden.
    L is number of properties  

Here the value is 560=100+23*.2*100). The reduction over the duplicated data is 75%.

Forces

There are many issues with having the same properties in multiple locations.  One obvious disadvantage, is adding a new property would require changing multiple files.  Also the files are larger and not really cohesive.   Tracking errors is also complicated, especially run-time errors due to configuration settings.  Any solution should not add its own complexity.  Thus, we should not require new types of configuration files and requirements.

Even when configuration could be isolated or systems modularized, there may be a need to have duplicated properties or reassignment to satisfy software development life cycle (SDLC).  A system will be different in dev, test, and production environments.

Solution

A hierarchy of property sources and a system that can load each source and override properties at a prior level will provide a partial solution.  This is the approach already taken in many systems.  For example, in software applications and servers, the final properties used are composed of those taken from various standard locations in a particular Operating System host.  In Windows for example, the Environment Variables are composed of those found in System and User name space.  In software tools, such as Mercurial DVCS, there is a command to show the results of the final configuration after all configuration sources are loaded: showconfig   show combined config settings from all hgrc files. In Git, one executes: git config -l.

Many build systems such as Ant and Maven, allow and expect the cascading of property files.  Note that in Ant, the inverse policy is used, the first property assignment wins.

And, of course, this cascading is seen in the Cascading Style Sheet (CSS) technology.

Consequences

There is now the requirement that the cascade order is well known, non-variable, and robust.  Any intervening file must be loaded, else the system may destabilize.  Adding a new property or changing a property will require extra care since an environment or subsystem may be effected if the file contents are not properly designed beforehand.

To provide management and debugging support the implementation should provide traceability of configuration.  Thus, during use, the system must report (log) what was overridden, not changed, what is missing, etc.

Implementation

Using this pattern is very easy.  One just has to determine how the system handles reassignment in the configuration loader subsystem.  If it allows resetting then the file hierarchy is from global to specific.

Where complexity may come is when one wants to satisfy more sophisticated requirements.  For example, a property may need to be appended.  In OS paths, for instance, paths are combined to create the final path.  Thus, the operations on properties could be a combination of:

  • create: initial assignment of a property
  • update:  a new value is assigned
  • delete:  the property is removed
  • append: append to existing value
  • merge:  use a pattern to merge a new value
  • locked: allow no change
  • fail on replace:  if a value already assigned throw an exception.

Code Example

In Java the java.util.Properties class will overwrite an existing property with the same key value.  Thus, to cascade properties, one just reuses the same instantiated Properties Object via the various load(…) methods.

Note that in some frameworks, like Ant, the reverse is true, properties do not get overwritten. So, the “root” properties must be loaded last in a cascade sequence.

Shown in listing one is a simple implementation written in Groovy, a dynamic language on the JVM.  The PropCascade class extends Properties and adds methods to load a list of sources.  The first source in the list is the root of the “cascade”.  In addition, a few methods explicitly specify the root source. From a traceability point of view, reusing java.util.Properties class may not be optimal. It gives no access to low level capture of the actual “put” action. Even with the use of Aspect Oriented Programming, with for example the AspectJ language, there is no joinpoint available to add it.

listing 1

/**
 * File: CascadedProperties.groovy
 * Date: 23OCT10T18:13-05
 * Author: JBetancourt
 */

import java.io.IOException;
import java.io.InputStream;
import java.util.List;
import java.util.Properties;

/**
 * An extension of Properties that adds Convenience
 * methods to load lists of sources.
 *
 * @author jbetancourt
 */
class CascadedProperties extends Properties {
	//private Properties rootProperties = new Properties();
	//private boolean firstWins = true;
	//private boolean failOnDuplicate = false;
	//private boolean isTrace = false;

	/**
	 * Load a list of properties sources.
	 * @param list
	 */
	public void load(List list) throws IOException, IllegalArgumentException {
		list.each {
			load(it)
		}
	}

	/**
	 * Explicit file path is specified.
	 * @param path
	 */
	public void load(String path) throws IOException, IllegalArgumentException {
		load(new File(path).newInputStream());
	}

	/**
	 * A load method that explicitly specifies the "default" source in
	 * the cascade order.
	 *
	 * @param inStream
	 * @param list
	 */
	public void load(InputStream inStream, List list) throws IOException, IllegalArgumentException {
		load inStream
		load list
	}

	/**
	 * A load method that explicitly specifies the "default" source in
	 * the cascade order.
	 *
	 * @param reader
	 * @param list
	 */
	public void load(Reader reader, List list) throws IOException, IllegalArgumentException {
		load reader
		load list
	}

	/**
	 * A load method that explicitly specifies the "default" source in
	 * the cascade order.
	 *
	 * @param path
	 * @param list
	 */
	public void load(String path, List list) throws IOException, IllegalArgumentException {
		load path
		load list
	}

} // end of CascadedProperties

In listing two, the JUnit test class is shown.
listing 2

/**
 * File: CascadedPropertiesTest.groovy
 * Date: 23OCT10T18:13-05
 * Author: JBetancourt
 */

import java.io.File;
import java.io.FileInputStream;
import java.util.List;

import org.junit.Before;
import org.junit.Test;

import groovy.util.GroovyTestCase;

/**
 * Test the {@link CascadedProperties} class.
 */
class CascadedPropertiesTest extends GroovyTestCase{
	private CascadedProperties cp;

	/** excuted before each test method run */
	public void setUp() throws Exception {
		cp = new CascadedProperties();
	}

	public void testloadListPaths() throws Exception {
		List list = new ArrayList();
		list.add path1
		list.add path2

		cp.load(list);

		assertEquals("v2",cp.get("k1"));
	}

	public void testloadListReaders() throws Exception {
		List list = new ArrayList();
		list.add reader1
		list.add reader2

		cp.load(list);

		assertEquals("v2",cp.get("k1"));
	}

	public void testloadListStreams() throws Exception {
		List list = new ArrayList();
		list.add iStream1
		list.add iStream2

		cp.load(list);

		assertEquals("v2",cp.get("k1"));
	}

	public void testloadStreamAndListStreams() throws Exception {
		List list = new ArrayList();
		list.add iStream2
		list.add iStream3

		cp.load(iStream1,list);

		assertEquals("v3",cp.get("k1"));
	}

	public void testloadPathAndListStreams() throws Exception {
		List list = new ArrayList();
		list.add iStream2
		list.add iStream3

		cp.load("data\\file1.properties",list);

		assertEquals("v3",cp.get("k1"));
	}

	public void testloadReaderAndListStreams() throws Exception {
		List list = new ArrayList();
		list.add reader2
		list.add reader3

		cp.load(reader1,list);

		assertEquals("v3",cp.get("k1"));
	}

	public void testPutAgain() {
		cp.put("k1", "v1")
		cp.put("k1", "v2")
		assertEquals(cp.get("k1"), "v2");
	}

	public void testLoadOneFilePath() throws Exception {
		cp.load("data\\file1.properties");
		assertEquals("v1",cp.get("k1"));
	}

	public void testLoadTwoFiles() throws Exception {
		cp.load(iStream1)
		cp.load(iStream2)

		assertEquals("v2",cp.get("k1"));
	}

	//
	// class fields
	//
	String path1 = "data\\file1.properties"
	String path2 = "data\\file2.properties"
	String path3 = "data\\file3.properties"
	File file1 = new File(path1)
	File file2 = new File(path2)
	File file3 = new File(path3)

	InputStream iStream1 = file1.newInputStream()
	InputStream iStream2 = file2.newInputStream()
	InputStream iStream3 = file3.newInputStream()
	Reader reader1 = file1.newReader()
	Reader reader2 = file2.newReader()
	Reader reader3 = file3.newReader()

}

Property Files Analysis

It is very difficult to eyeball a set of property files to see if there are many duplicates. I guess one could do some magic one line script that concatenates, sorts, prunes, etc. Here is an alternative using the Groovy language.
listing 3

/*
 * Script: AnalyizePropertyFiles.groovy
 * Author: J. Betancourt
 */
import java.util.Hashtable;
import java.util.Map.Entry;

/**
 * @author jbetancourt
 */
class AnalyizePropertyFiles {
	// these could have been created dynamically by reading the target folder.  But, that may pick up non-used files.
	static fileNames = []

	// where are the files?
	static basePath = ""

	// run analysis....
	static main(args) {
		def propList = [] // each property file will be load into a cell which contains a properties object.

		// put each file into properties object in propList.
		fileNames.each {
			def p = new Properties()
			p.load(new File(basePath + "\\" + it).newInputStream())
			propList.add(p)
		}

		// put all properties in one allProps object.
		Properties allProps = new Properties();
		propList.each {
			allProps.putAll(it)
		}// each propList
		
		def result = []
		
		int totalCount = 0
		
		// get how many times each property is used in all properties object
		allProps.each { prop ->
			String key = prop.key
			String value = prop.value
			def values = []
			int count = 0
			propList.each { p ->
				if(p.containsKey(key)){
					count++
					def curValue = p.get(key)
					if(!values.contains(curValue)){
						values.add(curValue)
					}								
				}
			} // each property file
				
			StringBuilder vb = new StringBuilder()
			values.each{ s ->
				vb.append("[" + s + "], ")
			}
			
			int numberValues = values.size()

			StringBuilder sb = new StringBuilder()			
			sb.append(count).append(",").append(values.size).append(", ").append(key).append(", ").append(vb.toString())
			result.add(sb.toString())	
			
			totalCount += count				
			
		} // each allProps property
		
		
		result.each{
			println(it)
		}
		
		println("Total number of times properties are repeated: " + totalCount)
		
	} // end main

} // end class

Related Patterns

Related Intellectual Property

Cascading configuration using one or more configuration trees“, U.S. Patent number 7760746, 30Nov2004, http://patft.uspto.gov/netacgi/nph-Parser?Sect2=PTO1&Sect2=HITOFF&p=1&u=%2Fnetahtml%2FPTO%2Fsearch-bool.html&r=1&f=G&l=50&d=PALL&RefSrch=yes&Query=PN%2F7760746


Further Reading


Thelonious Monk – ‘Round Midnight – 1966


Follow

Get every new post delivered to your Inbox.