Cascading Configuration Pattern

Synopsis

Many existing systems load configuration in cascade to reduce the use of duplicate properties and allow fine grained customization. This post expresses this common usage into a design pattern.

Keywords:   SCM, CM, Properties, Groovy, Config, CSS

Content:  Context, Forces, Solution, Consequences, Implementation, Code Example, Related Patterns, Related Intellectual Property, Further Reading

Context

A business system uses property (configuration) files to configure particular environments or subsystems.   Many environments share the same properties and values, however, some are different and crucial.  To avoid missing any properties, all the properties are duplicated in each file, and required differences are changed appropriately. 

For example, if there are 100 properties required and there are 23 possible environments, that is 2300 lines of source to manage.  If there are any duplicated properties that do not vary between environments, then there is an opportunity to simplify the configuration system.  In this case, if we make only a “root” file have the full set of properties,  the total size is given by: 

T = L+npL; 
where 
    T is total size, 
    n is number of files, 
    p is percent of each file that is overridden.
    L is number of properties  

Here the value is 560=100+23*.2*100). The reduction over the duplicated data is 75%.

Forces

There are many issues with having the same properties in multiple locations.  One obvious disadvantage, is adding a new property would require changing multiple files.  Also the files are larger and not really cohesive.   Tracking errors is also complicated, especially run-time errors due to configuration settings.  Any solution should not add its own complexity.  Thus, we should not require new types of configuration files and requirements.

Even when configuration could be isolated or systems modularized, there may be a need to have duplicated properties or reassignment to satisfy software development life cycle (SDLC).  A system will be different in dev, test, and production environments.

Solution

A hierarchy of property sources and a system that can load each source and override properties at a prior level will provide a partial solution.  This is the approach already taken in many systems.  For example, in software applications and servers, the final properties used are composed of those taken from various standard locations in a particular Operating System host.  In Windows for example, the Environment Variables are composed of those found in System and User name space.  In software tools, such as Mercurial DVCS, there is a command to show the results of the final configuration after all configuration sources are loaded: showconfig   show combined config settings from all hgrc files. In Git, one executes: git config -l.

Many build systems such as Ant and Maven, allow and expect the cascading of property files.  Note that in Ant, the inverse policy is used, the first property assignment wins.

And, of course, this cascading is seen in the Cascading Style Sheet (CSS) technology.

Consequences

There is now the requirement that the cascade order is well known, non-variable, and robust.  Any intervening file must be loaded, else the system may destabilize.  Adding a new property or changing a property will require extra care since an environment or subsystem may be effected if the file contents are not properly designed beforehand.

To provide management and debugging support the implementation should provide traceability of configuration.  Thus, during use, the system must report (log) what was overridden, not changed, what is missing, etc.

Implementation

Using this pattern is very easy.  One just has to determine how the system handles reassignment in the configuration loader subsystem.  If it allows resetting then the file hierarchy is from global to specific.

Where complexity may come is when one wants to satisfy more sophisticated requirements.  For example, a property may need to be appended.  In OS paths, for instance, paths are combined to create the final path.  Thus, the operations on properties could be a combination of:

  • create: initial assignment of a property
  • update:  a new value is assigned
  • delete:  the property is removed
  • append: append to existing value
  • merge:  use a pattern to merge a new value
  • locked: allow no change
  • fail on replace:  if a value already assigned throw an exception.

Code Example

In Java the java.util.Properties class will overwrite an existing property with the same key value.  Thus, to cascade properties, one just reuses the same instantiated Properties Object via the various load(…) methods.

Note that in some frameworks, like Ant, the reverse is true, properties do not get overwritten. So, the “root” properties must be loaded last in a cascade sequence.

Shown in listing one is a simple implementation written in Groovy, a dynamic language on the JVM.  The PropCascade class extends Properties and adds methods to load a list of sources.  The first source in the list is the root of the “cascade”.  In addition, a few methods explicitly specify the root source. From a traceability point of view, reusing java.util.Properties class may not be optimal. It gives no access to low level capture of the actual “put” action. Even with the use of Aspect Oriented Programming, with for example the AspectJ language, there is no joinpoint available to add it.

listing 1

/**
 * File: CascadedProperties.groovy
 * Date: 23OCT10T18:13-05
 * Author: JBetancourt
 */

import java.io.IOException;
import java.io.InputStream;
import java.util.List;
import java.util.Properties;

/**
 * An extension of Properties that adds Convenience
 * methods to load lists of sources.
 *
 * @author jbetancourt
 */
class CascadedProperties extends Properties {
	//private Properties rootProperties = new Properties();
	//private boolean firstWins = true;
	//private boolean failOnDuplicate = false;
	//private boolean isTrace = false;

	/**
	 * Load a list of properties sources.
	 * @param list
	 */
	public void load(List list) throws IOException, IllegalArgumentException {
		list.each {
			load(it)
		}
	}

	/**
	 * Explicit file path is specified.
	 * @param path
	 */
	public void load(String path) throws IOException, IllegalArgumentException {
		load(new File(path).newInputStream());
	}

	/**
	 * A load method that explicitly specifies the "default" source in
	 * the cascade order.
	 *
	 * @param inStream
	 * @param list
	 */
	public void load(InputStream inStream, List list) throws IOException, IllegalArgumentException {
		load inStream
		load list
	}

	/**
	 * A load method that explicitly specifies the "default" source in
	 * the cascade order.
	 *
	 * @param reader
	 * @param list
	 */
	public void load(Reader reader, List list) throws IOException, IllegalArgumentException {
		load reader
		load list
	}

	/**
	 * A load method that explicitly specifies the "default" source in
	 * the cascade order.
	 *
	 * @param path
	 * @param list
	 */
	public void load(String path, List list) throws IOException, IllegalArgumentException {
		load path
		load list
	}

} // end of CascadedProperties

In listing two, the JUnit test class is shown.
listing 2

/**
 * File: CascadedPropertiesTest.groovy
 * Date: 23OCT10T18:13-05
 * Author: JBetancourt
 */

import java.io.File;
import java.io.FileInputStream;
import java.util.List;

import org.junit.Before;
import org.junit.Test;

import groovy.util.GroovyTestCase;

/**
 * Test the {@link CascadedProperties} class.
 */
class CascadedPropertiesTest extends GroovyTestCase{
	private CascadedProperties cp;

	/** excuted before each test method run */
	public void setUp() throws Exception {
		cp = new CascadedProperties();
	}

	public void testloadListPaths() throws Exception {
		List list = new ArrayList();
		list.add path1
		list.add path2

		cp.load(list);

		assertEquals("v2",cp.get("k1"));
	}

	public void testloadListReaders() throws Exception {
		List list = new ArrayList();
		list.add reader1
		list.add reader2

		cp.load(list);

		assertEquals("v2",cp.get("k1"));
	}

	public void testloadListStreams() throws Exception {
		List list = new ArrayList();
		list.add iStream1
		list.add iStream2

		cp.load(list);

		assertEquals("v2",cp.get("k1"));
	}

	public void testloadStreamAndListStreams() throws Exception {
		List list = new ArrayList();
		list.add iStream2
		list.add iStream3

		cp.load(iStream1,list);

		assertEquals("v3",cp.get("k1"));
	}

	public void testloadPathAndListStreams() throws Exception {
		List list = new ArrayList();
		list.add iStream2
		list.add iStream3

		cp.load("data\\file1.properties",list);

		assertEquals("v3",cp.get("k1"));
	}

	public void testloadReaderAndListStreams() throws Exception {
		List list = new ArrayList();
		list.add reader2
		list.add reader3

		cp.load(reader1,list);

		assertEquals("v3",cp.get("k1"));
	}

	public void testPutAgain() {
		cp.put("k1", "v1")
		cp.put("k1", "v2")
		assertEquals(cp.get("k1"), "v2");
	}

	public void testLoadOneFilePath() throws Exception {
		cp.load("data\\file1.properties");
		assertEquals("v1",cp.get("k1"));
	}

	public void testLoadTwoFiles() throws Exception {
		cp.load(iStream1)
		cp.load(iStream2)

		assertEquals("v2",cp.get("k1"));
	}

	//
	// class fields
	//
	String path1 = "data\\file1.properties"
	String path2 = "data\\file2.properties"
	String path3 = "data\\file3.properties"
	File file1 = new File(path1)
	File file2 = new File(path2)
	File file3 = new File(path3)

	InputStream iStream1 = file1.newInputStream()
	InputStream iStream2 = file2.newInputStream()
	InputStream iStream3 = file3.newInputStream()
	Reader reader1 = file1.newReader()
	Reader reader2 = file2.newReader()
	Reader reader3 = file3.newReader()

}

Property Files Analysis

It is very difficult to eyeball a set of property files to see if there are many duplicates. I guess one could do some magic one line script that concatenates, sorts, prunes, etc. Here is an alternative using the Groovy language.
listing 3

/*
 * Script: AnalyizePropertyFiles.groovy
 * Author: J. Betancourt
 */
import java.util.Hashtable;
import java.util.Map.Entry;

/**
 * @author jbetancourt
 */
class AnalyizePropertyFiles {
	// these could have been created dynamically by reading the target folder.  But, that may pick up non-used files.
	static fileNames = []

	// where are the files?
	static basePath = ""

	// run analysis....
	static main(args) {
		def propList = [] // each property file will be load into a cell which contains a properties object.

		// put each file into properties object in propList.
		fileNames.each {
			def p = new Properties()
			p.load(new File(basePath + "\\" + it).newInputStream())
			propList.add(p)
		}

		// put all properties in one allProps object.
		Properties allProps = new Properties();
		propList.each {
			allProps.putAll(it)
		}// each propList
		
		def result = []
		
		int totalCount = 0
		
		// get how many times each property is used in all properties object
		allProps.each { prop ->
			String key = prop.key
			String value = prop.value
			def values = []
			int count = 0
			propList.each { p ->
				if(p.containsKey(key)){
					count++
					def curValue = p.get(key)
					if(!values.contains(curValue)){
						values.add(curValue)
					}								
				}
			} // each property file
				
			StringBuilder vb = new StringBuilder()
			values.each{ s ->
				vb.append("[" + s + "], ")
			}
			
			int numberValues = values.size()

			StringBuilder sb = new StringBuilder()			
			sb.append(count).append(",").append(values.size).append(", ").append(key).append(", ").append(vb.toString())
			result.add(sb.toString())	
			
			totalCount += count				
			
		} // each allProps property
		
		
		result.each{
			println(it)
		}
		
		println("Total number of times properties are repeated: " + totalCount)
		
	} // end main

} // end class

Related Patterns

Related Intellectual Property

Cascading configuration using one or more configuration trees“, U.S. Patent number 7760746, 30Nov2004, http://patft.uspto.gov/netacgi/nph-Parser?Sect2=PTO1&Sect2=HITOFF&p=1&u=%2Fnetahtml%2FPTO%2Fsearch-bool.html&r=1&f=G&l=50&d=PALL&RefSrch=yes&Query=PN%2F7760746


Further Reading


Thelonious Monk – ‘Round Midnight – 1966

About these ads

3 Responses to Cascading Configuration Pattern

  1. Link to ConfigureMe is outofdate, the proper link is http://www.configureme.org

  2. [...] of Unix Programming, Eric Steven Raymond. – ConfigParser – Work with configuration files – JSON – Cascading Configuration Pattern – java.util.Properties – RFC 4627 – XML-SW,a skunkworks project by Tim Bray. Brings a bunch of XML [...]

  3. [...] of Unix Programming, Eric Steven Raymond. – ConfigParser – Work with configuration files – JSON – Cascading Configuration Pattern – java.util.Properties – RFC 4627 – XML-SW,a skunkworks project by Tim Bray. Brings a bunch of XML [...]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: