Version control console using ReactJS

Created a web app to generate a report from the version control repository, Apache Subversion™. Similar approach is possible targeting a different repository, like Git.

TL;DR
Someone said a process we follow could not be automated. I took that as a challenge and created a proof of concept (POC) tool.

The final GUI using ReactJS is relatively complex: five data tables that hide/show/expand/collapse. Searches on those tables, sorting, navigation links, help page, Ajax requests to access Subversion repo data, export to CSV, report generation, and client side error handling. It started as just a GUI to a report, but since it was easy, added more features: Zawinski’s law.

To top it off, the app had to automatically invoke the default workflow or no one would use it.

Result?
1. It is a complex disaster that works. And works surprisingly fast. Using ReactJS and Flux made it into a fast elegant (?) disaster that works … kind of.
2. The app served as an example of a SPA in the our dev group. But, mostly to try out the ReactiveJS approach.
3. My gut feel is that there are some fundamental problems in the client side MV* approach which leads to control flow spaghetti (a future blog post).

Since the time I wrote that app I have noticed a mild push back on React that point out the hidden complexities. There are now new ideas and frameworks, like Redux or Cycle.js. Most recently, to tackle the action-coordination, there is much digital ink written on Redux Sagas, for example: “Managing Side Effects In React + Redux Using Sagas“.

Note, though there are critiques of the ReactJS approach or implementation, this does not imply that React was not a real breakthrough in front end development.

Report generation
Creating simple reports from a version control repository can be accomplished with command line tools or querying XML output from SVN log commands. In this case generating the criteria to generate the report was the hard part. Details are not relevant here: this web app would remove a lot of manual bookkeeping tasks that our group currently must follow due to a lot of branch merging and use of reports for error tracking, verification, and traceability. Yup, long ago legacy Standard Operating Procedures (SOP) of an IT shop.

Architecture
Server
A simple Java web app was created and deployed to a Tomcat server. A Java Servlet was used at the server to receive and send JSON data to the browser based client. This server communicates with the version control repository server.

Client
The browser is the client container with ReactJS as the View layer and Flux (implemented in the McFly library) as the framework client implementation. Dojo was used as the JavaScript tools library. Dojo supplied the Promise and other cross-browser capabilities. Why Dojo? That is already in use here. If we were using JQuery, that is what I would use.

Local application service
Performance note: Since the repo query and processing occurs at the server, multiple developers accessing the service would have a performance impact. A future effort is to deploy this as an runnable Jar application (Spring Boot?) that starts an embedded app server, like Tomcat or Jetty, at the developer’s workstation. The browser would still be used as the client.

Repository Query
Some options to generate SVN reports:

1. Use a high level library to access SVN information.
2. Export SVN info to a database, SQL or NoSQL.
3. Use an OS or commercial SVN report generator.
4. Use command line XML output option to create a navigational document object model (DOM)
5. Use SVN command line to capture log output, and apply a pipeline of Linux utilities.

This was a ‘skunkworks’ project to determine if some automation of a manual process could be done and most importantly, if doable, would the resulting tool be used? The first option, seemed easiest, and was chosen. The repo was accessed with the SvnKit Java library. (For Java access to a Git repo, JGit is available).

The process approach was to generate and traverse a Log collection. A simple rule engine was executed (just a series of nested conditionals) to determine what to add from the associated Revision objects.

This seemed like a workable idea until a requirement was requested after the POC was completed: instead of listing a particular source file once per report, show multiple times per each developer who made a commit to it. An easy change if this were implemented as an SVN log query sent to a pipe of scripts. However, with the design this required going into the nuts and bolts of the “rule engine” to add support for filtering, and further changes to the model.

Yup, a POC solution can be a big ball of mud, and unfortunately can be around a long time. Incidentally, this happened with Jenkins CI; where I …

Very recently a flaw in the design will force a revisit of the algorithm again. Instead of making the ‘rule engine’ more powerful, an alternative approach is to start from a Diff collection. The diff result would be used to navigate the Log collection. A similar approach is shown here: http://www.wandisco.com/svnforum/forum/opensource-subversion-forums/general-setup-and-troubleshooting/6238-svn-log-without-mergeinfo-changes?p=36934#post36934

But, already a problem was found with diff output. There is no command line or Java library support for pruning of deleted folders. For example, if a/b/c is a hierarchy, and you delete b, c is also deleted. Now if you generate a diff, the output would contain delete entries for: a/b and a/b/c. What was needed was just a/b. Simple, you say. Sure, but this information is a OOP object graph, so can be complicated.

I solved it: a diff list of 1800 folders was reduced to just 8 folders! I’m surprised a solution or something similar was not found in a web search. I wrote about this in “Can conditional state be used in RxJava Observable streams?”

Perhaps revisit the alternative approaches, like export to database? Not sure if this really would have simplified things, but instead just change where the complexity was located. Is the complexity of a software solution a constant?

Other systems take this export approach. One system I saw years ago, exports the version control history (it was CVS) into an external SQL database and then used queries to provide required views.

Client Single-Page Application
What to use as the browser client technology? From past experience, I did not want go down the path of using event handlers all over the place and complex imperative DOM updates.

Anyway, React seemed interesting and had a shorter learning curve. I looked at Angular, but it seemed to be the epitome of embedding the developer into the product (future blog post on the application development self-deception).

A few ReactJS components were created:

• BranchSelect
• CommentLines
• ControlPanel
• DiffTable
• ErrorPanel
• ExclusionRow
• ExclusionTable
• FilesRow
• FilesTable
• ManifestRow
• ManifestTable
• ProgramInfo
• ProjectPanel
• RevisionRow
• RevisionTable
• ViewController

Lessons Learned
This project progressed very quickly. React seemed very easy. But, that was only temporary. Until you understand a library or a paradigm, start with a smaller application. Really understand it. Of course, these too can fool you. For example, when this app first loads, I had to invoke the most likely use-case. There was a endless challenge of chicken/egg model flow disasters. Solved it, but can’t understand how I did it. Somehow I tamed the React flow callbacks. Or this is just a lull and will blow up in as a result of an unforeseen user interaction.

All the new cutting edge JavaScript front-end frameworks are very very complex for us average developers. Just the tooling is daunting if your coming from a vanilla JavaScript shop. Node dis! See this: ‘I’m a web developer and I’ve been stuck with the simplest app for the last 10 days‘

Next SPA project?
My next app will probably use Redux as the Flux framework. Or may leave Reactjs and go directly with Cycle.js which is looking very good in terms of ‘principles’ or conceptual flow and is truly Reactive, based on a ReactiveX library: RxJS.

Links

• Getting Out of JavaScript Fatigue – React Edition
• React.js Lifecycle Diagram
• Reactive client application state constraint contracts
• 19 Eponymous Laws Of Software Development
• What reporting is available for SVN
• SvnKit
• ReactJS
• McFly
• Use JAXB to export Subversion log to Java
• Run svn command from Java using Commons exec
• Getting “svn info” using Groovy
• RxJS
• I’m a web developer and I’ve been stuck with the simplest app for the last 10 days
• StatSVN
• Transform SVN log diff using Java XPath
• Easy stream parsing using Groovy, CVS example
• SVN log without mergeinfo changes?, accessed 9/12/2015.
• Facebook Open Sources Data Query Language GraphQL
• Why I No Longer Use MVC Frameworks

Prune deleted folders of repo diff report using Java

Sometimes the contents of deleted folders are not relevant. Just showing the top level deleted folder or dir is enough. For example, in a branch diff against trunk, folder moves are shown as adds and deletes. But in a diff report, the whole sub node of the deleted folder is output.

In the particular report I was creating everything under the deleted directories was just noise. How to prune that?

BTW, I searched the SVN docs and found no support for this kind of pruning. I also looked at the Git docs to see if it was different, but also found no mention of this use-case.

Example scenario:

d1 (modified)
├ d2 (deleted)
   ┝ d3 (deleted)

If we create a deleted prune report of the above, it would contain, d1 modified, and d1/d2 deleted. It would not contain d1/d2/d3.

The algorithm in psuedocode

Jan 8, 2015: I found a use-case where this algorithm does not work when used with an actual Subversion repository being accessed by SvnKit. Not due to SVNKit of course. Will update when fixed.

 

boolean deleting := false
String deletePath = “”

for each path{
    if deleting then 
            if path starts with deletePath then
                skip path
            else
                deleting = false
                deletePath = “”
                result.add(path)
            endif       
    else
        if path is directory and it is deleted then
            deleting = true
            deletePath = path
        endif
        
        result.add(path)
        
    endif
}

This can be implemented at the script level, for example, using Groovy or some other language.

SvnKit Scenario

In an Subversion Console application I wrote using ReactJS, I used SvnKit at the server side to interface to the Subversion server. Future blog post will discuss this single-page app. One of the React components was a Diff table.

Using the SvnKit library I generated an svn diff summary list. This is a list of SvnDiffStatus objects: List. Among other things, these object contain a path string of the respective resource.

To prune the excess deleted folders I first attempted to create an Object tree structure, then I gave up on that (too slow), finally a co-worker suggested I just sort the paths and do a simple contains comparison. Wow, how did I miss that? I guess I was sidetracked by “Objects” and didn’t use simple scripting techniques.

So to prune the folders we sort and prune the results of an SvnDiffSummary object, svnDiff::
List list = pruneDeletedFolders(sortDiffList(((Receiver)svnDiff.getReceiver()).getEntries()));

Where Receiver is an implementation of ISvnObjectReceiver.

(The SvnDiff entries may already be sorted, but the SvnKit API docs do not specify this).

Sort the diff list using Guava

	private List<SvnDiffStatus> sortDiffList(List<SvnDiffStatus> list) {
		return new Ordering<SvnDiffStatus>() {
			@Override
			public int compare(SvnDiffStatus left, SvnDiffStatus right) {
				return left.getPath().compareTo(right.getPath());
			}
		}.sortedCopy(list);
	}

prune the list

	private List<SvnDiffStatus> pruneDeletedFolders(List<SvnDiffStatus> list){
		String deletedPath = "";
		boolean isDeleting = false;
		
		List<SvnDiffStatus> prunedList = new ArrayList<>();

		for (Iterator<SvnDiffStatus> iterator = list.iterator(); iterator.hasNext();) {
			SvnDiffStatus diff = iterator.next();
			String path = diff.getPath();
			if (isDeleting) {
				if (path.startsWith(deletedPath)) {
					// skip this diff
				} else {
					isDeleting = false;
					deletedPath = "";
					prunedList.add(diff);
				}
			} else {
				if (isDirectory(diff) && isStatusDeleted(diff)) {
					isDeleting = true;
					deletedPath = path;
				}

				prunedList.add(diff);
			}
		}

		return prunedList;

	}

Links

• SVNKit
• ReactJS
• Guava
• Subversion
• Git

SvnKit E170001: Negotiate authentication failed: ‘No valid credentials provided’

In a new project, attempts to programmatically access our Subversion server using SvnKit fails with an E170001 error. But, only on one Windows 7 workstation.

After a lot of searching on web for answers finally found something that helped. I had to add system property: svnkit.http.methods=Basic,Digest,Negotiate,NTLM

So, using SvnCli, which I used to debug this, you add the property using the “-D” switch to the command line.

java -Dsvnkit.http.methods=Basic,Digest,Negotiate,NTLM -cp "SvnCli*" org.tmatesoft.svn.cli.SVN --username *** --password *** list

I also had to add this property to the Tomcat app server.

Solution?
While this does fix the problem in this instance, since only one workstation is effected, it is probably hiding an underlying configuration setup issue.

I wonder what percentage of the nation’s GDP is spent on configuration and its issues.

Original stacktrace:

Mar 18, 2015 11:40:31 AM org.tmatesoft.svn.core.internal.util.DefaultSVNDebugLogger log
SEVERE: CLI: svn: E170001: Negotiate authentication failed: 'No valid credentials provided'
org.tmatesoft.svn.core.SVNException: svn: E170001: Negotiate authentication failed: 'No valid credentials provided'
        at org.tmatesoft.svn.cli.AbstractSVNCommandEnvironment.handleWarning(AbstractSVNCommandEnvironment.java:401)
        at org.tmatesoft.svn.cli.svn.SVNListCommand.run(SVNListCommand.java:95)
        at org.tmatesoft.svn.cli.AbstractSVNCommandEnvironment.run(AbstractSVNCommandEnvironment.java:142)
        at org.tmatesoft.svn.cli.AbstractSVNLauncher.run(AbstractSVNLauncher.java:79)
        at org.tmatesoft.svn.cli.svn.SVN.main(SVN.java:26)
        at org.tmatesoft.svn.cli.SVN.main(SVN.java:22)
svn: E170001: Negotiate authentication failed: 'No valid credentials provided'

Environment

• Java 1.7
• SvnKit 1.8.8
• Tomcat 7

Links

• SVN authentication problem
• SvnKit

Getting “svn info” using Groovy

In a Subversion utility I had to the get the revision number of the HEAD of the current working copy. An answer found gave a Linux command line solution. Here is my Groovy version.

Groovy allows the ability to execute a String. This returns a Process instance, we wait for its termination, and then we query the process for the return code, the output to stderr, and to stdout.

Below I take the output and stick it into a Properties object, then dump the entries. Of course, in practice I will be using the Properties in the solution (should have used a Map).

def command = ['svn','info','-rHEAD']
def proc = command.execute()
proc.waitFor()

int code = proc.exitValue()
if(code != 0){
    println "code: '${code}'"
    println "stderr: ${proc.err.text}"
    return
}

def props = new Properties()

proc.in.eachLine{
    def m = (it =~ /^(.*?):(.*)$/)
    if(m){
        def matches = m[0]
        def key = matches[1].trim()
        def value = matches[2].trim()
        props.put(key,value)
    }else{
        println 'Could not parse the output of svn info'
    }
}

println 'Properties ...'
props.each{
    println "[${it.key}]=[${it.value}]"
}

Example output
Note: The brackets were added to provide a visual test of output, they are not stored. Also, the content was manually changed.

Properties ...
[Last Changed Date]=[201.............pr 2014)]
[Last Changed Rev]=[737]
[Path]=[Widget]
[Repository UUID]=[ec14......b668719305]
[Revision]=[744]
[Relative URL]=[^/widget/branches/skunk]
[Repository Root]=[https://widget/svn/widget]
[Last Changed Author]=[alfred e. newman]
[URL]=[https://widget/svn/widget/branches/skunk]
[Node Kind]=[directory]

Example error output
On error the output would be something like:

code: '1'
stderr: svn: E155007: 'C:temp' is not a working copy

Via SvnAnt?
An alternative is to use the svnant Ant task. This task has a subcommand ‘WcVersion’. It didn’t work for me.

Ant example is shown below, but you could of use the AntBuilder in Ant.

<target name="svn-info" depends="" description="- svn project info">
	<svn username="${svn.username}" password="${svn.password}">
		<wcversion path=".." prefix="svn" />
	</svn>		
</target>

Update
SvnAnt’s info command can be used to get this information. See http://subclipse.tigris.org/svnant/svntask.html#info

Links

• how can I get the SVN head version number in shell?
• Executing External Processes From Groovy
• svn info
• SvnAnt

Transform SVN log diff using Java XPath

Sometimes the output of a version control system’s tools or command options must be transformed for further use. For example, you generate a log report and must then import that into a spreadsheet or database.

Options
One of the options for doing this is with text based tools and scripting. Another approach is using the XML output of the version control system if available. With Subversion, XML can be used for the log and diff reports. Now you can use the various XML tools, such as XQuery, XPath, XSLT, and so forth.

DVCS?
Out of the box Git or Mercurial, two popular Distributed Version Control Systems (DVCS), do not output in XML as easily as SVN’s –xml option. To use XML, you have to define a ‘formatter’ or ‘pretty-print’ for them to use.

XPath
Below I use Java and XPath to transform the SVN output into Comma-separated values (CSV) files. One good reason for using an imperative approach is that this gives the opportunity to perform more complex transformations. A case in point is that in SVN the XML output format is only available for summary reports, and thus to create more comprehensive results or send to complex destinations, more processing would be required.

Source
The source is just a simple approach with minimal error handling and limited to CSV output. Originally I had two methods with internal loops that were almost identical, so I made the loop handling into a call-back that gets invoked by an Anonymous class. Kind of a simple Visitor or Strategy Pattern? Closures, as available in scripting languages like Groovy, would have been a simpler solution.

Of course a more comprehensive approach would abstract the output format or use other methods like POI to create a spreadsheet. Note that this approach requires loading the full XML into memory. For very large logs or diffs, a streaming approach would be needed.

Source also available here.

[expand title=”Example source SVN diff and log tranform to CSV”]

/**
 * 
 */
package com.octodecillion.utils;

import java.io.File;
import java.io.FileReader;
import java.io.IOException;
import java.io.PrintWriter;
import java.io.Reader;
import java.io.Writer;

import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;
import javax.xml.xpath.XPath;
import javax.xml.xpath.XPathConstants;
import javax.xml.xpath.XPathFactory;

import org.w3c.dom.Document;
import org.w3c.dom.Node;
import org.w3c.dom.NodeList;
import org.xml.sax.InputSource;

/**
 * Transform SVN diff and log XML output files.
 * 
 */
@SuppressWarnings("javadoc")
public class SvnOutputTransform {

	/** Example run */
	public static void main(final String[] args) {
		try {
			SvnOutputTransform svnTransform = new SvnOutputTransform();

			svnTransform.diffSummaryToCsv("data/diff-summary.xml",
					"bin/SvnDiff.csv");

			svnTransform.logSummaryToCsv("data/log-summary.xml",
					"bin/SvnLog.csv");

		} catch (Exception e) {
			e.printStackTrace();
		}

	}

	/** Call back interface */
	interface RowProcess {
		/** accept method of Vistor pattern */
		public void doRows(NodeList nodeList) throws Exception;
	}

	/**
	 * 
	 * @param dataFilePath
	 * @param reportFilePath
	 * @param nodeString
	 * @param processRows
	 */
	public void generateReport(final String dataFilePath,
			final String reportFilePath, final String nodeString,
			final RowProcess processRows) {

		try {
			NodeList nodeList = setup(dataFilePath, reportFilePath, nodeString);
			reportOut.println(HEADER_COLUMN);
			processRows.doRows(nodeList);
		} catch (Exception e) {
			throw new SvnTransformException("", e);
		} finally {
			finallyHandler(reportOut, fr);
		}
	}

	/**
	 * 
	 * @param dataFilePath
	 * @param reportFilePath
	 * @param xpathString
	 * @return
	 * @throws Exception
	 */
	private NodeList setup(final String dataFilePath,
			final String reportFilePath, final String xpathString) throws Exception {

		builder = DocumentBuilderFactory.newInstance().newDocumentBuilder();
		xp = XPathFactory.newInstance().newXPath();
		fr = new FileReader(dataFilePath);
		dom = builder.parse(new InputSource(fr));
		reportOut = new PrintWriter(new File(reportFilePath));

		Object nodes = xp.evaluate(xpathString, dom, XPathConstants.NODESET);
		NodeList nodeList = (NodeList) nodes;

		return nodeList;

	}

	/**
	 * Convert SVN generated diff summary xml file to CSV. 
	 * 
	 * The format of the input XML is:
	 * <diff>
	 *	<paths>
	 *		<path props="none" kind="file" item="modified">
	 *			full path of resource
	 *		</path>
	 *  </paths>
	 * </diff>
	 * 
	 * @param dataFilePath xml file generated by svn diff --xml --summary ....
	 * @param reportFilePath destination of CSV file
	 * @throws SvnTransformException 
	 */
	public void diffSummaryToCsv(final String dataFilePath,
			final String reportFilePath) {

		preconditionCheck(dataFilePath, reportFilePath);
		
		generateReport(dataFilePath, reportFilePath, "//diff/paths/path",
				new RowProcess() {
					@SuppressWarnings("synthetic-access")
					@Override
					public void doRows(final NodeList nodeList) throws Exception {
						for (int i = 0; i < nodeList.getLength(); i++) {
							Node node = nodeList.item(i);
							String kind = xp.evaluate(KIND_ATTR_NAME, node);
							String item = xp.evaluate(ITEM_ATTR_NAME, node);
							String pathEntry = xp.evaluate("text()", node);

							// row
							reportOut.println(String.format(
									"%s,%s,%s,%s,%s,%s,%s", DIFFUSER, revision,
									date, item, kind, pathEntry, message));
						}
					}
				});
	}

	/**
	 * Convert SVN generated log summary xml file to CSV.
	 * 
	 * <log>
	 * 		<logentry revision="10879">
	 * 			<author>T16205</author>
	 * 			<date>2013-03-15T18:10:07.264531Z</date>
	 * 			<paths>
	 * 				<path kind="file" action="A">
	 * 					/2013/Amica/branches/SOW114-LifeAPP/Test/Resources/Properties/Test/HomeQuotingFields.inix
	 * 				</path>
	 * 			</paths>
	 *      </logentry>
	 * </log>
	 * 
	 * @throws SvnTransformException 
	 */
	public void logSummaryToCsv(final String dataFilePath,
			final String reportFilePath) {
		preconditionCheck(dataFilePath, reportFilePath);

		generateReport(dataFilePath, reportFilePath, "//log/logentry",
				new RowProcess() {
					@SuppressWarnings("synthetic-access")
					@Override
					public void doRows(final NodeList nodeList) throws Exception {
						for (int i = 0; i < nodeList.getLength(); i++) {
							Node node = nodeList.item(i);
							String author = xp.evaluate(AUTHOR_NODE, node);
							String date = xp.evaluate(DATE_NODE, node);
							String revision = xp.evaluate("@revision", node);
							String message = """ + xp.evaluate(MSG_NODE, node) + """;

							NodeList paths = (NodeList) xp.evaluate(
									"paths/path", node, XPathConstants.NODESET);

							if (paths != null) {
								for (int k = 0; k < paths.getLength(); k++) {
									Node aPath = paths.item(k);

									String action = xp.evaluate("@action",
											aPath);
									action = actionToName(action);

									String filePath = xp.evaluate("text()",
											aPath);

									// row
									reportOut.println(String.format(
											"%s,%s,%s,%s,%s,%s", author,
											revision, date.split("T")[0],
											action, filePath, message));
								}
							}

						} // end each logentry		
					}
				});

	} // end logToCsv
	
	/**  */
	private String actionToName(final String n) {

		try {
			return ACTION.valueOf(n).getName();
		} catch (Exception e) {
			return n;
		}
	}

	private void finallyHandler(final Writer reportOut, final Reader fr) {
		if (reportOut != null) {
			try {
				reportOut.flush();
				reportOut.close();
			} catch (Exception e) {
				// 
			}
		}

		if (fr != null) {
			try {
				fr.close();
			} catch (IOException e) {
				// 				
			}
		}
	}

	/**
	 * @param dataFilePath
	 * @param reportFilePath
	 * @throws IllegalArgumentException
	 */
	private void preconditionCheck(final String dataFilePath,
			final String reportFilePath) throws IllegalArgumentException {
		if ((dataFilePath == null) || (reportFilePath == null)) {
			throw new IllegalArgumentException(String.format(
					"dataFilePath='%s',reportFilePath='%s'", dataFilePath,
					reportFilePath));
		}
	}

	/**
	 * SVN action codes.
	 * 
	 */
	enum ACTION {
		A("add"), M("modify"), D("delete"), Z("z");

		private final String name;

		private ACTION(final String name) {
			this.name = name;
		}

		public String getName() {
			return name;
		}
	}

	/** Svn transform runtime exception */
	static public class SvnTransformException extends RuntimeException {
		private static final long serialVersionUID = 1L;

		/**  */
		public SvnTransformException(final String message, final Throwable cause) {
			super(message, cause);
		}
	}

	private static final String HEADER_COLUMN = "Dev,Revision,Date,Action,Kind,Path";
	private static final String ITEM_ATTR_NAME = "@item";
	private static final String KIND_ATTR_NAME = "@kind";
	private static final String MSG_NODE = "msg";
	private static final String DATE_NODE = "date";
	private static final String AUTHOR_NODE = "author";
	private PrintWriter reportOut;
	private FileReader fr;
	private static final String DIFFUSER = "DIFF";
	private final String revision = "";
	private final String date = "";
	private final String message = "";
	private DocumentBuilder builder;
	private XPath xp;
	private Document dom;

}

[/expand]

 

Updates

• Dec 18, 2013: Just saw a blog post where the author also uses the XML output from subversion. Searching Subversion Logs with Groovy

Links

• Easy stream parsing using Groovy, CVS example
• Transform XML with Groovy and XMLTask
• Tool to generate change reports from Subversion
  logs
• svn2cl
• SvnStat
• CommitMonitor
• POI