Application logging using global inception identifier

March 7, 2015

For multithreaded programs that write to log files a best practice is to include a tracking ID. What should this ID be and how to use it? The following is presented as a ‘design pattern’.


For machine readability and tool use, a non-business related global inception ID (GIID) should be used for every log output. This ID is a combination of an ‘origin id’ and ‘unique id’. When each new business level or operational id is obtained or generated, it is logged to provide an ‘association’ with this GIID. This is similar to various existing practices so it is presented as a Software Design Pattern.

Author: Josef Betancourt, Jan 11, 2015. Originally posted here.

CR Categories: H.3.4 [Systems and Software]; D1.3 [Concurrent Programming]; D.2.9 [Management]; K.6 [MANAGEMENT OF COMPUTING AND INFORMATION SYSTEMS]; K.6.3 [Software Management]

  1. Context
  2. Forces
  3. Solution
  4. Consequence
  5. Implementation
  6. Appendix
  7. Further reading


App Logs

An application log file records runtime programmatic events: details, exceptions, debug, and performance related measures. This is different than specialized server log files, such as a webserver’s access logs, error.log, etc. The latter are more standardized, for example with W3C Extended Log File Format, and well supported.

App logs usually capture details at specific logging levels and named contexts. In the Java ecosystem there are plenty of libraries to support this and now the JDK supports this as part of the java.util.logging package.

Despite the advances made in logging APIs and support within operating systems and frameworks, app logs are at a primitive level of software maturity. What A. Chuvakin and G. Peterson describe as the “… horrific world of application logging” is composed of ad hoc, minimally documented, unstructured, untested, and under-specified delivered components.

Attempts to create widely used industry standards have failed and every business, software project, dev team, or industry reinvents and attempt to tackle the same problems.


In the context of server app logs, multiple sessions will output log information that can be intermixed. These sessions can be part of an ongoing process, such as a user interaction with a web site.

External integration points (web services, database, etc) may also be invoked. Unless each session is identified in the log and integration logs, subsequent support, debug, and auditing are very difficult.

The problem is not just tracking where and when ‘integration’ occurred or its non-functional integration criteria (i.e., timing), but the tracking of subsequent logging, if any, at that location.

App logs are used extensively during development. Their importance is illustrated by an old mime “debuggers are for wimps”. As such, logs with impeccable tracking used for design and test are a good Return On Investment (ROI).

The same is true for deployed systems. In many cases the only information available on historical behavior is in a log file.

This seems like a programming 101 lesson, but it is widely disregarded in practice. That log file output is a minor concern and sometimes not even part of a “code review” is puzzling.


1. A service must invoke a distributed call to another system. The service has retry logic, and logs each failure. If each log output does not identify the session or operation, the retries could get mixed with other threads. Identifying an end user’s request is a hit or miss bracketing of time stamps if the development team did not enough identifiable data in each log output.

2. A computer savvy end user or family may attempt to register into your system with multiple browsers simultaneously. This could cause problems if multiple logins are supported and an error occurs. How do you track this and debug it?

3. The app server makes a remote call to a service integration point and that service fails. How is the owner of that service informed as to the specific invocation? There are probably deployed systems where one would have to compare time stamps on log output to even coordinate where the two systems communicated and even then it is vague. Some systems may not even do any logging and the unless there is a fault of some kind.

4. You have to identify time periods based on hazy user complaints, search through multiple log files with regular expressions, then walk each output to recreate a specific error scenario. Isn’t this manual drudgery what computers were supposed to eliminate?


Global Inception ID

Logging with Unique identifiers is encouraged as a best practice:

“Unique identifiers such as transaction IDs and user IDs are tremendously helpful when debugging, and even more helpful when you are gathering analytics. Unique IDs can point you to the exact transaction. Without them, you might only have a time range to use. When possible, carry these IDs through multiple touch points and avoid changing the format of these IDs between modules. That way, you can track transactions through the system and follow them across machines, networks, and services.” —

This unique identifier is generalized so that on first entry into a system or the start of a process, A Global Inception ID (GIID), is assigned to distinguish that ongoing process from others. A more descriptive term would be a Global Tracking ID, but that conjures up privacy and security concerns and is already being used in commerce for a different purpose. But ‘inception ID’ brings up visions of barcodes on people’s foreheads. Ok, how about “bludzwknxxkjysjkajskjjj”?

The term “Global” is to indicate that this ID is unique in a specific enterprise system. The uniqueness comes from its creation on a first contact basis on a specific subsystem. In essence this is a log tracking ID.

For example, a web server or an app server would be the first point of contact or request from an external User. The GIID, consisting of a combination of origin id and a unique id, would be created at this point. GIID ::= originID uniqueID

In article “Write Logs for Machines, use JSON” Paul Querna uses the term “txnId” for this type of ID:

“… this is essentially a unique identifier that follows a request across all our of services. When a User hits our API we generate a new txnId and attach it to our request object. Any requests to a backend service also include the txnId. This means you can clearly see how a web request is tied to multiple backend service requests, or what frontend request caused a specific Cassandra query.”

Another term for this GIID, or ‘txnId’ is Correlation ID. This terminology is used in SharePoint.

The correlation ID is a GUID (globally unique identifier) that is automatically generated for every request that the SharePoint web server receives.

Basically, it is used to identify your particular request, and it persists throughout further communications, if any, between the other servers in the farm. Technically, this correlation ID is visible at every level in the farm, even at a SQL profiler level and possibly on a separate farm from which your SharePoint site consumes federated services. So for example, if your request needs to fetch some information from an application server (say, if you are using the web client to edit an Excel spreadsheet), then all the other operations that occur will be linked to your original request via this unique correlation ID, so you can trace it to see where the failure or error occurred, and get something more specific than “unknown error”. —

Various ‘Structured Logging’ efforts or syslog implementations already contain a ‘sending’ field specification. The GIID incorporates the sender id as the Origin ID, and this combination is more amendable to human and textual tools parsing.




A good candidate for a GIID must be large enough to satisfy uniqueness requirements. This could be, for example, a 36 character field. Where the log files are manually inspected with a text editor, this increases the log line which already contains many common fields like a time stamp.


Unintentionally, “bad” logging practices makes it harder to track and correlate personally identifiable information (PIN). With the use the trans-system GIID, correlation between various business related identifiers is made easier.

The correlation ID is not necessarily a secret, but like other tracking objects like cookies, can be used for information discovery or questionable information storage. But, if an attack can already access your log files, there are other more serious issues?


What determines the first contact subsystem? A true distributed system could be configured or dynamically configured so that any system could be the first contact system. If so, then each subsystem is creating GIID and passing that GIID to other systems that are themselves creating GIIDs.

One approach to handle this is that a system will only create a GIID if none is present in the incoming request.


For user interfaces, exposing the GIID or parts of it in exception situations can be beneficial:

“We also send the txnId to our user’s in our 500 error messages and the X-Response-Idheader, so if a user reports an issue, we can quickly see all of the related log entries.” —

Compare this to the Hunt The Wampus adventure in enterprises that only have an approximate time of an issue and must track this over multiple systems.


If a giid is part of a support system and as above the ID would be shared with Users in some way, would the value need some form of validity testing? Should it be tested that it is wellformed and include a checksum?

Example crc calculation for a UUID, based on textual version of id:

groovy -e " crc = new; crc.update(UUID.randomUUID().toString().getBytes());println Long.toHexString(crc.getValue())"



Origin ID

An OID uniquely identifies a system in an enterprise. This could be a web server or messaging system. Using a code for the actual system is recommended. Thus, instead of Acctsys, it would be a code, PE65K for example. Using a code is more ‘durable’ than a human readable name.

An extension is to also encode other information in the origin ID, such as application or subsystem identifiers.

Unique ID

This ID must not be a business related entity such as user id or account number. The simple reason is that these may occur in the logging record multiple times for different sessions or transactions. For example, user Jean Valjean with account number 24601 may log in multiple times into a web site. Tracking a specific session if a problem occurs is easier if we use a unique ID.

A business level ID may not even be relevant in another system that interacts with the origin point. In one system the ID could be one type of ID, and in the other the same party or user could be identified with a different ID.

Note that as soon as determined, accessed, or generated, a business level ID should be associated with the GIID. This could be a simple log output of that business ID which, since every log output has a GIID, will associate the business ID or value with the GIID.

Similarly when the same process communicates with another system, that system’s unique identifiers and related business IDs will also be associated with the GIID. For example, a web service will take the GIID and relate it to its own ID(s). Now a support engineer can follow the complete path of the system response to a user session.

ID creation

The easiest approach is to use the entry system’s session id created by the server. A potential problem is that this session id is not guaranteed to be unique and ends when the session expires. A UUID solves most problems.

Sample UUID generation in Groovy language:

groovy -e "println UUID.randomUUID().toString().replace('-','')"

If the system ID is 3491 then the above UUID is used to create the GIID and use in logging:

20110227T23:34:37,901; EV={_ID:”34911f788da1ac4a43bb82adb8e61cfcb205″, USERID:”felixthecat”, ….}

Alternative to UUID use?

A UUID is a 32 character string. Could something smaller be used? Perhaps, but eventually the complexity of threaded systems would make the uniqueness constraint of any ID approach a comparable length.

Other approaches are possible. Most of these will incorporate a timestamp in some way. Note that a UUID contains a timestamp.

An example of a ‘unique’ id is used by MongoDB’s ObjectID specification. That spec calls for a 12-byte BSON type of:
• a 4-byte value representing the seconds since the Unix epoch,
• a 3-byte machine identifier,
• a 2-byte process id, and
• a 3-byte counter, starting with a random value.
An example of an ObjectID string representation is ObjectId(“507f1f77bcf86cd799439011”)

Log Framework support for GIID

The use of a GIID is a ‘cross-cutting’ concern. Requiring programmatic use of this ID would be burdensome and error-prone, even if stored in a thread-safe context.

Some logging frameworks support the concept of “nested diagnostic contexts”. This is a way of storing an ID so that interleaved logging is properly identified. See for more information.

Example usage

In a conventional Java server application a JSP or template system would obtain a GIID and insert it into generated pages and associated client side scripts. That GIID would also be stored in the server side session. Since the GIID is stored at the session it is accessible to the various services and components on a server.

This ID is embedded in request to other distributed servers and provides event correlation. Thus the logging systems will have access to the GIID and Client or server side errors can then display or use the GIID for tracking and reporting to assist support engineers.

Since the client also has the GIID, it can display or use this ID for customer service processing.

Of course, this would make more sense if it is a part of a wider Application Service Management (ASM) system.

Standards for IDs

Though many standards specify ID handling, modern architectures, especially web based or distributed, emphasize a stateless protocol. A GIID requirement could be one of those legerdemain stateful practices.

Development impacts

If logging is a deployed feature of an application then it too needs testing. But, since log output is an integration point, it does not fall under the “unit” testing umbrella. There is even some doubt if this should even be tested! Here is one example: Unit Testing: Logging and Dependency Injection
If log files can contain security flaws, convey data, impact support, and impair performance, then they should be tested that they conform to standards. Project management spreadsheets needs to add rows for logging concerns.


Log output can be developer tested using the appropriate XUnit framework, like JUnit.
Mocking frameworks provide a means of avoiding actually sending the output of a logger to an external ‘appender’. “Use JMockit to Unit test logging output”.
In development of a project, the log output changes rapidly as the code changes. Selecting where in the software development life cycle (SDLC) to test logging or even specify what logs should contain is difficult.
One approach is that the deployed system will not do any app logging that was not approved by the stake holders. These must be “unit” tested, and all development support logging is removed or disabled except for use in a development environment.


There is no need to change every subsystem to use this log tracking approach. If the GIID is created somewhere in the “path” of a process or service, it adds value. Other systems can gradually start to use a tracking ID. Thus, the tools and training to use this capability can also be progressively introduced.

About this post

I was going to title this article ‘Logging in Anger’, as a response to my own experiences with application logging. Alas, there are so many issues that I had time to only focus on one as a result of a recent stint supporting an application that exhibits the typical logging anti-patterns. Example: it’s bad to get a null pointer exception, but to not know which argument to a function caused this?


Structured Logging

(this article was going to add more info on incorporating a GIID into a Structured Logging framework. This section is here for refernce)
Structured Logging is a type of app log file that is data based rather than prose based. Thus, it is machine readable and amendable to high-level tools, not just a text editor.

Treating logs as data gives us greater insight into the operational activity of the systems we build. Structured logging, which is using a consistent, predetermined message format containing semantic information, builds on this technique …. We recommend adopting structured logging because the benefits outweigh the minimal effort involved and the practice is becoming the default standard. —

An example, is a system where the log output uses a predetermined message format. An overview of such systems is found in chapter 5 of “Common Event Expression”,

Note this should not be confused with a similar sounding technology called “Log-structured file system”.

Further reading

  1. Log management and intelligence,
  2. Logging a global ID in multiple components,
  3. Application Service Management (APM) system
  4. Application performance management,
  5. The art of application logging,
  6. Patterns For Logging Diagnostic Messages,
  8. How to test valid UUID/GUID?
  9. Log Data as a Valuable Tool in the DevOps Lifecycle (and Beyond),
  10. OWASP – Logging Cheat Sheet,
  11. How to Do Application Logging Right,
  12. Request for comment Structured Logging,
  13. 6 – Logging What You Mean: Using the Semantic Logging Application Block,
  14. A Review of Event Formats as Enablers of event-driven BPM,
  15. Basic Android Debugging with Logs,
  16. Mapped diagnostic context vs Nested diagnostic context,
  17. Building Secure Applications: Consistent Logging,
  18. Log for machines in JSON,
  19. Logging Discussion,
  20. CEE,
  21. CEE is a Failure.,
  22. Centralized Logging Architecture,
  23. Centralized Logging,
  24. Logging and the utility of message patterns,
  25. Payment Application Data Security Standard,
    Payment application must facilitate centralized logging.
    Note: Examples of this functionality may include, but are not limited to:
    • Logging via industry standard log file mechanisms such as Common Log File System (CLFS), yslog, delimited text, etc.
    • Providing functionality and documentation to convert the application’s proprietary log format into industry standard log formats suitable for prompt, centralized logging.
    Aligns with PCI DSS Requirement 10.5.3
  26. NIST 800-92 “Guide to Computer Security Log Management”,
  27. UniqueID Generator: A Pattern for Primary Key Generation,
  28. java.util.logging,

Dedicating multiprocessors per OS structure

June 5, 2012

A few weeks ago something I’ve been thinking about for a while came back into my thoughts. What if instead of multithreading at the application level, we instead dedicate a chip of a multicore processor to a specific OS resource.

What kind of OS resource? I don’t know. Maybe four processors at ring zero, or at the IO subsystems. We could have application specific cores that are optimized for specific software tasks, just as the new graphic processors are used for APU.

I’m certainly not a system designer and the above is probably pure drivel. Like, how would context switching work and not slow everything down? However, there are possible reasons why perhaps its not.

  • Number of cores will keep growing.
  • GPUs will become more important and powerful.
  • Future applications will be distributed and require advanced resource handling, example, hierarchical robust device systems, like memory handling.
  • Thermal limitations will demand new forms of optimization.
  • AI and robotics applications will reach a tipping point and become ubiquitous, and even more performance will be demanded from computing systems

After thinking about this, I later started seeing articles on internet-on-a-chip, a new way of connecting multicores. Interesting.

Further reading

From Tags To Categories

February 5, 2012

Tags are usually non-hierarchical fine grained descriptors of a resource. They are the opposite of categories which are usually part of a semantic hierarchy. Categories are really old-school, killed by the effectiveness of Search and the expanding mash-up universe.

However, while tags provide rapid access to resources and easier sharing of them, tags do not enrich knowledge. We can see this if we consider knowledge as one of the tiers toward wisdom: data, information, knowledge, wisdom. Tags are data on information, metadata. Categories are information on knowledge.

Since data is used to create information, the data on information, tags, can be harnessed to increase the information on knowledge by the automated creation of categories. We simply create the directed cyclic graphs of tags around resources and detect clusters. The naming of categories come from the resources themselves.

This is how the internet will wake up. It will create an ontology and it will act. Hopefully, humans are part of a necessary category.

Some possible related reading

  1. Ontology on Wikipedia
  2. New Common Tag Format
  3. Semantic Web Use Cases and Case Studies. Case Study: Semantic tags
  4. Ontology is Overrated: Categories, Links, and Tags

State Machine discord as system error detection

October 24, 2011

It takes three subsystem state machines to create an error adaptive system.

This morning in the shower I was thinking about my last post on SCXML. What do you use as states in a system? Easy question, but then I thought, sure its internal states of a process, but how does the external “real” come into the picture, via guards? That could get messy in a very complex real world system. So, what if the external system is also modeled as a state machine?

Now we have two state machines, internal system and external system. What is the advantage? Well, each can have a peak into the other or using historical information, compute how the other works. Then if this prediction is not matched, it is an error. That is, if the internal machine is at state x and it knows that the external machine should be at state y at this point, and it is not, it is a system error. What to do?

This is where a third error machine would come into use. The function of this machine is to bring the other two into resonance, reduce the entropy, heat.

Depending on whether the machines are Mealy or Moore based, one would detect errors at particular points, events, transitions, or states.

I remember reading years ago about fault tolerant hardware systems. The minimum components was three; similar.

Hmmm. I don’t know what everything above really means, if anything. :)

For future thinking …


ATi Radeon 7970 [CAD Animation Benchmark]

Proactive Agents?

June 30, 2011

I read a blog post related to the topic I mentioned, “Synergistic Social Agent Network Cloud“.

The term Proactive Agents is mentioned by Carlos E. Perez, in “Software Development Trends and Predictions for 2011“:

Proactive Agents – For decades people have been forecasting the emergence of digital personal assistants that would actively react to the environment on one’s own behalf. The emergence of always present smart mobile devices and cloud computing shared spaces will be the catalyst for the developing of active agent based computing platforms and frameworks. At present, most computing is merely reactive, that is servicing web requests only on command of a user. Future computing will include a proactive aspect that suggests courses of actions to users. Semantic technologies like Zemanta and OpenCalais provide intelligence to writers by suggesting tags that are relevant to a written document.

Are current social networks, apps, and other mobile devices “proactive” yet?

Java Plain Old Concurrent Object

July 29, 2010


This blog post reiterates the existing critique of the concurrency support in the Java platform and language: that the low level thread and shared memory features should be improved upon with one or more high-level concurrency frameworks. Some of the competing frameworks are listed, and further references are given.

CR Categories: D1.3 [Concurrent Programming]; D.1.5 [Object-oriented Programming]; D.2.2 [Design Tools and Techniques]: Design Tools and Techniques — Software libraries;D3.2 [Language Classifications]: Concurrent, distributed, and parallel languages;D.3.3 [Language Constructs and Features]: Concurrent programming structures;

Keywords and Phrases: concurrency, Java, Actor Model, threads, Component based software engineering, Concurrent object-oriented programming, Distributed systems, Java CSP, CCS, multiprocessor

This is an excerpt from a document I wrote about three years ago.  The doc was in a Java Specification Request (JSR) format and from time to time I updated the link section as I came upon interesting references.   Maybe someone will find it interesting or it will spark a new idea for a project. In a later post I give some examples of concurrent code using the Groovy language.


The Java Platform has built-in support for concurrent programming. At the time of Java’s birth this was very big deal. But, is it now time to build upon this by the creation of a requirements document for high-level concurrency support in the Java language? Could this bring Java closer to a Concurrent Object-Oriented Language (COOL)?

Well, since I’m not a concurrency subject matter expert, just a working stiff developer, I will leave that to others, and just put my two cents here. What prompted me even looking into this subject was my experience on a few projects that required concurrency support.

What is needed is a survey or summary of what exactly are the issues, what is available, what their application areas are, and what are the development options. Some of initial tasks I see, in no particular order, are:

  • State the problem and application scope.
  • Provide use-cases.
  • Create project
    • wiki, forums, etc.
    • conference
  • Limit the scope of the effort.
  • Identify solution categories.
  • Identify solution selection criteria.
  • Testing and management requirements.
  • Measures, such as Performance and scalability.
  • Tools.

One possible conclusion is that there is no need for a ‘higher’ level concurrency support in Java or that this should evolve on its own before attempts at standardization are attempted.


One thing has been left out of the OO frenzy is the concept of a process. There are two manifestations of this: Methodology and Concurrency. In the methodological realm, ‘process’, which was part of data-flow analysis, is practically missing from modern analysis (as typified by OOP/UML derived methods). And even now, modular approaches, are still divorced from concurrency concerns. For example, OSGi fortunately brings back modularity to the Java environment. As stated by Kriens:

“Why is modular important? Well, if there is one consistent lesson in all our technology trends over the past 50 years then it is high cohesion and low coupling. Structured programming advocated this; OO forgot it for some time ….” — Peter Kriens,

Yet, even in OSGi, a module is still at the mercy of unconstrained concurrency effects:

“OSGi … does not provide a thread of execution to each bundle. Events are delivered to bundles through certain interfaces, but no guarantees are made about which thread event delivery occurs on. Generally, event callbacks are required to finish quickly and should not make calls back into the OSGi framework to avoid possible deadlock. It is common, therefore, for bundles to start one or more threads in order to get work done.” — Oliver Goldman, “Multithreading, Java, & OSGi”

Welch calls this “unconstrained OO”:

“In unconstrained OO, threads and objects are orthogonal notions — sometimes dangerously competitive. Threads have no internal structure and can cross object boundaries in spaghetti-like trails that relate objects together in a way that has little correspondence to the original OO design. ” – Peter H. Welch in Javadocs for jcsp.lang.CSProcess of the JCSP library

And, others give similar critiques:

All object methods have to be invoked directly (or indirectly) by an external thread of control – they have to be caller-oriented (a somewhat curious property of so-called object oriented systems). — P.H Welsh (“Process Oriented Design for Java: Concurrency for All”)

“In standard OO, if you hold a reference to friend, and you wish to invoke the borrowMoney() method, then the call friend.borrowMoney() is executed in your own thread of control, not in a separate thread (or in friend’s thread), thus breaking all similarity to the way the real world works.” — (Oprean and Pederson, 2008).

“Although the development of parallel languages began around 1972, it did not stop here. Today we have three major communication paradigms: monitors, remote procedures, and message passing. Any one of them would have been a vast improvement over Java’s insecure variant of shared classes. As it is, Java ignores the last twenty-five years of research in parallel languages.”
— JAVA’S INSECURE PARALLELISM by Per Brinch Hansen (1999)

“However, I think Java needs to change. When it was first released, its number one competitor was C++, and Java walked all over it in terms of support for concurrency. Threads in the standard library! Syntactic support for critical blocks! Implicit locks in every object! Now, Java’s number one competitor is C#, which is increasingly starting to look like a functional language. Also languages like Erlang and Haskell have concurrency primitives that walk all over Java’s (I particularly love Haskell’s composable memory transactions). I hope this is something that is addressed in Java 7, but I fear it won’t be…” — Neil Bartlett

A higher level concurrency support in Java will allow a more approachable use of concurrent development. Though the concurrency implementation in Java is a vast improvment over what was available in a popular language at the time, we are now seeing more interests in other approaches as shown in Scala, Erlang, Kilim, Akka, GPars, Haskell, and others.

A standard would then allow, within reason, the sharing of development expertise and common patterns and idioms among the different languages and frameworks. For example, could an Actor in Scala be behaviorially the same as an Actor in plain old Java?

We are also seeing multiple processor cores being used to continue the improvment in thruput since thermal and process limititions have reduced growth. These will also allow true parallism. A high-end actual product, Sparc T3, the Sun Microelectronics version of the chip multi-threaded (CMT) processor, will be capable of forming a quad core SOC (System on Chip) that offers 512 hardware threads. Even on personal desktop systems, we already have quad cores and soon hexacores will be commonplace in the high end, such as the Intel Nehalem-EX. (Note that concurrency and parallelism are not the same thing).

Update: This was written a while back. In the interim, Oracle bought Sun and in the process canceled the Rock high-end CMT based processor design. Yet, Oracle said they will continue to support the upcoming SPARC CMT versions. Interesting discussion is found here. Intel and AMD have not stood still. AMD, for example, is promising a 16 core “Interlagos” chip soon. On the mobile side, ARM and others are also producing multicore systems.

Plain Old Concurrent Component

Instead of just reusing existing terms, I can call this new concurrency support a Concurrent Component. A component is more similar to an OS process, whereas internally it may incorporate light weight threads.

A possible conceptual view of a Component or Plain Old Concurrent Object (POCO) is shown in figure 1 below . Note that this is intentionally reminiscent of a Programmable Logic Device (PLD) block diagram. Each block is an optional ‘concern’ that crosscuts each object that is part of the component. The nested objects can share state and optionally will execute within a fine-grain concurrency kernel. The external control communication is via interfaces that expose the Component API, whereas the actual messaging embeds the application API that the component instantiation provides. This is somewhat of the flavor of Cox’s Software IC concept (ref?). This is also similar to various old Microsoft COM models. In future, it’s even conceivable that each POCO could have its own core assigned in an dynamically created application specific multicore processor implemented in nanoprocessor FPGA.

And, yes the diagram is just my brain’s core dump on the subject. Would require more work to explore this further, and determine what would really make sense.


Concurrent Component

Why isn’t this need met by existing specifications?

Currently, developers can use the concurrency control constructs that are provided in the Java language itself. Many experts consider these too low level for some applications and for use by the average developer (in terms of development effort, correctness, and failure potentials):

Use Erlang-Style Concurrency. The Java concurrency primitives like locks and synchronized have been proven to be too low level and often to hard to use. There are better ways to write concurrent code. Erlang Style concurrency is one of them – in Java there are many ways to achive this in Java – I’ve written about them here. Newer ones are Akka and Actorom. You can also use Join/Fork or the myriad of data structures in java.util.concurrent. —- Stephan Schmidt in Go Ahead: Next Generation Java Programming Style

In Java 5 and above one uses the new java.util.concurrent utilities. These offer a powerful and relatively accessible API such as the Executor and the Fork/Join frameworks. However, though the latter were a significant improvement, their use still require advanced skills and really only offer the means to create application specific or JVM language based concurrency frameworks.

Even with concurrency updates in Java 6 and Java 7, the Java language doesn’t make parallel programming particularly easy. Java threads, synchronized blocks, wait/ notify, and the java.util.concurrent package all have their place, but Java developers pressed to meet the capacity of multi-core systems are turning to techniques pioneered in other languages.

— “A Java actor library for parallel execution

Some languages or libraries, such as Scala (?) and JCSP even use the underlying concurrency support in Java to create the respective ‘active’ object concurrency extensions. See for instance this presentation by Doug Lea, “Engineering Fine-Grained Parallelism Support for Java 7”.

Note that introducing a higher level of abstraction will not necessarily reduce the need for skilled engineering. In fact, it may increase that need until the patterns, methodologies, measurement, and tool sets catch up and support the new framework. Of course, many other concerns are important, such as performance.

Underlying technology or technologies:

There is a rich history in both industrial and academic research into the theory and practices of concurrency. In the fifty plus years of this, surely there must be more that can be used in Java then Monitors, semaphores and other low level tools.

Some examples are Communicating Sequential Processes (CSP), Calculus of communicating systems (CCS), and all the ‘proven’ concurrent support already found in other languages, among them, Occam, Go, Erlang, Haskell, Esterel, Scala, and many others in the academic research community. There is even one observation that the current Monitor implementation in Java itself could more fully embrace the original theoretical Monitor concept.

Of course, as in anything else, there is hype and fashion in software too. Thus, some approaches are touted as fixing deadlocks and other problems, when they don’t.

Related Projects:

· Disruptor – Concurrent Programming Framework
· LMAX: “The LMAX Architecture“, Fowler, M.,
· CQRS: CQRS, Fowler, M.,
· Akka:
· Multiverse:
· Staged event-driven architecture (SEDA)
· CSP:
· Kilim:
· Clojure:
· “Gpars:
· Groovy Parallel:
· Scala:
· Triveni:
· ActorFoundry project,
· “Jetlang, Message based concurrency for Java”
· Jsasb, adds event-driven programming paradigm to Java,
· Node.js:
. Er.js: Er.js: Erlang-in-JavaScript
· MPJ: Java for High Performance Computing,
· FunctionalJava, accessed on 3/2/2009 at
· JAVACT : a Java middleware for mobile adaptive agents, accessed on 3/2/2009 on 3/2/2009 at
· Threading in COM, accessed on 3/2/2009 at
· TBB:
· JConch:
· JAC:
· X10:
· BAM:
· Concurrent Programming Research Group,
· JCarder,
· Go,
· Panini:

Further Reading

Below are a few easily obtainable links found on-line. (To do: use standard style guide like IEEE Standard and Chicago).

· Coarse-grained locks and Transactional Synchronization explained

· Concepts compared

· “Joe Duffy on the Future of Concurrency and Parallelism

· Dr. Dobb’s Report: Real-Time Ready Java

· Actors Make Better Observers

· Talking about Akka, Scala and life with Jonas Bonér

· Miller, Alex; Comparing concurrent frameworks

· Lea, Doug; “Engineering Fine-Grained Parallelism Support for Java 7”,

· Milewsk, Bartosz; “Beyond Locks and Messages: The Future of Concurrent Programming”,

· Wiger, Ulf; “Death by Accidental Complexity”,

· “Erlang concurrency: why asynchronious messages?”,″>

· Merritt, Eric;”A Brief Overview of Concurrency.”,

· Kuppusamy, Rmsundar; “Is concurrency such a complex thing in real world applications?”,; Comments on the article are very informative.

· Kabutz, Heinz; “Deadlocks through Cyclic Dependencies”,

· Jackson, Joab, “Microsoft Windows architect says ditch the kernel”,

· Breshears, Clay, “Across the wide Concurrency”,

· Dibyendu Roy, “Rethinking Multi-Threaded Design Principles”, [Weblog entry.] 3 Mar 2010. ( 5 Mar 2010.

· Mordani, Rajiv; “Asynchronous Support in Servlet 3.0”,

· Michele Simionato, “Threads, processes and concurrency in Python: some thoughts”, [weblog entry.] The Explorer. 26 Jul 2010. ( 26 Jul 2010.

· Orlic, Bojan; “SystemCSP : a graphical language for designing concurrent component-based embedded control systems”. (2007) thesis.

· Jonas Bonér, “Introducing Akka – Simpler Scalability, Fault-Tolerance, Concurrency & Remoting through Actors”. 4 Jan 2010. ( 14 Mar 2010.

· Stephan Schmidt, “Actor Myths”, [Weblog entry.] Code Monkeying. Jul 21, 2010. (

· James Iry. “Erlang Style Actors Are All About Shared State”, [Weblog entry.] One Div Zero. 6 Apr, 2009. ( . 23 Jul 2010.

· Paul Chiusano, “Actors are not a good concurrency model”, [Weblog entry.] Prettt-tty, pretty, pretty good!. 15 Jan 2010. ( 2. Mar 2010.

· Welch, P.H.; “Process Oriented Design for Java: Concurrency for All”,

· Dimitris Andreou, “Thoughts on Actors”, [Weblog entry.] Code-o-matic. 8 Jan 2010. ( 23 July 2010.

· McDonald, Carol; “Some Java Concurrency Tips”,

· Oprean, George and Pedersen, Jan B., “Asynchronous Active Objects in Java”, Communicating Process Architectures, 2008.,

· sutter, Herb; “Prefer Using Active Objects Instead of Naked Threads”,;jsessionid=OCHVVY5DWODL3QE1GHRSKHWATMY32JVN

· Alex Tkachman, “Groovy concurrency in action: asynchronious resource pools with Groovy++”,

· Alex Tkachman, “Fast immutable persistent functional queues for concurrency with Groovy”, JavaLobby,

· Schumacher, Dale, “Composing Actors”, “blog It’s Actors All The Way Down”,

· “Objects as Actors?”, Debasish Ghosh,

· “Gartner Says as the Number of Processors Swells Inside Servers, Organizations May Not Be Able to Use All Processors Thrust on Them”,

· “Tilera vs. Godzilla”, Louis Savain,

· “Convergence in Language Design: A Case of Lightning Striking Four Times in the Same Place”, Peter Van Roy,

· “The Problem with Threads”, Edward A. Lee,

· “A Component-Oriented Language for Pointer-Free Parallel Programming”, Luc BlÄaser, Computer Systems Institute, ETH ZÄurich, Switzerland. Accessed on 3/10/2009 at

· Mauricio Arango, “Coordination in parallel event-based systems”,

· Irfan Pyarali, Tim Harrison, and Douglas C. Schmidt Thomas D. Jordan; “Proactor: “An Object Behavioral Pattern for Demultiplexing and Dispatching Handlers for Asynchronous Events”,

· “A Component-Oriented Model for the Design of Safe Multi-threaded Applications”, Reimer Behrends, R. E. K. Stirewalt and L. K. Dillon Dept. of Computer Science and Engineering Michigan State University, accessed on 3/10/2009 at

· “2028: Concurrent processing–from revolution to legacy”,

· “Join Patterns for Visual Basic”, accessed on 3/10/2009 at

· “ActorsModel”, at

· “Sharing Is the Root of All Contention”,Herb Sutter,

· Bryan Cantrill. “OpenSolaris Sewer Tour”. [weblog entry.] The Observation Deck. 14 Jun 05. ( 26 Jul 10.

· “Concurrent computing”,

Walsh, Matt; “SEDA: An Architecture for Highly Concurrent Server Applications”, Harvard University,

· “Multicore programming: Easy or difficult?”,

· “The universe model: An approach for improving the modularity and reliability of concurrent programs, Reimer Behrends”,

· Mitchell Pronschinske, “Concurrent Programming in Groovy”, JavaLobby,

· “Actors in Groovy”, Alex Miller,

· “Scala Actors vs Plain Old Java”, Duke Banerjee,

· Event-based actors in Groovy, Václav Pech, accessed on 3/4/09 at

· Pech, Václav; “Flowing with the data”,

· “We haven’t forgotten about other models – honest!”, posted to Maestro blog, accessed on 3/02/2009 at

· ” Infrastructure for the Synchronization and Coordination of Concurrent Java Component Programs”, Sandeep Mitra, Sudhir Aggarwal, accessed on 2/28/09 at

· Parallel Computing Developer Center,

· Search on CiteSeer for STM:;jsessionid=832BC25C661FF28D18EDAE10E725B301?q=software+transactional+memory&submit=Search&sort=rel

· “Synchronization and Time”, Duffy, Joe;

· “Concurrency’s Shysters”,

· “More thoughts on transactional memory”, Duffy, Joe;,guid,94778838-3e27-4788-a935-006ae9c8ef94.aspx

· “And now some Hardware Transactional Memory comments…”,Cliff Click Jr.,

· “Communicating Java Threads” (1997),,Gerald Hilderink, Jan Broenink, Wiek Vervoort, Andre Bakkers,

· “Actors That Unify Threads and Events”,Philipp Haller and Martin Odersky,

· Discussion at Lambda the Ultimate:

· “The Next Mainstream Programming Language”,

· “Actor Foundry”,”Rajesh Karmani, et al.”,

· “Implementing the π-Calculus in Java”,Liwu Li,

· “Tackling Concurrency Language or Library?”,

· “Programming in the Age of Concurrency: Software Transactional Memory”,

· “Communicating Process Architectures 2008”, Editors: Frederick R. M. Barnes, Jan F. Broenink, Alistair A. McEwan, Adam Sampson, G. S. Stiles, Peter H. Welch; IOS Press, Amsterdam, 978-1-58603-90. Accessed 26 Jul 10.

· “CSP for Java programmers, Part 1“,Abhijit Belapurkar,

· “CSP for Java programmers, Part 2“,Abhijit Belapurkar,

· “CSP for Java programmers, Part 3“,Abhijit Belapurkar,

· More Java Actor Frameworks Compared,, Sun Jan 04 2009 09:24:04 GMT-0500 (Eastern Standard Time)

· Concurrency, Actors and Kilim,, Sun Jan 04 2009 09:24:54 GMT-0500 (Eastern Standard Time)

· Event-Based Programming without Inversion of Control,
Philipp Haller and Martin Odersky, Proc. JMLC 2006, and. Accessed 2007 at

· Actors that Unify Threads and Events,
Philipp Haller and Martin Odersky, Proc. COORDINATION 2007. Accessed 2007 at

· “Integrating and Extending JCSP”, Peter WELCHa, Neil BROWNa, James MOORES b, Kevin CHALMERS c and Bernhard SPUTH. Communicating Process Architectures 2007. Accessed 2007 at xxxx.

· “Message Passing–Sync or Async?”, Bartosz milewski,

· A message thread about Erlang related questions,

· “Are Concurrent Programs That Are Easier to Write Also Easier to Check?”, Kedar S. Namjoshi, accessed 20081228 at

· Goldman, Oliver. “Multithreading, Java, & OSGi.”

· Vogel, Lars. “Java Concurrency / Multithreading – Tutorial”,

· “Thread-safety in OSGi”, Neil Bartlett, accessed at

· Ernst, Dan.”Parallelism Exposure – It’s Not That Hard!“,

· “Kilim: Isolation-typed actors for Java”, Sriram Srinivasan,

· “Wot, no chickens?”,

· “Java for High Performance Computing”,

· “4.5 Active Objects”,

· “Rock (processor)”,

· “Early Experience with a Commercial Hardware Transactional Memory Implementation”,Dave Dice, Yossi Lev, Mark Moir, and Dan Nussbaum,

· “Concurrency JSR-166 Interest Site”,

· “Actors Guild”,

· Synchronous Active Objects Home Page,

· “What after Java? From objects to actors”,Carlos A. Varela and Gul A. Agha,

· “Programming Java threads in the real world, Part 9”, Allen Holub,, 06/01/99,

· Actors Foundations for Open System:

· “Power Combination: SCA,, OSGi and Spring”, OSOA Collaboration,

· There Is a Silver Bullet, Brad J. Cox,

· Concurrency and Coordination Runtime,

· Yuri Gurevich, Wolfram Schulte, and Charles Wallace, “Investigating Java Concurrency using Abstract State Machines”. In Y. Gurevich, P. Kutter, M. Odersky, and L. Thiele, eds., Abstract State Machines: Theory and Applications, Springer LNCS 1912, 2000, 151-176.

· Understanding actor concurrency, Part 1: Actors in Erlang A new way to think about structuring concurrent applications By Alex Miller,, 02/24/09,

· “Understanding actor concurrency, Part 2: Actors on the JVM”,

· Byrne, Dennis; “Memory Barriers and JVM Concurrency”, InfoQ,

· Thread Safe Programming Language, Stefan Forster,

· Maestro: A Special-Purpose Language for Parallel Programming, Jonathan Erickson,

· Message Passing Interface (MPI),

· “Best programming approach/methodology to assure thread safety”,

· “RPC and its Offspring: Convenient, Yet Fundamentally Flawed”, Steve Vinoski”,

· “Bigraphs: a model for mobile agents”, Robin Milner, Accessed on 3/3/2009 at

· “Fine-grain concurrency”, Tony Hoare, accessed on 3/3/2009 at

· P. Brinch Hansen, A keynote address on concurrent programming. Keynote address for the IEEE Computer Software & Applications Conference, Chicago, IL, November 1978. Computer 12, 5 (May 1979), 50{56. Copyright °c 1979, Institute of Electrical and Electronics Engineers, Inc., accessed on 3/3/09 at

· “Concurrency: The Next Generation”, Damian Dimmich, Christian Jacobsen and Matthew Jadud, Computing Laboratory University of Kent Canterbury, CT2 7NZ, accessed on 3/3/09 at

· “Concurrency and Erlang” LCA2007 Andre Pang,

· “Concurrency in Erlang & Scala: The Actor Model”, Ruben Vermeersch,

· Erlang for Concurrent Programming, Jim Larson,

· How OSGi Changed My Life, Peter Kriens,

· “New parallel programming language design: a bridge between brain models and multi-core/many-core computers?”,

· Component-based software engineering,

· “Component-based approach for embedded systems”, Ivica Crnkovic Mälardalen University, Department of Computer Science and Engineering,

· Super Actors for Real Time (2001), G. Fortino, L. Nigro, F. Pupo, D. Spezzano, Proc. of the 6 th IEEE Workshop on Object-oriented Real-time Dependable Systems (WORDS’01

· “Fundamentals of Concurrent Programming for .NET” (2005), Charteris White Paper, Greg Beech, Papers – Fundamentals of Concurrent Programming for dotNET (Greg Beech) March 2005.pdf

· “A type safe state abstraction for coordination in Java-like languages”,Ferruccio Damiani, Elena Giachino, Paola Giannini, and Sophia Drossopoulou,Acta Informatica, Volume 45, Numbers 7-8 / December, 2008,


· The Art of Concurrency: A thread Monkey’s Guide to Writing Parallel Applications, Clay Breshears, O’Reilly Media, May 2009. 978-0-596-52153-0.

· Concurrency: State Models & Java Programs, Jeff Magee and Jeff Kramer

· Java Concurrency in Practice, Brian Goetz

· Concurrent Programming in Java, Doug Lea,

· Java Thread Programming, Paul Hyde,

· Java Threads, Oaks & Wong,

· Multithreaded Programming with Windows NT, Thuan Q. Pham and Pankaj K. Garg

· From “Industry Education Recommended Reading List”,

o C# Threading Handbook, Tobin Titus, Sandra Gopikrishna, Tejaswi

o Redkar, Srinivasa Sivakumar, Wrox Press 1861008295

o Computer Architecture: A Quantitative Approach 4th Edition John L. Hennessy and David Patterson Morgan Kaufmann 0123704901

o Multi-Core Programming Shameem Akhter, Jason Roberts Intel Press 0976483246

o Multithreading Applications in Win32: The Complete Guide to Threads Jim Beveridge, Robert Wiener Addison-Wesley Professional 0201442345

o Parallel Programming in C with MPI and OpenMP Michael J. Quinn McGraw-Hill Professional 0072822562

o Parallel Programming in OpenMP Rohit Chandra, Ramesh Menon, Leo Dagum, Dave Kohr, Dror Maydan, and Jeff McDonald Morgan Kaufmann 1558606718

o Using OpenMP: Portable Shared Memory Parallel Programming Barbara Chapman, Gabriele Jost, Ruud van der Pas, David J. Kuck MIT Press 0262533022

o Parallel Programming with MPI Peter Pacheco Morgan Kaufmann 1558603395

o Patterns for Parallel Programming Timothy G. Mattson, Beverly A. Sanders, Berna L. Massingill Addison-Wesley Professional 0321228111

o The Art of Concurrency Clay Breshears O’Reilly 9780596521530

o Programming with Hyper-Threading Technology Andrew Binstock, Richard Gerber Intel Press 0971786143

o Programming with POSIX® Threads David R. Butenhof Addison-Wesley Professional 0201633922

o The Software Optimization Cookbook, Second Edition Richard Gerber, Aart J.C. Bik, Kevin Smith, Xinmin Tian Intel Press 0976483211

o Optimizing Applications for Multi-Core Processors Stewart Taylor Intel Press 1934053015

o Intel Threading Building Blocks: Outfitting C++ for Multi-Core Processor Parallelism James Reinders O’Reilly 0596514808

o Threads Primer: A Guide to Multithreaded Programming Bil Lewis, Daniel J. Berg Prentice Hall 0134436989

Modular Continuously Shippable Applications

June 14, 2010

Long ago in 2000 at a job I suggested to the development teams that an approach we could use is to start on day one with a shippable product.  That is, using Stubs, Mocks, and fake data, we could create the complete app that is testable and shippable from day one.  Since the first customers are the investors and potential partners, they would get to see what we really intended to create up front, in working code, not fancy PowerPoint presentations.  The working code would have served as the presentation and the base on which to build the eventual product to ship to external customers.

This was not just a test-driven development (TDD) approach, I said. For example, we could use a Command Line Interface until we have the Rich Internet Application framework ready, we could use flat-files until we knew what the data model should be, etc.  The “product” was in flux based on potential customer feedback.  So, it was really to do the simplest thing that will get the application doing real work as soon as possible.

The advantages I thought could have been:

  • Faster to market
  • Better API (no fluff)
  • System always testable
  • Infrastructure for testing and QA
  • Usable for presentation
  • Modular

What was the response?  Nothing.  Oh well.  It would have been great to see what could have been accomplished.

I was reminded of the above as I stumbled upon “Architecture Review: Mock-Driven Three-Layer Architecture“ on James Shore’s The Art of Agile blog.

Now there has been much more creative work and new agile languages that would have made my idea more doable.  In fact, many of today’s successful frameworks are responses to the excess complexity that I had to deal with then.



“Potentially Shippable product – A Myth or Reality?”:

Continuous Integration:

“Learn how to Pivot — or you’re toast”:

Mock Object:

Mock-Driven Three-Layer Architecture”,

“Mocks Aren’t Stubs”,

“Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation (Addison-Wesley Signature Series (Fowler))”,



Get every new post delivered to your Inbox.