Is There a Return on Investment from Model-Based Systems Engineering (MBSE)? Part 1

Join us for a live webinar on this topic, “Is There a ROI From MBSE” on Thursday, October 17th at 11:00 am ET. Register Here.

This question is one that many people ask. In fact, the International Council on Systems Engineering (INCOSE) has that as one of its tasks for their Value Proposition Initiative. A group of systems engineers is trying to find evidence to prove that MBSE has value. However, that becomes very difficult for a concept that has only been around for a dozen years, when the lifecycle of many of the systems of interest are measured in several decades.

We can approach this question by inference. If there is a significant return on investment in systems engineering, then we can infer that there might be one for MBSE. Fortunately, we do have many decades of experience applying systems engineering to project since the 1950s (at least, depending on how you define systems engineering). One of the best analyses I have come across over the years was a very interesting piece of work by Werner M. Gruhl who was at the time the Chief of the Cost & Economics Analysis Branch at NASA. His work was published in a NASA technical paper entitled, “Issues in NASA Program and Project Management” (NASA SP-6101 (08), in 1994. In a paper by Ivy Hooks in this publication, she states that “if the program requirements are not well understood, there is not much hope for estimating the cost of a program.” She continues: “Werner Gruhl developed a history of NASA programs versus cost overruns” and cited the diagram below (redrawn to due to the poor quality of the document found – an old scanned PDF). She interpreted this chart as “if you have not done a good job in Phase A and B in defining and confining your program, you are going to encounter large numbers of changing requirements and the cost will go up accordingly.” Note the figure below indicates % Spent on Systems Engineering, which it is really Phase A and B in NASA terminology.

Thus, it’s clear that at least the combination of program management and systems engineering, which is what allows you to properly develop the set of requirements for the program, is required to keep the cost of the project from sky rocketing. Note that program management and systems engineering are flip sides of the same coin. The program manager optimizes cost, schedule, and performance, while mitigating risk in each of these areas. The systems engineer is tasked to do the same for the system. That’s why these two disciplines have been seen to overlap, which was recognized in a recent book by INCOSE and the Program Management Institute (PMI).

Another more recent attempt at qualifying the value of systems engineering came from the Software Engineering Institute. They conducted several studies to determine the value of systems engineering, including one documented in their November 2012 paper entitled “The Business Case for Systems Engineering Study: Results of the Systems Engineering Effectiveness Survey.” The authors “found clear and significant relationships between the application of SE best practices to projects and the performance of those project” as seen in the figure below.

Project Performance vs. Total SE Capability

In a later presentation by Mr. Joseph P. Elm, one of the authors of the 2012 paper, on “Quantifying the Effectiveness of Systems Engineering,” he cites a finding for a General Accountability Office (GAO) report (GAO-09-362T) that states:

“… managers rely heavily on assumptions about system requirements, technology, and design maturity, which are consistently too optimistic. These gaps are largely the result of a lack of a disciplined systems engineering analysis prior to beginning system development …”

So, it is recognized that there is great value in performing at least the “right amount” of systems engineering. If we use the Gruhl graph as a basis, we need to spend around 7-12% of the program’s budget on the combination of program management and systems engineering. Since the cost of the program could as much as double on average according to the chart, the return would be about 10 times the investment. For example, if we spent $100,000 on systems engineering and program management and the overall cost of the program was $1,000,000, then since the cost could have doubled to $2,000,000, we save $1,000,000.

So now that we agree there is a substantial return on investment in systems engineering, let’s get back to the question of ROI on MBSE. The question here is then, “does modeling help systems engineering.” Since we have always done modeling in systems engineering, I think that is clearly a part of good systems engineering. But the flavor of MBSE being pushed by many in the community has equated MBSE to SysML and many have also equated SysML as implemented by MagicDraw. But does SysML and MagicDraw® do all the things we need to do in systems engineering? In particular, do we obtain a good set of system requirements in a form easily used by all the stakeholders?

To begin to answer these questions, let’s go back to Mr. Elm’s paper. He states that the systems engineer must perform the following tasks:

  • Requirements Development
  • Requirements Management
  • Trade Studies
  • System Architecture Development
  • Interface Management
  • Configuration Management
  • Program Planning
  • Program Monitoring and Control
  • Risk Management
  • Product Integration Planning and Oversight
  • Verification Planning and Oversight
  • Validation Planning and Oversight

 

SysML consists of nine diagram type, most of which were derived from software engineering practices, not systems engineering. Yes, there is overlap between the two, but not as much as the overlap between systems engineering and program management. That becomes obvious from the task list above, many of which include explicitly the word “management.”

SysML also has proven to be very difficult for most other disciplines to understand, since they speak other languages. It also takes at least two large books, “A Practical Guide to SysML” and “SysML Distilled.” In comparison, the Lifecycle Modeling Language provides a whole systems engineering ontology and limited diagramming that completely subsume SysML. However, LML can still be explained in a very thin book, “Essential LML.” You can see the comparison in the picture below.

You probably are asking, “How is that possible that LML subsumes SysML?” By using an ontological approach that defines a set of entity classes and their relationships, along with the attributes on both, LML provides all the elements of a real language (nouns, verbs, adjectives and adverbs). This ontology can be used to capture the information easily and efficiently. Then that information can be displayed in many ways, including all the nine SysML diagram types.

The Innoslate® tool proves this assertion, as it produces all nine SysML diagrams (and many more) from this ontology as extended in Version 1.1 of the LML specification. In addition to the SysML diagrams, Innoslate produces the LML Action Diagram, that represents the same information as the SysML Activity Diagram, but in a significantly more understandable form. We can see this when we compare side by side the two type of diagrams, as shown below.

LML Action Diagram Example

SysML Activity Diagram

In the SysML diagram I need to know what the diamond and fork symbols mean. In the Action Diagram, I know exactly what they mean, because the words: OR, LOOP, and Decomposed, make their intent clear. In addition, in SysML I cannot just allocate the decision points to who or what performs them. I can in LML. Of course, if there were only two symbols I needed to decipher, then I would not care as much, but SysML has over 30 such symbols. You will need a “3-D decoder ring” to fully understand how to use all the symbols, hence the very large books and long training classes needed to try to learn SysML. This learning curve translates into a significant investment in the workforce to get them up to speed on this complex language. Of course, the electrical engineers, mechanical engineers, logistics experts, and all the other disciplines have their own languages and have no interest in learning something this complex.

You might say, “but of course MagicDraw overcomes these limitations in SysML?” The answer to that is not as well. In particular, if we go back to the list of systems engineering tasks, MagicDraw only does one “well:” System Architecture Development. Although MagicDraw has some limited requirements capability, almost everyone uses another requirements tool in conjunction with it. Innoslate by comparison has a robust Requirements View that includes automated requirements quality checking using the artificial intelligence technique of Natural Language Parsing (NLP). The requirements then can be directly traced to the diagram entities within the same tool, resulting in one database and no configuration management problems that you encounter having two databases. Innoslate also has a built in Test Center for the V&V activities. In addition, Innoslate provides Discrete Event and Monte Carlo simulators to verify that the Action (or Activity) Diagrams have been correctly done. We use that same approach to support Program Planning, Monitoring and Control.

So back to the ROI discussion. Can MBSE provide a healthy ROI? Only if we do all the things we need to do in systems engineering. In addition, if we can use modern technology to help automate these difficult tasks, we can provide even higher ROI than systems engineering by itself. Innoslate and LML provide a means to provide this higher ROI, while MagicDraw and SysML actually cost much, much more to implement and you end up with a poorer result. So, if you want ROI from your MBSE investment, use Innoslate and LML.

Join us for a live webinar on this topic, “Is There a ROI From MBSE” on Thursday, October 17th at 11:00 am ET. Register Here.