Blog Administration
Friday, December 29. 2006
GAO Releases Report on NMCI
Well, color us surprised. The Government Accountability Office released a not-so-flattering report on NMCI. In that standard GAO flair for the catchy title, the report is officially called, "Information Technology: DOD Needs to Ensure That Navy Marine Corps Intranet Program Is Meeting Goals and Satisfying Customers". Rolls, right off the tongue.
GCN (Article) called the conclusions "blunt". Blunt implies short and it was anything but short. Although it was relatively narrow in scope.
The report lifted the skirt of the those at the Navy-EDS dance and asks some tough questions. You can get a copy for yourself at http://www.gao.gov/cgi-bin/getrpt?GAO-07-51.
This was not the first visit to NMCI Fantasy Land for GAO. Nearly six months before the contract award GAO released "Defense Acquisitions: Observations on the Procurement of the Navy/Marine Corps Intranet, GAO/T-NSIAD/AIMD-00-116" and in 2002, "Information Technology: Issues affecting Cost Impact of Navy/Marine Corps Intranet need to be resolved, GAO-03-33". There goes GAO again with the catchy titles.
Let's start at the top.
What's this all about?
First, what this report was not. It was not a technical evaluation of alternatives or an engineering trade study, touching only briefly on the technical failings of NMCI. It did not, for example, discuss the rationale for choosing a particular software application or evaluate the network design or discuss the technical prowess of those operating and maintaining the intranet. The did not make any recommendations to that end either.
Instead, the report focused on the programmatics. It looked at the fiscal and administrative impact of the venture and asked, "Did the Navy get what it asked for and what it paid for?" Just as importantly, "Does the Navy know what it asked for and how would it know if it DID get it?" Remember, the GAO folks were working under the authority of the Comptroller General's office -- they weren't there to dig into bits and bytes.
So why did they do the report? Well, they lay it out very eloquently in the report summary. They were to determine:
(1) Whether the program is meeting its strategic goals
(2) The extent to which the contractor is meeting service level agreements
(3) Whether customers are satisfied with the program
(4) what is being done to improve customer satisfaction
GAO gets the facts
You don't need us to regurgitate the executive summary, its all there. Yet there are a few pearls that deserve highlighting. We'll also dig a bit deeper into the meat of the report in the next sections here. First, lets address those four objectives for the report.
Strategic Goals: "NMCI has not met its two strategic goals—to provide information superiority and to foster innovation via interoperability and shared services. The Navy’s mapping shows that NMCI has met only 3 of 20 performance targets (15 percent). This means that the mission-critical information superiority and operational innovation outcomes used to justify NMCI have yet to be attained." Oops. strike one.
Service Level Agreements: "Navy measurement of [service level] agreement satisfaction shows that performance needed to receive contractual incentive payments for the most recent 5-month period was attained for about 55 to 59 percent of all eligible seats, which represents a significant drop from the previous 9-month period." Strike two.
Customer Satisfaction: "...end user satisfaction surveys indicated that the percent of end users that met the Navy’s definition of a satisfied user has remained consistently below the target of 85 percent (latest survey results categorize 74 percent as satisfied). Given that the Navy’s definition of the term “satisfied” includes many marginally satisfied and arguably somewhat dissatisfied users, this percentage represents the best case depiction of end user satisfaction. Survey responses from the other two customer groups show that both were not satisfied." Strike three. But there's more.
Improving Customer Satisfaction: "...the Navy identified various initiatives that it described as completed, under way, or planned. However, the initiatives are not being guided by a documented plan(s), thus limiting their potential effectiveness. Without such an [effectively planned] approach, improvement efforts can be reduced to trial and error." And, strike four.
They sum this all up with this jewel that could have easily been taken from these very web pages:
"This means that after investing about 6 years and $3.7 billion, NMCI has yet to meet expectations, and whether it will is still unclear."
You gotta love that.
What they found
Let's look at some of the interesting things the GAO folks found during their investigation.
So we already let the cat out of the bag and told you that GAO found NMCI was not meeting its main strategic project goals (although the report says that program officials were quite happy to tell GAO that the program did meet them). Yet what exactly were these goals and how were they articulated by the Navy before the program was launched?
The first goal was "information superiority". Wait, don't laugh, it gets better. To obtain this goal, NMCI was to "create an integrated network in which connectivity among all parts of the shore establishment, and with all deployed forces at sea and ashore, enables all members of the network to collaborate freely, share information, and interoperate with other services and nations." Wow. Navy users don't even have access to the Marine Corps E-mail address book.
The second goal was to "foster innovation". They were going to obtain this goal by "providing interoperable and shared services environment that supports innovative ways of integrating doctrine and tactics, training, and supporting activities into new operational capabilities and more productive ways of using resources." Apparently using a seven year old OS with six year old hardware on a locked down network is a fostering environment. Once again we see flowery verbiage that falls short on every single measure.
The best part is the benefits that would flow forth from these goals: (1) uninterrupted flow of information; (2) improvements to interoperability, security, information assurance, knowledge sharing, productivity, and operational performance; and (3) reduced costs.
That, folks, is how they sold this whole mess to the people with the checkbook. After seeing those goals and reading those "benefits" that were used to birth NMCI is there any doubt that GAO was disappointed in what they found?
The vaunted super-double-secret user surveys that NMCI and the Navy refused to release (see: Our article on surveys and our other article) were printed in full and glorious detail in the GAO report, Appendix II. Apparently GAO didn't feel that releasing the question pool "compromised the integrity of the survey" like the Navy leadership did. Good for GAO, We didn't believe that nonsense either. What's more, the report published some surprising (at least to us) results of these prescient surveys.
The surveys are corralled into three groups: end user, commander and network operator. While we would all like to believe that it is only the worker-bee, end users screaming about the crippling failures of NMCI, apparently the "commanders" and "network operators" groups were even less satisfied. These groups got a different set of questions on the surveys but on a scale of 0 to 3 in satisfaction level, commanders averaged 0.8 and network operators averaged 0.3. So for everyone down in the trenches who believed that the leadership and backroom folks were brainwashed into believing the NMCI propaganda, relax. They know NMCI is a failure also. Nobody is satisfied with NMCI.
It was also interesting to note that the satisfaction levels varied with the workforce organizations. While the Navy System Commands (NAVAIR, NAVSEA) averaged 66% satisfied, Navy Installations averaged 84% satisfied. Similarly, "Aggregated Marines" (MARCORSYSCOM, HQMC, MCCDC) averaged 69% satisfied while MARFORRES averaged 77%. One could perhaps draw some conclusions from that but it is unclear what the cause and effect of those conclusions would be. It should be noted that the percentage of satisfied users includes those who GAO categorized as "marginally satisfied" so even these numbers can be called into question.
One of the biggest slaps in the face comes from the Marine Corps Network Operations and Security Command (MCNOSC) who's network operations group scored NMCI as zero on every single area the surveys addressed. There was no round-off error. They were scored zero on every single one for over a year.
GAO was shocked to find that several organizations had to continue to use their legacy systems because of the failings on NMCI. Don't be shocked. You could have read that here many months ago! People will find a way to get their job done even if it means working outside of a system that has failed them. Also, stop crediting these successes to NMCI. They are successful in spite of NMCI, not because of it.
NMCI "program officials" (we would be interested to know if these were contractor or Government officials) tried to defend their failure by stating that Government users are ignorant of the contract and what is provided. Implying that many of the complaints, continued need for legacy systems and perceived shortfalls of NMCI don't really exist, it is simply ignorant users who are not aware of what NMCI has to offer. They also stated that, "they have not been provided any data showing a drop in workforce productivity caused by NMCI." What? What!? We have been crying, "The emperor has no clothes!" but nobody is listening. GAO just handed you your data. Quit blaming Government employees for failures by the Navy and EDS leadership. Just because we refuse to pay obscene prices for substandard NMCI services and material doesn't mean we don't know about them.
Appendix III lists the SLA descriptions and performance levels for every single SLA established for the program. Perhaps one of the scariest SLA blunders is SLA 106-Information Assurance. Anyone familiar with DoD IA efforts knows they can be a virtual minefield of paperwork, bureaucracy and endless Orders, Instructions and Standards. Yet they are also some of the most important controls placed on any IT system. NMCI received "Not Met" ratings on 11 months of the 18 months for which they had data. Five other months are categorized as "No measurement taken." It is unclear why or how the program failed this important category but as it deals with "security event detection, security even reporting, security even response, and IA configuration management" lets hope the failures were minor.
Conclusions and Recommendations
We'd like to report that GAO recommended scrapping the entire NMCI effort but unfortunately their recommendations were a bit softer. However, it is clear that they are not enamored with the Navy's effort thus far and failed to fall victim to the blustering leaders and their smoke screens. Again, focusing on the programmatics, GAO reaches two main conclusions with some well-worded warnings for the Navy.
Goals and Measures: "...it is important for such programs to be grounded in outcome-based strategic goals that are linked to performance measures and targets, and it is important for progress against these goals, measures, and targets to be tracked and reported to agency and congressional decision makers. If such measurement does not occur, then deviations from program expectations will not become known in time for decision makers to take timely corrective action. The inevitable consequence is that program results will fall short of those that were promised and used to justify investment in the program."
Performance management and customer satisfaction: "despite investing in a range of activities intended to improve customer satisfaction, plans to effectively guide these improvement efforts, including plans for measuring the success of these activities, have not been developed. Given that the Navy reports that it has already invested about 6 years and $3.7 billion in NMCI, the time to develop a comprehensive understanding of the program’s performance to date, and its prospects for the future, is long overdue."
The report recommends:
"[The secretary of the Navy] ensure that the NMCI program adopts robust performance management practices that, at a minimum, include (1) evaluating and appropriately adjusting the original plan for measuring achievement of strategic program goals and provides for its implementation in a manner that treats such measurement as a program priority; (2) expanding its range of activities to measure and understand service level agreement performance to provide increased visibility into performance relative to each agreement; (3) sharing the NMCI performance results with DOD, Office of Management and Budget, and congressional decision makers as part of the program’s annual budget submissions; and (4) reexamining the focus, scope, and transparency of its customer satisfaction activities to ensure that areas of dissatisfaction described in this report are regularly disclosed to the aforementioned decision makers and that customer satisfaction improvement efforts are effectively planned and managed. In addition, ... take appropriate steps to ensure that the findings in this report and the outcomes from implementing the above recommendations are used in considering and implementing warranted changes to the NMCI’s scope and approach."
And the Navy's response?
The Navy was given a late draft version of the report so they could provide comments, corrections and it would seem, rebuttal. Their response would still have you believe the emperor has clothes. The final report as published has the Navy counter comments verbatim (in an appendix) and the GAO's counter counter comments (near the end of the report).
The Navy's response is summarized by GAO into five "points" or areas of disagreement. We'll distill these down even further (in detail, not quantity) here.
Navy: You talked to the wrong people.
GAO: We talked to the Navy and the Marine Corps. Who else is there?
Navy: NMCI is meeting its strategic goals... because we say so.
GAO: NMCI has met 3 of 20 performance categories associated with the goals. That's not a success.
Navy: The SLA data was misinterpreted.
GAO: Its the Navy's own interpretation. How else would we interpret it?
Navy: Customers can only be satisfied or dissatisfied. Therefor a response of 5.5 out of 10 is fully satisfied.
GAO: We'd like to think that 5.5 to 7 is "marginally satisfied" if that's o.k.
Navy: We adequately report to key program decision makers.
GAO: Apparently not. GAO was tasked to determine why NMCI is such a failure.
Did you actually expect anything else from the Navy?
GCN (Article) called the conclusions "blunt". Blunt implies short and it was anything but short. Although it was relatively narrow in scope.
The report lifted the skirt of the those at the Navy-EDS dance and asks some tough questions. You can get a copy for yourself at http://www.gao.gov/cgi-bin/getrpt?GAO-07-51.
This was not the first visit to NMCI Fantasy Land for GAO. Nearly six months before the contract award GAO released "Defense Acquisitions: Observations on the Procurement of the Navy/Marine Corps Intranet, GAO/T-NSIAD/AIMD-00-116" and in 2002, "Information Technology: Issues affecting Cost Impact of Navy/Marine Corps Intranet need to be resolved, GAO-03-33". There goes GAO again with the catchy titles.
Let's start at the top.
What's this all about?
First, what this report was not. It was not a technical evaluation of alternatives or an engineering trade study, touching only briefly on the technical failings of NMCI. It did not, for example, discuss the rationale for choosing a particular software application or evaluate the network design or discuss the technical prowess of those operating and maintaining the intranet. The did not make any recommendations to that end either.
Instead, the report focused on the programmatics. It looked at the fiscal and administrative impact of the venture and asked, "Did the Navy get what it asked for and what it paid for?" Just as importantly, "Does the Navy know what it asked for and how would it know if it DID get it?" Remember, the GAO folks were working under the authority of the Comptroller General's office -- they weren't there to dig into bits and bytes.
So why did they do the report? Well, they lay it out very eloquently in the report summary. They were to determine:
(1) Whether the program is meeting its strategic goals
(2) The extent to which the contractor is meeting service level agreements
(3) Whether customers are satisfied with the program
(4) what is being done to improve customer satisfaction
GAO gets the facts
You don't need us to regurgitate the executive summary, its all there. Yet there are a few pearls that deserve highlighting. We'll also dig a bit deeper into the meat of the report in the next sections here. First, lets address those four objectives for the report.
Strategic Goals: "NMCI has not met its two strategic goals—to provide information superiority and to foster innovation via interoperability and shared services. The Navy’s mapping shows that NMCI has met only 3 of 20 performance targets (15 percent). This means that the mission-critical information superiority and operational innovation outcomes used to justify NMCI have yet to be attained." Oops. strike one.
Service Level Agreements: "Navy measurement of [service level] agreement satisfaction shows that performance needed to receive contractual incentive payments for the most recent 5-month period was attained for about 55 to 59 percent of all eligible seats, which represents a significant drop from the previous 9-month period." Strike two.
Customer Satisfaction: "...end user satisfaction surveys indicated that the percent of end users that met the Navy’s definition of a satisfied user has remained consistently below the target of 85 percent (latest survey results categorize 74 percent as satisfied). Given that the Navy’s definition of the term “satisfied” includes many marginally satisfied and arguably somewhat dissatisfied users, this percentage represents the best case depiction of end user satisfaction. Survey responses from the other two customer groups show that both were not satisfied." Strike three. But there's more.
Improving Customer Satisfaction: "...the Navy identified various initiatives that it described as completed, under way, or planned. However, the initiatives are not being guided by a documented plan(s), thus limiting their potential effectiveness. Without such an [effectively planned] approach, improvement efforts can be reduced to trial and error." And, strike four.
They sum this all up with this jewel that could have easily been taken from these very web pages:
"This means that after investing about 6 years and $3.7 billion, NMCI has yet to meet expectations, and whether it will is still unclear."
You gotta love that.
What they found
Let's look at some of the interesting things the GAO folks found during their investigation.
So we already let the cat out of the bag and told you that GAO found NMCI was not meeting its main strategic project goals (although the report says that program officials were quite happy to tell GAO that the program did meet them). Yet what exactly were these goals and how were they articulated by the Navy before the program was launched?
The first goal was "information superiority". Wait, don't laugh, it gets better. To obtain this goal, NMCI was to "create an integrated network in which connectivity among all parts of the shore establishment, and with all deployed forces at sea and ashore, enables all members of the network to collaborate freely, share information, and interoperate with other services and nations." Wow. Navy users don't even have access to the Marine Corps E-mail address book.
The second goal was to "foster innovation". They were going to obtain this goal by "providing interoperable and shared services environment that supports innovative ways of integrating doctrine and tactics, training, and supporting activities into new operational capabilities and more productive ways of using resources." Apparently using a seven year old OS with six year old hardware on a locked down network is a fostering environment. Once again we see flowery verbiage that falls short on every single measure.
The best part is the benefits that would flow forth from these goals: (1) uninterrupted flow of information; (2) improvements to interoperability, security, information assurance, knowledge sharing, productivity, and operational performance; and (3) reduced costs.
That, folks, is how they sold this whole mess to the people with the checkbook. After seeing those goals and reading those "benefits" that were used to birth NMCI is there any doubt that GAO was disappointed in what they found?
The vaunted super-double-secret user surveys that NMCI and the Navy refused to release (see: Our article on surveys and our other article) were printed in full and glorious detail in the GAO report, Appendix II. Apparently GAO didn't feel that releasing the question pool "compromised the integrity of the survey" like the Navy leadership did. Good for GAO, We didn't believe that nonsense either. What's more, the report published some surprising (at least to us) results of these prescient surveys.
The surveys are corralled into three groups: end user, commander and network operator. While we would all like to believe that it is only the worker-bee, end users screaming about the crippling failures of NMCI, apparently the "commanders" and "network operators" groups were even less satisfied. These groups got a different set of questions on the surveys but on a scale of 0 to 3 in satisfaction level, commanders averaged 0.8 and network operators averaged 0.3. So for everyone down in the trenches who believed that the leadership and backroom folks were brainwashed into believing the NMCI propaganda, relax. They know NMCI is a failure also. Nobody is satisfied with NMCI.
It was also interesting to note that the satisfaction levels varied with the workforce organizations. While the Navy System Commands (NAVAIR, NAVSEA) averaged 66% satisfied, Navy Installations averaged 84% satisfied. Similarly, "Aggregated Marines" (MARCORSYSCOM, HQMC, MCCDC) averaged 69% satisfied while MARFORRES averaged 77%. One could perhaps draw some conclusions from that but it is unclear what the cause and effect of those conclusions would be. It should be noted that the percentage of satisfied users includes those who GAO categorized as "marginally satisfied" so even these numbers can be called into question.
One of the biggest slaps in the face comes from the Marine Corps Network Operations and Security Command (MCNOSC) who's network operations group scored NMCI as zero on every single area the surveys addressed. There was no round-off error. They were scored zero on every single one for over a year.
GAO was shocked to find that several organizations had to continue to use their legacy systems because of the failings on NMCI. Don't be shocked. You could have read that here many months ago! People will find a way to get their job done even if it means working outside of a system that has failed them. Also, stop crediting these successes to NMCI. They are successful in spite of NMCI, not because of it.
NMCI "program officials" (we would be interested to know if these were contractor or Government officials) tried to defend their failure by stating that Government users are ignorant of the contract and what is provided. Implying that many of the complaints, continued need for legacy systems and perceived shortfalls of NMCI don't really exist, it is simply ignorant users who are not aware of what NMCI has to offer. They also stated that, "they have not been provided any data showing a drop in workforce productivity caused by NMCI." What? What!? We have been crying, "The emperor has no clothes!" but nobody is listening. GAO just handed you your data. Quit blaming Government employees for failures by the Navy and EDS leadership. Just because we refuse to pay obscene prices for substandard NMCI services and material doesn't mean we don't know about them.
Appendix III lists the SLA descriptions and performance levels for every single SLA established for the program. Perhaps one of the scariest SLA blunders is SLA 106-Information Assurance. Anyone familiar with DoD IA efforts knows they can be a virtual minefield of paperwork, bureaucracy and endless Orders, Instructions and Standards. Yet they are also some of the most important controls placed on any IT system. NMCI received "Not Met" ratings on 11 months of the 18 months for which they had data. Five other months are categorized as "No measurement taken." It is unclear why or how the program failed this important category but as it deals with "security event detection, security even reporting, security even response, and IA configuration management" lets hope the failures were minor.
Conclusions and Recommendations
We'd like to report that GAO recommended scrapping the entire NMCI effort but unfortunately their recommendations were a bit softer. However, it is clear that they are not enamored with the Navy's effort thus far and failed to fall victim to the blustering leaders and their smoke screens. Again, focusing on the programmatics, GAO reaches two main conclusions with some well-worded warnings for the Navy.
Goals and Measures: "...it is important for such programs to be grounded in outcome-based strategic goals that are linked to performance measures and targets, and it is important for progress against these goals, measures, and targets to be tracked and reported to agency and congressional decision makers. If such measurement does not occur, then deviations from program expectations will not become known in time for decision makers to take timely corrective action. The inevitable consequence is that program results will fall short of those that were promised and used to justify investment in the program."
Performance management and customer satisfaction: "despite investing in a range of activities intended to improve customer satisfaction, plans to effectively guide these improvement efforts, including plans for measuring the success of these activities, have not been developed. Given that the Navy reports that it has already invested about 6 years and $3.7 billion in NMCI, the time to develop a comprehensive understanding of the program’s performance to date, and its prospects for the future, is long overdue."
The report recommends:
"[The secretary of the Navy] ensure that the NMCI program adopts robust performance management practices that, at a minimum, include (1) evaluating and appropriately adjusting the original plan for measuring achievement of strategic program goals and provides for its implementation in a manner that treats such measurement as a program priority; (2) expanding its range of activities to measure and understand service level agreement performance to provide increased visibility into performance relative to each agreement; (3) sharing the NMCI performance results with DOD, Office of Management and Budget, and congressional decision makers as part of the program’s annual budget submissions; and (4) reexamining the focus, scope, and transparency of its customer satisfaction activities to ensure that areas of dissatisfaction described in this report are regularly disclosed to the aforementioned decision makers and that customer satisfaction improvement efforts are effectively planned and managed. In addition, ... take appropriate steps to ensure that the findings in this report and the outcomes from implementing the above recommendations are used in considering and implementing warranted changes to the NMCI’s scope and approach."
And the Navy's response?
The Navy was given a late draft version of the report so they could provide comments, corrections and it would seem, rebuttal. Their response would still have you believe the emperor has clothes. The final report as published has the Navy counter comments verbatim (in an appendix) and the GAO's counter counter comments (near the end of the report).
The Navy's response is summarized by GAO into five "points" or areas of disagreement. We'll distill these down even further (in detail, not quantity) here.
Navy: You talked to the wrong people.
GAO: We talked to the Navy and the Marine Corps. Who else is there?
Navy: NMCI is meeting its strategic goals... because we say so.
GAO: NMCI has met 3 of 20 performance categories associated with the goals. That's not a success.
Navy: The SLA data was misinterpreted.
GAO: Its the Navy's own interpretation. How else would we interpret it?
Navy: Customers can only be satisfied or dissatisfied. Therefor a response of 5.5 out of 10 is fully satisfied.
GAO: We'd like to think that 5.5 to 7 is "marginally satisfied" if that's o.k.
Navy: We adequately report to key program decision makers.
GAO: Apparently not. GAO was tasked to determine why NMCI is such a failure.
Did you actually expect anything else from the Navy?
(Page 1 of 1, totaling 1 entries)
