NMCI and Other Nefarious Adventures - NMCI

Big Brother or Just Plain Incompetence

nospam@example.com (Charlie) — Sat, 03 Feb 2007 09:43:56 -0800

Some days it's hard to tell if the back-room folks at NMCI are doing something intentionally or if they are just bumbling around and do something by accident. Let's just say the latest networking oddity wasn't preannounced, unless we missed that memo.

It appears that they are blocking Google cache. Again, this "blocking" appears to the user as simply a network error. There is no announcement screen stating that the requested Google cache pages are blocked by policy. This effect also appears to come and go. Some days the cached information is available, some days it isn't.

The purpose for such a Big Brother Block is obvious. The cached pages on Google (other than imagery) allow users to circumvent any site-specific blocking done by NMCI administrators. Our site, for example, is very much available in Google cache, even to an otherwise blocked NMCI host.

Of course the legitimate uses for Google cache make this latest draconian restriction more than just a little annoying. Some sites are extremely slow or have recently gone down for some reason. Or some times all you need is the text from a site and don't want to wait for pictures to load -- the "cached text only" option on Google cache is great for this. Of course none of these things are available if the cache is blocked.

What's the solution? Simple. Find a different way to skin that cat! Use another search engine and access the cache information there. A great example is MyWay.com which provides a front-end to Google, Yahoo, Ask.com and others without annoying banner ads and pop-ups. Simply enter your search terms and let it do a default search (currently defaults to ask.com). When it pulls up the information, choose "Google" from the tabs at the top. This is the same information available at the regular Google.com page. Now all your cache information is available. Or at least the text is.

You could be reading this page right now from your NMCI seat! Happy Googling!

GAO Releases Report on NMCI

nospam@example.com (Charlie) — Fri, 29 Dec 2006 09:22:00 -0800

Well, color us surprised. The Government Accountability Office released a not-so-flattering report on NMCI. In that standard GAO flair for the catchy title, the report is officially called, "Information Technology: DOD Needs to Ensure That Navy Marine Corps Intranet Program Is Meeting Goals and Satisfying Customers". Rolls, right off the tongue.

GCN (Article) called the conclusions "blunt". Blunt implies short and it was anything but short. Although it was relatively narrow in scope.

The report lifted the skirt of the those at the Navy-EDS dance and asks some tough questions. You can get a copy for yourself at http://www.gao.gov/cgi-bin/getrpt?GAO-07-51.

This was not the first visit to NMCI Fantasy Land for GAO. Nearly six months before the contract award GAO released "Defense Acquisitions: Observations on the Procurement of the Navy/Marine Corps Intranet, GAO/T-NSIAD/AIMD-00-116" and in 2002, "Information Technology: Issues affecting Cost Impact of Navy/Marine Corps Intranet need to be resolved, GAO-03-33". There goes GAO again with the catchy titles.

Let's start at the top.

What's this all about?
First, what this report was not. It was not a technical evaluation of alternatives or an engineering trade study, touching only briefly on the technical failings of NMCI. It did not, for example, discuss the rationale for choosing a particular software application or evaluate the network design or discuss the technical prowess of those operating and maintaining the intranet. The did not make any recommendations to that end either.

Instead, the report focused on the programmatics. It looked at the fiscal and administrative impact of the venture and asked, "Did the Navy get what it asked for and what it paid for?" Just as importantly, "Does the Navy know what it asked for and how would it know if it DID get it?" Remember, the GAO folks were working under the authority of the Comptroller General's office -- they weren't there to dig into bits and bytes.

So why did they do the report? Well, they lay it out very eloquently in the report summary. They were to determine:

(1) Whether the program is meeting its strategic goals
(2) The extent to which the contractor is meeting service level agreements
(3) Whether customers are satisfied with the program
(4) what is being done to improve customer satisfaction

GAO gets the facts
You don't need us to regurgitate the executive summary, its all there. Yet there are a few pearls that deserve highlighting. We'll also dig a bit deeper into the meat of the report in the next sections here. First, lets address those four objectives for the report.

Strategic Goals: "NMCI has not met its two strategic goalsto provide information superiority and to foster innovation via interoperability and shared services. The Navys mapping shows that NMCI has met only 3 of 20 performance targets (15 percent). This means that the mission-critical information superiority and operational innovation outcomes used to justify NMCI have yet to be attained." Oops. strike one.

Service Level Agreements: "Navy measurement of [service level] agreement satisfaction shows that performance needed to receive contractual incentive payments for the most recent 5-month period was attained for about 55 to 59 percent of all eligible seats, which represents a significant drop from the previous 9-month period." Strike two.

Customer Satisfaction: "...end user satisfaction surveys indicated that the percent of end users that met the Navys definition of a satisfied user has remained consistently below the target of 85 percent (latest survey results categorize 74 percent as satisfied). Given that the Navys definition of the term satisfied includes many marginally satisfied and arguably somewhat dissatisfied users, this percentage represents the best case depiction of end user satisfaction. Survey responses from the other two customer groups show that both were not satisfied." Strike three. But there's more.

Improving Customer Satisfaction: "...the Navy identified various initiatives that it described as completed, under way, or planned. However, the initiatives are not being guided by a documented plan(s), thus limiting their potential effectiveness. Without such an [effectively planned] approach, improvement efforts can be reduced to trial and error." And, strike four.

They sum this all up with this jewel that could have easily been taken from these very web pages:
"This means that after investing about 6 years and $3.7 billion, NMCI has yet to meet expectations, and whether it will is still unclear."

You gotta love that.

What they found
Let's look at some of the interesting things the GAO folks found during their investigation.

So we already let the cat out of the bag and told you that GAO found NMCI was not meeting its main strategic project goals (although the report says that program officials were quite happy to tell GAO that the program did meet them). Yet what exactly were these goals and how were they articulated by the Navy before the program was launched?

The first goal was "information superiority". Wait, don't laugh, it gets better. To obtain this goal, NMCI was to "create an integrated network in which connectivity among all parts of the shore establishment, and with all deployed forces at sea and ashore, enables all members of the network to collaborate freely, share information, and interoperate with other services and nations." Wow. Navy users don't even have access to the Marine Corps E-mail address book.

The second goal was to "foster innovation". They were going to obtain this goal by "providing interoperable and shared services environment that supports innovative ways of integrating doctrine and tactics, training, and supporting activities into new operational capabilities and more productive ways of using resources." Apparently using a seven year old OS with six year old hardware on a locked down network is a fostering environment. Once again we see flowery verbiage that falls short on every single measure.

The best part is the benefits that would flow forth from these goals: (1) uninterrupted flow of information; (2) improvements to interoperability, security, information assurance, knowledge sharing, productivity, and operational performance; and (3) reduced costs.

That, folks, is how they sold this whole mess to the people with the checkbook. After seeing those goals and reading those "benefits" that were used to birth NMCI is there any doubt that GAO was disappointed in what they found?

The vaunted super-double-secret user surveys that NMCI and the Navy refused to release (see: Our article on surveys and our other article) were printed in full and glorious detail in the GAO report, Appendix II. Apparently GAO didn't feel that releasing the question pool "compromised the integrity of the survey" like the Navy leadership did. Good for GAO, We didn't believe that nonsense either. What's more, the report published some surprising (at least to us) results of these prescient surveys.

The surveys are corralled into three groups: end user, commander and network operator. While we would all like to believe that it is only the worker-bee, end users screaming about the crippling failures of NMCI, apparently the "commanders" and "network operators" groups were even less satisfied. These groups got a different set of questions on the surveys but on a scale of 0 to 3 in satisfaction level, commanders averaged 0.8 and network operators averaged 0.3. So for everyone down in the trenches who believed that the leadership and backroom folks were brainwashed into believing the NMCI propaganda, relax. They know NMCI is a failure also. Nobody is satisfied with NMCI.

It was also interesting to note that the satisfaction levels varied with the workforce organizations. While the Navy System Commands (NAVAIR, NAVSEA) averaged 66% satisfied, Navy Installations averaged 84% satisfied. Similarly, "Aggregated Marines" (MARCORSYSCOM, HQMC, MCCDC) averaged 69% satisfied while MARFORRES averaged 77%. One could perhaps draw some conclusions from that but it is unclear what the cause and effect of those conclusions would be. It should be noted that the percentage of satisfied users includes those who GAO categorized as "marginally satisfied" so even these numbers can be called into question.

One of the biggest slaps in the face comes from the Marine Corps Network Operations and Security Command (MCNOSC) who's network operations group scored NMCI as zero on every single area the surveys addressed. There was no round-off error. They were scored zero on every single one for over a year.

GAO was shocked to find that several organizations had to continue to use their legacy systems because of the failings on NMCI. Don't be shocked. You could have read that here many months ago! People will find a way to get their job done even if it means working outside of a system that has failed them. Also, stop crediting these successes to NMCI. They are successful in spite of NMCI, not because of it.

NMCI "program officials" (we would be interested to know if these were contractor or Government officials) tried to defend their failure by stating that Government users are ignorant of the contract and what is provided. Implying that many of the complaints, continued need for legacy systems and perceived shortfalls of NMCI don't really exist, it is simply ignorant users who are not aware of what NMCI has to offer. They also stated that, "they have not been provided any data showing a drop in workforce productivity caused by NMCI." What? What!? We have been crying, "The emperor has no clothes!" but nobody is listening. GAO just handed you your data. Quit blaming Government employees for failures by the Navy and EDS leadership. Just because we refuse to pay obscene prices for substandard NMCI services and material doesn't mean we don't know about them.

Appendix III lists the SLA descriptions and performance levels for every single SLA established for the program. Perhaps one of the scariest SLA blunders is SLA 106-Information Assurance. Anyone familiar with DoD IA efforts knows they can be a virtual minefield of paperwork, bureaucracy and endless Orders, Instructions and Standards. Yet they are also some of the most important controls placed on any IT system. NMCI received "Not Met" ratings on 11 months of the 18 months for which they had data. Five other months are categorized as "No measurement taken." It is unclear why or how the program failed this important category but as it deals with "security event detection, security even reporting, security even response, and IA configuration management" lets hope the failures were minor.

Conclusions and Recommendations
We'd like to report that GAO recommended scrapping the entire NMCI effort but unfortunately their recommendations were a bit softer. However, it is clear that they are not enamored with the Navy's effort thus far and failed to fall victim to the blustering leaders and their smoke screens. Again, focusing on the programmatics, GAO reaches two main conclusions with some well-worded warnings for the Navy.

Goals and Measures: "...it is important for such programs to be grounded in outcome-based strategic goals that are linked to performance measures and targets, and it is important for progress against these goals, measures, and targets to be tracked and reported to agency and congressional decision makers. If such measurement does not occur, then deviations from program expectations will not become known in time for decision makers to take timely corrective action. The inevitable consequence is that program results will fall short of those that were promised and used to justify investment in the program."

Performance management and customer satisfaction: "despite investing in a range of activities intended to improve customer satisfaction, plans to effectively guide these improvement efforts, including plans for measuring the success of these activities, have not been developed. Given that the Navy reports that it has already invested about 6 years and $3.7 billion in NMCI, the time to develop a comprehensive understanding of the programs performance to date, and its prospects for the future, is long overdue."

The report recommends:

"[The secretary of the Navy] ensure that the NMCI program adopts robust performance management practices that, at a minimum, include (1) evaluating and appropriately adjusting the original plan for measuring achievement of strategic program goals and provides for its implementation in a manner that treats such measurement as a program priority; (2) expanding its range of activities to measure and understand service level agreement performance to provide increased visibility into performance relative to each agreement; (3) sharing the NMCI performance results with DOD, Office of Management and Budget, and congressional decision makers as part of the programs annual budget submissions; and (4) reexamining the focus, scope, and transparency of its customer satisfaction activities to ensure that areas of dissatisfaction described in this report are regularly disclosed to the aforementioned decision makers and that customer satisfaction improvement efforts are effectively planned and managed. In addition, ... take appropriate steps to ensure that the findings in this report and the outcomes from implementing the above recommendations are used in considering and implementing warranted changes to the NMCIs scope and approach."

And the Navy's response?
The Navy was given a late draft version of the report so they could provide comments, corrections and it would seem, rebuttal. Their response would still have you believe the emperor has clothes. The final report as published has the Navy counter comments verbatim (in an appendix) and the GAO's counter counter comments (near the end of the report).

The Navy's response is summarized by GAO into five "points" or areas of disagreement. We'll distill these down even further (in detail, not quantity) here.

Navy: You talked to the wrong people.
GAO: We talked to the Navy and the Marine Corps. Who else is there?

Navy: NMCI is meeting its strategic goals... because we say so.
GAO: NMCI has met 3 of 20 performance categories associated with the goals. That's not a success.

Navy: The SLA data was misinterpreted.
GAO: Its the Navy's own interpretation. How else would we interpret it?

Navy: Customers can only be satisfied or dissatisfied. Therefor a response of 5.5 out of 10 is fully satisfied.
GAO: We'd like to think that 5.5 to 7 is "marginally satisfied" if that's o.k.

Navy: We adequately report to key program decision makers.
GAO: Apparently not. GAO was tasked to determine why NMCI is such a failure.

Did you actually expect anything else from the Navy?

Hang In There, Devil Dog

nospam@example.com (Charlie) — Fri, 13 Oct 2006 23:17:00 -0700

A sickeningly common report on the day-to-day lives of our active duty personnel and what they do to survive the NMCI assaults. Nice to see it in print.

Hang tough, Marine. We're all in this together. There are some things that can't be rectified with application of high explosives. As much as we'd like to try....

Congressional Scrutiny - Maybe

nospam@example.com (Charlie) — Sun, 01 Oct 2006 08:27:00 -0700

Federal Computer Week has an article that indicates some of the fallacies of NMCI may be surfacing to the people who hold the wallets.

A great, if short-sighted quote is offered from House Armed Services Committee showing that at least someone has noticed that "spin" and window dressing can only go so far. Addressing the spiraling costs and technical failures of NMCI, the committee expressed concern over "the enduring nature of legacy programs that a now mature NMCI was supposed to replace."

Perhaps the label "short sighted" is too harsh but we believe the cause and effect relationship that spawned this quote should lead one to dig deeper. The reason the legacy systems endure is because NMCI is not working correctly. Not working as advertised. Not working as promised. Perhaps this is self-evident to the committee and they did not want to draw attention to anyone's ugly baby.

Knock Knock

nospam@example.com (Charlie) — Sat, 02 Sep 2006 10:44:00 -0700

Oh, yes. There will be commentary.

No, the jack-booted thugs didn't come in the dark of night and beat us with a stick. We are still here.

Truth be told, the din of idiocy has simply numbed us over the past few months. Nothing manages to surprise us anymore so we have remained tacit more out of apathy than out of purpose. Most of the roll-out has been completed over the past couple years and although there are many new victims, the pain for all of us remains about the same.

We will be waiting with baited breath as the promised "tech refresh" rolls out. If it ever does. Many of the early deployment sites are well into four years on their hardware. This is hardware that was two years out of date when they originally received it. Of course Windows 2000 is going into it's seventh year of life, long ago eclipsed by Windows XP and soon to be further antiqued by Windows Vista. Inquiring minds would love to see the check that EDS writes to Microsoft every year for almost exclusive support of the outdated Windows 2000.

Alas, a faint whiff of hope crossed our sensors a couple weeks ago when stories about the "renegotiation" of the contract surfaced. The rumor mill is just spooling up on this so there isn't even any good calumniations to report but we hold out no hope for anything better than what we have now. The emperor still has no clothes and he's not about to come around after so many years of believing his own lies.

Spin Some More

nospam@example.com (Charlie) — Wed, 29 Mar 2006 17:47:00 -0800

In a flurry of feel-good quotes, the Navy extended the NMCI contract with another $3.1 billion. In for a penny, in for a billion. Once again, we turn to the good folks at Federal Computer Week for the compact version.

In a dizzying flourish of Clinton-esque "spin", EDS stated that the newly "crafted" contract extension "...ensures the long-term success of the Navy and Marine Corps mission...."

Wow. It ensures it. Can we maybe focus on the short-term success? Can we focus on any success at all?

Mighty Quiet Out There

nospam@example.com (Charlie) — Tue, 28 Feb 2006 22:45:10 -0800

Yep. A little too quiet....

Hear no evil, see no evil, speak no evil.

Censorship is not dead! It appears that NMCI began blocking network access to our site sometime around the middle of February. We are not sure about this but we've had reports from several people and the lack of NMCI gateways in our log files would appear to confirm this.

Am I upset about this? No way! I'm flattered! To think that someone at NMCI was so afraid of their users finding out the truth that they actually changed their configurations to block our site. I couldn't ask for better publicity!

Our readers will connect to us from their home ISPs or from legacy networks. Or they can simply read the cached articles on Google. Big brother hasn't made it into every aspect of our lives just yet but they are giving it the old college try.

At the risk of being too sanctimonious, let me offer the following:

We are not saying anything in these pages that isn't said out loud, every day, by thousands of victimized NMCI users. I could play a game of cat-and-mouse and move to a different IP address. I could do some tricky DNS stuff. But that would just be playing their game. No, we will be here... right here. We will continue to publish, even if nobody from the NMCI leadership wants to read it. Obviously they don't want you to read it.

Happy Birthday!

nospam@example.com (Charlie) — Fri, 17 Feb 2006 00:01:00 -0800

Say "Happy Birthday!" to your NMCI computer today. Your operating system is six years old.

Actually its well over six years old but that was the official birth date.

Are you patched? Could you do anything about it if you weren't?

Working Within the Software System

nospam@example.com (Charlie) — Wed, 18 Jan 2006 19:09:36 -0800

Let's be clear up front, we are not advocating anything below and doing anything proposed below may not be permitted by your local authority. However, we know of at least one instance where these types of things were permitted for certain mission requirements. We are also not sure if all of these would work as we have not tried them ourselves. Your mileage may vary. We'd be interested to hear any test results.

What if we could run a real application on our NMCI machines without violating the rules about installing software? Do you long to use Firefox or Thunderbird or a real FTP application? Do you need to use secure shell (SSH) to talk to devices or other servers?

Although it was rumored that you could install Firefox without modifying the registry at a non-S&T seat, doing so would still be prohibited by the "install no software" rule. Here is a way to use the full-blown Firefox and other applications without actually installing anything.

Portable Firefox is available which can be run off a USB drive. You can save all your settings and even import settings from your legacy machines.

They have several other applications available at the Portable Apps Site which should run fine on NMCI desktops and laptops.

What if you need to do something that just isn't possible in Windows 2000? Could you load Linux on an NMCI machine? No.

Well, maybe.

There is no reason that Damn Small Linux could not be booted from a CD. Or if you need more horsepower, Knoppix is certainly the king of bootable Linux CDs. It should recognize all the hardware on both NMCI laptops and NMCI desktops. It would leave the internal hard drives untouched. Leaving the NMCI machine connected to the NMCI network while booted into Linux would probably not be advisable.

Of course one extremely valuable use for a bootable Linux CD would be disaster recovery of a damaged NMCI Windows installation or a hard drive failure. Both Knoppix and DSL should mount both the internal hard drive and an external USB device. This would allow you to copy documents off the hard drive and on to the memory stick or other external drive. As the internal hard drives are (presumably) formatted as NTFS, you would not be able to write information to the internal hard drive of an NMCI computer. That's probably a good thing.... Although Linux is close to being able to mount an NTFS drive as read/write.

The Myth of the S&T Seat

nospam@example.com (Charlie) — Sun, 20 Nov 2005 22:43:08 -0800

The initial promise of the NMCI Science and Technology (or S&T) seat was great. Some call it a "developer's" account or "dot dev" after the login name but it is all the same thing. Users believed that they would have significant advantages over the standard NMCI user seat. Indeed, the S&T seat was conceived to answer the needs of the programmers, developers, engineers and technicians that required more control over their computing environment. What these power users received was something less than they expected.

Let's take a look at some of the common beliefs about the S&T seat and see what is true and what isn't true. We make no judgment on some of these policies as a few of them "come with the territory" when you subscribe to a managed IT paradigm and would probably be imposed on any managed IT user in any corporate setting. As always, we leave the cost vs. benefit evaluation of this managed IT world to the reader.

S&T users have administrative privileges on their computers
This is only partially true. One also has to place this within the context of other restrictions placed on the NMCI user, many of which will be discussed below. S&T users do indeed have administrative privileges however some of the common things that an administrator would want or need to do are not possible as an S&T user.

Simply logging in to the Microsoft Windows Update site to install security updates is not allowed. This is understandable when you consider that NMCI and the Navy have carefully carved their NMCI seats into a very custom build with some security/performance updates installed and others not installed. Although this is not a completely uncommon practice among managed IT systems, there is always something to be said for simply keeping everything current. Much of the tripe circulating in IT circles about "black Tuesday" upgrades breaking machines is nothing short of rumor and anti-Microsoft propaganda. Millions of fully updated users can prove this pick-and-choose methodology wrong every day. Trying to juggle various updates (for an outdated operating system such as Windows 2000) and installing some and not installing others proves to be very difficult and often needlessly exposes users to security vulnerabilities.

If an S&T user attempts to use Windows Task Manager to cancel a zombie (or annoying) process, there is no guarantee such an action will meet with success. Often such attempts will be met with "Operation could not be completed. Access is denied."

There are likely other Administrator actions, like Windows Update, that are physically blocked or broken to the S&T user. Yet by far, most of the capabilities of an Administrator user are blocked by policy and rule rather than hard, OS-based enforcement. Many of these are discussed in the following paragraphs.

S&T users can install any software they want
This is categorically false. Although the operating system, as configured, may not prevent an S&T user from installing software even if registry changes are required, it is against NMCI policy to do so. Even installing something as innocuous as an updated Microsoft hardware driver is not allowed.

Of course NMCI requires the computer user to have valid licenses for all installed software. This is nothing unusual. What is unusual is that even if the user has a boxed, licensed copy of a piece of software, there is no guarantee that they can install it. The software must be on a list of approved applications. The exact version the user wishes to install must also be on the list. If it isn't, the user's only recourse is to submit the desired software to NMCI for approval. The vaunted approval process has been tried by only a few hearty souls willing to endure the costly and lengthy process of having NMCI technicians test and endorse the software. One could say, "Well, of course they don't want users installing random software, it could break the OS/computer/network!" This would be true except for one thing: NMCI throws its S&T users to the wolves anyway. See S&T users can not call the help desk below.

So suppose an S&T user is lucky enough to find his desired application on the approved list. He still has to notify his local NMCI officials that he is installing the software and submit paperwork to do so. Any software found installed on a system that is not part of an official push or a documented subsequent S&T install is grounds for immediate revocation of user privileges and disciplinary action.

S&T seats are not allowed access to E-mail
This is true while logged in as the S&T user. Although we have reports of several S&T users who regularly access NMCI email (normally sent to their non-S&T accounts) this is still, as far as we know, prohibited under NMCI policy. This requires S&T users to log out of their S&T accounts and log back in as a non-S&T user just to check their E-mail, several times a day. One could speculate on the reason for this policy however we would hope that it is being done for security reasons.

S&T users can not do custom network applications
This is true at most sites on most network segments. The S&T user is subjected to the same draconian IP network filtering as the non-S&T user. There are no open ports except those that are proxied by NMCI proxy servers. This makes custom network applications impossible to test on the NMCI network. S&T seats could easily be placed on a VLAN to keep them separate from the protected non-S&T network and allow S&T users the flexibility they need to work with network-intensive applications.

S&T users have access to BuRAS
This is false. It is more false than it might first appear. S&T users are not only prohibited from using BuRAS while they are logged in as S&T users, they can't even use it while they are logged in as regular users. In fact, NMCI has refused to push the BuRAS software to any machine that is operated as an S&T seat. So even if the user has never logged in to his S&T account, he does not have the same BuRAS capability as the non-S&T user sitting next to him. This policy is supposed to change in the near future but as it presently stands, the S&T user is on the losing end again.

Software development is impossible on an S&T seat
This is true. This is another example where NMCI policy oversteps the physical limitations on the administrative user. Even if the user installs a compiler that is from the approved application list he is violating policy simply by compiling source code. Part of the agreement for S&T users is that they will not "execute, install or compile" non approved code. Obviously if they are developing new software, it is not on the approved application list.

S&T users can not call the help desk
This is partially -- the most important part -- true. Science and Technology users are welcome to call the help desk as long as it doesn't involve a software issue. It doesn't matter if the user has never installed anything on the computer. NMCI has deemed S&T users as experts and throws them to the wolves in terms of help desk support. The actual CLIN says, "Customer support will be limited to those services offered by EDS and not extend to software or hardware loaded and configured by the user." Yet in reality S&T users are told not to call the help desk for anything remotely related to software. So one could ask, "If they are not going to support me, why can't I install any software I want?" If the help desk is going to treat S&T users differently whether they install some bizarre piece of software or nothing at all, why put restrictions on the software?

There are probably other fine points of the Science and Technology lifestyle that we haven't discussed here but we hope the above items are enough food for thought. One now has to ask, "Is all this really worth an extra $25 per month?"

NMCI Hacked?

nospam@example.com (Charlie) — Fri, 11 Nov 2005 22:16:00 -0800

Several people have asked about NMCI being hacked on or around 20 October. Our site traffic stats also reflect an increased interest in this possible event. We can neither confirm nor deny any compromise of the NMCI infrastructure. We don't have a clue.

We have resisted posting anything here until now in an attempt to allow any security compromises to be handled or at least minimized. However Federal Times was not afraid to publish.

The truth is, we have no information. And if we did, we wouldn't post it here.

Thanks for asking though.

You didn't really want to read that anyway

nospam@example.com (Charlie) — Sun, 16 Oct 2005 01:29:00 -0700

Hold on folks, we're in for a bumpy ride.

Rumor has it that the vaunted Phase II of the Spam Flail-ex is about to begin. Several sources from an east coast facility say that "suspected spam" quarantining will start in the near future. As reported here earlier, users will have a fixed amount of time to respond to messages put into the quarantined area before it is deleted. If the user is unable, for whatever reason, to access his mailbox during the quarantine period, the E-mail is deleted. Messages determined to be "known spam", as opposed to suspected spam, will be dealt with even more swiftly -- it will be deleted immediately with no notification to the intended recipient.

We can only hope that the tagging debacle of a few months ago does not portend a quarantine storm where dozens of legitimate E-mails per day end up erroneously identified as spam.

Anyone who observed the monumental failure of spam tagging is surely holding their breath for the quarantine phase. We can only hope that the powers-that-be noticed the problems with tagging and have tweaked the spam identification process to prevent the same false-positive problem from occurring with this new phase.

We can always hope.

Ya Who Not You

nospam@example.com (Charlie) — Fri, 14 Oct 2005 01:25:00 -0700

Once again we see fallacy in action. NMCI has apparently started blocking web access to Hotmail, Yahoo Mail, Google Mail and other on-line, web-based mail sites. So what could their reason be for such a drastic measure?

Very simply this is DoN policy. That's right folks, don't blame the good folks at EDS for this one. The Navy has deemed access to unofficial E-mail as an unacceptable risk to the integrity of their internal networks. Certainly users could unsuspectingly download a virus, worm, Trojan, malware or other Nasty in the form of an E-mail or attachment.

Yet one needs to ask, how is downloading such a threat from an E-mail site any more likely than acquiring one from a regular website? What makes web E-mail any more of a threat?

In fact, there are several things that make such a stance even more illogical and lay fallacy to the idea that web-based E-mail is a higher threat.

First, most large web-based E-mail services do scanning on attachments. Yahoo, AOL, Juno and others all scan their user's E-mail for suspicious items. So most of these sites are probably less likely to have dangerous code on them than millions of other regular websites.

Secondly, in the same vein, how are the file links on a regular website checked when a user clicks on them? Perhaps in the NMCI HTTP proxy system, but does this checking occur for SSL sites? Point is, if there is a threat on web mail sites, then there is a threat on all web sites. Maybe they should block access to everything except .mil and .gov sites. I know. Don't give them any ideas.

Third, many of the recent vulnerabilities publicized for Microsoft products do not need a user-executed code vector to do their business, they exploit flaws in the browser itself. There are several easily exploited weaknesses that Microsoft has been quick to address in its current Internet Explorer products and Windows XP-SP2. Once again we see the problems with having an outdated OS and application software on NMCI seats. So if the exploit is on the webpage itself, and has nothing to do with a user downloading a malicious attachment, some of the safest sites on the net would be reputable sites like Hotmail, Yahoo and AOL.

Lastly, we wonder where the NMCI proxy administrators got their list of web-based E-mail sites. How complete could it be? How many thousands of Squirrelmail sites and small ISP sites are out there?

Their method for "blocking" these sights appears to be little more than DNS modification. So someone at the central site has gone through the domain name servers and put manual entries in for things like mail.yahoo.com and www.hotmail.com. These new entries redirect the user to a warning banner instead of taking them to the requested mail web page. Of course this means that any local user (S&T .dev account not required) can modify their own hosts table to override these same DNS entries. For Windows 2000 users this means simply going into c:\winnt\system32\drivers\etc\hosts and placing a few entries like:

66.218.75.184 mail.yahoo.com
64.233.185.83 mail.google.com

In reality it would prove a little more difficult as many mail sites use many different host names and jump around during the user session. It all comes down to following the rules -- the admins can make things more difficult and keep out the casual user but if someone insists on breaking the rules, they will.

There was also widespread speculation that the stance resulted from (or the fear of) users forwarding Government E-mail to non-Government servers -- for whatever reason. This opens up the possibility of FOUO, Privacy Act, Procurement Sensitive and even classified data (which would point to bigger problems) being sent to servers outside the Government's control. The thought of NMCI going to Yahoo and asking them to wipe their terrabytes of hard drives simply to remove one FOUO memo is not something anyone wants to deal with. It is still unclear how restricting web access to these remote sites has anything to do with someone writing an auto-forward rule in Outlook. Keeping them from reading it is different than keeping them from sending it.

At some point it all comes back to trusting the users to follow the policy. If the policy states that DoN users are not allowed to access personal E-mail from an NMCI seat, then at some point that has to be good enough. People are going to put infected floppies, USB drives and CDs into their seats. People are going to forward inappropriate mail to outside addresses. People are going to visit compromised web sites with an outdated browser. People are going to open infected attachments received through NMCI/Outlook. Web-based E-mail is probably the least of our worries. Blocking major websites is nothing more than window dressing. Once again....

The SPAM Debacle Appears To Be Over

nospam@example.com (Charlie) — Tue, 19 Jul 2005 19:50:44 -0700

"The day is ours, the bloody dog is dead." -- Mr. Shakespeare

In what can only be described as a tacit admission of utter and complete failure, NMCI administrators have disabled the tagging of spam in association with the "word lists" method described elsewhere in these pages.

You heard it here first. We told you it wouldn't work.

Not only did it not work to defeat spam, it caused horrendous disruption for thousands of Navy and Marine Corps users. Rampant false positives left a user's mailbox littered with erroneous warnings about "Sexual content", "profanity" and "Proprietary content".

The warnings were changed cosmetically early on in the debacle to include the words "May contain..." which was apparently done in an effort to make users feel better about false positives and the associated ineffective treatment of spam. Another cosmetic change was made from "sexual content" to "unauthorized content". This was not only completely useless, but was most likely technically incorrect for places like Naval hospitals where legitimate e-mails probably contain words that are both sexual in content and authorized.

One official from the East coast instructed managers to tell complaining users that, "...this is the solution to SPAM and unwanted e-mail users have been asking for." Really? We would be interested to see the requests to use such an ill-conceived method to address the spam issue.

The erroneous tagging was more than just annoying for the primary recipients. The tags, adulterating both the body of the E-mail and the subject line, had to be scrubbed before the message could be forwarded or replied to. Wasting the Government's time and resources once again, thank you NMCI.

No white-listing or "intelligent" source analysis was done on the E-mails. Simply confirming the message source (through IP address, not sender fields) as .mil or .gov addresses could have easily abated some of the mess and probably reduced loading on the systems parsing thousands of messages a day. Of course there never was any facility for the users to do custom white-lists of their own.

User's quarantined messages were to be deleted in seven days, regardless of activity from the intended recipient. So if a user was on TDY, vacation or away from an NMCI seat for more than seven days his E-mail would begin to be permanently purged having never seen the light of day. There was no provision for the user to extend this time period or disable the "feature" of quarantine all together.

We are still completely baffled why IronPort would cast its shadow on such an idiotic undertaking. SpamCop.net, Bonded Sender Program, Senderbase.org and C-Series Appliances all speak to IronPorts ability to provide very effective spam handling. What were they thinking?

Waging War on SPAM

nospam@example.com (Charlie) — Mon, 04 Jul 2005 21:01:00 -0700

In an effort to help wage the Holy War against UCE/SPAM, NMCI has announced that it will be partnering with IronPort Systems and Symantec. We applaud this effort and can only hope that NMCI will begin utilizing the excellent SpamCop real-time blacklist from IronPort Systems. NMCI press releases state that the new spam solution "will provide NMCI with advanced threat prevention, block SPAM, and enable effective DoN e-mail policy enforcement." This statement, of course, was written by a salesman or an accountant but by applying the nonsense filter, this appears to point in a hopeful direction.

One disturbing statement addresses the proposed method of identifying spam. "...the DoN and EDS have agreed to a set of business rules designed to identify specific words or phrases that are contained in SPAM messages." We can only hope that they do not actually intend to waste Government time and money on building a word/phrase list to identify SPAM. Such a sophomoric attempt at tagging spam might have worked ten years ago but "bad" word lists would be lucky to catch 5% of SPAM today. Bayesian filtering applies a statistically weighted version of a word list but needs to be customized for each application and the filter must "learn" from actual user E-mail for it to be effective. Spammers use dozens of methods to defeat simple word filters and even Bayesian filters. Certainly companies like Symantec and IronPort know this already. It is curious that NMCI would even mention such a useless task.

An intranet article referenced in the announcement explains that spam will be kept in a type of escrow account where the user will be able to review suspected spam and move it to their inbox or delete it (confirming that it is spam).

All of this, with the notable exception of word lists, sounds like an excellent but long-overdue treatment of spam on the NMCI network. Kudos to NMCI for the effort but we make the following suggestions:

1. Use real-time DNS blacklists at the mail gateway to block SMTP connections before they even connect to the server. Spamcop.net (an Ironport Systems project now) and Spamhaus.org (SBL-XBL) are two excellent choices if anyone is listening....

2. Abandon the use of "word and phrase lists" as they are completely ineffective and a waste of resources. The high number of false positives will cause user distrust and the use of pure word lists, as opposed to pattern matching, has been ineffective for the past several years. Smart matching from programs like Spam Assassin are an excellent alternative but come at the cost of processor loading.

3. Utilize a Bayesian database for each user to help build a statistical perspective that is surprisingly effective even with extensive Bayes poisoning campaigns now being waged by spammers.