Blog Administration |
Wednesday, February 23. 2005Pay No Attention to That Man Behind the Curtain
The Navy Marine Corps Intranet (NMCI) is a mandatory, omnibus information technology (IT) contract run by the U.S. Department of the Navy (DoN). The current prime contractor is EDS Corporation. All active duty military and DoD civilians within the Department of the Navy (U.S. Navy and U.S. Marine Corps) are obligated to use this contract for current and future technology needs such as networks, computers, servers, software and IT infrastructure.
Awarded in the year 2000, the first "seat" was activated in September of 2001. Over 500,000 users will be converted over from legacy systems by the end of fiscal year 2005. At least that’s the plan. Initial plans for the contract - or so it was rumored - included all voice and data requirements of the DoN. This would include management, design, acquisition, maintenance and operation of a huge host of services including voice telephone service, cellular phone and pager service, satellite connectivity, tactical communications, microwave and voice radio connectivity and, of course, anything in the computer and IT realm. This "super contract" was quickly scaled down to focus on IT. EDS Corporation won the bid for the contract that, at the time, was awarded for $6.9 Billion. One of the largest contracts of its kind in the history of the DoD. Mr. Robert Cringely has an excellent article on the finer points of the bidding process in his article which includes a great analysis of why and how EDS under-bid the contract. The Emperor Has No Clothes! Funding for the contract has bloated up from the initial $6.9 Billion to over $8.8 Billion as of 2004. The DoN continues to pour money into the contract in an effort to resuscitate both EDS and the ideal of outsourced IT. It is unclear what impact this expenditure -- the initial costs and the over-run -- will have on future DoN budgets as the contract was never approved by the U.S. Congress. Navy officials continue to tout the success of the contract despite huge outcries from the actual users who have to suffer with the equipment and services provided by the contract. In Cringely's second article he points out that one size does not fit all and it is folly to attempt this with such a large, diverse and technically skilled workforce. Mr. Cringely also highlights a trend that goes beyond the Navy and beyond NMCI. That is, the changing role of "outsourcing" in America. It used to be that a company who made Widget A decided that they wanted or needed to make Widget B. So they outsourced the production of Widget B so they could focus on what they do best: making Widget A. The trend in outsourcing now is different. Companies who make both Widget A and Widget B decide that they are doing a bad job with Widget B so, through outsourcing, they try to find someone else who screws it up less than they do. The Navy can do IT. EDS is not better at IT than the Navy. Someone just needed a plan. Instead they roped up $8.8 Billion and gave the job to someone else, hoping they would screw it up less. The Navy is better than EDS and any leader who doesn’t believe that should not be a leader. User "satisfaction surveys" are whitewashed or gilded to show what a great success the effort has been. A 55% satisfaction level was touted as a testimony for the great level of worker satisfaction with NMCI performance. In 2003 and 2004 there were a large number of laptop computers that began experiencing hard drive failures. NMCI refused to release the actual numbers of failures. Data began to be collected anecdotally at each facility in an effort to gain some visibility into the problem. Why would NMCI conceal the failure rates for equipment and force the Government to waste time trying to recreate statistics? One word: SLA. Well, that's an acronym... but it's kind of like a word. Service Level Agreements (SLA) are how EDS gets paid. It is how they make bonus. It is simply a metric used to determine if the contractor is meeting, failing or exceeding expectation thresholds. Obviously he who controls the data, controls his own fate. If you don’t release the results of surveys and tests, nobody can say you are not meeting your minimums. Additionally, the contract has an up-to-speed clause that states EDS is not required to meet any SLAs until they can demonstrate, for 90 consecutive days, that they can accomplish the minimum requirements of the contract. So as long as things don’t function correctly for 90 days at a time, they are not responsible for meeting SLAs. Excellent planning. Service response times are one of the most common complaints of NMCI users. The admirals can't figure this one out. Every time they call the help-desk on a computer issue, the technician is standing at the door before the admiral can put the phone back in the cradle. No service problems here. Imagine the sailor, underway on a Navy ship. He has no administrator privileges on his computer and his IT support staff is 300 miles away over open water. Giving the sailors the tools they need… or not. Come On In. The Water is Fine. In an effort to bolster support (or spread the misery) the Navy has tried to convince other organizations, from the U.S. Coast Guard to NASA, to adopt either the NMCI contract or a similar omnibus vehicle. Officially, these other departments are adopting a wait-and-see attitude. Unofficially they are running for the hills. The U.S. Air Force has a huge technology core to support such things as its fighter/bomber aircraft and its enormous space-based (satellite) groups. It has yet to belly up to the bar to drink from the bitter cup of NMCI despite the salesmanship of senior Navy officials. Snake Oil! Get Your Snake Oil here! When the work force was first told about NMCI, it was touted as a move towards a leaner, more efficient, standardized approach that was going to save the Navy money. A cost-cutting measure. In fact, that was the justification given for spending $6.9 Billion with little or no oversight. The funds would be repaid with the savings from leaning out the IT. Distributed help desks and network gurus could be removed from the hundreds of military bases and field sites and centralized into a core. Efficiency would skyrocket. After the initial "rollout" of seats began (and of course the contract had been awarded), people started to question the cost savings being touted. Inquiries to produce and explain the accounting behind these supposed cost savings were met with hand-waving and throat-clearing. EDS, from a business perspective, was nearly crushed. Huge cost overruns began to surface and the Navy and EDS started running out of fingers to plug holes in the dike. The impact to the Navy is incalculable in terms of lost work hours, reduced capability and general disruption. These affects are still being felt by many users trying to bring their NMCI computers up to speed just to do their job. Time to change gears. Forget that cost savings stuff. Advancement can be expensive. NMCI will be a secure network. Ya, that's it. It will be more secure than the existing, diverse legacy networks and equipment. Except for the fact that in 2003, thousands of supposedly secure machines and networks were infected with exactly the same worms as millions of other Windows PCs and servers around the world. In 2005, what was only rumored as, "NMCI was hacked" led to thousands of users being forced to change their passwords under emergency-like conditions. The details of this compromise were understandably never released but we hope NMCI administrators took notice. Additionally, one would also need to raise an eyebrow at the entire concept of allowing a contractor organization to be in control of Government data. Not necessarily from a classified/unclassified standpoint -- although that could warrant some thought also -- but more from the concept of impartiality. All the servers and desktop machines are under the control of EDS. Where does a procurement official store procurement-sensitive data? Would that procurement person even know if EDS, a contractor, was looking at their data? How are personnel, medical and financial data being secured? Since September 11th the whole concept of "sensitive" data has changed. Who has access to this data? O.k. so NMCI’s not so secure, let's try this again. We will achieve greatness through standardization. Ya, that's it. Everyone knows standardization is a good thing. Certainly in the mid 1990s, back when well-meaning leaders were nurturing the seeds of NMCI, there was reason to be concerned with the state of standardization. Word Perfect battled Microsoft Word. Lotus and Excel shot it out in the spreadsheet arena. Novell had no use for TCP/IP. VMS and Unix flexed their muscles.... If a command or organization did adopt a standard, it was invariably a different standard from the organization down the street. Yet as we fast forward to 2001 (and certainly 2005) we see the Navy has changed as much as every other technology-centric organization in the developed world. Microsoft suites are now standard. They are even dictated by things like the IT21 initiative. TCP/IP is the standard for networking and, again, concepts like ForceNet mandated it as a standard over technologies like ATM. PCs, running either Windows or Linux, are already standardized in the DoD and are even making their way into the traditionally proprietary world of embedded and tactical systems. How old is the "standardization" problem they are trying to solve? "Damn the torpedoes! Full speed ahead!" So, What's the Problem? So now that we’ve talked in generalities about unfounded claims of cost savings, security lapses and the quest for standardization, perhaps some more specific examples would be in order. Users are subjected to a managed Windows desktop. Managed, meaning they have very little control over not only the look of their Windows desktop but the functionality of the computer as a whole. Certainly not something that is reserved solely for NMCI, many corporate IT departments limit what end-users can install on their machines. Under many circumstances this makes sense to reduce the risk of virus infections and instability caused by users installing software packages they shouldn’t. But what do the users get? As of 2005: Windows 2000 (Phased out with the introduction of XP in 2001) Netscape 4.7 (Phased out in 1999) Internet Explorer 5.0 (Phased out in 2002) Office 2000 (Phased out in 2002) Why are all these items so outdated? Because EDS is on a three-year update cycle, at least with the hardware. Obviously the software and OS are on even longer cycles. Understandably, if you are buying 500,000 computer suites, you can't be expected to update everything the instant the OEM changes versions. However, many Navy customers were told (either officially or otherwise) that NMCI has adopted a "one behind" policy on software. That is, users will not get Windows XP until the rest of the world has moved on to XP's successor. There is no question that NMCI made an effort to downgrade many of the PCs supplied by Dell under the contract, uninstalling EOM software. NMCI laptop users can frequently turn their laptops over and find the OEM sticker for Windows XP on the bottom. Part of the promise of increased efficiency included disbanding local help desks and network support centers so that these tasks could be centralized. In fact many of the legacy support personnel were promised jobs with EDS once they were removed from their positions with the Navy. That benefit not withstanding -- in fact some reports are that these people were quickly let go after being hired -- it appears that many bases have quietly recreated their Government support infrastructure even though they are converted to NMCI. They either do it unofficially, assigning people "collateral duties" in addition to their regular tasks or they simply never disbanded their legacy support systems. These "shadow help desks" create duplication of effort and negate any purported benefit of NMCI centralization. Why do the Navy personnel do this when they could simply let NMCI fail? Because they are there to get the job done. Whatever it takes. The Dot Dev Account So, what if the user needs special application software or newer versions of popular applications that are not included as part of the managed desktop? Well, for a price, NMCI will sell you the administration rights to your own computer. It's called a "dot dev" or S&T (science and technology) or developer's seat. But there are several catches, even with this. As a .dev user, you do have the ability to install most software but the .dev user is still not granted full rights. A quick trip to the Windows on-line update site at windowsupdate.microsoft.com greets the .dev user with the warning: "Network policy settings prevent you from using Windows Update to download and install updates on your computer." But wait, there's still more. Just because you have the ability to install software through the OS doesn’t mean you have the permission to install it. If the software is not on the approved list of applications, it can not be installed. Users of .dev accounts sign a contract stating that they will not install, download or compile any software that has not been approved. One could ask what good such a system would do for a programmer. Every single time he changes code and recompiles he is essentially creating an unapproved piece of software on his NMCI seat. The .dev user has no access to E-mail while logged in to his development account. So any applications that would normally interface with Outlook (e.g. Palm Organizer or PDA software) are useless. Additionally, the .dev user has to log out of the development account and log in to his normal, non-development account several times a day just to check E-mail. For many E-mail-centric users, the .dev account is almost never used because of this reason. However he still gets charged extra, every day, just for having the privilege. Lastly, as a developer's seat, your help-desk support also gets placed into a special category. Unless you pay for it, you are not provided software support for any application, even those that are supplied as part of the basic package (called the Gold Disk). So if your MS Word application becomes corrupt and refuses to load, even if it was not the result of anything done by the .dev user, the NMCI help desk is not required to assist with this. We can do that! How Much Money Do You Have? Like most Firm Fixed Price contracts (if you can call $6.9B to $8.8B "fixed") NMCI has a Contract Line Item Number (CLIN) list. Each CLIN has an item number and the user can pick from this list just like a menu. If it isn’t on the list, they can't do it. There are some changes allowed to this list over the lifespan of the contract. However, it would be incorrect to call the pricing "dynamic". If the cost of a P4 laptop next year is half the price of this year's model, don't expect to see the NMCI CLIN for a laptop go down by half. There is no explicit requirement for the contractor to keep up with technology or prevalent pricing of that technology. So how bad is the pricing? The public CLIN list is available at http://www.nmci-isf.com/clinlist.pdf in PDF format or http://www.nmci-isf.com/clinlist.htm in HTML [Note: As of mid-2006 these URLs have been pulled by EDS]. So for about $4000 a year you can get a fairly capable machine (although the software is three years out of date). So not too bad. Ummm, remember, you have to pay $4000 every year and when you're done with that PC, NMCI gets it back. Let's see. You need to change cubicles and want to plug your PC in to the existing wall jack in your new cube? That's an "Add/Change/Move" action and you have to pay NMCI for that. When commands need to pay for NMCI, there appears to be no clear-cut method for funding such a transformation. There was certainly no core funding coming directly from CNO or the upper echelons of the DoN. Even commands like NAVAIR and NAVSEA left their subordinate bases to fend for themselves. Many sites chose to put a "tax" on all base funding sources in an effort to spread the misery. So Navy facilities, who were undertaking joint work with the Air Force or the Army, had to tax their Air Force customers to pay for NMCI. It is unclear how this is being reported to the Air Force or other non-DoN activities. This is especially difficult for large projects that never pre-estimated the cost of NMCI into their joint projects and are now looking at short-falls due to the unforeseen NMCI tax. Its Ours Now So what happened to all the PCs the Navy had before NMCI came along? They belong to EDS! As part of the contract, all existing IT assets became the property of EDS. The Navy, who had acquired billions of dollars worth of computers, networking equipment and infrastructure with taxpayer money, simply gave everything to a contractor. So if a tech-savvy command had just purchased the latest "loaded" PCs with huge amounts of memory, large hard drives and state of the art processors, it didn't matter. The computers were turned over to EDS and replaced with five-year old technology and six-year old software at twice the cost. Conversely, of course, if the command had been limping along with outdated, inadequate technology, NMCI could offer something better. Although people who felt their NMCI machines were "upgrades" are few and far between, if you trust many of the surveys in publications like Federal Computer Week. It would appear that administrative and secretarial users are happier with their NMCI seats than the engineers, scientists, analysts and project managers. One could ask which segment we are really trying to support with NMCI. Stop Your Whining. What's the Solution? The proponents of NMCI can point to some successes brought about by the NMCI contract. There is no doubt they exist. Yet if one looks closely, you can see that many of these "successes" are the result of Navy employees, not EDS and certainly not anything that NMCI brought to the party. Hard working Sailors and Marines using USB memory sticks, swapping hard drives and sneaking legacy laptops into their labs so they can do the work they need to do. Why don't they just let the mission fail and have NMCI take the justified heat? Because they care more about the Navy and Marine Corps than the NMCI leadership does. EDS is not a criminal. There is nothing, as of yet, to indicate corruption within the NMCI contract. EDS made some incredibly stupid mistakes. The Navy made some incredibly stupid mistakes. Unfortunately the American Taxpayer is now left to pay for those mistakes. A solution to the NMCI debacle could only come from a clear understanding of the initial problem that NMCI was introduced to solve. The problem statement that spawned NMCI has not been clearly communicated to the rank and file Navy employees. It certainly has not been communicated to Government leaders outside of the DoN – if they even knew to ask. Intentional obfuscation or negligent research, the outcome is the same. NMCI is a solution to a problem that doesn't exist. It is now a problem in its self. Make the NMCI contract optional. Let commands prove themselves without being forced to foot the bill for some else's ideal of standardization and efficiency. The foggy misdirection created by claims of improved efficiency do nothing to point to NMCI as a solution. In fact there is no evidence that NMCI is more cost effective or more efficient than anything the Navy could have done organically. It may require Congressional pressure to find out exactly what is right and what is wrong with NMCI. We can only hope that such an inquiry would go beyond the, "Pay no attention to that man behind the curtain!" shouts from the Navy leaders. The working Sailors, Marines and civilians know better. Trackbacks
Trackback specific URI for this entry
No Trackbacks
Comments
Display comments as
(Linear | Threaded)
Well said!!! No persons in my command are happier than when we had the IT under our responsibility. If we needed something to do our jobs, it was done - period!
At times, I interface with Army & Air Force in the performance of our job. At least once, these other departments wanted to set up networks so we could all "talk with each other" in a small center (on the Army base). This task required minor access to our "NMCI" computers. NMCI refused to grant these people even temporary access to allow them to hook us up in that network. It wasn't part of the contract. Screwed us!!!! The real denial of service of attack on the Navy is NMCI. We have been running Windows 2000 machines now for over 3 years with only 256MB of memory. Not a problem if only you were on the systems doing Outlook, but NMCI has so many background tasks running that we have disk swapping all the time which brings the system to its knees. When the anti-virus starts up everyday, then go have lunch because your system because useless. Try to run more than one MS Office application is a real killer too. Now today I was the victim of one of the many hard drive failures on Dell laptops. My co-worker in the office lost his a month ago. I now have to wait at least a week to get me computer fixed even though I was suppose to already have a replace computer since it has been over 3 years. I was lucky enough to have a backup on my own external hard drive, but a backup program or adequate backup space is not provided by NMCI. People are so frustrated with NMCI. Many of us are even using our own personal laptops because it is just too hard to get our jobs down using NMCI, especially on the road. I hope someone someday has the guts to kill this contract.
I could not agree more with what you guys have said. We have to call the NMCI Help Desk on a daily basis now. They sent out an e-mail a while back that said "due to the large volume of calls, we turned off the voice mail system. If you haven't received a call back in the last 2 weeks, call back and wait on the line." The wait times were over 2 hours at that time. No one wants us to log our down time due to NMCI. The cost would be staggering! Our "leaders" are too afraid to give an honest evaluation of the system. If you complain, you're not a "team player." We are engineers and we don't have the basic tools to perform our job and interface with the war fighters. It's back to pen and paper for us. The one program we had working was trashed last week due to a service pack that NMCI pushed.
The issues concerning NMCI developer S & T seats continues to be unresolved. As a result development has experienced serious impacts. Based on e-mails received thus far it appears that current NMCI policy is that NMCI will provide no direct support if a Navy required push/upgrade/fix causes software loaded by developers on their S & T seats to stop working. In addition, if a local fix or work around is developed, NMCI will not give the developers the authorization to install the fix to correct the problem. Specifically in this case an NMCI security push in late Dec 2005 broke Visual Studio 2003 development software. NMCI says the push does not break the newly released Visual Studio 2005 tool. As a result of the fact that our development is currently VS 2003 based, all development has had to be done on legacy equipment.
If you are reading this you are not using an NMCI workstation. NMCI blocks nmcistinks.com.
I am in touch with many people where I work and I do not know anyone who is happy with NMCI. Everyone hates NMCI. Everyone.
As of July 2006, we still have Windows 2000 and Office 2000. I do not personally care what version of software I use, but Excel 2000 has problems that are fixed with the next version, but we can't get the next version, so I have a whole stack of Excel spreadsheets that are corrupted. Many hours of work down the drain every month. We also have a number of legacy applications occasionally available via Citrix. For a particular application created in MS Access, we must first run Citrix, and then run RASS, and then from RASS we can select the MS Access application. The whole login process to start that string of applications is about 5 to 10 minutes (when it works). Last week we arrived at work on Monday, and the MS Access application was nowhere to be found! Just disappeared. It took the local IT help over 7 hours to find it and get it back online. Every single time we open a new document, or do something like "Save As", we have to wait 30 seconds while a lot of traffic goes back and forth over the network and freezes the application. Every time. Our work requires informational security, and the NMCI contract did not include the consideration for this. So we are set up with a situation where we can easily, accidently copy sensitive information to a network drive or email account that is not approved for sensitive information. When a hapless employee accidently falls victim to this setup-for-failure, he is reprimanded for carelessness and written up for causing a security violation. However, the idiots who established the unsafe network are blameless. We are still connected simultaneously to NMCI and to our legacy network. We have a common network legacy hard drive that is totally full. We are regularly requested to delete unnecessary files because the hard drive is full. One day I was actually unable to save a record to my database because the network drive was "full". I had to delete a file somewhere so I could add records to my database. We could get a new hard drive for $200 at the store, but NMCI owns everything and refuses to upgrade our legacy network drive. At the same time, our NMCI common drive with billions of gigabytes of storage remains unused. Why? They won't let us have access to it because nobody can decide how to fairly divide up the hard drive space among the different departments. So it goes unused. I have been therefore working off my C: drive and backing up to the network. Even working off the C: drive requires long waiting times when opening an application or doing a number of other operations. Ain't it great. "Go ahead, drink from the cool aid!"
I was so glad to happen upon this site. I have been working in military IT for 14 years, and I have"cutover" to NMCI on four different bases now. Every time it is the same situation you described. The only reason the cutover was successful (not that NMCI is a success) was because of the very hard work from the military members who busted their butts to make the transition happen. Telling our customers that we would do our best to take care of them through the transition put us in a sticky situation. NMCI, of course, made liars out of us. It is hard for our users to understand that, they are not "our" (the unit IT guy's) customers anymore. All we can do is convince them to "drink from the cool aid". Less security / False Claims
Before NMCI we could use our CAC certificates to encrypt and sign e-mail. Along comes NMCI and we no longer have access to versions of e-mail programs capable of this. Finally, NCMI upgrades to Outlook 2003. They then take credit for providing us with improved e-mail security! As a systems and software developer for the Navy, it is impossible for us to accomplish our tasks with NMCI alone. Legacy networks (RDT&E) have been allowed with reasonable justification and have allowed us to continue.
Recently, we've received notice that the Navy has a goal of 100% elimination of RDT&E networks. They are tightening the screws, forcing us to go through major justifications for these networks and requiring that we scan and update all equipment on these networks even if they're not hooked to the outside world. To do this would likely cause huge failures of custom software and hardware solutions and cost the government billions in lost productivity and irreparable software and hardware. I can only hope that something comes to a head before they do an audit to see if we are complying with this ill-conceived and misguided attempt at standardization and security. I love how my NMCI desktop has 256mb of RAM and about 340mb allocated in private bytes - mostly from Symantec crapware bogging the computer down. It takes 20 minutes to "run startup scripts". The hard disk is not going to survive another year. THE MOUSE HAS A BALL IN IT! Of course when I suggested getting a bit more RAM and shove it in the motherboard the local NMCI rep blew up.
You all need to stop your complaining- NMCI finally gave the have not commands equality- you all probably came from the Have Commands- the Commands that Have been used to having all of the newest gear- the highest speed Internet- having the newest appications and toys- I can show you so many instances of operational units who:
Had NT Had dial up connections Had 256MHz processors Had to scrounge just to keep PC's running Had to borrow from returing MEU to be outfitted Had to ask EDS for hand outs on the turn ins from the Have Commands- Sick and tired of the have hads crying when the operational forces have gone without for so long- well guess what, now we all equally have the same shitty gear- the way it should be. Semper Fi And I could show you hundreds of commands that got sent back in time. I don't think anyone here ever questioned that there were commands that benefited from NMCI. That's not the point. What you are arguing for is essentially what the communists argued for: There is no division between rich and poor -- we are all poor. Equality is good.
More to the point. NMCI is a huge waste of money. Those commands that needed help could have received that help a thousand other ways. Just because they got some good out of NMCI doesn't mean NMCI is a good thing. We didn't need to rape the American taxpayer just to bring a couple Devil Dogs into the 21st century. Financials aside, it is a technical failure also. Read the GAO report they have posted elsewhere on here. Wow, didn't even know most of these problem existed.
I joined the Navy in 2001 and went through the NMCI transition with my first command. We were one of the "have not" commands but while the hardware was a big step up, the service was not. Only one person would be allowed to log on each day because it would take upwards of twenty minutes to get onto the network. If somehow your security certificates became corrupted, it's a huge ordeal to get a new ID card and get the certificates back on. And if you have to call NMCI, it will take at least an hour to resolve any problem. The network has gotten a lot better. It takes about a minute to log on now. But the software is terribly outdated and the limitations on hardware are really starting to make it difficult to do our jobs. In my job, we are starting to do all paperwork online. This actually makes thing a lot easier but implementing something like this means you need a reliable network. With NMCI it's hit or miss. Lastly, the IT staff is all but impotent now. They still hear all the complaints, they just can't do anything about it while the NMCI support staff sits far away in a central location, occasionally dealing with the problem but more often than not, finding it's not their 'contract'. EDS is the fraud... I'm calling my congressman.. the whole project was a failure.. -
|
CategoriesArchivesSyndicate This Blog |
