Blog Administration
Tuesday, July 19. 2005
The SPAM Debacle Appears To Be Over
"The day is ours, the bloody dog is dead." -- Mr. Shakespeare
In what can only be described as a tacit admission of utter and complete failure, NMCI administrators have disabled the tagging of spam in association with the "word lists" method described elsewhere in these pages.
You heard it here first. We told you it wouldn't work.
Not only did it not work to defeat spam, it caused horrendous disruption for thousands of Navy and Marine Corps users. Rampant false positives left a user's mailbox littered with erroneous warnings about "Sexual content", "profanity" and "Proprietary content".
The warnings were changed cosmetically early on in the debacle to include the words "May contain..." which was apparently done in an effort to make users feel better about false positives and the associated ineffective treatment of spam. Another cosmetic change was made from "sexual content" to "unauthorized content". This was not only completely useless, but was most likely technically incorrect for places like Naval hospitals where legitimate e-mails probably contain words that are both sexual in content and authorized.
One official from the East coast instructed managers to tell complaining users that, "...this is the solution to SPAM and unwanted e-mail users have been asking for." Really? We would be interested to see the requests to use such an ill-conceived method to address the spam issue.
The erroneous tagging was more than just annoying for the primary recipients. The tags, adulterating both the body of the E-mail and the subject line, had to be scrubbed before the message could be forwarded or replied to. Wasting the Government's time and resources once again, thank you NMCI.
No white-listing or "intelligent" source analysis was done on the E-mails. Simply confirming the message source (through IP address, not sender fields) as .mil or .gov addresses could have easily abated some of the mess and probably reduced loading on the systems parsing thousands of messages a day. Of course there never was any facility for the users to do custom white-lists of their own.
User's quarantined messages were to be deleted in seven days, regardless of activity from the intended recipient. So if a user was on TDY, vacation or away from an NMCI seat for more than seven days his E-mail would begin to be permanently purged having never seen the light of day. There was no provision for the user to extend this time period or disable the "feature" of quarantine all together.
We are still completely baffled why IronPort would cast its shadow on such an idiotic undertaking. SpamCop.net, Bonded Sender Program, Senderbase.org and C-Series Appliances all speak to IronPorts ability to provide very effective spam handling. What were they thinking?
In what can only be described as a tacit admission of utter and complete failure, NMCI administrators have disabled the tagging of spam in association with the "word lists" method described elsewhere in these pages.
You heard it here first. We told you it wouldn't work.
Not only did it not work to defeat spam, it caused horrendous disruption for thousands of Navy and Marine Corps users. Rampant false positives left a user's mailbox littered with erroneous warnings about "Sexual content", "profanity" and "Proprietary content".
The warnings were changed cosmetically early on in the debacle to include the words "May contain..." which was apparently done in an effort to make users feel better about false positives and the associated ineffective treatment of spam. Another cosmetic change was made from "sexual content" to "unauthorized content". This was not only completely useless, but was most likely technically incorrect for places like Naval hospitals where legitimate e-mails probably contain words that are both sexual in content and authorized.
One official from the East coast instructed managers to tell complaining users that, "...this is the solution to SPAM and unwanted e-mail users have been asking for." Really? We would be interested to see the requests to use such an ill-conceived method to address the spam issue.
The erroneous tagging was more than just annoying for the primary recipients. The tags, adulterating both the body of the E-mail and the subject line, had to be scrubbed before the message could be forwarded or replied to. Wasting the Government's time and resources once again, thank you NMCI.
No white-listing or "intelligent" source analysis was done on the E-mails. Simply confirming the message source (through IP address, not sender fields) as .mil or .gov addresses could have easily abated some of the mess and probably reduced loading on the systems parsing thousands of messages a day. Of course there never was any facility for the users to do custom white-lists of their own.
User's quarantined messages were to be deleted in seven days, regardless of activity from the intended recipient. So if a user was on TDY, vacation or away from an NMCI seat for more than seven days his E-mail would begin to be permanently purged having never seen the light of day. There was no provision for the user to extend this time period or disable the "feature" of quarantine all together.
We are still completely baffled why IronPort would cast its shadow on such an idiotic undertaking. SpamCop.net, Bonded Sender Program, Senderbase.org and C-Series Appliances all speak to IronPorts ability to provide very effective spam handling. What were they thinking?
Monday, July 4. 2005
Waging War on SPAM
In an effort to help wage the Holy War against UCE/SPAM, NMCI has announced that it will be partnering with IronPort Systems and Symantec. We applaud this effort and can only hope that NMCI will begin utilizing the excellent SpamCop real-time blacklist from IronPort Systems. NMCI press releases state that the new spam solution "will provide NMCI with advanced threat prevention, block SPAM, and enable effective DoN e-mail policy enforcement." This statement, of course, was written by a salesman or an accountant but by applying the nonsense filter, this appears to point in a hopeful direction.
One disturbing statement addresses the proposed method of identifying spam. "...the DoN and EDS have agreed to a set of business rules designed to identify specific words or phrases that are contained in SPAM messages." We can only hope that they do not actually intend to waste Government time and money on building a word/phrase list to identify SPAM. Such a sophomoric attempt at tagging spam might have worked ten years ago but "bad" word lists would be lucky to catch 5% of SPAM today. Bayesian filtering applies a statistically weighted version of a word list but needs to be customized for each application and the filter must "learn" from actual user E-mail for it to be effective. Spammers use dozens of methods to defeat simple word filters and even Bayesian filters. Certainly companies like Symantec and IronPort know this already. It is curious that NMCI would even mention such a useless task.
An intranet article referenced in the announcement explains that spam will be kept in a type of escrow account where the user will be able to review suspected spam and move it to their inbox or delete it (confirming that it is spam).
All of this, with the notable exception of word lists, sounds like an excellent but long-overdue treatment of spam on the NMCI network. Kudos to NMCI for the effort but we make the following suggestions:
1. Use real-time DNS blacklists at the mail gateway to block SMTP connections before they even connect to the server. Spamcop.net (an Ironport Systems project now) and Spamhaus.org (SBL-XBL) are two excellent choices if anyone is listening....
2. Abandon the use of "word and phrase lists" as they are completely ineffective and a waste of resources. The high number of false positives will cause user distrust and the use of pure word lists, as opposed to pattern matching, has been ineffective for the past several years. Smart matching from programs like Spam Assassin are an excellent alternative but come at the cost of processor loading.
3. Utilize a Bayesian database for each user to help build a statistical perspective that is surprisingly effective even with extensive Bayes poisoning campaigns now being waged by spammers.
One disturbing statement addresses the proposed method of identifying spam. "...the DoN and EDS have agreed to a set of business rules designed to identify specific words or phrases that are contained in SPAM messages." We can only hope that they do not actually intend to waste Government time and money on building a word/phrase list to identify SPAM. Such a sophomoric attempt at tagging spam might have worked ten years ago but "bad" word lists would be lucky to catch 5% of SPAM today. Bayesian filtering applies a statistically weighted version of a word list but needs to be customized for each application and the filter must "learn" from actual user E-mail for it to be effective. Spammers use dozens of methods to defeat simple word filters and even Bayesian filters. Certainly companies like Symantec and IronPort know this already. It is curious that NMCI would even mention such a useless task.
An intranet article referenced in the announcement explains that spam will be kept in a type of escrow account where the user will be able to review suspected spam and move it to their inbox or delete it (confirming that it is spam).
All of this, with the notable exception of word lists, sounds like an excellent but long-overdue treatment of spam on the NMCI network. Kudos to NMCI for the effort but we make the following suggestions:
1. Use real-time DNS blacklists at the mail gateway to block SMTP connections before they even connect to the server. Spamcop.net (an Ironport Systems project now) and Spamhaus.org (SBL-XBL) are two excellent choices if anyone is listening....
2. Abandon the use of "word and phrase lists" as they are completely ineffective and a waste of resources. The high number of false positives will cause user distrust and the use of pure word lists, as opposed to pattern matching, has been ineffective for the past several years. Smart matching from programs like Spam Assassin are an excellent alternative but come at the cost of processor loading.
3. Utilize a Bayesian database for each user to help build a statistical perspective that is surprisingly effective even with extensive Bayes poisoning campaigns now being waged by spammers.
(Page 1 of 1, totaling 2 entries)
