by spam? Viruses? Afraid your employees are leaking confidential data?
We tested five e-mail firewalls designed to stop the enemies at the
BY Fred Avolio
E-mail is undoubtedly the most frequently used application in your business
infrastructure. From a security prospective, it’s also the one that introduces
the most headaches:
- Everyone has access to and uses e-mail, both
“approved” applications and Web-based e-mail.
- E-mail transports
all sorts of things, including documents with executable code–programs, scripts
and macros. E-mail continues to be the number one source of virus infections in
- Tools to write and launch e-mail attacks are
easily available and commonly used by attackers.
Unfortunately, neither desktop AV nor conscientious user-awareness training
is enough to make these problems go away. If you’re looking for defense in
depth, e-mail firewalls can add an extra layer of protection at your enterprise
gateway. In addition to screening incoming and outgoing mail for malware, spam
and questionable content, some e-mail firewalls provide data integrity, as well
as authentication of the organization host.
For this article, Information Security tested five leading e-mail firewalls.
Two are appliances: BorderWare’s MXtreme and CipherTrust‘s
others are software products running on Windows 2000: Clearswift‘s MAILsweeper
4.3 for SMTP, Elron
Software‘s Message Inspector 3.14 and Tumbleweed
Communications‘ Secure Mail 5.5. With their various
form factors and functions, these tools are representative of a broad range of
products and services designed to help you secure your mail servers (see “The
E-Mail Security Market”).
Why E-Mail Firewalls?
Most enterprises rely on Internet firewalls to
handle e-mail, and most of them do a decent job. But remember the adage that
“security and complexity are inversely proportional”–the more complex a system
is, the less we can trust its security. In this context, an application-specific
firewall, especially for an application as pervasive and risky as e-mail, makes
We evaluated each firewall in our lab test according to several criteria (see
“Evaluation Criteria,” below). We assigned a letter grade for each product in
each category–except antivirus and antirelay, since all five products do this
(see comparison chart). We didn’t grade cryptography–while we think it’s
important, most customers don’t–but we did give it some consideration in our
final grade. We also gave some weight to the two appliances’ hardened OSes and
“extra” features, such as Secure Mail’s support for X.509 certificates.
We used six primary criteria1 in evaluating the
e-mail firewalls selected for testing. We looked for flexibility in controls,
security and the ability to implement things that any reasonably secure
enterprise would care about.
- Configuration and administration: User interface, features,
product design and ease of configuration.
options: Configurable actions an e-mail firewall can take based on certain
conditions or events (e.g., a prohibited file type or a suspect combination
- Antivirus: Integration with at least one leading AV
- Antirelay: Confirmation that the e-mail firewall
couldn’t act as an open relay.
- Antispam: Ability to root
out spam using keyword and phrase searches. Also, integration with antispam
blacklists and other Web-based resources.
- Content filtering:
Effectiveness at flagging confidential and/or offensive material and
finding and stopping prohibited file types. Flexibility of rules regarding
attachments–what to allow and what to deny.
1While we examined each
product’s facility with encryption and digital signatures, it wasn’t a formal
To be considered for our tests, a product had to:
- Be e-mail
agnostic. In other words, we didn’t test any products that worked with only one
or two specific e-mail systems.
- Accept and generate SMTP
- Be installable behind an enterprise Internet firewall,
though appliances MXtreme and IronMail are on hardened platforms and would work
just fine in parallel with the firewall.
- Have no reliance on PKI. Some
will work fine with public keys on digital certificates, but the technology is
still not widely enough deployed.
- Include the three “anti’s”:
antispam, antirelay controls and integration with antivirus
- Perform content filtering to protect against dissemination
of confidential company information, as well as prevent inappropriate use
of corporate e-mail.
In addition, while we were looking for products that support both encryption
and digital signing of e-mail, we accepted products that didn’t.
Each software product had to come to our lab preinstalled on a vendor’s
computer. The lab environment was configured to represent a fairly secure,
vanilla installation. A notebook PC represented someone outside the test domain
and network address space. The SMTP server was a RedHat Linux box configured to
handle e-mail for the lab domain; it acted as the inside e-mail post office. The
e-mail firewall acted as the MX server (though we hard-coded these rules into
the machines in the lab). Most of the products can act as the post office
machine, supporting incoming POP3 and IMAP requests.
Configuration & Administration
A firewall–any firewall–is a gateway
standing between the good and the bad. It has to be easy to configure so that
policies can be quickly translated into active protection.
BorderWare’s MXtreme wins high marks for easy and feature-rich configuration
and administration. It uses a secure Web-based connection from a remote browser
for configuration. Protected over an SSL-encrypted connection with a
password-protected login prompt, the configuration and administrative controls
are easy to follow and understand, even without the manual. The rack-mountable
appliance presents the user with options for basic configuration, mail delivery,
user mailboxes and management.
Configuration is typically performed just once, barring significant changes
to your network. It involves inputting your administrators’ mailboxes, the
network setup (the system comes with multiple network interfaces), IP addresses
and domains served. Mail delivery has settings typical to any SMTP
server-domains handled, mapping virtual users and domains, aliases, delivery
settings, etc. MXtreme can be used as your post office system, allowing POP3 and
IMAP connections for users to retrieve e-mail stored on the appliance.
MXtreme can also be configured as a Web e-mail proxy, front-ending an Outlook
Web Access (OWA) or Lotus Notes system. The management interface provides access
to e-mail system management functions, such as queue control and management,
summary of quarantined messages and system logs.
CipherTrust’s IronMail, the other appliance in our test, also excels in this
category. It also uses a remote browser over an encrypted connection to access
the Web-based interface. Setup is more complex than the MXtreme system–you’ll
want to have your manual in hand as you walk through it. But the complexity
comes from its diverse feature set, including security tools, such as IDS and
vulnerability scanning, which we’ll discuss later. Its interface is just as easy
to navigate as MXtreme’s. IronMail can be configured as an SMTP e-mail server,
allowing you, as with any SMTP server, to specify domains served, maximum
message size allowed, authentication used, etc. This is also where you set up
Clearswift’s MAILsweeper has an easy-to-manage configuration interface,
allowing cut-and-paste and drag-and-drop creation and management of
The interface for Clearswift’s MAILsweeper looks and works just like Windows
Explorer (see screen, above). Processes are started and stopped outside this
interface, using Windows 2000 system service controls. No surprises there.
You’ll find a folder for management and another for policy.
The administrative interface for Elron Software’s Message Inspector is
simpler than the others. Its Internet Manager program can also configure and
administer add-on products from Elron: Web Inspector and Internet Manager
Anti-Virus. You can also check system reports, configure system values and read
In terms of configuration features, Tumbleweed’s Secure Mail is the best of
the software solutions. It’s managed through a Web page over an encrypted
connection (or on the same system). The initial setup is performed through the
installation program, which steps through the basic SMTP and network
configuration. The Web interface, running on a nonstandard port, is used for
All of the products have preconfigured actions to take
based on certain conditions. We’ll not present all possibilities–we’re talking a
very large matrix here. Rather, we’ll highlight some of the more interesting
possibilities among the different products.
MAILsweeper really stands out here. Policies allow for the creation of custom
queues for message handling, quarantining and multiple actions. It will also
send a notification, complete with an attached report describing what
triggered the event and where it was triggered (in the case of attachments, for
Policy wizards–unique to MAILsweeper–walk you through the process
and display a description of the policy you’ve created. Policies are stated in
terms of classifications and scenarios. Examples of classifications are “dirty
in” or “dirty out,” where the term “dirty” describes any undesirable message. An
example of a scenario is “block executables.” Inside that scenario, you’d
find the definitions of all of the things under that category the firewall would
MXtreme is the most limited of our test products in this category. For the
most part, you specify the basic actions of permit, block and quarantine, given
certain conditions. We wanted to do more: for example, to modify headers
(Subject: MAYBE-SPAM: …), and put different disclaimers on different messages
(e.g., to permit encrypted e-mail, but stick on a disclaimer saying it wasn’t
checked for viruses). And yet, what MXtreme provides is probably sufficient for
most first-time e-mail firewall buyers. Your best bet is to compare your policy
requirements against its features.
IronMail is also a bit limited in this department, but does allow policies
based on user groups and even time of day. For example, you might want to permit
only software developers to receive attachments with executables. Specific
notations or disclaimers can be added to messages. Also, IronMail can delete
expletives from messages.
Message Inspector’s policy section (see screen, below) provides a robust and
flexible set of rule and response options. You can designate rules for what’s
permitted and denied from whom to whom, when the rule is in effect, under what
conditions and which actions to take. You can also set up groups (users,
domains, IP addresses, etc.), shifts and related filters.
Elron’s Message Inspector offers a wide range of rules-based responses to
conditions or events, defined by groups and time frame.
Secure Mail also allows you to create granular policies and actions. You can
customize “tags” to associate with messages, used in decision making as the
message is processed. You can also customize “notations,” used in notification
messages sent to the postmaster, sender or recipient.
The only complaint we have with Secure Mail is its restricted options when a
queue’s time limit has been reached. For example, if a message is in a
quarantine queue for someone to review, one option is to leave it there until
someone comes and manually dispatches it. If you set a time limit on the queue,
the message is delivered after that time limit is up. We’d like the option of
moving it to another queue for different handling.
All of the products integrate with one or more leading antivirus
solutions. Typically, when a virus is flagged, AV scanners try to “clean” a
file. If it can’t be cleaned, the default is to quarantine the attachment, with
notification sent to the sender, recipient and postmaster. We tested this simply
by attempting to send a virus-infected message through each product.
MAILsweeper is ahead of the others in terms of AV flexibility. It integrates
with seven AV solutions, while the others integrate with only one or two. This
gives you more options to stick with the AV product you use on desktops, or to
use a different AV solution, if policy or preference requires it.
BorderWare’s MXtreme’s antispam capability utilizes the RBL blacklist as well
as content and header filtering and statistical analysis.
Antirelay is a straightforward operation. All products in our
test allow a setup for domains for which the e-mail firewall would relay. All
reject the relaying of e-mail in other circumstances.
Antispam is very interesting–and very difficult. All of the
products in our test make an attempt at this. Many claim patented or
patent-pending technology. None is 100 percent effective, but they don’t have to
be 100 percent to make a difference.
Many of the vendors claim 80-90 percent spam detection with no false
positives (that is, nonspam e-mail tagged as spam). These systems, of course,
require more than simple word searches. Some use services that act as spam
targets, harvesting and cataloging spam. Some make use of blacklist
services–lists of known spam-friendly ISPs or e-mail servers with open relays.
All make use of some “dirty word” lists to try to decide whether an e-mail is
spam or just offensive. All come preconfigured with word lists.
We tested this by running sample spam messages we had received through each
Tumbleweed Communication’s Secure Mail antispam interface allows users to
configure policy by selecting keywords and phrases and assigning “weights” to
IronMail was tops here. It uses words and phrases from the headers and the
message body. In addition, it makes use of the Realtime Blackhole List (RBL), keeping track of known
spam sources; Vipul’s Razor, aka “SpamNet;” and the Distributed Checksum Clearinghouse (DCC).
Drawing on the resources of these sites gives IronMail a robust antispam
capability, compared with products that may use a single source or none at
all. MXtreme (see screen) searches the subject header and message
body for suspect words and phrases. It also queries RBL. MXtreme was the only
product we saw that has a “training mode,” which inputs data to its Statistical
Token Analyzer. In this mode, MXtreme analyzes and catalogs all processed e-mail
for a period of time, and builds up a database of “normal” e-mail.
Mailsweeper, Message Inspector and Secure Mail all examine words and phrases
in headers and the message body. They determine the “spaminess” of a message
based on weighted values (see screen). You assign certain words
numeric values, and “hits” are tallied. You then can set up actions based on hit
thresholds. For example, you might decide that any e-mail containing “approved”
or “credit” or “$10,000” might be spam. But if all are present, it’s far more
likely to be e-junkmail.
Message Inspector uses RBL as well. However, MAILsweeper’s performance in
this category came up somewhat short.
The E-Mail Security Market
Several companies offer a range of e-mail security measures–antivirus,
content filtering, spam reduction–as managed services. The customer provides the
rules, and the vendor monitors alerts and reports, manages response to events,
updates AV definitions, etc. For some organizations, outsourcing is cost
effective, efficient and takes a big load off security personnel’s shoulders.
Among these service providers are:
In addition to the products in our test bed, a number of other e-mail
firewall products are available, with comparable functionality and feature sets.
All offer, at the very least, both AV and spam filtering and are mail-server
agnostic. Other products, not represented here, offer solutions for specific
mail servers, such as Exchange, and/or offer only AV capability. A
representative list of these products includes:
*TruSecure publishes Information Security.
Content filtering, unlike spam, addresses real security
concerns, such as the unauthorized release of proprietary information and
offensive messages that may constitute sexual harassment.
Simple keyword searching does not equate to full-fledged content inspection.
If your goal is to find messages that have offensive words in them, you have a
good chance of success. But if your goal is to find messages that might be
considered offensive, the task is more difficult. Also, we know that word lists
have to be contextualized. (A medical doctor might be talking about breast
cancer, rather than telling a sex-related story.) So, it all comes down to
expectation. As with spam filtering, combinations of words can produce more
You can design your security to make content filtering more effective at
blocking proprietary information from leaking out of the organization, either
deliberately or inadvertently. Your organization should have a security policy
dealing with classifying and tagging certain information. That tagging or
marking should carry over into electronic data, where it can simplify content
filtering and make it more effective.
Finally, content screening is where we screen attachment types. Most
organizations would measurably decrease virus attacks from the Internet if they
screened executables from e-mail at the firewall or mail server1.
All the test products look inside attachments to see if they are containers
(for example, zipped files) and examine the files inside the containers.
In our lab test, we sent e-mails and attachments of various types through
each product, including:
- E-mail with a project name in it
- E-mail with classification (“Top
- Attachments with project name, classification and
- Word attachment with project name in the “Properties”
- E-mail with obscenities.
- Executable attachment in
file with .exe extension.
- Executable attachment with
extension renamed .txt.
- Executable inside a zip file.
Word document with project name.
- PGP encrypted
- Word document with imbedded objects (AVI file).
CipherTrust’s IronMail adds an extra layer of self-defense with its
signature-based denial-of-service (DoS) detection and prevention, flagging and
blocking suspect traffic.
We then set each product to look for these things and take action on them. We
rated Secure Mail and MAILsweeper about equal at the top of this category. In
addition to searching keywords and phrases in both the body of the message and
attachments, MAILsweeper also finds embedded objects in documents (e.g.,
embedded AVI in Word documents), allowing you to apply controls even deeper into
the document. It makes decisions not just on the MIME-type/ extension, but on
analysis of the data. MAILsweeper was one of the best in this category.
Moreover, MAILsweeper integrates with Clearswift’s IMAGEmanager product to
discover inappropriate images. To test IMAGEmanager, we transmitted one e-mail
with a photo of a child’s face and a separate e-mail with a photo of a naked
woman. In our very brief test, IMAGEmanager made the correct decision, blocking
the latter while permitting the former.
Secure Mail ranks with MAILsweeper in content filtering effectiveness. It
also strips illegal attachments, analyzes embedded objects and makes decisions
not just on the MIME-type/extension, but on an analysis of the data.
MXtreme looks for words or phrases in the message body and in attachments. It
also scans for malformed MIME types (some e-mail systems are vulnerable to
buffer overflows or crashes because of bad values in MIME headers). We decided
to try and put a classification, “Top Secret,” which our policy said to catch,
in the “Properties” area of a Word document. We also put a project name,
“Foxtrot,” in the body. We set the policy to trigger if both were present.
Though it detected Foxtrot, MXtreme didn’t see the Top Secret property value.
IronMail does all these same things and also missed the classification in the
“Properties.” It will also strip the “bad” attachment–the one with a virus or
the one that is not allowed because of classification or file type–and send the
e-mail on with a text attachment explaining what happened.
Also, it will replace “bad words” with other text. On reflection, we’re not
sure receiving a message that says, in part, “Want a BIG #expletive deleted#?
Expand, Lengthen And Enlarge Your #expletive deleted#” is much of an improvement
over the original. But it’s a feature, and maybe you can use it.
Message Inspector searches the whole document for content, including the
Properties area. Message Inspector’s directory structure is particularly
well-designed. It’s hierarchical, allowing you to group words according to
various degrees of crudeness, spaminess or classification level.
It Starts With Policy
Before you buy any security technology, remember that it must support your
security policy. Network firewalls help implement your security policy, and an
e-mail firewall must support your e-mail security policy. Virus scanning is
always part of such a policy. Controls applied to e-mail attachments should also
be stated. For example, does everyone in your organization need to receive
executables in e-mail from the Internet? Almost certainly not. So, your policy
would block anything that can be executed. This includes much more than the
obvious “.exe” and “.bat” file extensions.
Do you have a policy that applies security controls on sending
e-mail to your competitors? Should you? Perhaps it would be useful to
examine outgoing e-mail to make sure someone is not accidentally or
intentionally, e-mailing company secrets to competitors or friends.
Your policy may require a disclaimer on every outgoing e-mail message. Do you
use classifications on documents? Do you use project names? (Shouldn’t you?)
Content filtering can look for bad things coming in and good things going
Work out your e-mail policy, if you don’t already have one, before you try or
buy an e-mail firewall. Then see what these products can do for you.
We would love if the entire world used encrypted
e-mail, which would protect e-mail from sender to recipient until opened. We can
force protection for e-mail by setting up our e-mail firewall to require SSL or
TLS (although this doesn’t address where e-mail is most vulnerable, on the
server). Many e-mail clients and gateways support this.
What we’d really like is for e-mail firewalls to both encrypt and digitally
sign messages on behalf of the corporate office. Only one product tested, Secure
Mail, will do this. Secure Mail, along with MXtreme and IronMail, support
encrypted e-mail tunnels between servers and from clients to servers. Both do
this using the TARTTLS standard extension for Internet e-mail (in which the
e-mail client and server set up an encrypted session using the TLS protocol).
Message Inspector and MAILsweeper lack this capability.
System self-defense must be considered, if not as part of
your purchase decision, certainly as part of your deployment plan.
The non-appliances–MAILsweeper, Message Inspector and Secure Mail–all rely on
the security of the Microsoft Windows system on which they are installed. This
means you must go through the process of hardening the host (turning off Telnet,
FTP, etc.) and applying system patches before installing the firewall
software. As appliances, MXtreme and IronMail already run on hardened
systems. You needn’t worry about dangerous network services running on them.
Further, they aren’t multipurpose, multiuser boxes, as is a generic Windows
Built by the company that makes the BorderWare Firewall, MXtreme benefits
from the experience and heritage of a government- and ICSA Labs-certified
IronMail adds a lot more security. When administering the box, the first
thing you’ll notice is the Mail-IDS control. IronMail’s database of signatures
detects attempted attacks. IronMail also checks system operation and file system
integrity, using cryptographically strong checksums of key system files and
directories. It also comes with vulnerability assessment and denial-of-service
(DoS) detection tools.
While some might contend this makes the box too complex, we liked this
All five products graded out well (“B” or better overall).
While we like appliances over software solutions for the self-defense factor, we
know that a software-only solution is attractive to many since it they can be
deployed on existing–perhaps last year’s–hardware.
Antivirus is critical, and all products supported at least one well-regarded
product. Is MAILsweeper the clear winner because it integrates with more
products? It depends. If your policy dictates deploying the same AV product
across all platforms–firewall, server, and desktops–and if MAILsweeper supports
your product and the others do not, that could dictate your choice. Your policy
could just as easily specify that different products should be used on different
platforms. There are security benefits to that. Again, MAILsweeper might
win in this analysis.
We suspect we rated the need for content filtering on outbound e-mail higher
than many readers. We see the attraction for screening in-bound e-mail for
abusive language, or other language that goes against corporate policies
(security, human resources, etc.). But the potential harm to an enterprise is
greater for outbound e-mail containing proprietary information.
Although we didn’t know going in that it would come out this way, we have an
appliance (IronMail) and a software product (Secure Mail) receiving top grades,
with the final caveat that users of the Secure Mail system must remember to
place it behind a firewall, and harden the host OS.
FRED AVOLIO is
president and founder of Avolio Consulting, a Maryland-based computer and
network security consulting firm. He writes the “Just the Basics” column for