As I was sitting down to write this month’s column, I received Bruce Schneier’s CRYPTO-GRAM in my e-mail. As is what you now read it is an electronic newsletter with information and opinion to educate the reader and promote and market a company. His most recent editorial was in response to an announcement from the Center for Internet Security announcing a product certification program. He doesn’t think such an endeavor is a good idea. I do. Our different opinions are probably rooted in different philosophies.

I pulled this unpublished opinion piece out to send to you this month, to present an opposing viewpoint. Even if you haven’t read Bruce’s editorial, I believe you will find this interesting. At the end of the original, I have added some more directed “rebuttals” to some of Bruce’s comments. I do this with respect, and I hope I make this clear. Bruce Schneier has written more books than I, and has contributed more to computer and network security than I. Oh, and let me also be clear on this: it is not my intention to turn this newsletter into a point/counterpoint forum… except just this once.

[Full Disclosure: ICSA Labs does product certification. In the past I have worked as a consultant for them, working on test criteria and certification descriptions. I did not develop this program with them. I was a vendor — TIS, Gauntlet — when I first was involved in product certification. I am currently working with them to develop their professional certification program.]

Computer Security Product Certification

I have the opportunity, as I teach at various computer and security venues, to discuss security product testing and certification with many different people. While most of us want to use tested and proven products, many of us are willing to use untested or uncertified products. Why? Well, the complexity of the problem is the issue: it is really difficult to test or certify products containing cryptographic systems, misuse detectors, biometric devices, or access controls. But there are ways…

There are four types of certification, and they are based on who does the certifying. They are vendor certification, market certification, user (self) certification, and independent certification.

Each of these requires:

access to the product,
test criteria,
a test environment,
expertise, and
time to test it.

Vendor Certification

Most vendors do some sort of product certification before shipping their product. Of course, we have no way of knowing whether a vendor has the expertise to produce a security product, let alone test it. But the real concern here is that a vendor is under enormous pressure – from customers, marketing, sales, investors – to ship a product, and to ship it “yesterday.” Also, the vendor has an investment in the outcome of the tests. Even a vendor with the highest integrity has to fight the subconscious push to have a product pass the vendor’s tests and start shipping.

Market certification

This simply means certification by other consumers. This is the most common type of computer software certification and often the safest. The complex product – we do this with operating systems, word processors, Internet security firewall, etc. — is used by so many people. You are not alone. We start thinking along these lines:

If there is something wrong with the product, I will not look stupid for buying it. Millions of others have also.
If there was something wrong with the product, one of those other millions of users would have already found the problem and reported it. If the problem is yet undiscovered, the probability of it biting me first is very low.

Looking at these two thoughts, we see that one, though common, has nothing to do with the way a product works, and the other is based on a false premise: that live security failures of security products will be publicized by end users.

Anyway, depending on this type of certification means we will always be waiting to use a product.

User (Self) certification

With this kind of certification, we (the users) test products before we purchase them. Simple, no? Well, no. User certification usually takes an incredibly large investment from the organization. It is time consuming and expensive.

You (the certification group) have to become expert in the area you are testing. Shouldn’t you become an expert to use a sophisticated security product, anyway? The knowledge needed to use a product safely and securely is much less than is needed to test it fully. (You have to know how to drive a car in order to test it, but the knowledge needed to safely drive and service a car is much less than is needed to certify it.)

Independent Certification

Independent certification involves an outside, vendor-independent organization that tests and certifies products. Independent certification – the one that I like and encourage – concentrates efforts, and therefore saves time and money. We users benefit from a test and certification organization’s ability to test many products from many vendors in the same way on the same test bed. Of course, vendors and consumers must trust the independent certification organization. Test criteria must be publicly examinable, for example.

There are two types of independent certification, public-sector and private-sector. The United Kingdom’s ITSEC scheme and the United States’ Common Criteria Project (CC) represent public-sector certification. ICSA is a private-sector certification group, as is West Coast.

I support and encourage private-sector certification. The reasons for this are:

The committee nature of the Common Criteria organization and control
Its size (the complete Common Criteria, as of Version 2.0, is 606 pages)
The nature of the CC itself: there is a potential for many different test criteria for the same type of product

Next Steps

Unless your company is very large and unless you find yourself with a lot of free time on the job, you will find that some sort of independent certification will do the job nicely for you. Check out the options for independent certification, read some of the test criteria, and decide which are credible and usable. You’ll then have a benchmark by which to measure products, and a tool to employ in the security product decision process.

Quotes from CRYPTO-GRAM

Quotes — yes, out of context, but the meaning’s not been touched — from Bruce Schneier’s column and my short responses:

“First, network security is too much of a moving target.”

Yes, it is. So you change the rules a bit. You have criteria that are good for a period of time … 6 months or a year … and you test against those criteria. The criteria are based on good practices. The result is a list of products that meet those criteria. Not as clean as being able to rate a safe a “TL-15” or rating a firewall — in building construction — as “2 hours”, but I think product certification is still a useful measure.

“Second, network security is much too hard to test.”

Granted, all one might be certifying is a firewall or antivirus product in one set configuration, surely matching no one individual’s set-up. Like EPA mileage numbers on automobiles (EPA is the US Environmental Protection Agency, which, among other things, publishes gasoline mileage numbers for new automiboles), your mileage may vary. But the certification is useful as a means of differentiation. And it causes vendors to make sure their products pass at least the “minimum” requirements.

“Third, I’m not sure how to make security ratings meaningful.”

To be meaningful we do not have to have hard numerical measurements. We do need test criteria and test results. Simple mechanisms — safes, fire doors — can have simple criteria. Complex mechanisms have complex, or more involved tests, and fuzzier results. Just as in security, we talk about assurances rather than “yes” or “no;” we settle for less determining, but still useful, criteria. “Pass” or “Fail” is meaningful and useful given meaningful criteria.

“Fourth, failures in network security are not always obvious.”

No argument here. For product certification, you test against products using known, good practices. When something new comes along, or you eventually find a hole in your logic, you change the criteria. It changes; products change; so retesting and re-certification is required.

“Fifth, I don’t see how a rating could take context into account.”

I do not believe it has to. Just as “your mileage may vary” doesn’t take your driving into account. The city/highway numbers in the EPA example attempt to capture this, but it still doesn’t really account for the infinite differences in driving styles, surfaces, conditions, etc. I maintain, certification does not have to be made up of numerically measurable criteria to be useful.

“Sixth, I don’t see how to combine this concept with security practices. Today the biggest problem with firewalls is not how they’re built, but how the user configures them.”

I agree that the test results — the certification — always assumes the product is configured and administered according to good industry practices. As long as this is understood, can’t we use the results? I think we can.

“And seventh, this kind of thing could easily fall into the trap of bashing small products and protecting large products.”

I recall this came up when I was a vendor and we were paying ICSA Labs money to test our firewall product. In practice, this was not a problem. It seems that if someone has the money to produce, sell, and market a product, the cost of certification is insignificant.

Bruce’s final concern is an important one to bring up. I paraphrase: the certifying or testing body will inadvertently become an extortionist. “Pay up and get certified or be branded as UNCERTIFIED.”

In practice, this has not happened. Not with ICSA Labs certification, not with Western Labs Checkmark certification, and probably will not with the Center for Internet Security. Why? I don’t know, but I have my hunches.

Links of interest:

Bruce Schneier’s latest CRYPTO-GRAM is at <http://www.counterpane.com/crypto-gram-0101.html#1>.

To subscribe to Bruce’s CRYPTO-GRAM, visit <http://www.counterpane.com/crypto-gram>.

ICSA Labs, <http://www.icsalabs.com/>

Common Criteria, <http://csrc.nist.gov/cc/info/cc-project.htm>

West Coast Labs, <http://www.check-mark.com/>

Self-promotions:

My speaking and teaching schedule can be found at <http://www.avolio.com/calendar.html>. I will continue to add to it throughout the year. I’m available to teach at your company. See my course descriptions at <http://www.avolio.com/CourseDescr.html>.

I recently posted another column called “The Ordo Cautela: Steps to Security” at <http://www.avolio.com/columns/ordocautela.html>.