Seriously? Product Security?

Seriously? You responded to my security due diligence question with that?

Hopefully, there’s a lesson in this tale of woe about what not to do when asked about product security?

This incident has been sticking in my craw for about a year. I think it’s time to get it off my chest. If for no other reason, I want to stop thinking about this terrible customer experience. And yes, for once, I’m going to name the guilty company. I wasn’t under NDA in this situation, as far as I know?

There I was, Enterprise Security Architect for a mid-size company (who shall not be named. No gossip, ever, from this blog). Part of my job was to ensure that vendors’ product security was strong enough to protect my company’s security posture. There’s a due diligence responsibility assigned to most infosec people. In order to fulfill this responsibility, it has become a typical practice to research software vendors’ product security practices.Based upon the results, either mitigate uncovered risks to policy and industry standards or raise the risk to organizational decision makers (and there are always risks, right?).

Every software vendor goes through these due diligence investigations on a regular basis. And I do mean “every”.

I’ve lived on both sides of this fence, conducting the investigations and having my company’s software go through many investigations. This process is now a part of the fabric of doing secure business. There should be nothing surprising about the questions. In past positions, we had a vendor questionnaire, a risk scale based upon the expected responses, and standards against which to measure the vendor. These tools help to build a repeatable process. One of these processes is documented in a SANS Institute Smart Guide released in 2011 and was published by Cisco, as well.

Now, I’m going to name names. Sorry, Google, I’m going to detail just what your Docs sales team said to me. Shame on you!

When I asked about Google Docs product security here is the answer, almost verbatim, that I received from the sales team:

“We’re Google. We can hire Vint Cerf if we want. That is enough.”

Need I point out to my brilliant readers that Dr. Vint Cerf, as far as I know, has never claimed to be an information security expert? I’m sure he knows far more about the design of TCP/IP than I? (but I remind readers that I used to write TCP/IP stacks, so I’m not entirely clueless, either). And, Dr. Cerf probably knows a thing or two about Internet Security, since he runs ICANN?

Still, I can tell you authoritatively that TCP/IP security and Domain Name Registry security are only two (fairly small) areas of an information security due diligence process that is now standard for software vendors to pass.

Besides, the sales team didn’t answer my questions. This was a classic “Appeal to Authority“. And, unfortunately, they didn’t even bother to appeal to a security authority. Sorry Vint; they took your name in vain. I suppose this sort of thing happens to someone of your fame all the time?

Behind the scenes, my counter-part application architect and I immediately killed any possible engagement with Google Docs. Google Sales Team, you lost the sale through that single response. The discussion was over; the possibility of a sale was out, door firmly closed.

One of the interesting results from the wide adoption of The Web has been the need for open and transparent engagement. Organizations that engage honestly gain trust through their integrity, even in the face of organizational mis-steps and faux pas. Organizations who attempt the old fashion paradigm, “control all communications”, lose trust, and lose it rapidly and profoundly. Commercial companies, are you paying attention? This is what democracy looks like (at least in part. But that’s a different post, I think?).

Product security is difficult and demanding. There are many facets that must compliment each other to deliver acceptable risk. And, even with the best intentions and execution, one will still have unexpected vulnerabilities crop up from time to time. We only have to look at development of Microsoft’s product security programme to understand how one of the best in the industry (in my humble opinion) will not catch everything. Do Microsoft bugs surface? Yes. Is the vulnerability level today anywhere near what it was 10 years ago? Not even close. Kudos, Microsoft.

It’s long past the time that any company can coast on reputation. I believe that Google do some very interesting things towards the security of their customers. But their  sales team need to learn a few lessons in humility and transparency. Brand offers very little demonstrable protection. Google, you have to answer those due diligence questionnaires honestly and transparently. Otherwise the Infosec person on the other side has nothing against which to base her/his risk rating. And, in the face of no information, the safest bet is to rate “high risk”. Default deny rule.

It’s a big world out there and if your undiscovered vulnerabilities don’t get’cha now, they will eventually. Beware; be patient; be humble; remain inquisitive; work slowly and carefully. You can quote me on that.



What Is Our Professional Responsibility?

What Is Our Professional Responsibility?Four times within about a month, I’ve had to deal with “security issues” that were reported as “emergencies”. These appeared as high priority vulnerabilities requiring immediate “fixing” from my team.

Except, none of these were really security issues. Certainly none of these was an emergency.

None of these were bugs or vulnerabilities. In fact, if the security engineer reporting the issue had done even a modicum of investigation, these would never have been reported. False positive.

In one instance, a security engineer had browsed information on a few web pages of a SaaS application and then decided, without any further investigation that the web product had “no security”. She or he even went to far as to “ban” all corporate use of that product. Wow! That’s a pretty drastic consequence for a product who’s security controls were largely turned off by that customer’s IT department. Don’t you think the security engineer should have checked with IT first? I do.

A few days later a competitor of that product pointed a security engineer to an instance that was also configured with few security controls by the customer. The competitor claimed that the “product has no security”. The engineer promptly reported a “security deficiency”.

Obviously, a mature product should have the capability to enforce the security policies of its users, whatever those happen to be. That’s one of the most important tenants of SaaS security: give the customer sufficient tools to enact customer policies. Do not decide for the customer what their appropriate policies must be; let the customer implement policy as required.

Since the customer has the power to enact customer policies, the chosen posture may be wide open or locked down. The security posture depends upon the business needs of each particular customer. Don’t we all know that? Isn’t this obvious? (maybe not?)

In both the cases that I’ve described (and the other two), I would have thought that the engineer would first investigate?

  • What is the actual problem?
  • How does the application work?
  • What are the application’s capabilities?
  • Is there a misconfiguration?
  • Is there a functionality gap?
  • Is there a bug?

When I was a programmer, the rule was, “don’t report a bug in infrastructure, library, or compiler until you have a small working program that positively demonstrates the bug1”. In other words, we had to investigate a problem, thoroughly understand it, isolate it, and provide a working proof before we could call technical support.

Apparently, some security engineers feel no compulsion for this kind of technical precision?

I went to the CSO and asked, “If I failed to investigate a problem, would you be upset? If I did it repeatedly, would you fire me?” Answer: “Yes, on all counts.”

Security managers, what’s our accountability? Are your engineers accountable for the issues that they report?

Are we so hungry for performance metrics that we are mistakenly tracking “incidents reported”? (which is, IMHO, not a very good measure of anything). To what are we holding security investigators accountable?

My understanding of my professional ethics requires me to be as sure as I possibly can before running an issue up the flag pole. Further, I like to present all unknowns as clearly as I can. That way, false positives are minimized. I certainly wouldn’t want to stake the precious trust that I’ve carefully built up on a mere assumption which easily might be a mistake.

Of course, it’s always possible to believe one has discovered a vulnerability that turns out to be misunderstanding or misconfiguration. That can happen to any one of us. Securing multiple technologies across multiple use cases, across many technologies is difficult. Mistakes are far too easy to make. Because of the ease of error, I expect to get one or two of poorly qualified vulnerabilities each year. But four in a month? What?

Let’s all try to be precise and not get carried away in the excitement of the moment. That holds true whether one thinks one has discovered a vulnerability or is taking a bug report. I believe that information security professionals should be seen as “truth tellers”. We must live up to the trust that is placed in us.

By the way, there’s a very exciting conference upcoming at the end of August (2011), Security Architecture: Baking Security into Applications and Networks 2011. This conference is particularly relevant for Security Architects and any practitioner who must design security into systems, or who is charged with building a practice for security architecture and designs. I’ll write more about this conference later. Stay tuned.



1) The small program could not include any of the functionality required by the program that was being written. It had to demonstrate the bug without any ancillary code whatsoever. That meant that one had to understand the bug thoroughly before reporting.