Why no “S” in IoT?

My friend, Chris Romeo, a security architect and innovator for whom I have deep respect and from whom I learn a great deal just posted a blog about the lack of security (“S”) in Internet of Things (IoT), “The S In IoT Stands For Security“.

Of course, I agree with Chris about his three problem areas:

  1. Lack of industry knowledge
  2. Developer’s lack of security knowledge and understanding
  3. Time to market pressure, particularly as it exists for startups

I don’t think one can argue with these; in fact, these problems have been and continue to be pervasive, well beyond and before the advent of IoT. Still, I would like to add two more points to Chris’ explanation of the problem space, if I may add to what I hope will be an ongoing and lively conversation within the security industry?

First, IoT products are not just being produced at startups; big players are building IoT, too. Where we place sensors and for what purposes is a universe of different solutions, all the way from moisture sensors in soil (for farming), through networkable household gadgets like light bulbs and thermostats, to the activity monitor on a human’s wrist, to wired up athletes, to the by now famous, mirai botnet camera.

Differing use cases involve varying types and levels (postures) of security. It is a mistake to lump these all together.

When at Intel, I personally reviewed any number of IoT projects that involved significant security involvement, analysis, and implementation. I’m going to guess that most of the major activity monitoring server-side components should have had quite a lot of basic web and cloud security in-built? At least for some products, interaction between the device (IoT sensor) and any intermediary device, like a smart phone or personal computer, has also been given some significant security thought – and it turns out, that this is an area where security has been improving as device capabilities have grown.

It’s hard to make an all out statement on the state of IoT security, though I believe that there are troublesome areas, as Chris points out, most especially for startups. Another problem area continues to be medical devices, which have tended to lag badly because they’ve been produced with more open architectures, as has been presented at security conferences time and again. Then, there’s the Jeep Wrangler remote control hack.

Autonomous autos are going to push the envelop for security, since it’s a nexus between machine learning, big (Big BIG) data, consumer convenience, and physical safety. The companies that put the cars together are generally larger companies, though many of these do not lead in the cyber security space. Naturally, the technology that goes into the cars comes from a plethora of concerns, huge, mid-sized, tiny, and startup. Security consciousness and delivery will be a range from clueful to completely clueless across that range. A repeating problem area for security continues to be complex integrations where differing security postures are pasted together without sufficient thought about how interactions are changing the overall security of the integrated system.

Given the foregoing, I believe that it’s important to qualify IoT somewhat, since context, use case, and semantics/structure matter.

Next, and perhaps more important (save the most important for last?) are the economics of development and delivery. This is highlighted best by the mirai camera: it’s not just startups who have pressure. In the camera manufacturer’s case (as I understand it) they have near zero economic incentive to deliver a secure product. And this is after the mirai attack!

What’s going on here? Importantly, the camaras are still being sold. The access point with a well-known, default password is presumably still on the market, though new cameras may have been remediated. Security people may tear their hair out with an emphatic, “Don’t ever do that!” But, consider the following, please.

  • Product occurs distant (China, in this case) from the consumption of the product
  • Production and use occur within different (vastly different, in this case) legal jurisdictions
  • The incentive is to produce product as cheaply as possible
  • Eventual users do not buy “security” (whatever that means to the customer) from the manufacturer
  • The eventual user buys from manufacturer’s customer or that customer’s customer (multiple entities between consumer and producer)

Where’s the incentive to even consider security in the above picture? I don’t see it.

Instead, I see, as a Korean partner of a company that I once worked for said about acquiring a TCP/IP stack, “Get Internet for free?” – meaning of course, use an open source network stack that could be downloaded from the Internet with no cost.

The economics dictate that, manufacturer download an operating system, usually some Linux variant. Add an working driver, perhaps based upon an existing one? Maybe there’s a configuration thingy tied to a web server that perhaps, came with the Linux. Cheap. Fast. Let’s make some money.

We can try to regulate this up, down, sideways, but I do not believe that this will change much. Big companies will do the right thing, and charge more proportionately. Startups will try to work around the regulations. Companies operating under different jurisdictions or from places where adherence is not well policed or can be bribed away will continue to deliver default passwords to an open source operating system which delivers a host ripe for misuse (which is what mirai turns out to be).

Until we shift the economics of the situation, nothing will change. In the case of mirai, since the consumer of the botnet camera was likely not affected during that attack, she or he will not be applying pressure to the camera manufacturer, either. The manufacturer has little incentive to change; the consumer has little pressure to enforce via buying choices.

By the way, I don’t see consumers becoming more security knowledgeable. Concerned about digital security? Sure. Willing to change the default password on a camera or a wifi router? Many consumers don’t even go that far.

We’re in a bind here; the outlook does not look rosy to me. I open to suggestions.

Thanks, Chris, for furthering this discussion.

cheers,

/brook

“We sell Hammers” – Not Security!

“We sell Hammers” – Not Security!

Several former Home Depot employees said they were not surprised the company had been hacked. They said that over the years, when they sought new software and training, managers came back with the same response: “We sell hammers.

Failure to understand the dependence that we have not just on our obvious digital devices – smart phone, laptop, tablet, fancy fitness bling on your wrist – but also on a matrix of interconnection tying all these devices and billions more together – will land you in the hot seat. For about three billion out of the seven billion people on this planet, we have long since passed the point where we are isolated entities who act alone and in some measure of unconnected global anonymity. For most of us, our lives are not just dependent upon technology itself, but also on the capabilities of innumerable, faceless business entities acting on our behalf.

Consider the following, common, but trivial example.

When I swipe my credit card at the pump to purchase petrol, that transaction passes through any number of computation devices and applications operated by a chain of business entities. The following is a typical scenario (an example flow – but not the only one, of course):

  • The point of sale device1, itself (likely supplied by a point of sale provider)
  • The networking equipment at the station2
  • The station’s Internet provider’s equipment (networking, security, applications – you have no idea!)
  • One or more telecom company’s networking infrastructure across the Internet backbone
  • The point of sale company or their proxy
  • More networking equipment and Internet providers
  • A credit card payment processor
  • More networking equipment and Internet providers
  • The card issuer who must validate the card and agree to pay the transaction for me

And so on…. All just to fill my tank up. It’s seamless and invisible – the communications between entities usually bring up an encrypted tunnel, though the protection offered is not as solid as you may hope – Invisible and seamless, except when the processing is not so invisible, like during a compromise and breach.

Every one of these invisible players has to have good enough security to protect me, and you, if you also use some sort of payment card for your petrol.

Home Depot, and Target before them, (and who knows who’s next?) failed to understand that in order to sell a hammer in the Internet world, you’re participating in this huge web of digital interconnection. Even more so, if you’re large enough, your business network will have become an eco-system of digital entities, many of whose security practices will affect your security posture in fairly profound ways. When 2 (or more) systems connect, each may affect the security posture of the other, sometimes in profound ways.

And there be pirates in them waters, Matey. As I wrote in the introduction to my next book, Securing Systems: Applied security architecture and threat models:

“…as of this writing, we are engaged in a cyber arms race of extraordinary size, composition, complexity, and velocity.”

One of the biggest problems for security practitioners remains that the cyber “arms race” isn’t just between a couple of nation-states. Foremost, the nation-state cyber war has to cross the same digital ocean that we use for our daily lives and digital entertainment. The shared web makes every digital citizen, potential “collateral damage”. But, there are more players than governments.

As can occur in a ground war, virtual “warlords” have private cyber armies marauding for loot, my loot, your loot. Those phishing spam don’t come from your friends, right3? Just trying to categorize the various entities engaged in cyber attacks could generate a couple of fine PhD theses and perhaps even provide years of follow-on papers? The number and varying loyalties of the many players who carry out cyber attacks increases the “size” of the problem, adds to the “composition”, and generates a great deal of “complexity”. It’s enough to make a well-meaning box-store retailer bury its collective head in the virtual sand. Which is precisely what happened to that hammer seller, Home Depot.

But answering the “who” doesn’t complete the picture. There’s the macro “how”, as well4. The Internet seems to suffer the “tragedy of the commons“.

In order to keep the Internet sufficiently interesting with compelling content such that we want to participate, it absolutely must remain neutral in character5. While Internet democracy certainly appears to be quite messy, the very thing that drives the diversity of content on the Internet is its level playing field6.

But leaving the Internet as an open field for all to enjoy means that some will take advantage of the many simply because the “pickings” are too rich to ignore. There is just too much to steal to let those resources lay untouched. And the pirates don’t! People actually do answer those “Nigerian Prince” scam emails. Really, someone does. People do buy those knock-off drugs. For the 3 billion of us who are digitally connected, it’s a dangerous digital day, every single day. Watch what you click!

In short, if you’re reading this on my blog site, you are perhaps an unwitting participant in that “cyber arms race of extraordinary size, composition, complexity, and velocity.” And so is every business that employs modern digital capabilities, whether for payments, or any other task. Failure to understand just how dependent a business is upon this matrix of digital interaction will make one a Target (pun intended). CEO’s, you may want to pay closer attention? Ignoring the current realities could cost you your job, perhaps even your career7!

If you think that you only sell objects and not some level of digital security, I fear that you are likely to be very sadly mistaken?

cheers,

/brook

  1. My friend and former colleague, Lucy McCoy, wrote the communications code in the first generation of gas pump payment terminals. At that time, terminals communicated via modem and phone line. She was a serial communications wiz. I remember the point of sale terminal laid out in her lab area. Lucy has since passed away. She was a brilliant engineer; she gave my code the best quality testing ever.
  2. The transactions have to get from station to payment processing, right? Who runs those cable modems and routers at the station? Could be the Internet provider, or maybe not. I run my own modem/routers/switches at home to which I have full admin access.
  3. I don’t know any spammers, as far as I know? Perhaps I make an unwarranted assumption that you don’t, either?
  4. The “what” and “why” of cyber attack seem pretty clear. Beyond attackers after money, they are after some other advantage: geo-political, business, just causes (pick your favourite or most hated cause), career enhancement, what-have-you. This is all pretty well documented. The security industry seems preoccupied with the “what”, i.e., the technical details of exploits. Again, these technical details seem pretty well documented.
  5. Imagine if your most hated or feared government had control over your Internet use, even the Internet itself, and proceeded to feed you exactly what they wanted you to know and prevented you from any other content. How would you like that?
  6. The richness and depth that is an emergent and continuing quality of the Internet, to me, demonstrates the absolute genius of the originators and early framers of the protocols and design.
  7. Of course, if I had a severance of $15.9 million, maybe I wouldn’t very much mind ending my career?

Heartbleed Exposure, What Is It Really?

Heartbleed Exposure, what is it really?

“Heap allocation patterns make private key exposure unlikely” Neel Mehta, discoverer of HeartBleed” 

In the media, there’s been a lot of discussion about what might be exposed from the heartbleed OpenSSL attack. It is certainly true that very sensitive items can be exposed. And over thousands of test runs, sensitive items like private keying materials and the like have been returned by the heartbleed buffer overread.

A very strong case can be made for doing exactly as industry due diligence suggests. Teams should replace private keys on servers that had been vulnerable, once these are patched. But should every person on the Internet change every password? Let’s examine that problems by digging into the details of exactly how heartbleed works.

First, heartbleed has been characterized as an “overflow” error: “Heartbleed is basically a buffer-overflow vulnerability”. This unfortunately is a poor descriptor and somewhat inaccurate. It may make better media copy, but calling heartbleed an “overflow” is a poor technical description upon which to base a measured response.

Heartbleed is not a classic buffer overflow. No flow control or executable code may be injected via heartbleed. A read of attacker chosen memory locations is not possible, as I will explain, below. A better descriptor of heartbleed is a “buffer over-read”. Unintentionally, some data from memory is returned to the attacker. To be precise, heartbleed is a data leak, not a flow control error.

In order to understand what’s possible to disclose, it’s key to understand program “heap” memory. The heap is an area of memory that programs use to store data. Generally speaking, well-written programs (like OpenSSL) do not to put executable code into heap (that is, data) memory[1]. Because data and execution are separated, the attacker has no way through this vulnerability to execute code. And that is key, as we shall see.

As a program runs, bits of data, large and small, temporary and more or less permanent for the run, are put into the heap[2]. Typically, data are put wherever is convenient at the moment of allocation, depending upon what memory is available.

Memory that’s been deallocated gets reused. If an available piece of memory happens to be larger than a requested size, the new sized piece will be filled with the new data, while adjacent to the new data will remain bits and pieces of whatever was there previously.

In other words, while not entirely random, the heap is filled with bits and pieces of data, a little from here, a little from there, a nice big chunk from this session, with a bit left over from some other session, all helter-skelter amongst each other. The heap is a jumble; taking random bits from the heap may be considered to be like attending a jumble sale.

Now, let’s return to heartbleed. The heartbleed bug returns whatever happens to be on the heap just above the 16 bytes that are required for the TLS heartbeat packet. The attacker may request as much as 64K bytes. That’s a nice big chunk of stuff from the heap; make no mistake about it. Anything might be in there. At the very least, decrypted  data intended for application processing will be returned to the attacker[3]. That’s certainly bad! It breaks the confidentiality supposedly gained through the TLS encryption. But getting a random bit is different than requesting an arbitrary memory location at the discretion of the attacker. And that is a very important statement to hold in mind as we respond to this very serious situation.

An analogy to Heartbleed might be a bit like going fishing. Sometimes, we fish where we can clearly see the fish (mountain streams) or signs of fish (clearer lakes), or with a “fish finder” appliance, that identifies fish  under the surface when the fish aren’t visible.

Heartbleed is a lot more like fishing for fish that are deep in a turbulent lake with no fish finding capability. The fisher is guessing. If she or he guesses correctly, fish for dinner. If not, it’s a long day holding onto the fishing rod.

In the same manner, the attacker, the “fisher” as it were, doesn’t know where the “fish”, the goodies are. The bait (the heartbleed request) is cast upon the “lake” (the program heap) in the hopes that a big fish will “bite” (secret “bytes” will get returned).

The attacker can heartbleed to her or his heart’s content (pun intended). That is, if left undiscovered, an attacker can continuously pound the other side of the connection with heartbleeds, perhaps thousands of times. Which means multiple chunks of memory will be returned to the attacker, as the heap allocates, deallocates, and moves data around.

Lots of different heap chunks will get returned. There will likely also be overlap between the chunks that are returned to the attacker. Somewhere within those memory chunks are likely to be some sensitive data. If the private key for a session happens to be in one of those chunks, it will be exposed to the attacker. If any particular session open through the OpenSSL library happens to a contain a password that had been transmitted, it’s been exposed. It won’t take an engineering genius to do an ASCII dump of returned chunks of memory in order to go poking about to find interesting bits.

Still, and nonetheless, this is hunting for goodies in a bit of a haystack. Some people are quite good at that. Let’s acknowledge that outright. But that’s very different than a directed attack.

And should a wise and prepared security team, making good use of appropriate security tools, notice a heartbleed attack, they will most likely kill the connection before thousands of buffers can be read. Heartbleed over any particular connection is a linear process, one packet retrieved at a time. Retrieving lots of data takes some time. Time to respond. Of course, an unprotected and unaware site could allow many sessions to get opened by an attacker, each linearly heartbled, thus revealing far more of what’s on the heap than a single session might. Wouldn’t you notice such anomalous behaviour?

It’s important to note that the returns in the heartbleed packets are not necessarily tied to the attackers’ session. Again, it’s whatever happens to be on the heap, which will contain parts of other sessions. And any particular heartbleed packet is not necessarily connected to the data in a previous or subsequent packet. Which means that there’s no continuity of session nor any linearity between heartbleed retrievals. All session continuity must be pieced together by the attacker. That’s not rocket science. But it’s also work, perhaps significant work.

I’ll reiterate in closing, that this is a dangerous bug to which we must respond in an orderly fashion.

On the other hand, this bug does not give attackers free reign to go after all the juicy targets that may be available on any host, server, or endpoint that happens to have OpenSSL installed. Whatever happens to be on the heap of the process using the OpenSSL library and that is adjacent to the heartbeat buffer will be returned. And that attack may only occur during a TLS session. Simply including the vulnerable library poses no risk, at all. Many programs make use of OpenSSL for other functionality beyond TLS sessions.

This bug is not the unfettered keys to the kingdom, unless a “key to the kingdom” just happens to be on the heap and happens to get returned in the over-read. What gets returned is entirely due to the distribution of the heap at the moment of that particular heartbeat.

Cheers,

/brook

These assertions have been demonstrated in the lab through numerous runs of the heartbleed attack by a  team who cannot be named here. My thanks to them for confirming this assessment. Sorry for not disclosing.

[1] There are plenty of specialized cases that break this rule. But typically, code doesn’t run from the heap; data goes onto the heap. And generally speaking, programs refrain from executing on the heap because it’s a poor security practice. Let’s make that assumption about OpenSSL (and there’s nothing to indicate that this is NOT true in this case), in order to make clear what’s going on with heartbleed.

[2] The libraries that support programs developed with the major development tools and running on the major operating systems have sophisticated heap management services that are consumed by the running application as it allocates and deallocates memory. While care must be exercised in languages like C/C++, the location of where data end up on the heap is controlled by these low-level services.

[3] That is, intended for the application that is using OpenSSL for TLS services.

Security Testing Is Dead. Rest In Peace? Not!

Apparently, some Google presenters are claiming that we can do away with the testing cycles in our software development life cycles? There’s been plenty of reaction to Alberto Savoia’s Keynote in 2011. But I ran into this again this Spring (2013)! So, I’m sorry to bring this up again, but I want to try for a security-focused statement…

The initial security posture of a piece of software is dependent upon the security requirements for that particular piece of software or system. In fact, the organizational business model influences an organization’s security requirements, which in turn influence the kinds of testing that any particular software or system will need before and after release. We don’t sell and deliver software in a vacuum.

Google certainly present a compelling case for user-led bug hunting. Their bounty programme works! But there are important reasons that Google can pull off user-led testing for some of their applications when other businesses might die trying.

It’s important to understand Google’s business model and infrastructure in order to understand a business driven bounty programme.

  • Google’s secured application infrastructure
  • Build it and they might come
  • If they come, how to make money?
  • If Google can’t monetize, can they build user base?

First and foremost, Google’s web application execution environment has got to have a tremendous amount of security built into it. Any application deployed to that infrastructure inherits many security controls. (I’ll exclude Android and mobility, since these are radically different) Google applications don’t have to implement identity systems, authorization models, user profiles, document storage protection, and the panoply of administrative and network security systems that any commercial, industrial strength cloud must deploy and run successfully. Indeed, I’m willing to guess that each application Google deploys runs within a certain amount of sandboxed isolation such that failure of that application cannot impact the security and performance of the other applications running on the infrastructure. In past lives, this is precisely how we built large application farms: sandbox and isolation. When a vulnerable application gets exploited, other applications sharing the infrastructure cannot be touched.We also made escape from the sandbox quite  difficult in order to protect the underlying infrastructure. Google would be not only remiss, but clueless to allow buggy applications to run in any less isolating environment. And, I’ve met lots of very smart Google folk! Scary smart.

Further, from what I’ve been told, Google has long since implemented significant protections for our Google Docs. Any application that needs to store documents inherits this document storage security. (I’ve been told that Google employ some derivation of Shamir’s Threshold Scheme such that unless an attacker can obtain M of N  stored versions of a document, the attacker gains no data whatsoever. This also thwarts the privileged insider attack)

My simple point is that Google is NOT entirely relying upon its external testers, its bug bounty programme. A fair amount of security is inherent and inherited by Google’s web application experiments.

And, we must understand Google’s business model. As near as I can tell from the outside, Google throws a lot of application “spaghetti” onto the Web. If an application “sticks”, that is, attracts a user base, then Google figure out how to monetize the application. If the application can’t be monetized, Google may still support the application for marketing (popularity, brand enhancement) purposes. Applications that don’t generate interest are summarily terminated.

In my opinion, Google’s business model leaves a lot of wiggle room for buggy software. Many of these experiments have low expectations, perhaps no expectation at the outset? This gives Google time to clean the code as the application builds user base and penetration. If nobody is dependent upon an application, then there’s not a very high security posture requirement. In other words, Google can take time to find the “right product”. This is entirely opposite for security function that must deliver protection independent of any support (like on an end point that can be offline). Users expect security software to be correct on installation: the product has to be built “right”, right from the start.

And, the guts of Google are most likely protected from any nasty vulnerabilities. So, user testing makes a lot of business sense and does not pose a brand risk.

Compare this with an endpoint security product from an established and trusted brand. Not only must the software actually protect the customer’s endpoint, it’s got to work when the endpoint is not connected to anything, much less the Internet (i.e., can’t “phone home”). Additionally, security software must meet a very high standard for not degrading the posture of the target system. That is, the software shouldn’t install vulnerabilities that can be abused alongside the software’s intended functionality. Security software must meet a very high standard of security quality. That’s the nature of the business model.

I would argue that security software vendors don’t have a great deal of wiggle room for user-discovery of vulnerabilities. Neither do medical records software, nor financials. These applications must try to be as clean as possible from the get go. Imagine if your online banking site left its vulnerability discovery to the user community. I think it’s not too much of a leap to imagine the loss in customer confidence that this approach might entail?

I’ll state the obvious: different businesses demand different security postures and have different periods of grace for security bugs. Any statement that makes a claim across these differences is likely spurious.

Google, in light of these obvious differences, may I ask your pundits to speak for your own business, rather than assuming that you may speak for all business models, rather than trumpeting a “new world order”? Everyone, may I encourage us to pay attention to the assumptions inherent in claims? Not all software is created equally, and that’s a “Good Thing” ™.

By the way, Brook Schoenfield is an active Google+ user. I don’t intend to slam Google’s products in any manner. Thank you, Google, for the great software that I use every day.

cheers

/brook

 

 

 

Seriously? Product Security?

Seriously? You responded to my security due diligence question with that?

Hopefully, there’s a lesson in this tale of woe about what not to do when asked about product security?

This incident has been sticking in my craw for about a year. I think it’s time to get it off my chest. If for no other reason, I want to stop thinking about this terrible customer experience. And yes, for once, I’m going to name the guilty company. I wasn’t under NDA in this situation, as far as I know?

There I was, Enterprise Security Architect for a mid-size company (who shall not be named. No gossip, ever, from this blog). Part of my job was to ensure that vendors’ product security was strong enough to protect my company’s security posture. There’s a due diligence responsibility assigned to most infosec people. In order to fulfill this responsibility, it has become a typical practice to research software vendors’ product security practices.Based upon the results, either mitigate uncovered risks to policy and industry standards or raise the risk to organizational decision makers (and there are always risks, right?).

Every software vendor goes through these due diligence investigations on a regular basis. And I do mean “every”.

I’ve lived on both sides of this fence, conducting the investigations and having my company’s software go through many investigations. This process is now a part of the fabric of doing secure business. There should be nothing surprising about the questions. In past positions, we had a vendor questionnaire, a risk scale based upon the expected responses, and standards against which to measure the vendor. These tools help to build a repeatable process. One of these processes is documented in a SANS Institute Smart Guide released in 2011 and was published by Cisco, as well.

Now, I’m going to name names. Sorry, Google, I’m going to detail just what your Docs sales team said to me. Shame on you!

When I asked about Google Docs product security here is the answer, almost verbatim, that I received from the sales team:

“We’re Google. We can hire Vint Cerf if we want. That is enough.”

Need I point out to my brilliant readers that Dr. Vint Cerf, as far as I know, has never claimed to be an information security expert? I’m sure he knows far more about the design of TCP/IP than I? (but I remind readers that I used to write TCP/IP stacks, so I’m not entirely clueless, either). And, Dr. Cerf probably knows a thing or two about Internet Security, since he runs ICANN?

Still, I can tell you authoritatively that TCP/IP security and Domain Name Registry security are only two (fairly small) areas of an information security due diligence process that is now standard for software vendors to pass.

Besides, the sales team didn’t answer my questions. This was a classic “Appeal to Authority“. And, unfortunately, they didn’t even bother to appeal to a security authority. Sorry Vint; they took your name in vain. I suppose this sort of thing happens to someone of your fame all the time?

Behind the scenes, my counter-part application architect and I immediately killed any possible engagement with Google Docs. Google Sales Team, you lost the sale through that single response. The discussion was over; the possibility of a sale was out, door firmly closed.

One of the interesting results from the wide adoption of The Web has been the need for open and transparent engagement. Organizations that engage honestly gain trust through their integrity, even in the face of organizational mis-steps and faux pas. Organizations who attempt the old fashion paradigm, “control all communications”, lose trust, and lose it rapidly and profoundly. Commercial companies, are you paying attention? This is what democracy looks like (at least in part. But that’s a different post, I think?).

Product security is difficult and demanding. There are many facets that must compliment each other to deliver acceptable risk. And, even with the best intentions and execution, one will still have unexpected vulnerabilities crop up from time to time. We only have to look at development of Microsoft’s product security programme to understand how one of the best in the industry (in my humble opinion) will not catch everything. Do Microsoft bugs surface? Yes. Is the vulnerability level today anywhere near what it was 10 years ago? Not even close. Kudos, Microsoft.

It’s long past the time that any company can coast on reputation. I believe that Google do some very interesting things towards the security of their customers. But their  sales team need to learn a few lessons in humility and transparency. Brand offers very little demonstrable protection. Google, you have to answer those due diligence questionnaires honestly and transparently. Otherwise the Infosec person on the other side has nothing against which to base her/his risk rating. And, in the face of no information, the safest bet is to rate “high risk”. Default deny rule.

It’s a big world out there and if your undiscovered vulnerabilities don’t get’cha now, they will eventually. Beware; be patient; be humble; remain inquisitive; work slowly and carefully. You can quote me on that.

cheers,

/brook

Missed Points, Wrong Message?

Today, as every day, I was bombarded with articles to read. You, too?

Today, Testing for the Masses: More affordable assessment services reveal a new focus on application security caught my eye.

As some of my readers may know, I’ve spent a lot of time and effort on re-creating the customer web application security testing viewpoint. And, of course, Markus and I often don’t agree. So, I trundled off to read what he’d opined on the subject.

Actually, it’s great to see that ideas that were bleeding edge a few years ago are now being championed in the press!

* Build security in the design
* Code securely
* Verify both the design and the code

The “design” part of the problem requires reasonable maturity of a security architecture practice.

And, web application code must be tested and fixed before deployment, just as this article suggests. All good. If we could achieve these goals on a broad scale it would be a much better world that what we’ve got, which has been to deploy more than 100 millon lines of vulnerable code to the web (see Jeremiah Grossman’s white papers on this topic)

Further, the writers suggested that price points of web application testing tools must fall. And here’s where the ball got dropped. Why?

The question of price is related to one of the main limitations on getting at least the simple and egregious bugs out of our web code.

The tool set has until recently been made for and focused to people like Markus Ranum – security experts, penetration testing gurus. These tools have emphasized broad coverage and deep analysis. And that emphasis has been very much at the expense of:

* tool cost
* tool complexity
* noisy results that must be hand qualified by an expert

There aren’t that many Markus Ranum’s in the world. In fact, as far as I know, only one! Seriously, there are not that many experienced web application vulnerability analysts. My estimate of approximately 5000 was knocked down by some pundit (I don’t remember who?) at a SANs conference at which I was speaking. His guesstimate? 1500. Sigh. Imagine how many lines of code each has to analyze to even make a dent in those 100 million lines deployed.

Each of the major web application vulnerability scanners assumed that the user was going to become an expert user in the tool. User interfaces are complex. Setting the scope and test suite are a seriously non-trivial exercise.

And finally, there is the noise in the results – all those false positives. These tools all assume that the results will be analyzed by a security expert.

Taken together, we have what I call the “lint problem”. For those who remember programming in the C language, there is a terrifically powerful analyzer called “lint”. This tool can work through a body of code and find all kinds of additional errors that the compiler will miss. Why doesn’t everyone use lint, then? Actually, most C programmers never use lint.

Lint is too hard and resource intensive to tune. For projects that are more complex than one of two source files, the cost of tuning lint far out weighs it’s value. So, only a few programmers ever used the tool. On a 3 month project, just about the time the project is over is when lint is finally returning just the useful results. Not very timely, I’m afraid.

And so, custom web application vulnerability testing has remained in the hands of experts, people like Markus Ranum, who, indeed, makes his living by reviewing and fixing code (and rumour has it, a very good living, indeed).

So, of course, Markus probably isn’t going to suggest the one movement that will actually bring a significant shift this problem.

But I will.

The vulnerability check must be put into the hands of the developer! Radical idea?

And, that check can’t be a noisy, expert driven analysis.

* The scan must be no more difficult to run than a compiler or linker. (which are, after all, the tools in use by many web developers every day)
* The scan should integrated seamlessly into the existing workflow
* The scan must not add a significant  time to the developer’s workflow
* The results must be easy to interpret
* The results must be as accurate as a compiler. E.g., the developer must be able to trust the results with very high confidence

Attack these bugs at the source. And, to achieve that, give the developer a tool set that s/he can use and trust.

And, the price point has to be in accordance with other, similar tools.

I happen to know of several organizations that have either experimented with this concept or have put programs into place based on these principles (can’t name them, sorry. NDA) And guess what? It works!

Hey, toolmakers, the web developer market is huge! There’s money to be made here, folks.

Markus, I think you missed a key point, and are pointing in the wrong direction (slightly)

cheers

/brook

What’s Occurred in Web Years?

What’s Occurred in Web Years?

I was trying to explain to a colleague some of the sea change that has already taken place on the web (IMHO) – “Web 2.0” if you will?

I was explaining that the typical large enterprise (and really, many small ones, too) have become an intersection of clouds.

We used to use the image of the “good” inside protecting itself from the hostile (“bad”) Internet. The Internet was the cloud, the internal network a known and fixed space, “the network”. And, indeed, when I got started in Information Security, that is pretty much how things were.

But where I work, nobody knows all the networks that are interconnected. Acquisitions, test networks, global points of presence (POPs), all work to make a complex of interconnected networks. It’s too complex to hold in the mind. A mapping exercise some years ago revealed connections that nobody had documented – “discovered lands!”

In fact, the enterprise network seems a mini-internet or cloud that is probably less hostile than the Internet. But it’s certainly not trusted like the little “Class C” (anybody remember classed networks?) that I use to be responsible for where I knew everyone on the network and a lot of what they were using the network for.

But the enterprise cloud is not the only cloud at play.

Within the business eco-system are many similar interconnected network spaces of varying regulation and relationship, from close to pretty distant and untrusted. To be sure, a lot of these connect via the Internet. But certainly not all. Most enterprises have private connections with partners. The partners are literally cross-connected to the “internal” network.

There may be (should be?) network and other restrictions in place between the networks. Still, traffic flows and presumably some portion of that traffic is likely hostile (hopefully a minute portion, well monitored?)

The need to model business relationships via the network has caused an explosion of interconnected clouds.

Basically, the perimeter means much less than it once did – perhaps even “The perimeter is dead. Long live the perimeter!”

Coupled with this change has been the rapid growth of software tools on the network that model human relationship graphs and which allow a much greater degree of participation.

So, while the internal network has been growing in complexity and opening up to interconnections, usage patterns  (and business demand, it turns out) have been driving from inside to out.

These two forces have already occurred. And they happened, to my observation, faster than the growth of Web 1.0. Social networking seems to me to have taken off in months. The blogosphere has well been in place for several years. When is the last time you bought an expensive item about which you were uncertain without first checking online reviews and perhaps the blogosphere? I almost always (always?) do this before uncertain purchases.

What does this all have to do with security architecture?

These changes shift things fundamentally.

Network controls are now a tool – not the basis for one’s information security posture. We are using them around critical assets, but they no longer divide the “good” inside from the “bad” outside.

Meanwhile, data is moving out of the inner cloud and is a “must” for business agility. We can’t control our data by keeping our hot little security controls (“ACLs” – smile) around it.

The old security paradigm is obsolete. We need a new one.

And all this, in my opinion, transpired in web years while many of us were sleeping away building better network castles.

cheers,

/brook

(from the Denver airport)

Web Years

“Web Years”

I’ve been using this term lately to describe ever increasing pace of change. I started using the term as a jest, “dog years” are faster than human years. “web years” are acceleratingly shorter.

I arrogantly imagined that I’d coined the term. Not a chance, 1996.

I blog about this to introduce the term in the context of an accelerating pace of change that is fundamental to thoughts I’ve been having about Web 2.0 changes.

I want to emphasize web years in order for me to explain my thoughts about Web 2.0/Social Computing/Communication Convergence.

If web years are getting faster, then the time we have to react is getting shorter. Worse, how do we get in front of trends?

A lot of folks that I’ve been talking to about security issues of Web 2.0 are discussing the “future”.

But Web 2.0 happened (note the past tense) in web years. The original DARPANET mesh built out over a period of years. Web 1.0 in at most, a few years (I experienced the change happening in just about a year).

Social computing happened in a few months! (of course, some of the underlying tools have been around a lot longer)

Yep, accelerating.

And the change has already happened. It’s done. Organizations are catching up with plans. Some security folks are still worrying about the perfect network perimeter. LOL!

That doesn’t mean that I think organizations have implemented Web 2.0. Please don’t mistake me. I simply believe that the way people use the web has already changed. The tools are there. We’re grappling with how to make use of these changes and how to secure them.

The perimeter is gone, gone, gone. That is not to say that we should throw away our network controls. These are a part of our security toolkit, absolutely. But information security has not equaled network security for years.

I will be blogging about changes with which I think we security folk have to catch up in another edition.

take care,

/brook

The SANS Application Security Summit

Two days, about 20 speakers, all the major tool makers, pundits, researchers, penetration testers, this summit was almost a who’s who of application security. SANS wanted to kick-off their new GIAC application coding certification. And, along with that, they wanted to take a pulse of the industry. I was privileged to be included on a couple of panels – there were frighteningly smart people speaking – country mouse playing with the city cats, definitely!

A couple of my colleagues sat for the exam. I once held GIAC Intrusion Detection certification (#104?) but I let it lapse. (It’s not really relevant to what I do now) My buddies said it was a reasonable exam. The exam covered implementing security controls from the language APIs (java – JAAS calls) and coding securely.

The larger issue is what can we make out of this certification? How much effect will it have? Will getting coders certified change the landscape significantly?

There was plenty of FUD from the toolmakers. Yes, the entire web is vulnerable, apparently. Ugh! You probably knew that already, huh?

The numbers of applications out there is staggering. Estimates were running in the 100 millions. That’s a lot of vulnerable code. And, considering that my work’s DMZ takes a 6 million attack pounding every 24 hours, it’s not too far a stretch to assume that at least a few of those attacks are getting through. We only know about the incidents that are reported or that cause damage (as pointed out in the Cenzic report released at the Summit)

And, it seems like the financial incentives for exploiting vulnerabilities are maturing? (a google search will reveal dozens of financially motivated hack reports) If so, cash incentives will likely increase the number of incidents for all of us. Another big sigh.

One of the most interesting speakers to me at the conference was Dinis Cruz, CTO of Ounce Labs. A couple of times, he pointed out that while yes, we are being hacked, there’s no blood on the floor. Nobody’s been killed from a web hack (thank goodness!) We (inductries that use the web heavily) are losing money doing damage control, making up losses (especially, the financial industry), and doing remediations. Absolutely true.

But has anyone done an analysis of what the risk picture looks like? Is web exposure worth it? Are we making more than we’re losing? So much more that, like actuaries, the risks are worth it?

I know this question may seem crazy for someone from inside the information security industry to ask? But, anyone who knows me, knows that I like to ask the questions that aren’t being asked, that perhaps might even be taboo? We security folk often focus on reducing risk until we feel comfortable. That “warm and fuzzy”. Risk is “bad”, and must be reduced. Well, yeah. I fall into that trap all too often, myself.

It’s important when assessing a system to bring its risk down to the “usual and acceptable” levels of practice and custom of one’s organization. That’s at least part of my daily job.

But appropriate business risk always includes space for losses. What’s the ratio? Dinis definitely set me to considering the larger picture.

My work organization takes in more than 90% of its revenue through web sites. As long as loss is within tolerable limits, we should be ok, right? The problem with this statement comes with a few special classes of data compromise. These can’t be ignored quite so easily (or, the risk picture needs to be calculated more wholistically)

• Privacy laws are getting tougher. How does one actually inform each of 38 States Attorneys-General first? And, the Japan law makes this problem seem trivial.
• Privacy breaches are really hard on customer goodwill. Ahem
• Internet savvy folks are afraid of identity theft (again, a personally identifying information (PII) issue). The aftermath from identity theft can last for years and go way beyond the $50 credit card loss limit that makes consumer web commerce run.

I’ve got friends working for other companies that are dealing with the fall-out from these sorts of breaches. It ain’t pretty. It’s bad for organizations. The fall-out sometimes dwarfs the asset losses.

And, if I think about it, about all the vulnerabilities out there, are we each just one SQL injection exposure away from the same? Forget the laptop on the car seat. PII is sitting on organization database servers that are being queried by vulnerable web applications.

Considering this possibility brings me back to the importance of application security. Yes, I think we have to keep working on it.

And, our tool set is immature. As I see it, the industry is highly dependent upon people like Dinis Cruz to review our code and to analyze our running applications. I don’t think Dinis is a cheap date!

So, what to do?

I’ll offer what John Chambers, CEO of Cisco Systems, Inc., told me. (no, I don’t know him. I happened be in the same room and happened to have him take my question and answer it. Serendipitous circumstances). “Think architecturally”

That is, I don’t think we’re going to train and hire our way out of the vulnerability debt that we’ve dug for ourselves with 100 million apps on the web being vulnerable. Ahem. That’d take a lot of analysis. And, at the usual cost of $300/hour, who’d have the resources?

I’ll offer that we have to stop talking about and to ourselves about the problem and get it in front of all the stake holders. Who are those?

• The risk holders (i.e., executive management)
• Our developer communities (SANS certification is certainly a start. But I think that security has to be taught as a typical part of proper defensive programming. Just like structuring code or handling exceptions, input must be validated.)
• Security as an aspect of system design and architecture. Not that cute box along the side labeled “security” in the logical architecture diagram. Rather, we need to treat each security control as components of the system, just like the other logical functions.

Which brings me back to the SANS Summit.

My sense of this summit is that we haven’t yet reached the tipping point. I didn’t speak to everyone there. But I spoke to a fair share (I won’t call it a sample. My conversations were hardly statistical in nature!)

Most folks with which I chatted were like me, in the trenches. We had a few formidable notables in the room, and, of course, I think there were a few beginners, as well. My guess is that most branches of the industry were represented: tool makers, consultants, Information security folk, with a sprinkling of governmental folk thrown in. The folks that I spoke to understood the issues.

If I could hazard a guess, the folks who made the time to come are the folks concerned and/or charged with, or making money out of, application security. In other words, the industry. We were talking to ourselves.

That’s not necessarily a bad thing. I’m guessing that most of us have been pretty lonely out there? There’s validation and solidarity (leftists, please forgive me for using that term here – but it does fit!) when we get together. We network. We test our ideas and our programs. That’s important. And, I heard a few really great ideas that were fresh to me.

But I think we need to take this out a good deal further, to a “tipping point” if you will, in order to move the state-of-the-art. And, I don’t think we’re there yet. Consider, if you will, how many programmers are writing web code right now?

What do you think?

cheers

/brook

A Beginning

Sitting on a plane from Washington DC, USA, to my home in Oakland, California, I’m thinking about the SANS Application Security Summit that I just attended. What are the implications of this gathering, at this time? This seems like a propitious time to open a personal blog on information security. Some new winds may be bBrook, in the Netherlandslowing? Perhaps this summit is a the beginning of a sea change?

I’ve repeatedly thought that I’d like to share thoughts on the development of my industry. It’s exciting to me, certainly frustrating, sometimes even frightening.

Perhaps like many of you readers who work in Information Security, I spend my days  helping folks manage their digital risks? And probably, like you, I’m not always successful? Perhaps IT can’t field the technology required? Or, providing security requirements is seen by as an undue burden that cannot be borne at this time?

Still, when I understand that the stake holders feel that due diligence has been served, that an appropriate risk posture has been taken, it’s a good day. Small victories, even though our technologies are often immature or mis-applied, our processes insufficient, and our art, developing. And, of course, very occasionally, I help to identify and eventually close a major gap. Job satisfaction, absolutely.

Does any of this ring any bells or resonate for you?

Occasionally, a flash of incite will come to me. And clear as mud, I suddenly sense a possibility for us to perhaps advance our art just a wee bit. I’ll share those here for your consideration and comment. While I do occasionally publish papers and speak at conferences (as I did these last 2 days), I intend to use this forum for my tentative possibilities, not for my certainties, which are generally few, anyway.

Months will go by in the daily round of meetings, risk assessments, security requirements, system architectures, and design comments. During these periods, I may choose to be quiet, waiting for some inspiration to strike. Please stay tuned.

Perhaps you’ll appreciate knowing that I’m working through the same issues as you? Or, maybe you’ll comment that I’m way off course? I don’t know. I welcome the interchange, in any event. Through dialog, I learn as much, probably more, than I give.

I’ll write more about the SANS Summit in a subsequent entry. But, here’s a beginning…

cheers,

/brook