Heartbleed Exposure, What Is It Really?

Heartbleed Exposure, what is it really?

“Heap allocation patterns make private key exposure unlikely” Neel Mehta, discoverer of HeartBleed” 

In the media, there’s been a lot of discussion about what might be exposed from the heartbleed OpenSSL attack. It is certainly true that very sensitive items can be exposed. And over thousands of test runs, sensitive items like private keying materials and the like have been returned by the heartbleed buffer overread.

A very strong case can be made for doing exactly as industry due diligence suggests. Teams should replace private keys on servers that had been vulnerable, once these are patched. But should every person on the Internet change every password? Let’s examine that problems by digging into the details of exactly how heartbleed works.

First, heartbleed has been characterized as an “overflow” error: “Heartbleed is basically a buffer-overflow vulnerability”. This unfortunately is a poor descriptor and somewhat inaccurate. It may make better media copy, but calling heartbleed an “overflow” is a poor technical description upon which to base a measured response.

Heartbleed is not a classic buffer overflow. No flow control or executable code may be injected via heartbleed. A read of attacker chosen memory locations is not possible, as I will explain, below. A better descriptor of heartbleed is a “buffer over-read”. Unintentionally, some data from memory is returned to the attacker. To be precise, heartbleed is a data leak, not a flow control error.

In order to understand what’s possible to disclose, it’s key to understand program “heap” memory. The heap is an area of memory that programs use to store data. Generally speaking, well-written programs (like OpenSSL) do not to put executable code into heap (that is, data) memory[1]. Because data and execution are separated, the attacker has no way through this vulnerability to execute code. And that is key, as we shall see.

As a program runs, bits of data, large and small, temporary and more or less permanent for the run, are put into the heap[2]. Typically, data are put wherever is convenient at the moment of allocation, depending upon what memory is available.

Memory that’s been deallocated gets reused. If an available piece of memory happens to be larger than a requested size, the new sized piece will be filled with the new data, while adjacent to the new data will remain bits and pieces of whatever was there previously.

In other words, while not entirely random, the heap is filled with bits and pieces of data, a little from here, a little from there, a nice big chunk from this session, with a bit left over from some other session, all helter-skelter amongst each other. The heap is a jumble; taking random bits from the heap may be considered to be like attending a jumble sale.

Now, let’s return to heartbleed. The heartbleed bug returns whatever happens to be on the heap just above the 16 bytes that are required for the TLS heartbeat packet. The attacker may request as much as 64K bytes. That’s a nice big chunk of stuff from the heap; make no mistake about it. Anything might be in there. At the very least, decrypted  data intended for application processing will be returned to the attacker[3]. That’s certainly bad! It breaks the confidentiality supposedly gained through the TLS encryption. But getting a random bit is different than requesting an arbitrary memory location at the discretion of the attacker. And that is a very important statement to hold in mind as we respond to this very serious situation.

An analogy to Heartbleed might be a bit like going fishing. Sometimes, we fish where we can clearly see the fish (mountain streams) or signs of fish (clearer lakes), or with a “fish finder” appliance, that identifies fish  under the surface when the fish aren’t visible.

Heartbleed is a lot more like fishing for fish that are deep in a turbulent lake with no fish finding capability. The fisher is guessing. If she or he guesses correctly, fish for dinner. If not, it’s a long day holding onto the fishing rod.

In the same manner, the attacker, the “fisher” as it were, doesn’t know where the “fish”, the goodies are. The bait (the heartbleed request) is cast upon the “lake” (the program heap) in the hopes that a big fish will “bite” (secret “bytes” will get returned).

The attacker can heartbleed to her or his heart’s content (pun intended). That is, if left undiscovered, an attacker can continuously pound the other side of the connection with heartbleeds, perhaps thousands of times. Which means multiple chunks of memory will be returned to the attacker, as the heap allocates, deallocates, and moves data around.

Lots of different heap chunks will get returned. There will likely also be overlap between the chunks that are returned to the attacker. Somewhere within those memory chunks are likely to be some sensitive data. If the private key for a session happens to be in one of those chunks, it will be exposed to the attacker. If any particular session open through the OpenSSL library happens to a contain a password that had been transmitted, it’s been exposed. It won’t take an engineering genius to do an ASCII dump of returned chunks of memory in order to go poking about to find interesting bits.

Still, and nonetheless, this is hunting for goodies in a bit of a haystack. Some people are quite good at that. Let’s acknowledge that outright. But that’s very different than a directed attack.

And should a wise and prepared security team, making good use of appropriate security tools, notice a heartbleed attack, they will most likely kill the connection before thousands of buffers can be read. Heartbleed over any particular connection is a linear process, one packet retrieved at a time. Retrieving lots of data takes some time. Time to respond. Of course, an unprotected and unaware site could allow many sessions to get opened by an attacker, each linearly heartbled, thus revealing far more of what’s on the heap than a single session might. Wouldn’t you notice such anomalous behaviour?

It’s important to note that the returns in the heartbleed packets are not necessarily tied to the attackers’ session. Again, it’s whatever happens to be on the heap, which will contain parts of other sessions. And any particular heartbleed packet is not necessarily connected to the data in a previous or subsequent packet. Which means that there’s no continuity of session nor any linearity between heartbleed retrievals. All session continuity must be pieced together by the attacker. That’s not rocket science. But it’s also work, perhaps significant work.

I’ll reiterate in closing, that this is a dangerous bug to which we must respond in an orderly fashion.

On the other hand, this bug does not give attackers free reign to go after all the juicy targets that may be available on any host, server, or endpoint that happens to have OpenSSL installed. Whatever happens to be on the heap of the process using the OpenSSL library and that is adjacent to the heartbeat buffer will be returned. And that attack may only occur during a TLS session. Simply including the vulnerable library poses no risk, at all. Many programs make use of OpenSSL for other functionality beyond TLS sessions.

This bug is not the unfettered keys to the kingdom, unless a “key to the kingdom” just happens to be on the heap and happens to get returned in the over-read. What gets returned is entirely due to the distribution of the heap at the moment of that particular heartbeat.

Cheers,

/brook

These assertions have been demonstrated in the lab through numerous runs of the heartbleed attack by a  team who cannot be named here. My thanks to them for confirming this assessment. Sorry for not disclosing.

[1] There are plenty of specialized cases that break this rule. But typically, code doesn’t run from the heap; data goes onto the heap. And generally speaking, programs refrain from executing on the heap because it’s a poor security practice. Let’s make that assumption about OpenSSL (and there’s nothing to indicate that this is NOT true in this case), in order to make clear what’s going on with heartbleed.

[2] The libraries that support programs developed with the major development tools and running on the major operating systems have sophisticated heap management services that are consumed by the running application as it allocates and deallocates memory. While care must be exercised in languages like C/C++, the location of where data end up on the heap is controlled by these low-level services.

[3] That is, intended for the application that is using OpenSSL for TLS services.

The “Real World” of Developer-centric Security

My friend and colleague, Dr. James Ransome, invited me late last Winter to write a chapter for his 10th book on computer security, Core Software Security(with co-author, Anmol Misra published by CRC Press. My chapter is “The SDL In The Real World”, SDL = “Secure Development Lifecycle”. The book was released December 9, 2013. You can get copies from the usual sources (no adverts here, as always).

It was an exciting process. James and I spent hours white boarding possible SDL approaches, which was very fun, indeed*. We collectively challenged ourselves to uncover current SDL assumptions, poke at the validity of these, and find better approaches, if possible.

Many of you already know that I’ve been working towards a different approach to the very difficult, multi-dimensional and multi-variate problem of designing and implementing secure software for a rather long time. Some of my earlier work has been presented to the industry on a regular basis.

Specifically, during the period of 2007-9, I talked about a new (then) approach to security verification that would be easy for developers to integrate into their workflow and which wouldn’t require a deep understanding of security vulnerabilities nor of security testing. At the time, this approach was a radical departure.

The proving ground for these ideas was my program at Cisco, Baseline Application Vulnerability Assessment, or BAVA, for short (“my” here does not exclude the many people who contributed greatly to BAVA’s structure and success. But it was more or less my idea and I was the technical leader for the program).

But, is ease and simplicity all that’s necessary? By now, many vendors have jumped on the bandwagon; BAVA’s tenets are hardly even newsworthy at this point**. Still, the dream has not been realized, as far as I can see. Vulnerability scanning still suffers from a slew of impediments from a developer’s view:

  • Results count vulnerabilities not software errors
  • Results are noisy, often many variations of a single error are reported uniquely
  • Tools are hard to set up
  • Tools require considerable tool  knowledge and experience, too much for developers’ highly over-subscribed days
  • Qualification of results requires more in-depth security knowledge than even senior developers generally have (much less an average developer)

And that’s just the tool side of the problem. What about architecture and design? What about building security in during iterative, fast paced, and fast changing agile development practices? How about continuous integration?

As I was writing my chapter, something crystalized. I named it, “developer-centric security”, which then managed to get wrapped into the press release and marketing materials of the book. Think about this:  how does the security picture change if we re-shape what we do by taking the developer’s perspective rather than a security person’s?

Developer-centric software security then reduced to single, pointed question:

What am I doing to enable developers to innovate securely while they are designing and writing software?

Software development remains a creative and innovative activity. But so often, we on the security side try to put the brakes on innovation in favour of security. Policies, standards, etc., all try to set out the rules by which software should be produced. From an innovator’s view at least some of the time, developers are iterating through solutions to a new problem while searching for the best way to solve it. How might security folk enable that process? That’s the question I started to ask myself.

Enabling creativity, thinking like a developer, while integrating into her or his workflow is the essence of developer-centric security. Trust and verify. (I think we have to get rid of that old “but”)

Like all published works, the book represents a point-in-time. My thinking has accelerated since the chapter was completed. Write me if you’re intrigued, if you’d like more about developer-centric security.

Have a great day wherever you happen to be on this spinning orb we call home, Earth.

cheers,

/brook

*Several of the intermediate diagrams boggle in complexity and their busy quality. Like much software development, we had to work iteratively. Intermediate ideas grew and shifted as we worked. a creative process?

**At the time, after hearing BAVA’s requirements, one vendor told me, “I’ll call you back next year.”. Six months later on a vendor webcast, that same vendor was extolling the very tenets that I’d given them earlier. Sea change?

RSA 2013 & Risk

Monday, February 25th, I had the opportunity to participate in the Risk Seminar at RSA 2013.

My co-presenters/panelists are pursuing a number of avenues to strengthen not only their own practices, but for all. As I believe, it turns out that plenty of us practitioners have come to understand just how difficult a risk practice is. We work with an absence of hard longitudinal data, with literally thousands of variant scenarios, generally assessing ambiguous situations. Is that hard enough?

Further complicating our picture and methods is the lack of precision when we discuss risk. We all think we know what we mean, but do we? Threats get “prioritized” into risks. Vulnerabilities, taken by themselves with no context whatsoever, are assigned “risk levels”. Vulnerability discovery tools ratings don’t help this discussion.

We all intuitively understand what risk is; we calculate risk every day. We also intuitively understand that risk has a threat component, a vulnerability component, vulnerabilities have to be exploited, threats must have have the capacity and access to exploit. There are mitigation strategies, and risk always involves some sort of  impact or loss. But expressing these relationships is difficult. We want to shorthand all that complexity into simple, easily digestible statements.

Each RSA attendee received a magazine filled with short articles culled from the publisher’s risk archives. Within the first sentences risk was equated to threat (not that the writer bothered to define “threat”), to vulnerability, to exploit. Our risk language is a jumble!

Still, we plod on, doing the best that we can with the tools at hand, mostly our experience, our ability to correlate and analyze, and any wisdom that we may have picked up along the journey.

Doug Graham from EMC is having some success through focusing on business risk (thus bypassing all those arguments about whether that Cross-site Scripting Vulnerability is really, really dangerous). He has developed risk owners throughout his sister organizations much as I’ve used an extended team of security architects successfully at a number of organizations.

Summer Fowler from Carnegie-Mellon reported on basic research being done on just how we make risk decisions and how we can make our decisions more successful and relevant.

All great stuff, I think.

And me? I presented the risk calculation methodology that Vinay Bansal and I developed at Cisco, based on previous, formative work by Catherine Blackadder Nelson and Rakesh Bharania. Hopefully, someday, the SANS Institute will be able to publish the Smart Guide that Vinay and I wrote describing the method?*

That method, named “Just Good Enough Risk Rating” (JGERR) is lightweight and repeatable process through which assessors can generate numbers that are comparable. That is one of the most difficult issues we face today; my risk rating is quite likely to be different than yours.

It’s difficult to get the assessor’s bias out of risk ratings. Part of that work, I would argue, should be done by every assessor. Instead of pretending that one can be completely rational (impossible and actually wrong-headed, since risk always involves value judgements), I call upon us all to do our personal homework by understanding our own, unique risk preferences. At least let your bias be conscious and understood.

Still, JGERR and similar methods (there are similar methods, good ones! I’m not selling anything here) help to isolate assessor bias and at the very least put some rigour into the assessment.

Thanks  to Jack Jones’ FAIR risk methodology, upon which JGERR is partly based, I believe that we can adopt a consistent risk lexicon, developing a more rigorous understand of the relationships between the various components that make up a risk calculation. Thanks, Jack. Sorry that I missed you; you were speaking at the same time that I was in the room next door to mine. Sigh.

As Chris Houlder, CISO of Autodesk, likes to say, “progress, not perfection”. Yeah, I think I see a few interesting paths to follow.

cheers

/brook

*In order to generate some demand, may I suggest that you contact SANS and tell them that you’d like the guide? The manuscript has been finished and reviewed for more than a year as of this post.

Alan Paller’s Security Architecture Call To Action

I was attending the 2nd SANS What Works in Security Architecture Summit, listening to Alan Paller, Director of Research at the SANS Institute, and Michele Guel, National Cybersecurity Award winner. Alan has given us all a call to action, namely, to build a collection of architectural solutions for security architecture practitioners. The body of work would be akin to Up To Date for medical clinicians.

I believe that creating these guides is one of the next critical steps for security architects as a profession. Why?

It’s my contention that most experienced practioners are carrying a bundle of solution patterns in their heads. Like a doctor, we can eyeball a system, view the right type of system diagram, get a few basic pieces of information, and then begin very quickly to assess which of those patterns may be applicable.

In assessing a system for security, of course, it is often like peeling an onion. More and more issues get uncovered as the details are uncovered. Still, I’ve watched experienced architects very quickly zero in to the most relevant questions. They’ve seen so many systems that they intuitively understand where to dig deeper in order to assess attack vectors and what is not important.

One of the reasons this can happen so quickly is almost entirely local to each practitioner’s environment. Architects must know their environments intimately. That local knowledge will eliminate irrelevant threats and attack patterns. Plus, the architect also knows  which  security controls are pre-existing in supporting systems. These controls have already been vetted.

Along with the local knowledge, the experienced practioner will have a sense of the organization’s risk posture and security goals. These will highlight not only threats and classes of vulnerabilities, but also the value of assets under consideration.

Less obvious, a good security architecture practitioner brings a set of practical solutions that are tested, tried, and true that can be applied to the  analysis.

As a discipline, I don’t think we’ve been particularly good at building a collective knowledge base. There is no extant body of solutions to which a practioner can turn. It’s strictly the “school of hard knocks”; one learns by doing, by making mistakes, by missing important pieces, by internalizing the local solution set, and hopefully, through peer review, if available.

But what happens when a lone practioner is confronted with a new situation or technology area with which he or she is unfamiliar? Without strong peer review and mentorship, to where does the architect turn?

Today, there is nothing at the right level.

One can research protocols, vendors, technical details. Once I’ve done my research, I usually have to infer the architectural patterns from these sources.

And, have you tried getting a vendor to convey a product’s architecture? Reviewing a vendor’s architecture is often frought with difficulties: salespeople usually don’t know, and sales engineers just want to make the product work, typically as a cookie cutter recipe. How many times have I heard, “but no other customer requires this!”

There just aren’t many documented architectural patterns available. And, of the little that is available, most has not been vetted for quality and proof.

Alan Paller is calling on us to set down our solutions. SANS’ new Smart Guide series was created for this purpose. These are intended to be short, concise security architecture solutions that can be understood quickly. These won’t be white papers nor the results of research. The solutions must demonstrate a proven track record.

There are two tasks that must be accomplished for the Smart Guide series to be successful.

  • Authors need to write down the architectures of their successes
  • The series will need reviewers to vet guides so that we can rely on the solutions

I think that the Smart Guides are one of the key steps that will help us all mature security architecture practice. From the current state of Art (often based upon personality) we will force ourselves to:

  • Be succinct
  • Be understandable to a broad range of readers and experience
  • Be methodical
  • Set down all the steps required to implement – especially considering local assumptions that may not be true for others
  • Eliminate excess

And, by vetting these guides for worthwhile content, we should begin to deliver more consistency in the practice. I also think that we may discover that there’s a good deal of consensus in what we do every day**. Smart Guides have to juried by a body of peers in order for us to trust the content.

It’s a Call to Action. Let’s build a respository of best practice. Are you willing to answer the call?

cheers

/brook

** Simply getting together with a fair sampling of people who do what I do makes me believe that there is a fair amount of consensus. But that’s entirely my anecdotal experience in the absence of any better data

SANS Security Architecture

There are plenty of security conferences, to be sure. You can get emersed in the nitty gritty of attack exploits, learn to better analyze intrusion logs, even work on being a better manager of information security folks (like me).

In the plethora of conferences trying to get your attention, there is one area that doesn’t get much space: Security Architecture.

While the Open Group has finally recognized the Security Architect discipline, conferences dedicated to the practice have been too few and too far between. The SANS Institute hosted one last year in Las Vegas. It was such a success that they’ve decided to do it again.

SANS Security Architecture: Baking Security Into Applications And Networks

This year’s conference is intended to bring security architecture to the practical. The conference will be chock full of approaches and techniques that you can use “whole hog” or cherry pick for those portions which make sense for your organization.

Why do security architects need a conference of their own?

If you ask me (which you didn’t!), security architecture is a difficult, complex practice. There are many branches of knowledge, technical, organizational, political, personal that get brought together by the successful architect. It’s frustrating; it’s tricky. It’s often more Art than engineering.

We need each other, not just to commiserate, but to support each other, to help each other take this Art and make what we do more repeatable, more sustainable, more effective. And, many times when I’ve spoken about the discipline publicly, those who are new to security architecture want to know how to learn, what to learn, how to build a programme. From my completely unscientific experience, there seems a strong need for basic information on how to get started and how to grow.

Why do organizations need Information Security Architects? Well, first off, probably not every organization does need them. But certainly if your organization’s security universe is complex, your projects equally complex, then architecture may be of great help. My friend, Srikanth Narasimhan, has a great presentation on “The Long Tail of Architecture”, explaining where the rewards are found from Enterprise Architecture.

Architecture rewards are not reaped upfront. In fact, applying system architecture principles and processes will probably slow development down. But, when a system is designed not just for the present, but the future requirements; when a system supports a larger strategy and is not entirely tactical; when careful consideration has been given to how seeming disparate systems interact and depend, as things change, that’s where architecting systems delivers it’s rewards.

A prime example of building without architecture is the Winchester Mystery House in San Jose, California, USA. It’s a fun place to visit, with its stairwells without destination, windows opening to walls, and so on. Why? The simple answer is, “no architecture”.

So if your security systems seem like a version of the “Mystery House”, perhaps you need some architecture help?

Security Architecture brings the long tail of reward through planning, strategy, holism, risk practice, “top to bottom, front to back, side to side” view of the matrix of flows and system components to build a true defense in depth. It’s become too complex to simply deploy the next shiney technology and put these in without carefully considering how each part of the defense in depth depends upon and affects, interacts, with the others.

The security architecture contains a focus that addresses how seeming disparate security technologies and processes fit into a whole. And, solutions focused security architects help to build holistic security into systems; security architecture is a fundamental domain of an Enterprise Architecture practice. I’ve come to understand that security architecture has (at least) these two foci, largely through attending last year’s conference. I wonder what I’ll gain this year?

Please join me  in Washington DC, September 28-30, 2011 for SANS Security Architecture.

cheers

/brook

 

Architecture or Engineering?

It’s not exactly the great debate. There may not be that many folks on the planet who care what we call this emerging discipline in which I’m involved? Still, those of us who are involved are trying to work through our differences. We don’t yet have a clear consensus definition. Though I would offer that a consensus on the field of security architecture is beginning to coalesce within the practitioner community. (call the discipline what-you-will)

One of the lively discussions here at the SANS What Works In Security Architecture is, “What do we call ourselves?”

Clearly, there are at least two threads. One thread (my practice) is to call what I do “Information Security Architecture”. The other thread is “Systems Security Engineer”.

I read one of the formative documents from the “Systems Security Engineer” position last month (it’s not yet announced; so I’m sure if I can say from whom, yet?)  And what was described was pretty close to what I’ve been calling “Security Architecture”.

Here’s the interesting part to me: that no matter the name, collectively, we’re realizing that we require this set of functions to be fulfilled. I’ll try to make a succinct statement about the consensus task list that I think is emerging:

  • Study of a proposed system architecture for security needs
  • Discussion with the proposers of the system (“the business” in parlance) about what the system is meant to do and the required security posture
  • Translation of required security posture into security requirements
  • Translation of the security requirements into specific security controls within a system design
  • Given an architecture, assessing the security risk based upon a strong understanding of information security threats for that organization and that type of architecture

In some organizations, the security architect task is complete at delivery of the plan. In others, the architect is expected to help design the specific controls into the system. And then, possibly, the architect is also responsible for the due diligence task of assuring that the controls were actually implemented. In my experience, I am responsible for all of these tasks and I’m called a “Security Architect”.

From the other perspective, the role is called “Systems Security Engineer”. And, it is also expected that the upfront tasks of discussion and analysis are performed by a security architect who is no longer engaged after delivery of the high level items (listed above).

That’s not how it works in my world. And, I would offer that this “leave it after analysis” model might lead to “Ivory Tower” type pronouncements. I’ll never forget one of my early big errors. I was told that “all external SSL communications were required to be bi-directionally authenticated” OK – I’d specify that for every external SSL project. Smile

Then projects started coming back to me, sometimes months later, having been asking for bi-directional authentication, and being told that on the shared java environment,”no can do”.

The way that this environment was set up, it didn’t support bi-directional authentication at an application granularity. And that was the requirement that I had given, not once but several times. I was “following the established standard”, right?

I submit to your comment and discussion that

  1. There’s more than one way to achieve similar security control (in this case, bi-directionally authenticated encrypted tunnels)
  2. It’s very important to specify requirements that can be met
  3. A security architect must understand the existing capabilities before writing requirements.

Instead of ivory tower pronouncements, if I believe that there is a technology gap, my job is to surface and make the gap as transparent as possible to the eventual owners of the system under analysis (“the business”).
How can I do this job if all I have to do is work from paper standards  and then walk away. Frankly, I’d lose all credibility. The only thing that saved me early on was my willingness to admit my errors and then pitch in wholeheartedly to solve any problems that I’d help create (or, really, anything that comes up!)

Some definitions for your consideration:

  • Architecture: the structure and organization of a computer’s hardware or system software
  • Engineering:  the discipline dealing with the art or science of applying scientific knowledge to practical problems

Engineers apply say, cryptography, an algorithm to solve problems. Architects plan that cryptography will be required for a particular set of transmissions or for storage of a set of data.

At least, these are my working definitions for the moment.

So, after listening to both sides, in my typically equivocal manner, I’m wondering if this role that I practice is really a good and full slice of architecture, combined with a good and full slice of engineering?

Practitioners, it seems to me from my experience, need both of these if they are to gain credibility with the various groups with whom they must interact. And “no”, I don’t want to perpetuate any ivory towers!

Send me your ideas for a better name, a combinatorial name. what-have-you

cheers,

/brook

from Las Vegas airport