Category Archives: Web

Is API Authentication Enough?

Posted on by

Reading a recent article, Five Common Cloud Security Threats and Data Breaches“, posted by Andreas Dann I came across some advice for APIs that is worth exploring.

The table stakes (basic) security measures that Mr. Dann suggests are a must. I absolutely support each of the recommendations given in the article. In addition to Mr. Dann’s recommendations, I want to explore the security needs of highly exposed inputs like internet-available APIs.

Again and again, I see recommendations like Mr. Dann’s: “…all APIs must be protected by adequate means of authentication and authorization”. Furthermore, I have been to countless threat modelling sessions where the developers place great reliance on authorization, often sole reliance (unfortunately, as we shall see.)

For API’s with large open source, public access, or which involve relatively diverse customer bases, authentication and authorization will be insufficient.

As True Positives‘ AppSec Assurance Strategist, I’d be remiss to misunderstand just what protection, if any, controls like authentication and authorization provide in each particular context.

For API’s essential context can be found in something else Mr. Dann states, “APIs…act as the front door of the system, they are attacked and scanned continuously.” One of the only certainties we have in digital security is that any IP address routable from the internet will be poked and prodded by attackers. Period.

That’s because, as Mr. Dann notes, the entire internet address space is continuously scanned for weaknesses by adversaries. Many adversaries, constantly. This has been true for a very long time.

So, we authenticate to ensure that entities who have no business accessing our systems don’t. Then, we authorize to ensure that entities can only perform those actions that they should and cannot perform actions which they mustn’t. That should do it, right?

Only if you can convince yourself that your authorized user base doesn’t include adversaries. There lies “the rub”, as it were. Only systems that can verify the trust level of authorized entities have that luxury. There actually aren’t very many cases involving relatively assured user trust.

As I explain in Securing Systems, Secrets Of A Cyber Security Architect, and Building In Security At Agile Speed, many common user bases will include attackers. That means that we must go further than authentication and authorization, as Mr. Dann implies with, “…the API of each system must follow established security guidelines”. Mr. Dann does not expand on his statement, so I will attempt to provide some detail.

How do attackers gain access?

For social media and fremium APIs (and services, for that matter), accounts are given away to any legal email address. One has only to glance at the number of cats, dogs, and birds who have Facebook accounts to see how trivial it is to get an account. I’ve never met a dog who actually uses, or is even interested in Facebook. Maybe there’s a (brilliant?) parrot somewhere? (I somehow doubt it.)

Retail browsing is essentially the same. A shop needs customers; all parties are invited. Even if a credit card is required for opening an account, which used to be the norm, but isn’t so much anymore, stolen credit cards are cheaply and readily available for account creation.

Businesses which require payment still don’t get a pass. The following scenario is from a real-world system for which I was responsible.

Customers paid for cloud processing. There were millions of customers, among which were numerous nation-state governments whose processing often included sensitive data. It would be wrong to assume that nation-states with spy agencies could not afford to purchase a processing account in order to poke and prod just in case a bug should allow that state to get at the sensitive data items of one of its adversaries. I believe (strongly) that it is a mistake to assume that well-funded adversaries won’t simply purchase a license or subscription if the resulting payoff is sufficient. spy agencies often work very slowly and tenaciously, often waiting years for an opportunity.

Hence, we must assume that our adversaries are authenticated and authorized. That should be enough, right? Authorization should prevent misuse. But does it?

All software fails. Rule #1, including our hard won and carefully built authorization schemes.

“[E]stablished security guidelines” should expand to a robust and rigorous Security Development Lifecyle (SDL or S-SDLC), as James Ransome and I lay out in Building In Security At Agile Speed. There’s a lot of work involved in “established security”, or perhaps better termed, “consensus, industry standard software security”. Among software security experts, there is a tremendous amount of agreement in what “robust and rigorous” software security entails.

(Please see NIST’s Secure Software Development Framework (SSDF) which is quite close to and validates the industry standard SDL in our latest book. I’ll note that our SDL includes operational security. I would guess that Mr. Dann was including typical administrative and operational controls.)

Most importantly, for exposed interfaces, I continue to recommend regular, or better, continual fuzz testing.

The art of software is constraining software behaviour to the desired and expected. Any unexpected behaviour is probably unintended and thus, will be considered a bug. Fuzzing is an excellent way to find unintended programme behaviour. Fuzzing tools, both open source and commercial, have gained maturity while at the same time, commercial tools’ price has been decreasing. The barriers to fuzzing are finally disappearing.

Rest assured that you probably have authorized adversaries amongst your users. That doesn’t imply that we abandon authentication and authorization. These controls continue to provide useful functions, if only to tie bad behaviour to entities.

Along with authentication and authorization, we must monitor for anomalous and abusive behavour for the inevitable failures of our protective software. At the same time, we must do all we can to remove issues before release, while at the same time, using techniques like fuzzing to identify those unknown, unknowns that will leak through.
 

Cheers,

/brook

Continuing Commentary on NIST AppSec Guidance

Posted on by

Continuing with a commentary on NIST #appsecurity guidance, again, NIST reiterates what we recommended in Core Software Security and then updated (for Agile, CI/CD, DevOps) in Building In Security At Agile Speed:

 

Threat Model

Automate as much as possible

Static analysis (or equivalent) SAST/IAST

DAST (web app/api scan)

Functional tests

Negative testing

Fuzz

Check included 3rd party software

 

NIST guidance call’s out heuristic hard coded secrets discovery. Definitely a good idea. I maintain that it is best to design in protections for secrets through threat modelling. Secrets should never be hardcoded without a deep and rigorous risk analysis. (there are a few cases I’ve encountered where there was no better alternative. these are few and very far between.)

NIST guidance is essentially the same as our Industry Standard, Generic Security Development Life cycle (SDL).

There has been an implicit industry consensus on what constitutes effective software security for quite a while. Let’s stop arguing about it, ok? Call the activities whatever you like. but do them.

Especially, fuzz all inputs that will be exposed to heavy use (authenticated or not!) I’ve been saying this for years. I hope that fuzzing tools are finally coming up to the our needs?

https://www.nist.gov/itl/executive-order-improving-nations-cybersecurity/recommended-minimum-standard-vendor-or-developer

Security Research Is Threat Modeling’s Constructive Criticism

Posted on by

(the following is partially excerpted from my next book)

Adam Shostack, author of Threat Modeling: Designing for Security has succinctly described a process of constructive critique within engineering:

“The back and forth of design and critique is not only a critical part of how an individual design gets better, but fields in which such criticism is the norm advance faster.”

The Spectre/Meltdown issues are the result of a design critique such as Shostack describes in his pithy quote given above.  In fact, one of the fundamental functions that I believe security research can play is providing designers with a constructive critique.

Using Spectre and Meltdown as an example of engineering critique, let’s look at some of the headlines from before the official issue announcement by the researchers:

“Kernel-memory-leaking Intel processor design flaw forces Linux, Windows redesign”, John Leyden and Chris Williams 2 Jan 2018 at 19:29, The Register, https://www.theregister.co.uk/2018/01/02/intel_cpu_design_flaw/

“A CRITICAL INTEL FLAW BREAKS BASIC SECURITY FOR MOST COMPUTERS”, Andy Greenberg, January 3, 2018, Wired Magazine, https://www.wired.com/story/critical-intel-flaw-breaks-basic-security-for-most-computers/

There were dozens of similar headlines (many merely repeating the first few, especially, The Register’s), all declaiming a “flaw” in CPUs. I want to draw the reader’s attention to the word, “flaw”. Are these issues “flaws”? Or, are they the result of something else?

The Register headline and that first article were based upon speculation that had been occurring amongst the open source community supporting the Linux Kernel. A couple of apparently odd changes that had been made to kernel code. But, the issues to which these changes responded were “embargoed”, that is, the reasoning behind the changes was known only to those making the changes.

Unlike typical open source changes, whose reasoning is public and often discussed by members of the community, these kernel changes had been made in opaquely without public comment, which of course, set concerned kernel community members wondering.

To observers not familiar with the reasoning behind the changes, it was clear that something was amiss and likely in relation to CPU functions; anxious observers were guessing what might be the motivation for those code changes.

Within the digital security universe, there exists an important dialog between security research aimed at discovering new attack techniques and the designers of the systems and protocols upon which that research is carried out. As Adam noted so very wryly, achieving solid designs, even great ones, and most importantly, resilient designs in the face of omnipresent attack requires an interchange of constructive critique. That is how Spectre and Meltdown were discovered and presented.

Neither of this collection of (at the time of announcement) new techniques involved exercising a flaw, that is, a design error – in other words, the headlines quoted just above were erroneous and rather misleading[2].

Speculative execution and the use of kernel mapped user memory pages by operating systems were intentional design choices that had been working as designed for more than 10 years. Taken together, at least some of the increases in CPU performance over that period can directly be tied to speculative execution design.

Furthermore, and quite importantly to this discussion, these design choices were made within the context of a rather different threat landscape. Some of today’s very active threat actors didn’t exist, or at least, were not nearly as active and certainly not as technically sophisticated circa 2005 as they are today, May 2018.

If I recall correctly (and I should be able to remember, since I was the technical lead for Cisco’s Web infrastructure and application security team at that time), in 2005, network attacks were being eclipsed by application focused attack methods, especially, web attack methods.

Today, web attacks are very “ho, hum”, very run of the ordinary, garden variety. But in 2005, when the first CPU speculative execution pipelines were being released, web applications were targets of choice at the cutting edge of digital security. Endpoint worms and gaining entrance through poor network ingress controls had been security’s focus up until the web application attack boom (if I may title it so?). At about that time, targeting web applications was fast displacing network periphery concerns. Attackers were in the process of shifting to targets that used a standard protocol (HTTP) which was guaranteed to pass through organizational firewalls. Many of the new targets’ were becoming always  available via the Public Internet.

Since the web application attack boom, attacks and targets have continued to evolve. The threat landscape changed dramatically over the years since the initial design of speculative execution CPUs. Alongside the changes in types of attackers as well as their targets, attacker and researcher sophistication has grown, as has an attackers’ toolbox. 2018 is a different security world than 2005. I see no end to this curve of technical growth in my crystal ball.

The problem is, when threat modeling, whether in 2005 or 2018, one considers the attacks of the past, those of moment, and then one must try one’s best to project from current understanding to attacks that might arise within the foreseeable future. Ten or twelve years seems an awfully long horizon of prescience, especially when considering the rate at which technical change continues to take place.

As new research begins to chew at the edges of any design, I believe that the wise and diligent practitioner revisits their existing threat models in light of developments.

If I were to fault the CPU and operating system makers whose products are subject to Spectre or Meltdown, it would be for a failure to anticipate where research might lead, as research has unfolded. CPU threat modelers could have taken into account advances in research indicating unexpected uses of cache memory.

Speculative execution leaves remnants of a speculated execution branch in cache memory when a branch has not been taken. It is those remnants that lie at the heart of this line of research.

A close examination of the unfolding research might very well have led those responsible for updating CPU threat models to consider the potential for something like Spectre and Meltdown. (Or, perhaps the threat models were updated, but other challenges prevented updates to CPU designs? CPU threat modelers, please tell us all what the real story is)

I’ve found a publication chain during the 3 years previous that, to me, points towards the new techniques. Spectre and Meltdown are not stand-alone discoveries, but lie on a body of CPU research that had been published regularly for several years.

As I wrote for McAfee’s Security Matters blog in January of 2018 (as a member of McAfee’s Advanced Threat Research Team),

“Meltdown and Spectre are new techniques that build upon previous work, such as “KASLR”  and other papers that discuss practical side-channel attacks. The current disclosures build upon such side-channels attacks through the innovative use of speculative execution….An earlier example of side-channel based upon memory caches was posted to Github in 2016 by one of the Spectre/Meltdown researchers, Daniel Gruss.” Daniel Gruss is one of the Spectre and Meltdown paper authors.

Reading these earlier papers, it appears to me that some of the parent techniques that would be used for the Spectre and Meltdown breakthroughs could have been read (should have been read?) by CPU security architects in order to re-evaluate the CPU’s threat model. That previously published research was most certainly available.

Of course, hindsight is always 20/20; I had the Spectre and Meltdown papers in hand as I reviewed previous research. Going the other way might be more difficult?

Spectre and Meltdown did not just spring miraculously from the head of Zeus, as it were. They are the results of a fairly long and concerted effort to discover problems with and thus, hopefully, improve the designs of modern processors. Indeed, the researchers engaged in responsible disclosure, not wishing to publish until fixes could be made available.

To complete our story, the driver that tipped the researchers to an early, zero-day disclosure (that is, disclosure without available mitigations or repairs) were the numerous speculative (if you’ve forgive the pun?) journalism (headlines quoted above) that gained traction based upon misleading (at best) or wrong conclusions. Claiming a major design “flaw” in millions of processors is certainly a reader catching headline. But, unfortunately, these claims were vastly off the mark since no flaw existed in the CPU or operating system designs.

While it may be more “interesting” to imagine a multi-year conspiracy to cover up known design issues by evil CPU makers, no such conspiracy appears to have taken place.

Rather, in the spirit of responsible disclosure, the researchers were waiting for mitigations to be made available to customers; CPU manufacturers and operating system coders were heads down at work figuring out what appropriate mitigations might be, and just how to implement these with the least amount of disruption. None of these parties was publicly discussing just why changes were being made, especially to the open source Linux kernel.

Which is precisely what one would expect in order to protect millions of CPU users: embargo the technical details to foil attackers. There is actually nothing unusual about such a process; it’s all very normal and typical, and unfortunately for news media, quite banal[3].

What we see through the foregoing example about Spectre and Meltdown is precisely the sort of rich dialog that should occur between designers and critics (researchers, in this case).

Designs are built against the backdrop and within the context of their security “moment”. Our designs cannot improve without collective critique amongst the designers; such dialog internal to an organization or at least, a development team is essential. I have spoken about this process repeatedly at conferences: “It takes a village to threat model,” (to misquote a famous USA politician.)

But, there’s another level, if you will, that can reach for a greater constructive critique.

Once a design is made available to independent critics, that is, security researchers, research discoveries can and I believe, should become part of an ongoing re-evaluation of the threat model, that is, the security of the design. In this way, we can, as an industry, reach for the constructive critique called for by Adam Shostack.

[1]In my humble experience, Adam is particularly good at expressing complex processes briefly and clearly. One of his many gifts as a technologist and leader in the security architecture space.

[2]Though salacious headlines apparently increase readership and thus advertising revenue. Hence, the misleading but emotion plucking headlines.

[3]Disclosure: I’ve been involved in numerous embargoed issues over the years.

Why no “S” in IoT?

Posted on by

My friend, Chris Romeo, a security architect and innovator for whom I have deep respect and from whom I learn a great deal just posted a blog about the lack of security (“S”) in Internet of Things (IoT), “The S In IoT Stands For Security“.

Of course, I agree with Chris about his three problem areas:

  1. Lack of industry knowledge
  2. Developer’s lack of security knowledge and understanding
  3. Time to market pressure, particularly as it exists for startups

I don’t think one can argue with these; in fact, these problems have been and continue to be pervasive, well beyond and before the advent of IoT. Still, I would like to add two more points to Chris’ explanation of the problem space, if I may add to what I hope will be an ongoing and lively conversation within the security industry?

First, IoT products are not just being produced at startups; big players are building IoT, too. Where we place sensors and for what purposes is a universe of different solutions, all the way from moisture sensors in soil (for farming), through networkable household gadgets like light bulbs and thermostats, to the activity monitor on a human’s wrist, to wired up athletes, to the by now famous, mirai botnet camera.

Differing use cases involve varying types and levels (postures) of security. It is a mistake to lump these all together.

When at Intel, I personally reviewed any number of IoT projects that involved significant security involvement, analysis, and implementation. I’m going to guess that most of the major activity monitoring server-side components should have had quite a lot of basic web and cloud security in-built? At least for some products, interaction between the device (IoT sensor) and any intermediary device, like a smart phone or personal computer, has also been given some significant security thought – and it turns out, that this is an area where security has been improving as device capabilities have grown.

It’s hard to make an all out statement on the state of IoT security, though I believe that there are troublesome areas, as Chris points out, most especially for startups. Another problem area continues to be medical devices, which have tended to lag badly because they’ve been produced with more open architectures, as has been presented at security conferences time and again. Then, there’s the Jeep Wrangler remote control hack.

Autonomous autos are going to push the envelop for security, since it’s a nexus between machine learning, big (Big BIG) data, consumer convenience, and physical safety. The companies that put the cars together are generally larger companies, though many of these do not lead in the cyber security space. Naturally, the technology that goes into the cars comes from a plethora of concerns, huge, mid-sized, tiny, and startup. Security consciousness and delivery will be a range from clueful to completely clueless across that range. A repeating problem area for security continues to be complex integrations where differing security postures are pasted together without sufficient thought about how interactions are changing the overall security of the integrated system.

Given the foregoing, I believe that it’s important to qualify IoT somewhat, since context, use case, and semantics/structure matter.

Next, and perhaps more important (save the most important for last?) are the economics of development and delivery. This is highlighted best by the mirai camera: it’s not just startups who have pressure. In the camera manufacturer’s case (as I understand it) they have near zero economic incentive to deliver a secure product. And this is after the mirai attack!

What’s going on here? Importantly, the camaras are still being sold. The access point with a well-known, default password is presumably still on the market, though new cameras may have been remediated. Security people may tear their hair out with an emphatic, “Don’t ever do that!” But, consider the following, please.

  • Product occurs distant (China, in this case) from the consumption of the product
  • Production and use occur within different (vastly different, in this case) legal jurisdictions
  • The incentive is to produce product as cheaply as possible
  • Eventual users do not buy “security” (whatever that means to the customer) from the manufacturer
  • The eventual user buys from manufacturer’s customer or that customer’s customer (multiple entities between consumer and producer)

Where’s the incentive to even consider security in the above picture? I don’t see it.

Instead, I see, as a Korean partner of a company that I once worked for said about acquiring a TCP/IP stack, “Get Internet for free?” – meaning of course, use an open source network stack that could be downloaded from the Internet with no cost.

The economics dictate that, manufacturer download an operating system, usually some Linux variant. Add an working driver, perhaps based upon an existing one? Maybe there’s a configuration thingy tied to a web server that perhaps, came with the Linux. Cheap. Fast. Let’s make some money.

We can try to regulate this up, down, sideways, but I do not believe that this will change much. Big companies will do the right thing, and charge more proportionately. Startups will try to work around the regulations. Companies operating under different jurisdictions or from places where adherence is not well policed or can be bribed away will continue to deliver default passwords to an open source operating system which delivers a host ripe for misuse (which is what mirai turns out to be).

Until we shift the economics of the situation, nothing will change. In the case of mirai, since the consumer of the botnet camera was likely not affected during that attack, she or he will not be applying pressure to the camera manufacturer, either. The manufacturer has little incentive to change; the consumer has little pressure to enforce via buying choices.

By the way, I don’t see consumers becoming more security knowledgeable. Concerned about digital security? Sure. Willing to change the default password on a camera or a wifi router? Many consumers don’t even go that far.

We’re in a bind here; the outlook does not look rosy to me. I open to suggestions.

Thanks, Chris, for furthering this discussion.

cheers,

/brook

“We sell Hammers” – Not Security!

Posted on by

“We sell Hammers” – Not Security!

“Several former Home Depot employees said they were not surprised the company had been hacked. They said that over the years, when they sought new software and training, managers came back with the same response: “We sell hammers.””

Failure to understand the dependence that we have not just on our obvious digital devices – smart phone, laptop, tablet, fancy fitness bling on your wrist – but also on a matrix of interconnection tying all these devices and billions more together – will land you in the hot seat. For about three billion out of the seven billion people on this planet, we have long since passed the point where we are isolated entities who act alone and in some measure of unconnected global anonymity. For most of us, our lives are not just dependent upon technology itself, but also on the capabilities of innumerable, faceless business entities acting on our behalf.

Consider the following, common, but trivial example.

When I swipe my credit card at the pump to purchase petrol, that transaction passes through any number of computation devices and applications operated by a chain of business entities. The following is a typical scenario (an example flow – but not the only one, of course):

  • The point of sale device1, itself (likely supplied by a point of sale provider)
  • The networking equipment at the station2
  • The station’s Internet provider’s equipment (networking, security, applications – you have no idea!)
  • One or more telecom company’s networking infrastructure across the Internet backbone
  • The point of sale company or their proxy
  • More networking equipment and Internet providers
  • A credit card payment processor
  • More networking equipment and Internet providers
  • The card issuer who must validate the card and agree to pay the transaction for me

And so on…. All just to fill my tank up. It’s seamless and invisible – the communications between entities usually bring up an encrypted tunnel, though the protection offered is not as solid as you may hope – Invisible and seamless, except when the processing is not so invisible, like during a compromise and breach.

Every one of these invisible players has to have good enough security to protect me, and you, if you also use some sort of payment card for your petrol.

Home Depot, and Target before them, (and who knows who’s next?) failed to understand that in order to sell a hammer in the Internet world, you’re participating in this huge web of digital interconnection. Even more so, if you’re large enough, your business network will have become an eco-system of digital entities, many of whose security practices will affect your security posture in fairly profound ways. When 2 (or more) systems connect, each may affect the security posture of the other, sometimes in profound ways.

And there be pirates in them waters, Matey. As I wrote in the introduction to my next book, Securing Systems: Applied security architecture and threat models:

“…as of this writing, we are engaged in a cyber arms race of extraordinary size, composition, complexity, and velocity.”

One of the biggest problems for security practitioners remains that the cyber “arms race” isn’t just between a couple of nation-states. Foremost, the nation-state cyber war has to cross the same digital ocean that we use for our daily lives and digital entertainment. The shared web makes every digital citizen, potential “collateral damage”. But, there are more players than governments.

As can occur in a ground war, virtual “warlords” have private cyber armies marauding for loot, my loot, your loot. Those phishing spam don’t come from your friends, right3? Just trying to categorize the various entities engaged in cyber attacks could generate a couple of fine PhD theses and perhaps even provide years of follow-on papers? The number and varying loyalties of the many players who carry out cyber attacks increases the “size” of the problem, adds to the “composition”, and generates a great deal of “complexity”. It’s enough to make a well-meaning box-store retailer bury its collective head in the virtual sand. Which is precisely what happened to that hammer seller, Home Depot.

But answering the “who” doesn’t complete the picture. There’s the macro “how”, as well4. The Internet seems to suffer the “tragedy of the commons“.

In order to keep the Internet sufficiently interesting with compelling content such that we want to participate, it absolutely must remain neutral in character5. While Internet democracy certainly appears to be quite messy, the very thing that drives the diversity of content on the Internet is its level playing field6.

But leaving the Internet as an open field for all to enjoy means that some will take advantage of the many simply because the “pickings” are too rich to ignore. There is just too much to steal to let those resources lay untouched. And the pirates don’t! People actually do answer those “Nigerian Prince” scam emails. Really, someone does. People do buy those knock-off drugs. For the 3 billion of us who are digitally connected, it’s a dangerous digital day, every single day. Watch what you click!

In short, if you’re reading this on my blog site, you are perhaps an unwitting participant in that “cyber arms race of extraordinary size, composition, complexity, and velocity.” And so is every business that employs modern digital capabilities, whether for payments, or any other task. Failure to understand just how dependent a business is upon this matrix of digital interaction will make one a Target (pun intended). CEO’s, you may want to pay closer attention? Ignoring the current realities could cost you your job, perhaps even your career7!

If you think that you only sell objects and not some level of digital security, I fear that you are likely to be very sadly mistaken?

cheers,

/brook

  1. My friend and former colleague, Lucy McCoy, wrote the communications code in the first generation of gas pump payment terminals. At that time, terminals communicated via modem and phone line. She was a serial communications wiz. I remember the point of sale terminal laid out in her lab area. Lucy has since passed away. She was a brilliant engineer; she gave my code the best quality testing ever.
  2. The transactions have to get from station to payment processing, right? Who runs those cable modems and routers at the station? Could be the Internet provider, or maybe not. I run my own modem/routers/switches at home to which I have full admin access.
  3. I don’t know any spammers, as far as I know? Perhaps I make an unwarranted assumption that you don’t, either?
  4. The “what” and “why” of cyber attack seem pretty clear. Beyond attackers after money, they are after some other advantage: geo-political, business, just causes (pick your favourite or most hated cause), career enhancement, what-have-you. This is all pretty well documented. The security industry seems preoccupied with the “what”, i.e., the technical details of exploits. Again, these technical details seem pretty well documented.
  5. Imagine if your most hated or feared government had control over your Internet use, even the Internet itself, and proceeded to feed you exactly what they wanted you to know and prevented you from any other content. How would you like that?
  6. The richness and depth that is an emergent and continuing quality of the Internet, to me, demonstrates the absolute genius of the originators and early framers of the protocols and design.
  7. Of course, if I had a severance of $15.9 million, maybe I wouldn’t very much mind ending my career?

Heartbleed Exposure, What Is It Really?

Posted on by

Heartbleed Exposure, what is it really?

“Heap allocation patterns make private key exposure unlikely” Neel Mehta, discoverer of HeartBleed” 

In the media, there’s been a lot of discussion about what might be exposed from the heartbleed OpenSSL attack. It is certainly true that very sensitive items can be exposed. And over thousands of test runs, sensitive items like private keying materials and the like have been returned by the heartbleed buffer overread.

A very strong case can be made for doing exactly as industry due diligence suggests. Teams should replace private keys on servers that had been vulnerable, once these are patched. But should every person on the Internet change every password? Let’s examine that problems by digging into the details of exactly how heartbleed works.

First, heartbleed has been characterized as an “overflow” error: “Heartbleed is basically a buffer-overflow vulnerability”. This unfortunately is a poor descriptor and somewhat inaccurate. It may make better media copy, but calling heartbleed an “overflow” is a poor technical description upon which to base a measured response.

Heartbleed is not a classic buffer overflow. No flow control or executable code may be injected via heartbleed. A read of attacker chosen memory locations is not possible, as I will explain, below. A better descriptor of heartbleed is a “buffer over-read”. Unintentionally, some data from memory is returned to the attacker. To be precise, heartbleed is a data leak, not a flow control error.

In order to understand what’s possible to disclose, it’s key to understand program “heap” memory. The heap is an area of memory that programs use to store data. Generally speaking, well-written programs (like OpenSSL) do not to put executable code into heap (that is, data) memory[1]. Because data and execution are separated, the attacker has no way through this vulnerability to execute code. And that is key, as we shall see.

As a program runs, bits of data, large and small, temporary and more or less permanent for the run, are put into the heap[2]. Typically, data are put wherever is convenient at the moment of allocation, depending upon what memory is available.

Memory that’s been deallocated gets reused. If an available piece of memory happens to be larger than a requested size, the new sized piece will be filled with the new data, while adjacent to the new data will remain bits and pieces of whatever was there previously.

In other words, while not entirely random, the heap is filled with bits and pieces of data, a little from here, a little from there, a nice big chunk from this session, with a bit left over from some other session, all helter-skelter amongst each other. The heap is a jumble; taking random bits from the heap may be considered to be like attending a jumble sale.

Now, let’s return to heartbleed. The heartbleed bug returns whatever happens to be on the heap just above the 16 bytes that are required for the TLS heartbeat packet. The attacker may request as much as 64K bytes. That’s a nice big chunk of stuff from the heap; make no mistake about it. Anything might be in there. At the very least, decrypted  data intended for application processing will be returned to the attacker[3]. That’s certainly bad! It breaks the confidentiality supposedly gained through the TLS encryption. But getting a random bit is different than requesting an arbitrary memory location at the discretion of the attacker. And that is a very important statement to hold in mind as we respond to this very serious situation.

An analogy to Heartbleed might be a bit like going fishing. Sometimes, we fish where we can clearly see the fish (mountain streams) or signs of fish (clearer lakes), or with a “fish finder” appliance, that identifies fish  under the surface when the fish aren’t visible.

Heartbleed is a lot more like fishing for fish that are deep in a turbulent lake with no fish finding capability. The fisher is guessing. If she or he guesses correctly, fish for dinner. If not, it’s a long day holding onto the fishing rod.

In the same manner, the attacker, the “fisher” as it were, doesn’t know where the “fish”, the goodies are. The bait (the heartbleed request) is cast upon the “lake” (the program heap) in the hopes that a big fish will “bite” (secret “bytes” will get returned).

The attacker can heartbleed to her or his heart’s content (pun intended). That is, if left undiscovered, an attacker can continuously pound the other side of the connection with heartbleeds, perhaps thousands of times. Which means multiple chunks of memory will be returned to the attacker, as the heap allocates, deallocates, and moves data around.

Lots of different heap chunks will get returned. There will likely also be overlap between the chunks that are returned to the attacker. Somewhere within those memory chunks are likely to be some sensitive data. If the private key for a session happens to be in one of those chunks, it will be exposed to the attacker. If any particular session open through the OpenSSL library happens to a contain a password that had been transmitted, it’s been exposed. It won’t take an engineering genius to do an ASCII dump of returned chunks of memory in order to go poking about to find interesting bits.

Still, and nonetheless, this is hunting for goodies in a bit of a haystack. Some people are quite good at that. Let’s acknowledge that outright. But that’s very different than a directed attack.

And should a wise and prepared security team, making good use of appropriate security tools, notice a heartbleed attack, they will most likely kill the connection before thousands of buffers can be read. Heartbleed over any particular connection is a linear process, one packet retrieved at a time. Retrieving lots of data takes some time. Time to respond. Of course, an unprotected and unaware site could allow many sessions to get opened by an attacker, each linearly heartbled, thus revealing far more of what’s on the heap than a single session might. Wouldn’t you notice such anomalous behaviour?

It’s important to note that the returns in the heartbleed packets are not necessarily tied to the attackers’ session. Again, it’s whatever happens to be on the heap, which will contain parts of other sessions. And any particular heartbleed packet is not necessarily connected to the data in a previous or subsequent packet. Which means that there’s no continuity of session nor any linearity between heartbleed retrievals. All session continuity must be pieced together by the attacker. That’s not rocket science. But it’s also work, perhaps significant work.

I’ll reiterate in closing, that this is a dangerous bug to which we must respond in an orderly fashion.

On the other hand, this bug does not give attackers free reign to go after all the juicy targets that may be available on any host, server, or endpoint that happens to have OpenSSL installed. Whatever happens to be on the heap of the process using the OpenSSL library and that is adjacent to the heartbeat buffer will be returned. And that attack may only occur during a TLS session. Simply including the vulnerable library poses no risk, at all. Many programs make use of OpenSSL for other functionality beyond TLS sessions.

This bug is not the unfettered keys to the kingdom, unless a “key to the kingdom” just happens to be on the heap and happens to get returned in the over-read. What gets returned is entirely due to the distribution of the heap at the moment of that particular heartbeat.

Cheers,

/brook

These assertions have been demonstrated in the lab through numerous runs of the heartbleed attack by a  team who cannot be named here. My thanks to them for confirming this assessment. Sorry for not disclosing.

[1] There are plenty of specialized cases that break this rule. But typically, code doesn’t run from the heap; data goes onto the heap. And generally speaking, programs refrain from executing on the heap because it’s a poor security practice. Let’s make that assumption about OpenSSL (and there’s nothing to indicate that this is NOT true in this case), in order to make clear what’s going on with heartbleed.

[2] The libraries that support programs developed with the major development tools and running on the major operating systems have sophisticated heap management services that are consumed by the running application as it allocates and deallocates memory. While care must be exercised in languages like C/C++, the location of where data end up on the heap is controlled by these low-level services.

[3] That is, intended for the application that is using OpenSSL for TLS services.

Security Testing Is Dead. Rest In Peace? Not!

Posted on by

Apparently, some Google presenters are claiming that we can do away with the testing cycles in our software development life cycles? There’s been plenty of reaction to Alberto Savoia’s Keynote in 2011. But I ran into this again this Spring (2013)! So, I’m sorry to bring this up again, but I want to try for a security-focused statement…

The initial security posture of a piece of software is dependent upon the security requirements for that particular piece of software or system. In fact, the organizational business model influences an organization’s security requirements, which in turn influence the kinds of testing that any particular software or system will need before and after release. We don’t sell and deliver software in a vacuum.

Google certainly present a compelling case for user-led bug hunting. Their bounty programme works! But there are important reasons that Google can pull off user-led testing for some of their applications when other businesses might die trying.

It’s important to understand Google’s business model and infrastructure in order to understand a business driven bounty programme.

  • Google’s secured application infrastructure
  • Build it and they might come
  • If they come, how to make money?
  • If Google can’t monetize, can they build user base?

First and foremost, Google’s web application execution environment has got to have a tremendous amount of security built into it. Any application deployed to that infrastructure inherits many security controls. (I’ll exclude Android and mobility, since these are radically different) Google applications don’t have to implement identity systems, authorization models, user profiles, document storage protection, and the panoply of administrative and network security systems that any commercial, industrial strength cloud must deploy and run successfully. Indeed, I’m willing to guess that each application Google deploys runs within a certain amount of sandboxed isolation such that failure of that application cannot impact the security and performance of the other applications running on the infrastructure. In past lives, this is precisely how we built large application farms: sandbox and isolation. When a vulnerable application gets exploited, other applications sharing the infrastructure cannot be touched.We also made escape from the sandbox quite  difficult in order to protect the underlying infrastructure. Google would be not only remiss, but clueless to allow buggy applications to run in any less isolating environment. And, I’ve met lots of very smart Google folk! Scary smart.

Further, from what I’ve been told, Google has long since implemented significant protections for our Google Docs. Any application that needs to store documents inherits this document storage security. (I’ve been told that Google employ some derivation of Shamir’s Threshold Scheme such that unless an attacker can obtain M of N  stored versions of a document, the attacker gains no data whatsoever. This also thwarts the privileged insider attack)

My simple point is that Google is NOT entirely relying upon its external testers, its bug bounty programme. A fair amount of security is inherent and inherited by Google’s web application experiments.

And, we must understand Google’s business model. As near as I can tell from the outside, Google throws a lot of application “spaghetti” onto the Web. If an application “sticks”, that is, attracts a user base, then Google figure out how to monetize the application. If the application can’t be monetized, Google may still support the application for marketing (popularity, brand enhancement) purposes. Applications that don’t generate interest are summarily terminated.

In my opinion, Google’s business model leaves a lot of wiggle room for buggy software. Many of these experiments have low expectations, perhaps no expectation at the outset? This gives Google time to clean the code as the application builds user base and penetration. If nobody is dependent upon an application, then there’s not a very high security posture requirement. In other words, Google can take time to find the “right product”. This is entirely opposite for security function that must deliver protection independent of any support (like on an end point that can be offline). Users expect security software to be correct on installation: the product has to be built “right”, right from the start.

And, the guts of Google are most likely protected from any nasty vulnerabilities. So, user testing makes a lot of business sense and does not pose a brand risk.

Compare this with an endpoint security product from an established and trusted brand. Not only must the software actually protect the customer’s endpoint, it’s got to work when the endpoint is not connected to anything, much less the Internet (i.e., can’t “phone home”). Additionally, security software must meet a very high standard for not degrading the posture of the target system. That is, the software shouldn’t install vulnerabilities that can be abused alongside the software’s intended functionality. Security software must meet a very high standard of security quality. That’s the nature of the business model.

I would argue that security software vendors don’t have a great deal of wiggle room for user-discovery of vulnerabilities. Neither do medical records software, nor financials. These applications must try to be as clean as possible from the get go. Imagine if your online banking site left its vulnerability discovery to the user community. I think it’s not too much of a leap to imagine the loss in customer confidence that this approach might entail?

I’ll state the obvious: different businesses demand different security postures and have different periods of grace for security bugs. Any statement that makes a claim across these differences is likely spurious.

Google, in light of these obvious differences, may I ask your pundits to speak for your own business, rather than assuming that you may speak for all business models, rather than trumpeting a “new world order”? Everyone, may I encourage us to pay attention to the assumptions inherent in claims? Not all software is created equally, and that’s a “Good Thing” ™.

By the way, Brook Schoenfield is an active Google+ user. I don’t intend to slam Google’s products in any manner. Thank you, Google, for the great software that I use every day.

cheers

/brook

 

 

 

Seriously? Product Security?

Posted on by

Seriously? You responded to my security due diligence question with that?

Hopefully, there’s a lesson in this tale of woe about what not to do when asked about product security?

This incident has been sticking in my craw for about a year. I think it’s time to get it off my chest. If for no other reason, I want to stop thinking about this terrible customer experience. And yes, for once, I’m going to name the guilty company. I wasn’t under NDA in this situation, as far as I know?

There I was, Enterprise Security Architect for a mid-size company (who shall not be named. No gossip, ever, from this blog). Part of my job was to ensure that vendors’ product security was strong enough to protect my company’s security posture. There’s a due diligence responsibility assigned to most infosec people. In order to fulfill this responsibility, it has become a typical practice to research software vendors’ product security practices.Based upon the results, either mitigate uncovered risks to policy and industry standards or raise the risk to organizational decision makers (and there are always risks, right?).

Every software vendor goes through these due diligence investigations on a regular basis. And I do mean “every”.

I’ve lived on both sides of this fence, conducting the investigations and having my company’s software go through many investigations. This process is now a part of the fabric of doing secure business. There should be nothing surprising about the questions. In past positions, we had a vendor questionnaire, a risk scale based upon the expected responses, and standards against which to measure the vendor. These tools help to build a repeatable process. One of these processes is documented in a SANS Institute Smart Guide released in 2011 and was published by Cisco, as well.

Now, I’m going to name names. Sorry, Google, I’m going to detail just what your Docs sales team said to me. Shame on you!

When I asked about Google Docs product security here is the answer, almost verbatim, that I received from the sales team:

“We’re Google. We can hire Vint Cerf if we want. That is enough.”

Need I point out to my brilliant readers that Dr. Vint Cerf, as far as I know, has never claimed to be an information security expert? I’m sure he knows far more about the design of TCP/IP than I? (but I remind readers that I used to write TCP/IP stacks, so I’m not entirely clueless, either). And, Dr. Cerf probably knows a thing or two about Internet Security, since he runs ICANN?

Still, I can tell you authoritatively that TCP/IP security and Domain Name Registry security are only two (fairly small) areas of an information security due diligence process that is now standard for software vendors to pass.

Besides, the sales team didn’t answer my questions. This was a classic “Appeal to Authority“. And, unfortunately, they didn’t even bother to appeal to a security authority. Sorry Vint; they took your name in vain. I suppose this sort of thing happens to someone of your fame all the time?

Behind the scenes, my counter-part application architect and I immediately killed any possible engagement with Google Docs. Google Sales Team, you lost the sale through that single response. The discussion was over; the possibility of a sale was out, door firmly closed.

One of the interesting results from the wide adoption of The Web has been the need for open and transparent engagement. Organizations that engage honestly gain trust through their integrity, even in the face of organizational mis-steps and faux pas. Organizations who attempt the old fashion paradigm, “control all communications”, lose trust, and lose it rapidly and profoundly. Commercial companies, are you paying attention? This is what democracy looks like (at least in part. But that’s a different post, I think?).

Product security is difficult and demanding. There are many facets that must compliment each other to deliver acceptable risk. And, even with the best intentions and execution, one will still have unexpected vulnerabilities crop up from time to time. We only have to look at development of Microsoft’s product security programme to understand how one of the best in the industry (in my humble opinion) will not catch everything. Do Microsoft bugs surface? Yes. Is the vulnerability level today anywhere near what it was 10 years ago? Not even close. Kudos, Microsoft.

It’s long past the time that any company can coast on reputation. I believe that Google do some very interesting things towards the security of their customers. But their  sales team need to learn a few lessons in humility and transparency. Brand offers very little demonstrable protection. Google, you have to answer those due diligence questionnaires honestly and transparently. Otherwise the Infosec person on the other side has nothing against which to base her/his risk rating. And, in the face of no information, the safest bet is to rate “high risk”. Default deny rule.

It’s a big world out there and if your undiscovered vulnerabilities don’t get’cha now, they will eventually. Beware; be patient; be humble; remain inquisitive; work slowly and carefully. You can quote me on that.

cheers,

/brook

Missed Points, Wrong Message?

Posted on by

Today, as every day, I was bombarded with articles to read. You, too?

Today, Testing for the Masses: More affordable assessment services reveal a new focus on application security caught my eye.

As some of my readers may know, I’ve spent a lot of time and effort on re-creating the customer web application security testing viewpoint. And, of course, Markus and I often don’t agree. So, I trundled off to read what he’d opined on the subject.

Actually, it’s great to see that ideas that were bleeding edge a few years ago are now being championed in the press!

* Build security in the design
* Code securely
* Verify both the design and the code

The “design” part of the problem requires reasonable maturity of a security architecture practice.

And, web application code must be tested and fixed before deployment, just as this article suggests. All good. If we could achieve these goals on a broad scale it would be a much better world that what we’ve got, which has been to deploy more than 100 millon lines of vulnerable code to the web (see Jeremiah Grossman’s white papers on this topic)

Further, the writers suggested that price points of web application testing tools must fall. And here’s where the ball got dropped. Why?

The question of price is related to one of the main limitations on getting at least the simple and egregious bugs out of our web code.

The tool set has until recently been made for and focused to people like Markus Ranum – security experts, penetration testing gurus. These tools have emphasized broad coverage and deep analysis. And that emphasis has been very much at the expense of:

* tool cost
* tool complexity
* noisy results that must be hand qualified by an expert

There aren’t that many Markus Ranum’s in the world. In fact, as far as I know, only one! Seriously, there are not that many experienced web application vulnerability analysts. My estimate of approximately 5000 was knocked down by some pundit (I don’t remember who?) at a SANs conference at which I was speaking. His guesstimate? 1500. Sigh. Imagine how many lines of code each has to analyze to even make a dent in those 100 million lines deployed.

Each of the major web application vulnerability scanners assumed that the user was going to become an expert user in the tool. User interfaces are complex. Setting the scope and test suite are a seriously non-trivial exercise.

And finally, there is the noise in the results – all those false positives. These tools all assume that the results will be analyzed by a security expert.

Taken together, we have what I call the “lint problem”. For those who remember programming in the C language, there is a terrifically powerful analyzer called “lint”. This tool can work through a body of code and find all kinds of additional errors that the compiler will miss. Why doesn’t everyone use lint, then? Actually, most C programmers never use lint.

Lint is too hard and resource intensive to tune. For projects that are more complex than one of two source files, the cost of tuning lint far out weighs it’s value. So, only a few programmers ever used the tool. On a 3 month project, just about the time the project is over is when lint is finally returning just the useful results. Not very timely, I’m afraid.

And so, custom web application vulnerability testing has remained in the hands of experts, people like Markus Ranum, who, indeed, makes his living by reviewing and fixing code (and rumour has it, a very good living, indeed).

So, of course, Markus probably isn’t going to suggest the one movement that will actually bring a significant shift this problem.

But I will.

The vulnerability check must be put into the hands of the developer! Radical idea?

And, that check can’t be a noisy, expert driven analysis.

* The scan must be no more difficult to run than a compiler or linker. (which are, after all, the tools in use by many web developers every day)
* The scan should integrated seamlessly into the existing workflow
* The scan must not add a significant  time to the developer’s workflow
* The results must be easy to interpret
* The results must be as accurate as a compiler. E.g., the developer must be able to trust the results with very high confidence

Attack these bugs at the source. And, to achieve that, give the developer a tool set that s/he can use and trust.

And, the price point has to be in accordance with other, similar tools.

I happen to know of several organizations that have either experimented with this concept or have put programs into place based on these principles (can’t name them, sorry. NDA) And guess what? It works!

Hey, toolmakers, the web developer market is huge! There’s money to be made here, folks.

Markus, I think you missed a key point, and are pointing in the wrong direction (slightly)

cheers

/brook

What’s Occurred in Web Years?

Posted on by

What’s Occurred in Web Years?

I was trying to explain to a colleague some of the sea change that has already taken place on the web (IMHO) – “Web 2.0” if you will?

I was explaining that the typical large enterprise (and really, many small ones, too) have become an intersection of clouds.

We used to use the image of the “good” inside protecting itself from the hostile (“bad”) Internet. The Internet was the cloud, the internal network a known and fixed space, “the network”. And, indeed, when I got started in Information Security, that is pretty much how things were.

But where I work, nobody knows all the networks that are interconnected. Acquisitions, test networks, global points of presence (POPs), all work to make a complex of interconnected networks. It’s too complex to hold in the mind. A mapping exercise some years ago revealed connections that nobody had documented – “discovered lands!”

In fact, the enterprise network seems a mini-internet or cloud that is probably less hostile than the Internet. But it’s certainly not trusted like the little “Class C” (anybody remember classed networks?) that I use to be responsible for where I knew everyone on the network and a lot of what they were using the network for.

But the enterprise cloud is not the only cloud at play.

Within the business eco-system are many similar interconnected network spaces of varying regulation and relationship, from close to pretty distant and untrusted. To be sure, a lot of these connect via the Internet. But certainly not all. Most enterprises have private connections with partners. The partners are literally cross-connected to the “internal” network.

There may be (should be?) network and other restrictions in place between the networks. Still, traffic flows and presumably some portion of that traffic is likely hostile (hopefully a minute portion, well monitored?)

The need to model business relationships via the network has caused an explosion of interconnected clouds.

Basically, the perimeter means much less than it once did – perhaps even “The perimeter is dead. Long live the perimeter!”

Coupled with this change has been the rapid growth of software tools on the network that model human relationship graphs and which allow a much greater degree of participation.

So, while the internal network has been growing in complexity and opening up to interconnections, usage patterns  (and business demand, it turns out) have been driving from inside to out.

These two forces have already occurred. And they happened, to my observation, faster than the growth of Web 1.0. Social networking seems to me to have taken off in months. The blogosphere has well been in place for several years. When is the last time you bought an expensive item about which you were uncertain without first checking online reviews and perhaps the blogosphere? I almost always (always?) do this before uncertain purchases.

What does this all have to do with security architecture?

These changes shift things fundamentally.

Network controls are now a tool – not the basis for one’s information security posture. We are using them around critical assets, but they no longer divide the “good” inside from the “bad” outside.

Meanwhile, data is moving out of the inner cloud and is a “must” for business agility. We can’t control our data by keeping our hot little security controls (“ACLs” – smile) around it.

The old security paradigm is obsolete. We need a new one.

And all this, in my opinion, transpired in web years while many of us were sleeping away building better network castles.

cheers,

/brook

(from the Denver airport)