Subresource Integrity is coming to the modern browsers near you

Great news - just a couple of months ago (on the 5th of May) W3C has delivered a working draft for Subresource Integrity (SRI) specification. From the abstract:

This specification defines a mechanism by which user agents may verify that a fetched resource has been delivered without unexpected manipulation.

Why am I exciting about this announcement? Well, the key phrase here is "verify that ... delivered without manipulation". This feature alone is not a panacea for all the bad stuff happening on the Internet. But it is an excellent defence in depth measure that (in many cases) doesn't cost too much time and effort to implement.

Many web sites embed resources like css or javascript. Sometimes those resources are hosted on the 3rd party web sites. E.g. you may find it easier to reference a boostrap CSS file from http://www.bootstrapcdn.com/ or the latest version of jQuery from http://code.jquery.com/jquery-git2.min.js. But what if one of the sites hosting those resources gets compromised? The security of your web site will be affected.

We have also seen cases of content delivered across the wire being modified on the way to the end users (e.g. ISPs or WiFi hotspot operators injecting ads or governments stealing credentials). By supporting SSL/TLS and loading websites via HTTPS we can protect the content of the web pages. SRI helps to further improve security by allowing a server to supply an additional piece of information to the client (browser) to ensure that this particular resource hasn't been modified/tampered with.

This additional piece of information is officially called "integrity metadata". This is just a base64-encoded hash of the resource. The specification says that servers/clients MUST support SHA-2 hashes (i.e. SHA-256, SHA-384, SHA-512) and MAY support other cryptographic functions. By supplying a hash we can (almost - aside from a hash collision scenario) guarantee that the resource hasn't been modified from the moment when the hash has been generated.

Note: if an attacker controls the web server then he/she can produce valid hashes.

Now putting it all together - this is how it will most likely look like:

<script src="https://analytics-r-us.com/v1.0/include.js"
        integrity="sha256-SDfwewFAE...wefjijfE"
        crossorigin="anonymous"></script>

Here we can see a standard script tag that embeds an external include.js javascript file and a newly introduced "integrity" attribute, that specifies a SHA-256 hash of the include.js file. A client side (browser) now has the ability to download this resource, recalculate the hash and compare the result with the value supplied in the integrity attribute. If 2 values don't match then this resource needs to be discarded (it can't be trusted).

It will also be possible to specify multiple hash values for the same resource

<script src="hello_world.js"
   integrity="sha256-+MO/YqmqPm/BYZwlDkir51GTc9Pt9BvmLrXcRRma8u8=
              sha512-rQw3wx1psxXzqB8TyM3nAQlK2RcluhsNwxmcqXE2YbgoDW735o8TPmIR4uWpoxUERddvFwjgRSGw7gNPCwuvJg=="
   crossorigin="anonymous"></script>

In this scenario client will be able to choose the strongest supported hash function.

Note: some examples that you might find on the Internet use an older syntax (notice the "ni" part that stands for "named information" as defined in RFC6920):

integrity="ni:///sha-256;C6CB9UYIS9UJeqinPHWTHVqh_E1uhG5Twh-Y5qFQmYg?ct=application/javascript"

Around January 2015 the specification has been updated to adopt the same format as CSP Level 2 for the hash format. So the "ni" part is no longer required.

In addition to link (css) and script tags the future versions of SRI will support other types of resources - e.g. file downloads referenced in A tags or even iframes.

From the information I have, it looks like SRI will be fully supported in Firefox v.42.

It is currently "under consideration" for Microsoft Edge. Most likely it won't be implemented in IE11.

In conclusion I would like to share 2 links with you:

SRI hash generator - will make it easier to calculate hashes

W3C SRI test - will run the test and show how well SRI is supported in your browser of choice.

KCodes NetUSB vulnerability (CVE-2015-3036) and a short-term fix for TP-Link

There was a vulnerability disclosed by SEC Consult earlier today that affects a significant number of SOHO routers. NetUSB is a technology that provides a "USB over IP" functionality. It was developed by a company KCodes and since then has been adopted by many popular network device manufactures (including Netgear and TP-Link). 

NetUSB runs as a Linux kernel driver. When it is available it launches a server on TCP port 20005 (that is typically accessible on the LAN only). I have seen some reports already claiming that some (mis)configurations were exposing port 20005 on the WAN side (i.e. to the Internet) as well. And this is bad news because according to the advisory NetUSB suffers from a remote stack buffer overflow. And being kernel driver it means that remote attacker by exploiting this vulnerability can gain admin privileges on the affected device.

The AES keys are static - that's not great, as it means they are useless. They won't be able to stop the attackers as the keys are already known to them. And all they need to do is to send a computer name longer than 64 bytes to cause an overflow. This feels like 90s again.

If port 20005 is not accessible from the outside then this reduces the risk but it still leaves this network vulnerable to the attacks from the inside.

I've got my hands on one of the affected models - TP-Link Archer D9.

A quick test to connect to port 20005: 

telnet 192.168.0.1 20005

reveals that it does indeed listen on port 20005 in the default configuration (i.e. I was able to connect).

A web management interface has this section under USB Management -> Print server:

As you can see by default the Print Server is turned on.

Let's click the Stop button...

... and try to connect again:

Ah, much better.

I am not sure if this approach fully mitigates this issue but it certainly makes the overall situation better.
An updated firmware version with a fix from TP-Link is expected around the 25th of May. Until then I would recommend you to stop the Print Server.

Integers - when size does matter

I have just read about an issue affecting all Boeing 787 airplanes:

"We have been advised by Boeing of an issue identified during laboratory testing. The software counter internal to the generator control units (GCUs) will overflow after 248 days of continuous power, causing GCU to go into failsafe mode. If the four main GCUs (associated with the engine mounted generators) were powered up at the same time, after 248 days of continuous power, all four GCUs will go into failsafe mode at the same time, resulting in a loss of all AC electrical power regardless of flight phase.

Wow, this is scary - especially the "regardless of flight phase" bit. I've done some research and it turns out that the probability is VERY low for a given aircraft to remain powered for 248 days in a row.

In the same document FAA (as an interim measure) adds a requirement:

This AD requires a repetitive maintenance task for electrical power deactivation

This essentially means - each plane must be periodically powered off (obviously with the frequency < 248 days). We all recognise this pattern - a periodic application restart or server reboot when dealing with the misbehaving applications (memory leaks etc)

In fact, there are several relevant and related to IT moments that caught my attention.

The "magic" 248 days number

A 32 bit signed (i.e. we can only use 31 bit) integer can store a maximum value of 2147483647.

2,147,483,647 / (24 hours*60 min*60 sec) = 24,855

Many sensors connected to the ARINC-429 bus have a 100Hz data sampling rate.

Dividing 24,855 by 100Hz we get 248.55 days needed to overflow this integer.

We have many examples in IT, where integer overflows cause all sorts of troubles (ranging from availability to security)

But I wanted to mention one issue that's worth keeping an eye on. Have you ever seen an error message like this?

Server: Msg 8115, Level 16, State 1, Line 1
Arithmetic overflow error converting IDENTITY to data type int.
Arithmetic overflow occurred.

SQL Server will generate this error when it detects an IDENTITY column overflow.

I used to use a script that looked very similar to the one provided by Red-gate. Give it a go - who knows, you might be able to discover an identity column approaching the limit and prevent an outage.

If the four <redundant devices> were powered up at the same time...

This is another interesting issue. If we have N redundant devices but they all share the same common (time based) problem then there is a chance that the fault across all N devices will happen at the same time. This means that a fault will escalate to a system-wide failure (i.e. outage). 

It is quite common in IT to implement a staggered approach when introducing changes (OS patches, code rollout etc) - "touching" all systems at the same time is not desirable.

There are certain events that bring application restarts and server reboots back in "sync". E.g.

• A predictable Patch Tuesday and default settings will result in many computers applying patches and rebooting around 3 AM (in a given time zone). Note: Microsoft decided to move away from the monthly cycle and release patches as they become available - this is a good move in my view.
• A vulnerability in a common library might force many systems administrators to patch affected software almost at the same time around the world. Plus SSL certificates might need to be reissued too - we've seen an April spike caused by Heartbleed 

This is something that we usually don't think about when patching our systems. So take this into the consideration next time you apply patches to a multi-node cluster or a group of network devices operating in HA mode.

Aussie banks security (HTTP headers edition)

Introduction

Troy Hunt has recently published a blog post, where he analysed a current state of affairs with regards to SSL/TLS support and (in)secure configurations in the Australian banking industry. This is an interesting overview and a great mechanism for raising awareness, which (judging by the updates) has already prompted some banks to make changes and improve their ratings.

I was curious - how would the situation with SSL correlate with another publicly available piece of information - HTTP headers (especially the security related subset). These headers could provide some valuable insights (although in my view in general the impact of SSL misconfiguration could be bigger compared to the presence/absence of a particular header).

So I performed a mini-research, focusing on the same set of Australian banks. I used https://securityheaders.io to simplify the collection of all headers. The headers I was interested in were (group 1):

• Strict-Transport-Security
• Content-Security-Policy
• Public-Key-Pins
• X-Frame-Options
• X-Xss-Protection
• X-Content-Type-Options

When I started analysing captured results I decided to add the following headers into the mix (group 2):

• Server
• X-Powered-By
• X-AspNet-Version

Surprisingly, I haven't observed the 3rd "usual suspect" - x-aspnetmvc-version. Perhaps banks haven't upgraded their web applications to the MVC versions yet?

Also worth mentioning that X-AspNet-Version will only be emitted by the ASP.Net powered application - i.e. this header won't be relevant for other platforms.
All results have been collected in early May 2015 - the situation may have changed since then. It will be interesting to perform periodic checks to observe the dynamics.

Methodology

Once the raw data were captured in a table I started thinking how I could rank these findings. Here's a set of rules I came up with (I'd be very keen to hear suggestions for improvement):

• The initial state - everyone starts with 0 (zero) points - neutral state 
• If Strict-Transport-Security is present - add 2 points. HSTS is too good to be ranked equally with the other headers
• If any other header from group 1 is present - add 1 point for each header
• If "Server" header is present
• Deduct 1 point if a full/non-obfuscated name is found (marked red in the table below)
• Otherwise deduct 0.5 points (e.g. "CUA Server" doesn't disclose much and is much more benign compared to "Microsoft-IIS/7.5" - marked as orange)
• If "X-Powered-By" header is present
• Deduct 1 point if a well-known framework is disclosed
• Otherwise deduct 0.5 points (marked as orange)
• If "X-AspNet-Version" header is present - deduct 1 point
• I took a sum for each bank and listed it as a "Score"
• Then I assigned banks to the corresponding groups based on that score
• Green: greater than 0
• Yellow: 0
• Orange: -0.5
• Red: less than -0.5

Results

Bank	Score	Strict-Transport-Security	Content-Security-Policy	Public-Key-Pins	X-Frame-Options	X-Xss-Protection	X-Content-Type-Options	Server	X-Powered-By	X-AspNet-Version
Bank West	5	Yes	Yes	No	Yes, SAMEORIGIN	Yes	No	No	No	No
CUA	1.5	Yes	No	No	No	No	No	Yes, CUA Server	No	No
Commonwealth Bank	1	Yes	No	No	No	No	No	Yes, Apache/2.2.3 (Red Hat)	No	No
ING Direct	1	No	No	No	Yes, SAMEORIGIN	No	No	No	No	No
AMP	0.5	No	No	No	Yes, SAMEORIGIN	No	No	Yes, IBM_HTTP_Server	No	No
St George	0.5	No	No	No	Yes, SAMEORIGIN	No	No	Yes, Apache	No	No
Bankmecu	0	No	No	No	No	No	No	No	No	No
Bendigo Bank	0	No	No	No	No	No	No	No	No	No
Beyond	0	No	No	No	No	No	No	No	No	No
Greater	0	No	No	No	No	No	No	No	No	No
Heritage	0	No	No	No	No	No	No	No	No	No
Macquarie	0	No	No	No	No	No	No	No	No	No
People's Choice Credit Union	0	No	No	No	No	No	No	No	No	No
Suncorp	0	No	No	No	Yes, SAMEORIGIN	No	No	No	Yes, ASP.NET	No
IMB	-0.5	No	No	No	No	No	No	Yes, Sandstone Framework	No	No
Westpac	-0.5	No	No	No	No	No	No	No	Yes, Servlet/3.0	No
Newcastle Permanent	-1	No	No	No	No	No	No	No	Yes, ASP.NET	No
ANZ	-2	No	No	No	No	No	No	Yes, Microsoft-IIS/6.0	Yes, ASP.NET	No
Bank of Queensland	-2	No	No	No	No	No	No	No	Yes, ASP.NET	Yes, 2.0.50727
P&N	-2	No	No	No	No	No	No	Yes, Microsoft-IIS/7.5	Yes, ASP.NET	No
Teachers Mutual	-3	No	No	No	No	No	No	Yes, Microsoft-IIS/8.0	Yes, ASP.NET	Yes, 4.0.30319

Update 1 - 14 June 2015: Bank West has addressed a few issues and contacted me to update the results. A quick check revealed a massive improvement - going from a "-2" score all the way to the top of the leaderboard! Congratulations to the team - thank you for taking time to improve your ranking.
This also gave me an opportunity to review the situation with the other banks from this list. Newcastle Permanent dropped 1 point down for the presence of the X-Powered-By header. ANZ did the same for the Server header (IIS 6, really???). Heritage went 1 point up (removed Server header)

Key findings

• Only Strict-Transport-Security and X-Frame-Options security headers have been observed in the wild
• Surprisingly - clear lack of wider adoption of security headers (group 1) by the banks.
• Only 2 banks out of 21 use HSTS - kudos to CUA and Commonwealth Bank;
• Only 4 banks  use X-Frame-Options
• The situation with the group 2 headers was better, most of the banks had none of them. Some banks had several Asp.Net related headers (which took them to the bottom of the list). This situation is especially surprising given how easy it is to remove these headers.
• Weak correlation between SSL and HTTP headers results
• ING Direct is the only bank that has managed to reach the top (green) categories in both tests

And the last point that I would like make - some headers (which are not directly related to security per se) might still leak some useful to the attackers information. This includes:

• Set-Cookie: f5_cspm=[skipped]; - indicates presence of an F5 appliance
• Set-Cookie: citrix_ns_id=[skipped] - indicates presence of a Citrix Netscaler appliance
• Set-Cookie: ASP.NET_SessionId=[skipped]; path=/; HttpOnly - indicates an ASP.Net application even if all other headers (from group 2) have been removed

To summarise - it is a bit disappointing that we don't see a wider adoption of the HTTP security headers by the Australian banking industry. Some of these headers are trivial to implement (e.g. X-Frame-Options) and yet they provide a valuable protection layer.

I would like to encourage everyone reading this blog to get a better understanding of what all of these headers do and to start implementing at least some of these headers on your own web sites. This will make Internet as a whole more secure.

The wonderful pipe "|" character

Today I want to talk about a pipe character (a.k.a vertical bar). Or more specifically - a special role it plays in probing the quality of web applications. The pipe character is classified as unsafe in RFC 1738 - this puts it in a "grey area" hence a good candidate for testing how various frameworks and applications handle this case. When URLencoded this character is represented by the %7C sequence.

I love this character when it comes to testing error handling/custom errors setup in the .Net applications.

A standard ASP.Net web application uses *.aspx file extension by default. As a test I like adding a pipe character in the page name (preserving the aspx extension to make sure the request is still sent to the .Net handler). It's worth noting that a similar approach quite often works for the "extensionless" ASP.Net MVC URLs too.

I will demonstrate a few possible scenarios.

Let's take a login page and add our pipe character:

http://<mysite>/login|.aspx

A well-behaving application correctly sends me to the error page:

But notice that the error page is also a .Net page. What if we try to trigger the same error again in the error page itself? This is actually a very common scenario when the initial exception will be caught and handled properly but the error page won't be able to defend itself.

http://<mysite>/Error|.aspx?aspxerrorpath=/login|.aspx

Boom!

We have generated an unhandled exception but because customErrors were turned on in web.config we've got the page above. Not ideal but it can be worse.

Let's try the same approach against a test vulnerable application courtesy Acunetix:
http://testaspnet.vulnweb.com/login%7C.aspx

We get an "Illegal characters in path" exception. We see that this is an ASP.Net v2 application. Notice that we only have the System namespaces - user code hasn't been invoked yet!

For completeness here is the same problem in ASP.Net v4 application (with a slightly different stack trace):

And this is why I love the pipe character. It helps uncovering various interesting scenarios - when customErrors are not setup correctly, when global exception handler (Application_Error) is missing, or when error pages themselves can't handle exceptions properly.

This is a very simple test. So go on - give it a go, see how your application handles the pipe and leave a comment if you found anything interesting or if it helped you to make your application more robust.

See this article for more information how to configure exception handling properly.

P.S. another good character to play with is a tilde "~"

Engineer for resilience

I actually didn't plan to write this blog post. I was reading my twitter feed when I spotted a discussion about an Indian web site TRAI that publicly exposed several thousands of e-mail addresses (net neutrality related e-mails). Another comment was made that the web site was down because of the amount of people trying to check if their addresses were in those published lists. And then I noticed this screenshot:

Makes sense - a massive (and unexpected) spike in workload and it is understandable that the backend database server couldn't handle the load. I am obviously speculating here but it doesn't really matter for the purpose of this discussion.

Before we move on - displaying detailed error messages like this one (containing a stack trace, .Net version etc) is bad from the security perspective. Troy Hunt has a great (and very detailed) explanation how to set up custom error pages properly.

But what really caught my attention was this line:

   banner.proc_Display_banner(String param) +26

A "display banner" function tries to perform an action against the database. This request fails and the whole page "explodes" and displays a detailed error page. I will speculate again - a banner may mean 2 things in my view. It is either a topbar of the web site or an embedded advertisement of some sort.

In both cases I would argue that this banner is most likely not essential for the core functionality of the web site. And that brings me to the topic that I am very passionate about - engineering for resilience. I've seen it many times - it is way too easy to just throw an exception and give up at this point (hoping that this exception will bubble up and will be handled somewhere - or maybe not).

Think about it - a function call to display a banner fails and takes down the whole page. Or we might even argue - takes down the whole web site as the error happens on the homepage. What would you prefer - an operational web site that lacks a topbar (or doesn't display an advertisement) or a web site that is broken?

For some reason I see this times and times again - developers focus on the "success" story - i.e. the code performs what it's supposed to do in the ideal conditions when all databases/services/endpoints are available. And they don't consider scenarios when some of these building blocks they rely on in their code become unavailable. It's convenient to just assume that this database will always be there, right? And it will be the case in 99.9% of the time. The SLA of "three nines" is not ideal but not unheard of (especially for the single instance setups). 99.9% also means roughly 40 minutes of downtime each month. How will your code behave during those 40 minutes? The key point I'd like to make - we need to expect failures and engineer for resilience. As we move from the monolithic systems to distributed/microservices based architectures, inevitably we will be relying more and more on various external (to our code) APIs, various endpoints and databases. And usually we don't have any information about the availability of a particular endpoint when we are about to call it.

There is nothing wrong with failures per se. You pick up the phone, call you colleague and receive a busy signal. In a way it is a failure. You failed to connect and invite your colleague for a coffee. But is it a problem? No, as most likely you will "retry" a few minutes later. 

Retry is a great (and simple) way to recover from failure. Just retry the last failed call and see if it finishes successfully this time. Limit the number of retries to make sure you don't create and endless retry loop and that you don't overload/DOS the system that might already be struggling.

So what can we do if we retried but were not able to achieve a desired outcome?

Consider a degrade approach. (Think - going for a coffee alone without your colleague). In many cases it is possible to reduce full functionality of a system - somehow simplify it - without losing it (functionality) completely. E.g. in the case of a banner we could:

• Replace a banner that we wanted to display (and couldn't retrieve from the database) with a hardcoded version (that doesn't require access to the database).
• Replace this banner with an blank image of the right size
• Don't display the module that failed at all. This might break the HTML layout but the main page will still be alive.

By wrapping the database call in a try-catch block obviously would have allowed a developer to catch this failure condition and make decision what to do next - how to handle this failure scenario.

I will write another complementary blog post regarding how "engineering for resilience" is done in space technology - approach/ideas/principles, which can teach us - IT people - a few things.

Please make sure you consider all possible outcomes - no matter how improbable they might be. Databases go down, processes run out of threads and memory, network failures/packet drops happen. We need to be prepared for all of these scenarios. Code that can survive failures is a sign of a high class/experienced developer.

MS15-034 (http.sys)

There has been a lot written about this (quite nasty) vulnerability since the latest "Patch Tuesday". My understanding is that so far only DoS (a simple crash) has been reliably confirmed, although I've seen some reports that an RCE (remote code execution) exploit is being sold on black market for slightly over $100.

I can also see scanning requests in the wild. I think it's just a matter of a few days for us to start seeing a world wide spike in those probing requests. That spike will last for a couple of weeks (attackers going after the low hanging fruit). We have seen the same behaviour with CVE-2014-6271 (Bash Command Injection). This is all fine and kind of expected.

What I find surprising is that bugs of that calibre can actually be uncovered in 2015. I will explain what I mean:

First of all - this bug can be triggered by a request like this:

curl -v http(s)://hostname/ -H "Host: hostname" -H "Range: bytes=0-18446744073709551615" -k

So all we need to do is send an HTTP (or HTTPS - doesn't matter) request to a server with a specific Range header. What is so special about 18446744073709551615? It's 2^64-1. So MS15-034 is essentially an integer overflow bug.

HTTP is one of the most popular protocols. And a Range header has been added to the HTTP/1.1 version (which was around for quite some time now - RFC 2068 - January 1997). People have been fuzzing protocols (including obviously HTTP) for ages. In fact I will quote the "Fuzzing: Brute Force Vulnerability Discovery" book By Michael Sutton, Adam Greene, Pedram Amini

"Any and all request headers can and should be fuzzed" indeed. A value of 2^64-1 (or MAXINT-1 on some platforms) should be in the Top 10 of any integer fuzzer. Microsoft lists Windows 7 as affected meaning this vulnerability has been around for quite some time. This is why I am surprised - how come that this vulnerability has only been found now? It should've probably been even picked up by the internal QA team. And certainly with so many researchers running fuzzers (which kind of makes security research less exciting - topic for another blog post) how did we not find it earlier?! And more importantly - how many other bugs of this calibre are still out there?

Gentlemen, start your fuzzers!

Update:
Using a WAF to block MS15-034 attack pattern is a great (and simple) way to protect your environment while systems administrators continue their assessment and rollout of the patch to all servers. Many major vendors (Incapsula, Cloudflare, ModSecurity, Akamai etc) have already created custom rules for their customers.
I've seen some people suggesting a WAF rule to block requests that match on "0-18446744073709551615" when examining the Range header. Be careful - "0" was used for harmless probing requests. The actual DoS requests contain a non-zero value (I've seen 18 and 20 so far). So it would be better to match and block requests that match a broader pattern: [\d]+-18446744073709551615