My first 2 months in America - part 1

It's been 2 months since I left Melbourne, Australia and moved to California, USA. 2 months is not enough to fully understand and get a feeling of a new country but it is certainly enough to make some observations. I wanted to capture these first impressions while they are still fresh.

California (Bay Area) is surprisingly very similar to Melbourne both from the climate and landscape perspectives - very similar temperatures, drought-like conditions during the summer months, ocean proximity, hills surrounding the area.

But there are certainly some differences too. For some reason a large part of these differences revolves around cars and driving for me. So let's begin:

After 12 years of driving on the left side of the road in Australia, I am back to driving on the right side. It is fairly easy to adjust, just need to keep thinking when making turns for the first couple of weeks. What made it easier for me is that I was driving a European car in Australia with controls (wipers, indicators) already in the same arrangement, so I avoided the usually inevitable wipers instead of indicators when making a turn.

Petrol stations!

First of all, it is not "petrol" anymore. It's "gas". An engineer in me cringes but I do realise that it is a contraction of "gasoline".

Octane numbers are different. And the reason for this is that Australia uses RON (Research Octane Rating) while in the US it is (R+M)/2 (an average of RON and MON). In Australia we had 91, 95, 98 (plus you could get a 100 racing grade up until recently). Here in the US it's 87, 89, 91.

The actual process of how you buy petrol is different too. In Australia you put the nozzle in and start pumping petrol straight away. You can either fill a full tank or use one of the presets. Then you go inside and pay to a person. In US you can pay to a person too but the usual way is to swipe your card right at the pump. And the biggest surprise - a requirement to enter your postcode (at the pump!) when using a credit card (even when using a debit card in a credit card mode). Apparently this is to reduce the amounts of fraud and stop people from using stolen credit cards but I found this unusual. For a debit card you will just be asked for a pin number.

Driving style had its own share of surprises. It is very common here for drivers to change lanes and turn without signalling. They just turn. Speeding is another common issue. As an example - the speed limit on the freeways is 65mph but hardly anyone drives at that speed. I estimate that on average people will exceed this limit by 10-15mph (driving at ~75mph or ~120km/h).

The Stop signs are there but the vast majority of people don't stop. They just slow down and roll through. This is called a "California stop".

Another mildly confusing finding was how the right of way is implemented in California when crossing an intersection. The rules are similar with one exception: "yield to the vehicle or bicycle that arrives first". This is so arbitrary and quite confusing to me. I am used to certain road rules. And here you need to pay attention to who arrives to the intersection first (even by a second or two) because some of the car moves may surprise you. People may turn in front of you even if you are driving straight.

I liked the idea of slip lanes in Australia that allow a car to turn without entering the intersection. Slip lanes do exist here in US but they are not very common. Instead, there is a rule that allows cars to turn right on the red signal. So far I have to overcome a sort of psychological barrier every time I execute a turn like this.

Carpool lanes - or officially high-occupancy vehicle (HOV) lanes - are very similar to transit lanes (T2, T3) in Australia. You can use these lanes during the certain (peak) hours if you have 2 or more people in your car. And unlike Australia, there are special overhead cameras that monitor all passing cars. Another difference is that there is also a possibility to pay a fee to use these lanes if you travel alone and require faster commute (similar to the paid roads with the eTag). And local version of the eTag is called FasTrack.

When I first saw Australian roads I was pleasantly surprised by the quality of the road surface and the surrounding infrastructure. I can't say the same thing about the roads in California. The road surface on the major freeways is uneven, with patches and cracks. And there is rubbish on the sides of these roads.

Many people use an app called Waze to navigate around. It's based on a crowd sourced model with the actual drivers providing updates about police sightings, objects on the road and various other hazards.

The fuel economy is measured in miles per gallon (MPG), which is an inverse approach to the usual litres per 100km.

And speaking about "strange" measurement units - cars' power is measured in horse powers (HP) - not kilowatts but it was OK for me because the same unit is used in Russia. But "pound foot" (lb·ft) used to measure torque instead of Newton meters is a complete mystery for me. I don't "feel" these values.

I guess that's enough talking about cars.

Let's talk about finances. Credit rating(s) is such a visible and important part of one's financial life. These ratings do exist in Australia too and are used to assess the borrowing power (creditworthiness), various risks etc but it's all kind of hidden. But not in the US. There are credit bureau agencies that keep track of your credit history, there are apps that can display your credit score, every time you do anything remotely related to money/finances you can be sure your credit score will be examined at that point. This leads to situations, where people actively working on improving their score. This also created a bizarre (in my view) type of a credit card called "secured credit card". In Australia usually you can have a bank (debit) card (where you use your own money) or a credit card (where you use bank's money for a period of time for free - 44 or 55 days - and then you need to repay money back or you are going to incur some interest on the amount owed to the bank). Both types do exist in the US too but you cannot get a proper credit card if your credit score is low. A secured credit card can be used in this situations as a way to repair/improve your credit score. It uses your own money under the hood but acts as a credit card. E.g. you can put a $500 deposit and the bank will issue you a secured credit card with the same amount allocated to your "credit line". So how is it different to a standard debit card (which also can act as a VISA card)? Apparently the difference is - when you use your  debit cards it only affects you and your bank. It helps building your relationship with the bank but this is where it stops. With the secured credit card your activity (late or on-time repayments, the amount owed etc) is fed/reported to the credit bureaus, which directly affects your credit score. So the theory is that if you need to improve your credit score then it's a low risk for the bank to issue you such card (afterall it's your own money) but all sensitive operations are tracked and it allows you to demonstrate that you are sensible type when it comes to managing finances and ultimately improves your creditworthiness.

Well, this is it so far. I will continue writing about my US experiences as I tackle and learn new things.

Stay tuned!

What is wrong with this code?

Every now and then I run training sessions for the dev teams, where we go through the small code samples and I ask audience a question - "what's wrong with this piece of code?". Sometimes it could be a trivial security blunder that leads to a SQL injection or XSS. In other cases the answer could be less obvious (e.g. a security weakness). And sometimes the question should really be "What CAN go wrong with this code" depending on certain implementation details. The purpose of these training sessions is to raise security awareness by demonstrating real security issues that I came across during my career (originating from multiple sources - code reviews, bug bounties, social media etc)

I would like to share these samples with you and hopefully together we can make Internet a safer place. Please let me know if you decide to use any of these samples as part of your own training sessions - I'd be very keen to know how it goes and to receive any feedback.

Note for security professionals - these examples are very simple. They are not meant to be hard, they are just a starting point and in most cases I seek a nearly immediate response from the audience.

"Forewarned is forearmed!"

Question 1

"SELECT ItemID, CONVERT(varchar(20), SubmitDate ,6) As SubmitDate FROM tblItem
WHERE LoginID = " & CStr(getCookie("Myapp", "iUserID")) & "
ORDER BY SubmitDate desc"

Answer 1

Potential SQL injection via getCookie("Myapp", "iUserID"). Cookies = untrusted input. Avoid constructing dynamic SQL statements (string concatenation) - this coding style often leads to SQL injections.

Also it looks like iUserID is an Integer. I always recommend (where possible) to constrain input for length, range, format, and expected data type (i.e. where we know expected type upfront). By wrapping getCookie() in either CInt() or CLng() before feeding it into the SQL statement we can essentially eliminate the risk of SQL injection. 

A second issue to consider (especially if iUserID is an integer) is that it might be possible to supply another user ID and bypass security controls to gain access to someone else's data.

---

Question 2

Code behind:
if(Request.QueryString.Get("ver") != null)
  ItemVersion.Text = Request.QueryString.Get("ver").ToString();
Page:
<asp:label id="ItemVersion" runat="server"></asp:label>

Answer 2

XSS (cross-site scripting) via the "ver" parameter.
Label.Text is unfortunately unsafe - by default it sets HTML markup. An attacker can supply a value/payload similar to this: ver=<marquee>xss</marquee> or ver=<script>alert(1)</script>

---

Question 3

From /myapp/logout.asp
' Get User ID. if found then log the user out of My App
if getCookie("Myapp", "iUserID")<>"" then
  [skipped]
  Set mySession = Server.CreateObject("Myapp.Session")
  mySession.SessionID = getCookie("Myapp", "SessionID")
  if mySession.Delete(connStrDB) then
   writeCookie "Myapp", "", "iUserId"
   writeCookie "Myapp", "", "sUserName"
   Response.Buffer
   SafeRedirect "/myapp/logout.asp?Success=True"
   Response.End
  [skipped]
  end if
end if

Answer 3

It is possible to cause a denial of service - to log out any session. getCookie("Myapp", "SessionID") is untrusted input and the code never checks that this SessionID actually belongs to this user (iUserID). If session IDs are easy to guess (e.g. integers) then it is trivial to iterate through and log out all active users.

---

Question 4

sUploadedFileName = Mid(fUpload.UserFilename, InstrRev(fUpload.UserFilename, "\") + 1)
If InStr(sUploadedFileName, ".gif") <= 0 And Instr(sUploadedFileName, ".jpg") <= 0 Then
            DisplayErrorPage "Only accept GIF or JPEG files, please try again"
            Response.End
End If

Answer 4

The idea is to allow uploading only *.gif and *.jpg files. But it is possible to upload any file as long as it has ".jpg" or ".gif" somewhere in the name. E.g. MyEvilFile.jpg.SomeOtherText.MyExt

Weak validation of this kind often leads to hackers being able to upload executable files (or shells)  - especially if uploaded "images" are accessible from the web (i.e. the Upload directory is under the web root) - e.g. http://mysite.com/UploadedImages/MyEvilFile.jpg.php

---

Question 5

sCustMediaDir = sCustHomeRoot & "\" & Replace(getCookie("Myapp", "sCustName"), " ", "_") & "\media\"

sNewFileName = sCustMediaDir & sUploadedFileName

fUpload.Form("dlgFile").SaveAs sNewFileName

Answer 5

The problem is in the way how we construct the sCustMediaDir string. sCustName is untrusted input and it can contain anything (including paths like "..\..\mypath\"). As a minimum this bug allows the attacker to rewrite files that belong to other customers (by changing the sCustName cookie value to "..\customer2"). Also if file system permissions for the web application user are weak and allow writing outside of the sCustHomeRoot directory then it could be possible to create or overwrite other files on this drive (the "\media\" part will make it less useful though)

---

Question 6

'if website address contains http:// strip it out
if inStr(strWebsiteAddress, "http://") > 0 then
                strWebsiteAddress = Right(strWebsiteAddress, len(strWebsiteAddress) - 7)
End if

Answer 6

This attempt to strip out "http://" can be bypassed by supplying "http://http://mysite.com"
It is also worth noting that "https://" is not stripped out and potentially can be used as a bypass too.

---

Question 7

http://mysite.com/embed.aspx?frameSrc=/mypath/campaign1.htm

inside embed.aspx
<iframe id="theFrame" name="theFrame" width="780" scrolling="auto" frameborder="no" border="0" scrolling="no" src="<% Response.Write(System.Web.HttpUtility.UrlEncode(Request.QueryString["frameSrc"])); %>" ></iframe>

Answer 7

An attempt is made to load local content (relative path) into an iframe. Unfortunately a developer here forgot that the value in frameSrc is untrusted and can be controlled by the attacker.

E.g. we can supply an external malicious page http://mysite.com/embed.aspx?frameSrc=//www.externalevilsite.com/evilpage.php which will be rendered/embedded into the web page. This approach can be leveraged in phishing campaigns etc

I would recommend to avoid referencing pages by name/URL directly and instead have a whitelist or a resource map, where each allowed page should have a corresponding ID associated with it.

E.g. if we have an internal map matching /mypath/campaign1.htm to "myresource123" then we can request it i na safe way as http://mysite.com/embed.aspx?resourceID=myresource123

---

Question 8

isAdministrator = (getCookie("Myapp",  "iEditedBy") = "1")

Answer 8

A classic insecure cookie handling vulnerability. The presence of a cookie iEditedBy with a value of "1" means you are an admin! And apparently these types of issues are quite common.

---

Question 9

' check the incoming remote address

if 0 < Instr(Request.ServerVariables("REMOTE_ADDR"), "10.11.12") then
 ' all OK
 [skipped]
else
 ' not allowed
 errorMessage = ERROR_PREFIX & " IP address is not allowed"
 displayErrorMessage(errorMessage)
End if

Answer 9

An attempt is made to only allow IP addresses from the 10.11.12.0/24 range (from 10.11.12.0 to 10.11.12.255). Unfortunately the way this filtering is implemented will also allow IP addresses that follow this pattern: xxx.10.11.12, which is most likely undesired.

---

Question 10

http://www.somesite.com/Search.asp?query=SELECT+cname%2C+sname%2C+description%2C+pid%2C+picture+FROM+tblCategory+c%2C+tblSubcategory+s%2C+tblItem+i+WHERE+i.cid%3D5+AND+i.sid%3D31+AND+i.cid%3Dc.id+AND+i.sid%3Ds.id+ORDER+BY+cname%2C+sname%2C+description%2C+pid

Answer 10

This is just for the giggles. But this is a real example (real web site) that I came across a few years ago. I wouldn't even call it a SQL injection. It is more than that. These guys allow anyone to execute any SQL statement on their web site. And surprisingly they are not alone. I see this approach time and time again in the old ASP and PHP based web sites.

In fact, you can run a Google search (aka Google dork) similar to this one to see what I mean: inurl:"query="+inurl:SELECT+inurl:FROM+inurl:WHERE+inurl:"order by"

Another web site had 2 separate parameters for the "where" clause and the "order by" part of the query but the "idea" remains the same:

www.somesite.com/mypath?where_clause=+item_status_id+in+(select+status_id+from+item_statuses+where+category_id=1)+and+item_type_id+in+(1,2)&order_by_clause=ORDER+BY+create_date+DESC

---

Question 11

       customerIDs = Convert.ToString(Request.QueryString[CustomerIDParam]);
       if (customerIDs == null || !Regex.IsMatch(customerIDs, "[0-9,]+"))

Answer 11

The weakness is that this RegEx checks that we have digits or commas but it doesn’t prevent an attacker from entering other characters (like an apostrophe as an example) as long as there IS at least one digit or comma.

Once accepted these values are fed into a SQL query. A defence in depth principle dictates that we should try to defend our systems at each level. The application level is certainly capable of performing some input parameter validation. In order to fix this particular weakness we can make a regex tighter:

       customerIDs = Convert.ToString(Request.QueryString[CustomerIDParam]);
       if (customerIDs == null || !Regex.IsMatch(customerIDs, "^[0-9,]+$"))

---

Question 12

www.somesite.com/mypath?stdTextCol=&linkTextCol=&p1imageLoc=http://db1.img.somesite.com/i/123.jpg&p1thumbImageLoc=http://db2abcde01:83/i/456.jpg

Answer 12

I can see several potential issues with this request URL:

1. We are allowing images to be loaded from a different domain (p1imageLoc and p1thumbImageLoc parameters). An attacker can supply their image to alter the look of the web site (and potentially use this in a phishing style attack)
2. Look at the difference in the way how image server is specified in p1imageLoc and p1thumbImageLoc. In the first case this is just a normal domain name but in the second case we see an internal server name followed by a non-standard port (!!!). What happens next really depends on the implementation.

As a minimum we leak information that an attacker might find useful (as part of their reconnaissance effort). They now know that there is an internal server called db2abcde01 that runs a web service on port 83.

But the situation can actually be worse. This URL can potentially give an attacker a leg into the internal network (again - depending on how much information is actually returned back to the attacker - e.g. as part of the detailed error messages).

E.g. an attacker may try to perform a port scan by iteration through the port numbers (83, 84, 85 etc)

Or try a different URI scheme (ftp://, file://, telnet:// or even svn:// ;) )

Or execute an admin request (db2abcde01:83/admin/SensitiveOperation) - either unauthenticated by themselves or embedding this URL somewhere waiting for a logged in person with admin privileges to inadvertently execute this request.

Or an attacker may try to find other servers on the internal network. What if they try db2abcde02? Or SuperSecretServer01?  

Final words

12 questions should be enough for the first blog post of the series. I've got a lot more examples and I am sure this is not the last post of this kind. I might even try something different next time. I can post just questions (avoiding the most trivial ones) and let the audience try their "hacker" thinking and then publish the answers a week later. What do you think?

The case of slow API connections and TCP retransmission

For years I've been a big fan of Mark Russinovich's "The case of" blog posts. So I decided to do a similar post this time. A couple of months ago my team was troubleshooting an issue related to slow responses from a 3rd party API. This particular API is located in the US and our code runs in Australia. Typically we saw response times of a few hundreds of milliseconds (which includes time to establish connection, round-trip to a different continent and back plus processing time). Everything worked well until suddenly one day our monitoring systems picked up a significant increase in request processing time. It looked like this:

message time	elapsed time (ms)
12/18/2015 10:15:31.938 +1100	10085
12/18/2015 10:15:24.107 +1100	10114
12/18/2015 10:15:17.490 +1100	9924
12/18/2015 10:15:11.704 +1100	9991
12/18/2015 10:15:05.796 +1100	9953
12/18/2015 10:14:50.723 +1100	9964
12/18/2015 10:14:49.815 +1100	9911
12/18/2015 10:14:40.021 +1100	10140
12/18/2015 10:14:29.147 +1100	10151
12/18/2015 10:14:28.646 +1100	9937

Everything still worked fine but instead of sub-second responses we saw requests taking 9-10 seconds to complete. Further investigation was required. We performed the usual troubleshooting steps but still could not figure out what was going on there. We had to go deeper and deeper in our analysis - eventually all the way to the network packet capture. In fact, it's the packet capture that gave us the first hint of what the problem was. We saw a lot of the TCP retransmissions.

Two things were clear for us now.
Firstly, we noticed that only the SYN packets had delivery problems and had to be retransmitted. SYN packet is the first packet of a 3 packet "handshake" used to establish a TCP/IP connection. We saw that once the connection was established there were no more retransmissions during the session/data transfer.

Secondly, we could see where all those extra seconds were coming from!
See how there is a 3 seconds difference between the initial SYN packet (packet 22921) and the retransmission in line 23003 which is 3 seconds after (103.53… and then 106.54…)

And then we retransmit again 6 seconds later (packet 23087).
After that the connection is finally established but we’ve just lost 3+6=9 seconds during the TCP handshake.

Another interesting observation was that when we retransmit for the second time (packet 23087) we remove the ECN and CWR flags.

We performed several packet captures and it became clear that our SYN packets were not reaching  the destination and we had to retransmit them (or their SYN/ACK packets were not reaching us)

SYN packet retransmission (at least on Windows) by default works like this:

“The retransmission timer is initialized to three seconds when a TCP connection is established. However, it is adjusted on the fly to match the characteristics of the connection by using Smoothed Round Trip Time (SRTT) calculations as described in RFC793. The timer for a given segment is doubled after each retransmission of that segment. By using this algorithm, TCP tunes itself to the normal delay of a connection”

This is where we get 3 seconds (initial retransmission delay) plus 6 seconds (3 seconds doubled for the second retransmission).

Also given that “Max SYN Retransmissions” is set to 2, the system will only retransmit the SYN packet twice hence the ~9 seconds delay we see in the worst cases. The initial retransmission timer value is set in the "Initial RTO" parameter (see the screenshot below). To test this theory we decided to change this value from 3 seconds to 2 seconds:

This can be achieved by running this command:

netsh int tcp set global initialRto=2000

Once this change went live, straight away we saw request processing time decreasing from 9-10 seconds down to ~6 seconds. We knew we were on the right track.

Another suspicious finding (as mentioned above) was that most of the SYN packets with the ECN and CWR flags were dropped while SYN packets without these flags were going through.

ECN (Explicit Congestion Notification) is an interesting protocol extension defined in the RFC 3168. In the TCP/IP world the standard way for the receiver to "notify" sender of network congestion is to drop packets. This behaviour obviously can have a significant impact on the overall network performance. ECN (when supported and negotiated by both ends) allows signalling/notification of network congestion to happen without dropping packets.

Windows had ECN for TCP support since Windows Server 2008 and Vista (but it was disabled by default). But it is enabled in Windows 2012. (Linux passively supports ECN - will negotiate if asked by the other end)

ECN support has improved significantly since the introduction 15 years ago but apparently some issues still exist.

The next step for us was to try to disable ECN to see if this was the culprit.
ECN capability can be turned off by executing this command:

netsh int tcp set global ecncapability=disabled

Once this change was applied, all TCP Retransmissions disappeared and request processing time was back to a few hundred milliseconds.

We contacted the API vendor and they reassured us that their end had proper ECN support. The fact that not all of the SYN packets with ECN flag had this issue (but most of them), led us to believe that we saw a "Path-dependent connectivity dependency" as described on slide 6. This is also indirectly supported by the fact that some of the BGP routes changed roughly around the same time when we started experiencing this issue.

We were happy to see this issue resolved. Hope this blog post will help someone in a similar situation.

Keywords: Max SYN Retransmissions, maxsynretransmissions, slow connection, ECN, ecncapability, TCP retransmission

Resilience - Part 3 - The Aerospace Industry Approach

In Part 2 we've discussed SLAs. As an IT engineer I think of a 99.95% SLA (for a single instance) as a pretty good one. But you probably already know by now that I like to compare IT with the aerospace industry. As a passenger - would you be happy if say a flight computer in your jet was allowed to malfunction for ~21 minutes in a given month? I'd be scared. And our experience tells us that this is not the case in real world. So how do they achieve high availability in the aerospace industry?

Aerospace industry approach

I wanted to write about this for quite some time now. In fact, I've been thinking about it since May 2015 after reading a fascinating presentation by Peter Seiler and Bin Hu called "Design and Analysis of Safety Critical Systems". I reached out to Peter Seiler (Assistant Professor from the Department of Aerospace Engineering and Mechanics, University of Minnesota) and asked for permission to reuse some of the slides from this presentation. Thank you Peter!

Let's take Boeing 777-200 as an example (Boeing's first fly-by-wire aircraft). "Fly by wire" is defined by the Dictionary of Aeronautical Terms as: 

Fly-by-wire (FBW) is a system that replaces the conventional manual flight controls of an aircraft with an electronic interface. The movements of flight controls are converted to electronic signals transmitted by wires (hence the fly-by-wire term), and flight control computers determine how to move the actuators at each control surface to provide the ordered response. The fly-by-wire system also allows automatic signals sent by the aircraft's computers to perform functions without the pilot's input, as in systems that automatically help stabilize the aircraft, or prevent unsafe operation of the aircraft outside of its performance envelope.

To put it simply - pilots (their controls) are no longer directly connected to the control surfaces (ailerons, rudder etc). Instead pilot actions are sent to the computers, which then "move" the control surfaces accordingly. This arrangement obviously makes these flight control computers critical to the overall safety of the aircraft.

Consequently, the reliability requirements are very strict: less than 10^-9 catastrophic failures per hour.

Modern flight control systems are very complex but at their heart they have a simple classic feedback loop:

Simple indeed. But with just a single flight computer we will probably be only achieving availability similar to the cloud instances above - far cry from the required reliability targets.

What can we do to increase reliability? Based on the information from Part 1, we know that we can add redundant components to improve the fault tolerance of the system.

In IT world if we add one extra server we can have a 2-node cluster (with the active/active or active/passive arrangements). If we add more nodes then it can get more complicated. One of the approaches is the Majority Node Set. The triple modular redundancy approach is very common in the aerospace industry. By having 3 redundant components we arrive to the classic "Triplex" architecture:

Instead of one we have 3 independent components for each critical subsystem. These components are involved in the voting process to work out the correct result/action. 

But even this is not enough and aerospace engineers go further. The 1996 "Triple-Triple Redundant 777 Primary Flight Computer" paper by Y.C. (Bob) Yeh describes 5 principles of Boeing's 777 FBW (fly by wire) design philosophy/design constraints related to safety:

1. Common Mode/Common Area Faults
2. Separation of FBW Components
3. FBW Functional Separation
4. Dissimilarity
5. FBW Effect on Structure

This is where we see the triple modular redundancy evolving into the triple-triple architecture. We have 3 similar/identical channels (left, centre, right)...

For obvious reasons physical electrical wires for different channels will be located as far from each other as possible (to satisfy the second "separation of components" principle).

... and 3 dissimilar lanes in each channel (one in command, the other 2 functioning as monitors). 

I was fascinated by the dissimilarity principle. There were many methods used in 777 architecture to satisfy this principle but the IT engineer in me was really impressed by this particular approach:

Dissimilar Microprocessor and Compilers (with Common software)

Or quoting [Yeh, 96]:

The microprocessors are considered to be the most complex hardware devices. The INTEL 80486, Motorola 68040 and AMD 29050 microprocessors were selected for the PFCs (Primary Flight Computers - DK). The dissimilar microprocessors lead to dissimilar interface hardware circuitries and dissimilar ADA compilers.

How cool is that?!!!

Intel 80486 (that powered PCs around the world in the 90s), 68040 (Macintosh Quadra 700 anyone?)... ah, memories!

So the designers selected 3 different CPU architectures. This means 3 different versions of machine code. So we need 3 different ADA compilers for each platform to provide triple dissimilarity. Wow... What an incredible level of assurance this approach provides!

When I was reading about this approach I was also contemplating an idea of having 3 independent groups of software developers implementing the same project requirements to avoid (or at least reduce the probability of) producing the same bugs... 

Anyway, I hope you enjoyed this overview. While researching this topic I have certainly felt a lot of respect to aerospace designers, architects, and engineers. They produce highly reliable systems that make it safe for all of us to fly. And we (IT people) can certainly learn a few tricks there (especially in the mission critical systems).

Resilience - Part 2 - SLAs explained

SLAs

In Part 1 we've covered the basics. Now let's talk about the real world situations. In IT world we often talk about Service Level Agreements (SLAs).

An SLA is an agreement/document that describes the expected level of service (including specific metrics used to measure provided quality of service and potentially penalties for not meeting the expectations i.e. not achieving agreed levels of service).

SLAs can be both formal (e.g. between an external 3rd party service provider and a client) and informal (e.g. between 2 internal departments or teams within the organisation)

Service providers usually have an option to choose different levels of service quality/uptime. It is natural for customers to expect to pay more for higher levels of system availability.

How do we specify SLAs?

E.g. you might get a server running in a data centre and your hosting provider will promise a 99.9 SLA. This is a typical SLA for a single server setup. But what does it mean? How reliable is this server going to be? "99.9" ("three nines") means that the hosting provider guarantees that this server will be up and running (i.e. will be available) 99.9% of the time. 

If we take a "standard" month that consists of 30 days then all these "nines" can be translated in real terms of downtime as:

SLA	Downtime
99% (two nines)	7 hours 12 minutes
99.9% (three nines)	43 minutes 12 seconds
99.95%	21 minutes 36 seconds
99.99% (four nines)	4 minutes 19 seconds
99.999% (five nines)	26 seconds
99.9999% (six nines)	3 seconds

You can use a very convenient uptime calculator if you want to experiment with some other numbers.

To give you a few examples let's see what some of the most popular cloud providers commit to. For simplicity let's check the SLAs for single instances/VMs

	AWS	Azure
Service commitment	99.95% during any monthly billing cycle	99.95%
Service credit	<99.95% - 10% <99% - 30%	<99.95% - 10% <99% - 25%
The actual SLA	AWS SLA	Azure VM SLA

As a side note - it is also interesting to note how AWS and Azure define "downtime" or being "unavailable".

AWS: 

"Unavailable" and "Unavailability" mean:
For Amazon EC2, when all of your running instances have no external connectivity.

Azure:

Downtime - The total accumulated minutes that are part of Maximum Available Minutes that have no External Connectivity.

So both vendors define being unavailable as having no external connectivity.

I'd like to mention another consideration that I was made aware of while visiting Telstra's GSOC in Melbourne. Imagine if a telco dropped just 1 packet in a whole month. So a particular client just hasn't received one single packet. The telco might think their availability was nearly 100% for that month. But from the client's perspective this same situation may result in a very different outcome. Some (especially old legacy) systems cannot tolerate a single packet loss and enter the error state. In order to recover, engineers on the client site might be forced to go through an hour long process of restarting their systems in a predefined order to recover from this failure. Just think about it - a single lost packet can cause an hour long outage on the client side (straight away - this client won't be able to achieve a 99.9% SLA for that month). This may sound like an extreme case but trust me - these things do happen in the real world.

Defence in Depth - Use cloud based security solutions

I have recently attended the Akamai Edge conference in Miami. I was invited by Akamai (which sponsored my trip) to participate in the Cloud Security CAB (client advisory board) and to take part in one of the panel discussions. I have really enjoyed my time in Miami - thank you Akamai!

I decided to write a quick blog post based on my notes, that I used to prepare for the "Application Security Multi-Layer Defense" panel discussion.

First of all - you need to know what assets you are trying to protect. This may sound trivial but we really don't want to miss anything. Imagine having a 3 years old unpatched CMS server somewhere in the "far corner" of your environment that you are not aware of. Or (as an alternative) a web site, deployed by the Marketing team somewhere in the cloud (credit card purchase/shadow IT anyone?). This won't end up well.

Application security is not an end goal. You can't just tick a box. It's an ongoing process.

I like approaching web application security from the defence in depth perspective. We all know the egg analogy - being hard on the outside, but once this first shell layer is penetrated - it's all soft and squishy inside. Good security means being like a swamp, where it is harder and harder to take the next step, so that the attacker eventually gives up.

Defence in depth means that we wrap our application in multiple protective layers. Once a new layer is introduced, we assess the residual risk to see whether we satisfied business requirements. A good practice is to keep the bad guys as far away from your core systems as possible. This means that the outer layers should be quite broad but shallow. They should remove most of the noise. Usually they are application agnostic. This is like a funnel - we become more application specific as we move to the inner protective layers.

If you've seen online attack monitors (like the one from Norse) you may have noticed that there are a lot of different attacks happening "in the wild". What I usually do when I talk about this - is keep the monitor running for a few minutes to collect some stats. Here's a random sample that I've collected a few minutes ago:

You can see the origins and destinations of the attacks. But most importantly you can see which ports are being attacked most. Telnet (port 23) leads with the 2,200+ hits and so on. But the fascinating thing for me is that HTTP (port 80) is only at the 9th place (with just 35 hits) and HTTPS (port 443) is not even in Top 10. Obviously different samples will have slightly different distributions but the overall picture is always the same. There is a lot of "rubbish" packets hitting public IP addresses. But under "normal" circumstances HTTP/HTTPS attack traffic only constitutes a small percentage of the overall noise. So it makes sense to stop all/most of this noise at the perimeter - as far away as possible from your environment.

This is what cloud based security solutions (e.g. cloud based WAF) allow you to achieve with ease by providing an additional protective layer for your environment. By only sending us traffic on ports 80 and 443 and stopping EVERYTHING ELSE at the perimeter, cloud based solutions provide an extremely efficient way of reducing the noise/malicious traffic hitting your servers. In the example above - only 35 HTTP requests (in the worst case - if we don't block any of them) would've been passed through to your servers. The rest (a couple of thousands of malicious packets) would've been stopped at the far reaches.

There are multiple benefits - your own firewalls will have some free capacity as they won't need to deal with these extra packets. Your internet link will have more spare bandwidth as it won't be occupied by the malicious packets (and you may even pay less traffic charges). Also cloud based solutions usually can absorb/defend against significantly larger volumetric attacks (think DDoS). Attacks generating several hundreds of gigabits/sec are becoming more common. Not that many companies can afford to have that much free internet capacity to withstand such attack.

There are several players in this market. Do your homework, choose vendor/solution that meets your objectives and add the cloud based security solution as a defensive layer for your environment/application.

Aussie banks security (Login form edition)

Introduction

You may have seen my previous blog post from the "Aussie banks security" series, where we analysed the situation with the security related HTTP headers. What prompted me to write this new post was the fact that I have recently read a couple of articles about password managers and certain design considerations for web sites (especially around UI design and the underlying HTML layout) to support the use of password managers.

One of the advantages of using password managers is that you can select much longer and stronger passwords. E.g. there is nothing wrong in having a password 100 characters long. This is a great measure against brute force attacks against your password in cases when a corresponding membership database is compromised. But in order to do so the web site should actually be able to accept long passwords.

One of the common issues is web sites applying strict limits on the maximum number of characters that a user can enter into the password field:

<input type="password" size="10" maxlength="6" name="authKey" autocomplete="off">

In the case above a UI will limit the maximum possible password length to just 6 characters.

So I decided to use the same Aussie banks test set to check what they do with their login forms. Are they ready to support password managers? Are there any other interesting observations we can make by looking at the login forms' HTML? And of course it will be interesting to come up with a new rating - who is going to be at the top this time?

Methodology

• I decided to focus on the internet banking login forms (for private customers if there was a distinction between private and business customers)
• I have captured 3-4 lines of raw HTML for each bank (the form tag itself and the username/password input fields)

Something that looked similar to this:

<form name="loginform" action="/daib/processlogon.asp" method="post" autocomplete="OFF">

<input type="text" name="mn" value="">

<input type="password" name="pwd" value="" maxlength="16">

• I have also tried to click the submit buttons as in several cases the error messages were quite descriptive and revealed useful additional information about the accepted data quality, which in turn has allowed me to further clarify and improve my findings.

Here is an example of such error message:

• I put everything into the spreadsheet and analysed the raw data
• For the purpose of ranking I only used information related to passwords (but I will provide additional observations related to other fields too)

For ranking/scoring I used the following approach:

• No password length limitations - green
• Password length up to 16 characters - yellow
• Max password length between 8 and 12 characters - orange
• And there is a special category reserved for the banks that force max password length to be precisely 6 characters - red

Before we proceed to the results, I just want to make it absolutely clear - this is not a "hacking" exercise of some kind. All information is publicly available in HTML or in the error message supplied back by the systems after clicking the login button.

Results

Bank	Username	Password	Autocomplete	Comment
AMP	No limits	No limits	Off	-
Bank of Queensland	1000	No limits	Off (both form and elements)	Passwords (personal access codes) are case-sensitive
Beyond	No limits	No limits (min 6)	Off (both form and elements)	Username: member number - digits only
Greater	12	No limits	-	-
Heritage	2-16	No limits	Off (on the form elements)	Username: member number - digits only (2-16). Passwords can be be typed in manually or entered via a virtual keyboard. Virtual keyboard only contains upper-case letters and digits
P&N	No limits	No limits	Off (on the form elements)	Password entered via keypad (proper size with digits,upper/lower case/special characters)
People's Choice Credit Union	No limits	No limits (client side), min 4 server-side	Off	No limitation on the client side. Server side requires: member number - digits only, password min 4 characters
Suncorp	10	No limits	Off (on the form elements)	Username: customer id, digits only. Additional security token - digits only
Teachers Mutual	No limits	No limits	Off	Passwords submitted as SHA1 hashes calculated on the client
ANZ	19	No limits (in HTML). Client side validation requires 8-16	Off (on the form elements)	No password limitation on the form but a pop up warning says 8-16 chars, needs to contain 1 digit and 1 letter. Client side validation won't allow to submit otherwise
Bank West	8	16	Off	Username (PAN) is digits only [0-9]*
Bankaust	No limits	16	Off (both form and elements)	Username: client number - only digits
Commonwealth Bank	8	6-16	Off	Username: client number (8 digits). Password 6-16
IMB	9	8-16	Off (both form and elements)	Passwords are case-sensitive
Bendigo Bank	12	8	Off (on the form elements)	Username: Access Id - digits only. Can support 2FA (6 digits authentication key)
Macquarie	8	8	Off (both form and password element)	Username: Macquaries access code (MAC)
St George	16	12	Off	Card number: 16 digits auto-separated by dashes (19 chars). Additional "security number" field (6 digits).
CUA	16	6-6	Off (on the form elements)	Password is called WAC (exactly 6 digits on the numpad)
ING Direct	8	6-6	Off	Username: client number. Password: Access code on the numpad, 8 places in UI but only accepts 6 digits.
Newcastle Permanent	8	6-6	Off (both form and elements)	Password (access code) is precisely 6 digits. Password submitted encrypted (?) (txtPassword_RSA)
Westpac	12	6-6	Off (both form and elements)	Password (via keypad) - can only be 6 chars. Only 0-9,a-z

Observations

• Only 10 out of 21 banks from this list don't appear to apply password length limitations, making them properly compatible with the password managers. It doesn't mean that you can't use password managers for other banks but the potential quality of passwords will suffer as a result.
• Most of the banks in the "red" category not only limit passwords to precisely 6 characters, some of them go even further and only allow digits in the password field. In 2015 this is just terrible.
• All but 1 banks turn autocomplete off (either on the form itself or on the individual elements). This is interesting because many modern browsers ignore this setting for the login fields.
• Many banks utilise numeric logins (user names). They give them different names (member number, client number, client id, card number etc) but in the essence it is a relatively short sequence of digits. I was surprised by this fact. "Integer" user names are quite weak (predictable, can be iterated through etc)
• Several banks had a 3rd field available for a two-factor authentication process (2FA - tokens etc). I know that some other banks do support 2FA too but it is great to see this functionality on the page, meaning that more people will be made aware of it and hopefully this will lead to the higher adoption rates.
• Some banks use virtual keyboards for entering passwords. Again the terminology varies (numpad, keypad, virtual keyboard etc). Generally speaking this is a better option as it makes it harder for some banking trojans to steal passwords. But I didn't like the fact that some virtual keyboards had a severe negative impact on password quality.

Compare the following examples:

The first one only allows entering passwords that consists of uppercase letters, digits and a space character. This limitation significantly reduces password quality! While the second keyboard provides a much wider variety (uppercase/lowercase, digits, a few special characters).

Or consider the following example:

Only digits and uppercase letters. Combine it with the fact that this bank limits all passwords to precisely 6 characters and you have a situation where a login process is quite weak.

• Special cases - client side processing to avoid submitting passwords in clear text

To be honest I didn't expect that. I spotted a couple of cases where client side javascript was used to "transform" clear text passwords to another representation before submitting it to the server (there might be more - I haven't paid much attention to the way how passwords were submitted to the server as long as it was done via a secure connection).

In one case I think it was a form of RSA encryption and in another case it was a SHA-1 hash.

The SHA-1 example is an interesting one. So the application takes a SHA-1 hash (no salt) of a user supplied password and submits it to the server as part of the login process instead of a clear text original. My initial reaction was "Oh, gee, this is bad!". Indeed, by sending a SHA-1 hash to the application means that:

• The application has to store/rely on this SHA-1 hash (and SHA-1 gets easier to attack/brute force these days)
• It will be very difficult to change the hashing algorithm if they decide to do so (as there will be nothing to compare the new hash against on the server end)
• An attacker knows the length (160bit or 20bytes) and format of every password in the system (essentially a string that consists of 40 hex-digits)

On the positive side - clear text passwords don't traverse the wire.

But then I thought - what they've done is transformed ANY password to a 40 character string (0-9,a-f). Depending on how bad the original password was, this may actually improve password quality for some of them. The hash itself doesn't have to be stored as is. Instead they might use another form of hashing (or encryption) to store those user supplied hashes on the server end. We just don't know. But I still don't like the idea. I'd be keen to know what other security professionals think about this approach.