Practical HTTP Header Smuggling: Sneaking Past Reverse Proxies to Attack AWS and Beyond

[Music] hello and welcome to this session on practical http header smartplane where we're going to look at some of the ways we can sneak http request headers past reverse proxies and some of the fun vulnerability exploitation we can get as a result of that but before we jump into that we need to have the mandatory who am i side so hello i'm daniel thatcher i work at a company called intruder we're a cloud-based vulnerability scanner focused on making continuous monitoring easy i work there as a researcher as a penetration tester as well as providing some of the security expertise for the platform a lot of my research is focused on black box scenarios as these are the ones i find myself working most often and to me they tend to be the most interesting i've also got some links to our social media and our research blogs which you will be available at the end of the presentation as well so what are we going to talk about today we're going to start a review of a review of the model that we're using to describe a modern worm applications architecture we're then going to move on to discussing what is actually header smuggling some talk about the detection of header smuggling and then a look at some applications finally we'll finish off talking about how to defend against head smuggling and some conclusions and talk of where the research can go in the future so how is a modern web application set up well we're going to model it in this talk as a front-end and a back-end server the front-end server is typically responsible for duties such as caching load balancing or acting as a web application firewall whereas the backend server is what actually runs the application's code and logic is responsible for accessing all the data of the application now it's really important to mention that this is just a model and in most scenarios including the ones we'll see today it's actually a massive oversimplification of reality however this model allows us to understand a lot of behavior we see and allows us to move forward with exploitation by assuming the setup is like this but if you find yourself in a situation where you're trying to understand some behavior this is an assumption worth challenging as usually the front-end servers could be a chain of more servers they could be multiple back-end servers or the setup could be more complex than this in ways you couldn't even imagine now generally front-end servers use http headers to pass information to back-end servers a great example of this is the exported four-header which front-end servers typically put the client ip address in so the back end servers can know the initial ip address from which the request originated this makes it quite important for front-end servers to actually filter out headers such as these from the client's request otherwise as a user i'd be able to put whatever value i wanted into the exported four header and spoof my ip address so if the information these headers is to be trusted the front end server needs to validate it and filter some information out if it was provided by the clients we also get some interesting situations when the servers disagree about either the value or the existence of a header so one we've seen a lot of in recent years is request smuggling when servers disagree about either the value or the existence the content length and transfer encoding headers this can often lead to request smuggling similarly we'll also look at the situation in this talk where this leads to cash poisoning when two servers disagree on the value of the header now with that in mind what actually is header smuggling simply put it's the process of disguising a header so that it will be passed differently by different web servers and in this research we're going to focus on the situation where the front-end server misses a header or doesn't pass it in the way that the back-end server does and then the backend server passes it as we want it to without the front end server modifying it how do we discuss headers well we're doing what i terms of mutation which is simply a modification you can make to a header to cause different servers to pass it differently so some examples of mutations include the identity mutation where we do nothing to the header so this x my header test header here will look pretty much as you'd expect a standard 80 to be headed look however if you use underscores in header names it's quite common for some servers to process it as it was the x my header test header we had on the last slide whereas some will see it as a different header similarly when we see a lot in request smuggling is putting white space in places around the header for example here before the column and it's worth noting at this point that a lot of these mutations come from request smuggling research as you may have recognized already another one we see quite commonly is white space at the start of the header and one we're going to see a lot of in this talk is including some random junk in the header name after a space so some servers will pass this header as a header called x my header abcd whereas others will just pass it as a header called x my header with a value of test so this research set out to find a methodology for detecting header smuggling and there were two key aims of this methodology the first is that it would identify mutations in a generic way where you find a mutation independently of a header so once a mutation has been found we can apply it to any header to sneak it through to the back end server without being recognized by the front-end server the second aim was that it would work in black box scenarios as i mentioned at the top of the talk these are the scenarios that i find myself in most often and personally for my research find most interesting so these are the ones i chose to focus on now i'm going to explain the methodology through the use of an example so the methodology works by testing one mutation at a time and trying to determine whether it can be used to achieve header smuggling so in this example we're looking at the mutation which adds an abcd after the space to the header name and we're going to see if the front-end server will process this as a header other than a header called x header while the back end will still see it as the x header header so to start off the detection process we use what may look like a fairly normal request the only thing that might stand out to you about slightly strange in this request is that it is a get request but it has a content length header in there and the value of the content length header is set to zero so we're actually not including a body now most servers are completely fine with this so as we can see here we get to 200 okay indicating the response we really expected for a get request to the home page the next step is to put an invalid value in that content length header and as we can see here unsurprisingly this causes an error this indicates that one of the servers in the chain is trying to pass the content length header and when it sees an invalid value such as the z here it's going to throw an error now we repeat this process with our mutated header so here again we're putting this content length value of zero in the mutated content length at this time and we get a response that we expect without any errors now when we put an invalid value in there we again get an error so we know that some server in the chain is passing this now there's a couple of key things to look at here to determine whether we have header smuggling the first is a comparison between our requests that set the content length to zero it's quite important that these set the same response and using this we can know that our mutated header hasn't been so malformed that one of the servers in the chain has rejected it or refused to pass the request completely every server has continued to pass the request as a valid http request and return the response we expect we also need to compare the errors now importantly here with our mutated unmutated header when we put an invalid value in them we get different errors and this suggests that the errors are coming from different servers which are erroring when they see an invalid value in the content length header so one of our servers is reading the content length unmutated header whereas another one it seems to be passing our mutated header now as we know that the server which is likely the front-end server which is passing the unmutated header is not throwing an error for our mutated header as we're getting a different error back this suggests that that second error of the mutated header is coming from a backend server which is reading this while the front end is ignoring that mutated header so we have a mutation which allows us to sneak a header through to the back end without the front end seeing it where do we get from here well sometimes we might know a header we want to test already and sneak through a great example of this is the exported for header where if we're coming across some ip restrictions this might be a great candidate to try and sneak through to the back end to bypass these restrictions however often we don't know a specific header which will be interesting for our target system and we just want to try guess as many headers as possible now for this in my research i've been using an updated version of james kettle's pram miner extension which i will be releasing as well as part of the research at the time of the conference what this updated version will do is it will search for mutations which allow us to sneak a header through to the back end and then with each of those mutations it finds it will try every header that paraminer will try anyway and see if it can trigger some interesting behavior so we're going to look at some examples now and the first one of these examples is in aws api gateway now api gateway is a service from aws which allows you to build http apis built on lambda functions which are aws version of serverless functions in essence you run the code for your function without having to maintain any of the infrastructure and you only pay for the execution time of your function now if you want to limit ip access to one of these functions or one of these apis you use what's called a resource policy which you apply to your api gateway service i have an example of the resource policy here which may be of interest to some of you more familiar with aws but the important part here is that only the ip addresses 1.2.3.4 and the range 100 8 are allowed to make requests to this service unfortunately i don't own the ip address 1.2.3.4 and can't send requests from it and in this example we're sending requests from outside the customers cloud account so there's no way i can send from that internal range either now if we set up an api with api gateway and apply this resource policy and send a request to it we unsurprisingly get a 403 forbidden because we're not requesting from an allowed ip address a perhaps slightly naive approach might be to just include the exported 4 header in our request with an ip address on the white list and unsurprisingly this doesn't bypass aws's restrictions would be nice and it's always worth testing but i don't think that's going to come as a shock to anyone however through some testing i discovered that i could sneak headers through to the back end by using this mutation of adding some junk after a space in the header name and the back end would still pass it as though that space and training characters weren't there so i tried the following requests and i got through to my api bypassing all the restrictions so that's quite a nice start but often we won't actually know an ip address which is allowed through to the api we might be able to work it out but we'll be blind guessing quite a bit and that's quite difficult so i moved on to trying something perhaps more stupid but putting an imbalanced value in there that can't be passed as an ip address in this case a letter z and quite fortunately this also worked so to summarize that we could bypass bypass ip restrictions in api gateway resource policies by adding this header and the slide to the request another similar example that i have also reported to aws was incognito which is used for identity management now cognito on a penetration test that i was working on their cognito instance was limiting access to the clients whereas rate limiting you so if you send too many requests i believe was five within a short period of time your ip address will get blocked for a while however i've discovered that if you add the x folded four header with that's a vertical tab after the colon you can see in the slide there and then any other value your you would no longer be blocked and you'd get five more requests it's important to note here that you couldn't cycle the bank of this header including valid ip addresses and you wouldn't get any more requests so this is a very minor bug that takes your limit from five requests to 10 total within a short period of time however i feel it's quite a nice demo of what can be done and even though the impact is minimal i think it's worth mentioning just as an example another example we're going to look at today is cash poisoning so i reported the issue with bypassing the resource policy ipa address restrictions to aws and they fairly promptly fixed it however i noticed that i could still sneak headers through to back-end servers using that same mutation of adding a space and some trading characters so i started to think about what else could i add here that could be interesting now there's probably a lot of headers that turn onto api gateway that would be interesting if you knew them but i couldn't find any however one that perhaps is quite interesting is the host header as it's a bit more fundamental to the request so i set up two apis to test with the first one you can see here is this victim api and a request to slash a on this will just give you real response a the second api is this attacker api and for reasons we'll see in a second this responds with poisoned a when you request it so the question is what happens when i send the following request here we have two host headers an unmutated one which is for the victim api and the mutated header which is for the attacker api well it turns out api gateway will give us the response for the attacker api this poisoned a response so this becomes interesting when we introduce a cache to test i set up cloudfront in front of my api gateway instance and i set the all viewer request policy which effectively means that cloudfront will forward all headers to api gateway i then sent the request from the previous slide now cloudfront doesn't pass this mutated host header so it sees the host header for the victim api and it thinks the url is for the victims api however api gateway for this request gives us a response from the attackers api which the attacker controls in this scenario what happens as a result here is cloudfront will cache the result under the victim's api cache but the response will be completely controlled by the attacker this allows an attacker to target a victim using cloudfront or any other caching proxy which will forward these headers in front of api gateway and just overwrite any responses in their cache with any value completely under the attacker's control now we're also going to look at cl.cl request smuggling i.e request smuggling based on two content length headers now a lot of requests only recently has been based on the use of a transfer encoding and a content length header however at blackhat usa 2020 amic client presents debug in squid and abyss which was based on two content length headers one of them being mutated now he detected this through white box testing but i was curious about whether it was possible to detect these sorts of bugs through black box testing and importantly whether it was possible to do it safely so those of you familiar with james kettle's detection techniques based on timeouts for request smuggling might be wondering why we can't just adapt those directly the issue is when you're looking at cl.cl request smuggling if you don't know which server is going to pass which header out of the front end of backend server you have a 50 chance of causing a timeout in the vulnerable system but you also have a 50 chance of leaving some data in the socket and affecting another user's request and this is something we generally want to avoid so just to recap the setup we have squid acting as a front end doing some caching in front of abyss now you can exploit this bug to do more than caching you can directly affect other users requests however i'm showing caching here as i think it's the nicest sort of simple way to show this bug now so the bug that amicline presented looks like this we start with a fairly normal post request shown here in green the only slightly strange thing about it is we have two content length headers one unmutated content length header which says the length of the body is 33 bytes and one mutated content length header which says the length of the body is zero bytes now what's going to happen is our front-end server squid is going to read our unmutated content length header and believe that this is a post request with a body of length 33 bytes so it sees two requests in this stream the first being what we have in green without post request and the body of this post request is shown in blue it then thinks the second request in this stream starts in red there and is a request and is a get request to the slash doesn't exist page so it's going to cache the result of this request for the path slash doesn't exist however the back end server abyss is going to read on mutated content length header which says that the length of the request is 0 bytes the length of the body is 0 bytes even so it thinks that the second request in the stream starts in blue there that request to slash a.html with a header called something and a value it just doesn't really care about and then again it's a request as before to squidoo1 to rs lab so just to show this in a diagram we have squid on the front end reading the unmutated content length header i'm thinking the second request is to the slash doesn't exist path and abyss on the back end reading our mutated content length header and thinking the second request is two slash a.html so squid is going to cache the value of a.html under a different path and this is the simple demo to show that cache poisoning exists so how do we detect this black box well we're working with detecting mutations here and that's something we've already looked at so we just need to modify our methodology slightly so this might look fairly similar to before we've got a post request and the slightly strange thing about it is we have two content length headers and one of them is mutated now we start off by setting the value of both of these to zero and we get a 200 okay and the response we expect for this post request so we know that this is being passed as normal and we haven't broken anything then we put an invalid value inside the content length header that isn't mutated so this content length z you see here and unsurprisingly we get an error in this case a 411 length required as one of the servers in the chain is trying to look at the content length of this post request and not be able to find it due to our invalid value uh so similarly we try again with the mutated content length header now i'm showing this request with zero bytes in both headers again for comparison but when we push an invalid value in our mutated content length header we also get an error however it's a different error to what we saw last time so we can infer that this error has been generated by a different server in the chain so we now know that we have two servers server a and server b server a is reading our unmutated content length header and server b is reading our mutated content length header we don't actually know at this point which is the front-end server and which is the back-end server and this is very important to know this before we continue testing so that we can continue testing safely and minimize the danger of affecting another user's request so to find out which is a front-end server we do something quite simple we just send a post request with only one content length header not mutated and put an imbalanced value in it we expect the front end server to pass this request as we expect any server to pass it and we see a 411 length required which we can assume is from the front end server comparing to our testing earlier this is the same as when we put this value in the unmutated header before so we can assume that when we send two headers it's the front-end server in this case squid which is reading the unmutated content length header and therefore the back end server abyss reasonably mutated one so we've got quite a good chance of request smuggling here as we can send two content length headers and specify different content lengths in them and this and the two servers that we're looking at will think the request has a different body to each other however this doesn't guarantee request modeling that's quite important to highlight you actually need to continue testing it's quite common in this sort of scenario for the back-end server to actually be looking at both content length headers and continuing as normal if they have the same value but as soon as you put a different value in those headers it will complain and throw an error at you so you can generate a timeout in a similar way to james kettle's research by essentially starving the back end of content and full details of that will be in the white paper as well as some exploitation scripts but the general advice from here is that you need to confirm beyond this and often you have to actually just try exploit the system which obviously you have to be quite careful about so we've looked at some header smuggling and the question is how do we defend against this well part of the defense i recommend is to use the tooling i'm going to be releasing as part of this research and scan any systems you're conf you're concerned about another piece of defense i consider key is on front-end servers don't forward malformed or suspicious headers so cloudflare don't afford headers with a space in for example and that saved them from being used in the example earlier with api gateway and the cache poisoning and this is also the approach that aws are taking to resolve this issue within api gateway they're writing a lot of tests to test the services with an api gateway to make sure they don't forward these malformed headers and there's also this concept that you should be quite liberal about passing protocols such as http however if it's for a back-end server and you're expecting the request to be filtered or normalized by a front-end server this doesn't necessarily apply as you won't be seeing requests from all sorts of different implementations of http you should only be seeing requests from one implementation so you can be a lot more strict and that can help defend against these vulnerabilities now in terms of future research i think there are plenty more applications to be found i'd recommend that you just go and scan as many systems as you can with my modified turbo intruder find interesting behavior that you probably can't predict right now and go play with it and see if you can exploit something we've of course also seen a lot of http 2 request smuggling research this year focused on http 2 to 81 downgrades and this has also led to lots of new ways to sneak headers through to the back end with the focus on the transfer and coding header in general so these techniques i haven't looked at as part of my research but that could definitely provide some interesting results if you go and look at those i've also made some assumptions in my research which are probably worth addressing i've assumed that front and back end servers will give different errors to each other i've also assumed that all headers are passed the same so if you find one mutation that works with a content length header we're assuming it works with every other header which may not be true although it generally seems to be and i've also assumed that all servers pass the content length header and get requests which is mostly true but there are definitely some servers i've seen which don't do this so those will be missed by the methodology of this research now there are some references here which i'd encourage you to go check out these have really been informative in this research there's also some other resources which again you can find in the slides and if you found this talk interesting i think you should probably have a look at these as you'll enjoy them as well and thank you for coming so links to our blogs there i'd like to say a massive thank you to the aws security team who've been really helpful and really good to work with throughout this research and also we're hiring so you have a bit of experience come and talk to us and are there any questions [Music]