haproxy,amazon-elb , HAProxy 1.4: how to replace X-Forwarded-For with custom IP

HAProxy 1.4: how to replace X-Forwarded-For with custom IP


Tag: haproxy,amazon-elb

I have an HAProxy 1.4 server behind an AWS ELB. Logically, the ELB sends the users IP in the X-Forwarded-For header. My app reads that header and behaves differently based on the IP (country).

I want to test that behavior overriding the X-Forwarded-For with custom IPs, but the AWS ELB appends my custom value with my current IP (X-Forwarded-For:,

What I have been trying to do is to send another custom header X-Force-IP and once it gets into HAproxy, delete X-Forwarded-For headers and use reqirep to change the name X-Force-IP to X-Forwarded-For

This is how my config chunk looks like

acl custom-ip hdr_cnt(X-Force-IP) 1
reqidel ^X-Forwarded-For:.* if custom-ip
reqrep X-Force-IP X-Forwarded-For if custom-ip

but when it gets into my app, the app server (lighttpd) rejects it with "HTTP 400 Bad Request" as if it were malformed.

[[email protected]]$ curl -I -H "X-Forwarded-For: 123.456.7.12" "http://www.example.com"
HTTP/1.1 200 OK
Set-Cookie: PHPSESSID=mcs0tqlsg31haiavqopdvm02i6; path=/; domain=www.example.com
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Content-Type: text/html; charset=UTF-8
Date: Sun, 11 Jan 2015 02:57:34 GMT
Server: beta

[[email protected]]$ curl -I -H "X-Forwarded-For: 123.456.7.12" -H "X-Force-IP: 321.456.7.12" "http://www.example.com"
HTTP/1.1 400 Bad Request
Content-Type: text/html
Content-Length: 349
Date: Sun, 11 Jan 2015 02:57:44 GMT
Server: beta

From the previous it looks like the ACL is working. I checked with tcpdump in the app server and it seems that it has deleted the X-Forwarded-For header but also deleted the X-Force-IP instead of replacing it.

[[email protected] ~]# sudo tcpdump -A -s 20240 'tcp port 80 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)' | egrep --line-buffered "^........(GET |HTTP\/|POST |HEAD )|^[A-Za-z0-9-]+: " | sed -r 's/^........(GET |HTTP\/|POST |HEAD )/\n\1/g' 
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 20240 bytes

GET / HTTP/1.1
User-Agent: curl/7.38.0
Host: www.example.com
Accept: */*
Connection: close

HTTP/1.1 400 Bad Request
Content-Type: text/html
Content-Length: 349
Connection: close
Date: Sun, 11 Jan 2015 02:56:50 GMT
Server: beta

The previous was with the X-Force-IP, and the following without it:

GET / HTTP/1.1
User-Agent: curl/7.38.0
Host: www.example.com
Accept: */*
X-Forwarded-For: 123.456.7.12
Connection: close

HTTP/1.1 200 OK
X-Powered-By: PHP/5.3.4
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Content-Type: text/html; charset=UTF-8
Connection: close
Transfer-Encoding: chunked
Date: Sun, 11 Jan 2015 02:57:02 GMT
Server: beta

^C71 packets captured
71 packets received by filter
0 packets dropped by kernel

Any help? I was expecting to have "X-Force-IP: 321.456.7.12" converted into "X-Forwarded-For: 321.456.7.12"

Thanks! Ignacio


The regex matching provided here doesn't do simple substitution. It's quite a bit more powerful, and has to be used accordingly.

reqrep ^X-Force-IP:(.*) X-Forwarded-For:\1 if custom-ip

The reqrep (case sensitive request regex replace) and reqirep (case insensitive request regex replace) directives operate at the individual request header level, replacing the header name and its value with the 2nd argument, if the 1st argument matches... so if there's information you want to preserve (such as the value) you need one or more capture groups, such as (.*), in the 1st arg, and a placeholder \1 in the 2nd arg, in order to do the preserve the data.

Your current configuration does indeed invalidate the request, by creating a malformed/incomplete header line.

Also, you should anchor the pattern to the left side of the header name with ^. Otherwise, the expression could match more headers than you expect.


Will haproxy ever retry a request where the upstream request has timed out waiting for a response?

I am trying to find the exact conditions when haproxy (version 1.4) in layer 7 mode will retry an HTTP request. Its documentation for retries states: It is important to understand that this value applies to the number of connection attempts, not full requests. When a connection has effectively been...

which ssl certificate is the right one for aws elastic load blancer and cloudfront? ssl certificate compatibility

Can I use any ssl certificate for aws services? Are there any prerequiries? There isn't any kind of aws ssl certificate compatibility list. Does any certificate work for cloudfront (sni support) and elastic load balancer. Do I have to get an special certificate or can I just use any? Are...

Logging in to private docker registry v2 behind haproxy

I am trying to set up a new Docker Registry (v2) with HAProxy. For the Docker Registry I am using the image from the docker hub and running it with docker run -d -p 5000:5000 -v /path/to/registry:/tmp/registry registry:2.0.1. And this is a subset of my HAProxy configuration: global chroot /var/lib/haproxy...

HAproxy heartbeat throws “Task Cancelled exception” constantly when hitting API method

We are using HAproxy with the httpchk to verify that our websites are up on each server. The controller method for the httpchk method is as follows: [Route("heartbeat")] [HttpGet] [AllowAnonymous] public async Task<IHttpActionResult> Heartbeat() { return Ok(); } We see the following in the error log tons of times each...

HAProxy SSL termination + client certificate validation + curl / java client

I would like to have SSL termination on HAProxy, using my own self-signed certificates, and to validate client access using client certificates I create. I create the server (which is also the CA) certificates the following way: openssl genrsa -out ca.key 1024 openssl req -new -key ca.key -out ca.csr openssl...

Do I need to create a endpoint to use Elastic Load Balancer on my VPC?

I'm creating a VPC to host a web app at AWS, and I want to use load balancers. Do I need to create a endpoint for elb like I have to s3?

Linking containers with Mesos JSON file

I am trying to link my application container to my DB container when sending the JSON task description to the Marathon framework. I've read up on this question from last year: Linked Docker Containers with Mesos/Marathon The question is: I know Fig will be able to help me with this...

How to redirect to 404 when url is pointing to an existing folder (no trailing slash). HAproxy or mod-rewrite related

I am using HAProxy listening on 80 to send requests to a node server (port:3000) or php server (4000); I also have CSF installed which have ports 3000 and 80 available. It works okay when I browse a page at http://example.com/forum/1/page.php, but sometimes when I accidentally enter example.com/forum/1 (no trailing...

Can Haproxy use endpoint to determine load balancing?

Is it possible for haproxy to ask a (REST http) Endpoint to determine, if a specific server is available for loadbalancing? See example below for our setup. If 'a' is down, the haproxy 'ha' should not use the way over 'x'. x--a / \ ha out \ / y--b ...

HAProxy is not load balancing due to persistent connections

We have a web server and a client, both written in go, that interact with each other. We want HAProxy to load balance requests between several instance of the server, but it's not working. The client will always connect to the same server while it's still up. If I look...

How to fail over node.js timer on amazon load balancer?

I have setup 2 instance under aws load balancer. I have deployed node.js web services + mongodb in both instance. load balancer works fine with web services. But, Problem is I have one timer service (node.js service only). the behavior of this timer is updating my mongodb based on some...

Haproxy configuration with sticky `app set` cookie

I have a cookie setup by my app that represents a room. I want all users belonging to the same room to connect to the same server. With the classic cookie sticky scheme I do not want Haproxy to write a cookie for me or use the cookie value as...

redispatch request to default server or other server in haproxy?

I am working on haproxy configuration for my tomcats, I have two backend group , 1st backend is having two tomcat and serving request in round robin method and other backend is having only 1 tomcat. Now I want if request goes to second backeng and it is down ,...

Map domain name to multiple AWS Elastic Beanstalk instances

The company I work for has multiple elastic beanstalk applications. We have setup a domain (eg. http://applications.ourcompany.com ) that currently directs to one of these instances. This works fine, the user enters http://applications.ourcompany.com and they end up at this ELB instance. However what we would really like to to is...

Is ELB draining tcp based?

I am facing an interesting problem. We run an HTTPS based application that is state based. State is maintained based on session cookie. The application is designed in such a way that if the session gets terminated abruptly, the application is reverted to home screen any un-saved data is lost....

How can I have a custom restart script for runit?

I'm using runit to manage an HAProxy and want to do a safe restart to reload a configuration file (specifically: haproxy -f /etc/haproxy/haproxy.cfg -sf $OLD_PROCESS_ID). I figure that I could run sv restart haproxy and tried to add a custom script named /etc/service/haproxy/restart, but it never seems to execute. How...

how to connect to specific server behind elb?

I'm running a chat application which allows clients to create rooms using websockets. I want to use aws elastic load balancing, but I'm not sure how to make sure two clients who want to chat end up on the same server. Specifically avoiding the following scenario: Client A creates a...

Global Variables not updated due to load balancing

There are two servers running a web service and the servers are load balanced using HAProxy. Web Service does a post and updates a global variable with a certain value and another application running on the same servers will read this global value and does some processing. My issues is...

What is difference b/w HAProxy and Apache server

Hi actually we are working on load balancing and cluster environment. So we have some confusion is this both are same or any difference between them.

Multiple websites with Single CloudFront Distribution does not get routed

I have a few websites - around 20. All of them are php application are currently hosted on a single machine in apache. Now with virtual host configurations I can host all of them with one single ELB (unless they need to be over HTTPS). I can easily route the...

AWS calculator, data processed by ELB

Can somebody explain what is a "Total Data Processed by all ELBs" in AWS calculator(http://calculator.s3.amazonaws.com/index.html). What exactly "data processed" means ? What is counted ? Headers ? Body ? What about HTTP POST file uploads ? ...

How do I get tutum and haproxy to use SSL?

I just purchased an SSL Certificate. It came with 1 server certificate, 1 intermediate certificate and 1 root certificate. I have my *.csr file and *.key used to create the certificates. I am using Tutum with HAProxy and there is a SSL_CERT environment variable. I read the documentation but could...