Connection Pool vs Firewall

As some of you might have noticed, I’m busy at work to create a new reverse proxy environment to publish our websites. The whole set-up is very simple and commonly used in the world, so I wasn’t expecting a lot of problems. A short summary:

We have a ISA array set-up as reverse proxy and behind the ISA there are the webservers. The webservers get there data from a database server. In our test environment all these three machines are in a different subnet. The database was placed in an “old” subnet which is deprecated after the completion of the project.So we would not harm the production environment and the webservers connect to that subnet. The ISA runs a Form Based Authentication (FBA) against an Active Directory in the same subnet as the webservers.

About a week ago, we got into a beta setup, to let some well choosen customers test the environment. And since the first day we got some tips that a customer has to push twice the logon button to get actually logged in. The biggest problem for us, that this “twice” login problem just occured a few times a day, which makes it hard to find the actual problem. First we thought this problem was caused by the ISA’s, so we checked every option of the publish rule, but without any luck.  The next thing we started to do was to capture all the packets on all the interfaces, which took a lot of time, cause we needed a capture of the moment that the problem occured. Just to get a good capture took us at least a day. After searching a lot through the capture files, we finally find out, that there went something wrong between the webserver and the database communication. This is a part of the communication between the webserver and databaserver, this capture is made on the webserver. ]

no_syn_synack

As you can see the webserver started immediately with PSH, ACK instead of building up a connection with SYN, etc. The big question for us was why and this was a bit of puzzling, but it tuns out that the firewall between these subnets was terminating the connections. So when the webserver was in need to contact the DB server, it takes a look in its connection pool and picked an connection which was (by his thoughts) open, so no SYN was sent. You can guess what happened, the firewall has closed that particular connection and denied the connection, cause no SYN packet was sent. So whenever you want to set-up a connection pool, make sure there is no Firewall between them or that the time-out for IDLE connections are properly configured on both sides. This can save you a lot of time :-)

Rob Maas
Rob Maas
Technical Challanger at ON2IT

If it is broken, fix it! If it ain’t broken, make it better!

Related