Load Balancing an SMTP gateway can be tricky if the platform used is not well understood. In this post we will go over the iterations of load balancing I had to go through protect an SNMP Gateway that had two servers.
The Problem: Critical vulnerabilities identified with TLS v1, v1.1 and v3. Our SMTP gateway has a web portal for end-users to manage their spam, it also uses TLS over port 25 SMTP (Not identified on initial discovery) The SMTP gateway did not provide a method to address the vulnerabilities in TLS. They had to be addressed in networking.
The original configuration had the MX record on DNS pointed to three A records, two of the A Records pointed to the same IP Address.
The firewall configuration in place allowed for a direct Public IP to Server DMZ IP address translation to happen directly. PUBLIC IP1 Translated to NODE1 IP and PUBLIC IP2 translated to NODE2 IP.
Whenever TLS was used either for HTTP or SMTP the vulnerability was hit because weak ciphers configured on the NODE 1 and NODE 2 servers.
Solution: Place SMTP gateway servers behind an F5 Application Delivery Controller. Proxy all connections for TLS (443) and SMTP (TCP 25).
In our initial discovery of the SMTP Gateway operation we identified that two servers were present and due to lack of documentation it was presumed that both served HTTPS and SMTP traffic. An additional requirement was that we could not modify public DNS records since that could result in extended outages due to DNS Propagation.
NOTE: In addition to the F5 Application Delivery Controller a Firewall was present for public NAT and additional threat prevention.
The first iteration of the solution utilized the following:
- An http/https virtual server. We used an iApp to create the server and load the certificates and create the redirection from http to https. SSL Bridging was used.
- Manually create a SMTP virtual server.
- Retain the original NAT for PUBLIC IP2 in order to make sure some form of mail delivery continued while the transition to the load balanced virtual server took place.
- Used the same server pools for HTTP/HTTPS and SMTP.
The solution was immediately rolled back when the following issues were identified:
- NODE 1 and NODE 2 did not have the same capabilities. NODE 2 could not serve HTTP Requests.
- It was identified that NODE 2 also allows STARTTLS over SMTP. This required SSL bridging in order to also protect communication to NODE 2 and NODE 1 SMTP Communication.
- It was identified that both NODE 1 and NODE 2 require DNS and SMTP Outbound access to be able to send email out.
The next steps to address the issues identified were the following:
- Create a new pool in which only NODE 1 was included in order to handle the HTTPS requests.
- Add STARTTLS Support for the SMTP virtual server. This required loading the certificates and attaching the client SSL profile and server SSL profile to the virtual server. In addition an SMTPS profile was tied to the virtual server.
- New SMTPS Profile with Activation Mode set as “Required”
- New SMTPS Virtual server configuration including the addition of the SSL Profiles and the SMTPS profile.
Once the Source Network Address Translations were added for DNS and Outbound SMTP the configuration ended as follows:
This configuration was considered stable enough to remain in production but another set of issues were identified that needed to be resolved.
- SNMP STARTTLS connection was not working properly for the SMTP virtual server. The connection was not completing and since STARTTLS was required, clear text was failing.
- The SMTP Gateways were not able to perform IP Reputation checking. Since “SNAT Automap” was used the connection source IP was being masked as the Floating IP Address of the F5.
To solve the issues above the following was done:
- F5 Support Recommended the removal of the Server SSL Profile. This way TLS will be offloaded and not bridged, connections to the SMTP Gateways would be unencrypted while connections to the F5 Virtual server will be encrypted.
- Removed SNAT Automap the virtual server and set the configuration of Source Address translation to “None” In addition the SMTP Gateway servers had their default gateway configured with the F5 Floater IP.
- The last change was the modification of the STARTTLS Action Mode option under the SMTPS profile. The option was to set to “Allow” instead of “Require”
The modification of the Source Address Translation setting helped with the IP Reputation issue but broke direct communication to the SMTP Gateway server since the IP Default Gateway was changed to the F5. This configuration was rolled back
The solution to the default gateway issue was the configuration of an F5 IP Forwarding Virtual Server.
The function of an IP Forwarding Virtual Server is to respond to IP traffic for which the F5 does not have a socket (IP and Port) configured. This allows the F5 to respond to communications from the nodes to the rest of the network as if it was the Default Gateway.
Below is the configuration of the IP or L3 forwarding virtual server:
A new issue came up after configuring the L3 Forwarding virtual server and configuring the SMTP Gateway with their Default Gateway as the F5. Some of the communications were still failing. We could no longer manage the SMTP Gateways over the network.
Doing research we found the following:
If a different router exists on any directly connected network, you may need to create a custom fastL4 profile with “Loose Initiation” & “Loose Close” enabled to prevent LTM from interfering with forwarded conversations traversing an asymmetrical path.
The new custom FastL4 profile was configured as shown below and applied to the L3 Forwarding Virtual Server, this profile includes the “Loose initiation” and “loose close” settings enabled.
Once all these elements were in place the SMTP Gateway were able to communicate properly in their load balance and secure configuration. The final diagram of the communication flow ended up looking as shown below: