Block Instant Messaging Traffic Using Forefront TMG's HTTP Filter

Forefront Threat Management Gateway (TMG) 2010 is a powerful application-layer firewall and web proxy server. TMG also includes advanced web protection technologies for providing essential security for clients accessing resources on the Internet. To provide deep application layer traffic inspection, Forefront TMG includes many web and application filters assigned to various protocols to parse traffic for inspection.

For HTTP traffic, Forefront TMG’s HTTP filter is responsible for ensuring that all HTTP communication is valid and RFC compliant. If the TMG firewall processes a request on TCP port 80 (the default port for HTTP) the filter is invoked and the application layer data stream is fully inspected. This ensures that any non-HTTP traffic, or HTTP traffic that doesn’t conform to RFC 2616, will be denied by the firewall. This prevents a malicious user or an attacker from tunneling encrypted communication or non-web based traffic over TCP port 80.

Blocking HTTP Applications with Forefront TMG

In certain scenarios the TMG administrator may wish to block specific types of valid HTTP traffic. For example, most popular instant messaging (IM) applications today leverage HTTP for communication. This can be problematic for security administrators tasked with enforcing a corporate security policy that dictates that only approved corporate IM applications like Microsoft Lync Server be used for instant messaging. Since TCP port 80 is typically open outbound to the Internet, and the IM application is using RFC compliant HTTP communication, this traffic would normally be allowed by firewall policy.

Thankfully, the Forefront TMG HTTP filter is configurable and allows us the flexibility to selectively deny HTTP traffic based on application layer content. For example, the HTTP filter can be configured to identify and block instant messaging application traffic.

Identifying Instant Messaging Traffic

If you want to prevent Windows Live Messenger communication, start by identifying something in the HTTP communication that uniquely identifies this traffic. Sometimes referencing documentation for the product, or consulting an online database like useragentstring.com can help determine this. Often it requires doing a bit of investigation and observing client communication using a protocol analyzer such as Microsoft Network Monitor, or perhaps other tools such as HTTPWatch or Fiddler.

I like to use Network Monitor because it is excellent for this type of work, as by default it breaks out network communication by process, which allows you to easily distinguish the traffic being generated by the application in question. By selecting a frame where the client generates an HTTP request and expanding the frame details I’ve observed that Windows Live Messenger’s complete User-Agent string is:

UserAgent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; Windows Live Messenger 15.4.3555.0308)

The idea here is to identify something unique to this application in the User-Agent request header, so I’ve highlighted what I am planning to filter on, which is the string Windows Live Messenger.

Configuring Forefront TMGs HTTP Filter

To configure Forefront TMG to deny Windows Live Messenger traffic:

1. Open the Forefront TMG management console and highlight the Firewall Policy node in the navigation tree.
2. Right-click the first access rule that allows outbound HTTP communication and choose Configure HTTP.
3. Select the Signatures tab and click Add.
4. Enter a name for the filter and select Request headers in the drop-down box for the Search in field
5. Specify User-Agent: for the HTTP header.
6. Finally, enter Windows Live Messenger in the Signature field and click Ok.

User-Agent not UserAgent

The HTTP header field is not case sensitive, but it is context sensitive. That is, it must be entered exactly as the header appears in the application layer data stream. As much as I love Network Monitor, it presents a problem here because it identifies this particular header incorrectly as UserAgent:. However, if you look at the raw data you’ll see the header is actually User-Agent:. This is what you’ll need to enter in the HTTP header field above.

Save and apply the configuration above, then once the configuration is fully synchronized you can begin testing. If all goes well, Windows Live Messenger communication should be blocked by the Forefront TMG firewall.

More Effective Blocking

A word of caution here: altering the User-Agent string is trivial and this should not be considered a 100% effective way to limit communication from certain applications - see Etienne Liebetrau's article on User Agents and how easily they can be spoofed. However, in general it can be an effective deterrent in most environments.

A comprehensive plan to effectively control undesired communication will include URL filtering (blocking specific categories), additional firewall rules (deny specific hosts, domain names, or network blocks), and making certain that any other ports required for communication are not allowed. Also, many IM applications today will use SSL/TLS to protect their communication, which means that unless you have enabled HTTPS inspection on the Forefront TMG firewall, you will not be able to filter out this traffic.

Of course it goes without saying that knowing what types of communication are taking place on your network is vital to controlling unwanted traffic. This requires detailed log analysis, and I can think of no better tool for this discovery than TMG Reporter from Fastvue. Download an evaluation of TMG Reporter and begin assessing your traffic profile today. You might be surprised to see what’s actually going on!