Everything You Need To Know About User Agents

Fastvue TMG Reporter provides a Top Applications chart in the Bandwidth Dashboard, which is built using the User Agent field in Forefront TMG's proxy log file (referred to as ClientAgent or c-agent depending on your log file format)

It is usually easy to identify these applications

• Internet Explorer
• Adobe Acrobat Updater/10.0.0 CFNetwork/520.2.5 Darwin/11.2.0
• Skype/5.8.0.1027 CFNetwork/520.2.5 Darwin/11.2.0 (x86_64) (iMac12%2C2)

However, some are fairly obscure and don’t give you much indication.

• NSPlayer/12.0.7601.17514
• VCSoapClient

What Are User Agent Strings?

The data that makes up the application identifier is called the User Agent String. The user agent is a field in the HTTP request header. The request header is sent in the initial GET request and the User Agent field is often used to determine the capabilities or limitations of the application.

The format of the User Agent field is specified in RFC 2616 Section 14.43.

14.43 User-Agent

The User-Agent request-header field contains information about the user agent originating the request. This is for statistical purposes, the tracing of protocol violations, and automated recognition of user agents for the sake of tailoring responses to avoid particular user agent limitations. User agents SHOULD include this field with requests. The field can contain multiple product tokens (section 3.8) and comments identifying the agent and any subproducts which form a significant part of the user agent. By convention, the product tokens are listed in order of their significance for identifying the application.

As a rule almost all browser user agent strings look as follows:

Mozilla/

[version] ([system and browser information]) [platform] ([platform details]) [extensions].

Most browsers start with Mozilla to indicate compatibility, which is a hangover from the mid 1990’s browser wars. Here are some Internet Explorer User Agent examples with various versions and extensions:

• Internet Explorer 4 on windws 98 - Mozilla/4.0 (compatible; MSIE 4.01; Windows 98; Hotbar 3.0)
• Internet Explorer 7 - Mozilla/5.0 (Windows; U; MSIE 7.0; Windows NT 6.0; el-GR)
• Internet Explorer 9 - Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)

Looking at these you can see that a common field value is MSIE. We can reasonably assume that all user agents containing this value represent a version of Internet Explorer. Similar common values exist for other browsers such as Firefox and Chrome.

TMG Reporter does a nice job of representing these user agents with common values into more readable Applications. For all others, the raw user agent is displayed.

Why Do User Agent Strings Matter?

In addition to identifying the browser or application being used, the user agent strings sent by popular browsers usually include information about the host operating system. This is particularly useful when using a website traffic analysis tool such as Google Analytics as it can help target users more accurately.

For instance, if a retailer notices a large amount of Windows XP traffic they could start pushing more Windows 7 upgrade sales. Likewise, if it detects traffic coming from an Apple OSX device it would exclude this promotion completely.

Web developers can also use the User Agent string to dynamically format web page content for smart phones, tables and desktops (although using css media queries is widely considered a better approach).

This of course raises some questions about privacy and security since a large amount of user machine information is revealed. For example, if you know that a connection is being made from a machine with IE6 on Windows XP, you know what potential vulnerabilities there are on that system.

How can Forefront TMG Admins Use User Agents?

All Internet applications should identify themselves with a unique user agent string. We can therefore use this to determine if certain applications are drawing large amounts of traffic unbeknownst to users. An easy example of this is Adobe updater.

Adobe Acrobat Updater/10.0.0 CFNetwork/520.2.5 Darwin/11.2.0

You could use a time based access rule to block traffic to the Adobe Update servers during peak office hours. TMG Reporter will help you identify this too.

Since machines effectively advertise what they are with the agent string you can use this to track down non-compliant machines.  As an example setting a filter for agent type containing MSIE6 will help you find machines on your network that are far out of date.

The user agent string can also be useful for blocking applications that communicate over HTTP on port 80 using Forefront TMG's HTTP Filter.  For a good example, see Richard Hicks' post, Block Instant Messaging Traffic using Forefront TMG's HTTP Filter

Searching User Agents in Forefront TMG to Identify Risks

You can use TMG’s log monitoring to track down machines running user agents that could pose a risk to your network.

• From the TMG management console select Logs & Reports
• Right-click the column header and click Add / Remove Columns
• Add Client Agent and move it to the top of the list
• This will help you identify all the agents that are active on your network.
• You can also click Edit Query and use the Client Agent field to refine your results.

Spoofing User Agents

Certain sites use the user agent string to respond with or render a particular version of the site.  I am going to use a local site in the example.  Using a Firefox plugin called TamperData we can intercept and edit the initial GET request and replace Firefox's user agent string with that of a mobile platform.

We now get redirected to the .mobi site as opposed to the full site.

Some browsers are including this type of spoofing as a feature in order to allow the user to set site version preferences. For example, the Dolphin browser has a setting to enable desktop mode.  This will force sites such as Facebook to render the full version desktop version rather than the mobile version.

Browser developer tools such as the ones included in Internet Explorer or Chrome also often provide the functionality to spoof older versions of the browser to test backwards compatibility.

And unfortunately certain types of Malware also use user agent spoofing to avoid detection.

Conclusion

User agent strings can be very useful, but as they are very easily 'spoofed', they cannot be considered 100% accurate or “honest”.  Having a good understanding of them can however give you a valuable insight in managing your environment's Internet consumption and identifying potential problem areas.

More Information

• Useragentstring.com: A very useful User Agent related site. The default page will tell you what user agent and browser information you are presenting.  This is particularly interesting with various mobile browsers.
• Mozilla User Agent Specification
• Understanding User Agent Strings (includes Microsoft Internet Explorer User Agent Specification)
• User Agents (wikipedia)