User Agent Detection Use Cases

What is The User Agent String?

Browsers (user agents in general) provide a way to identify themselves on the network through a HTTP header: User-Agent: <identification-string>. This string is sent with each requests the browser is sending to the server. The string is defined in HTTP 1.1bis as

The "User-Agent" header field contains information about the user agent originating the request, which is often used by servers to help identify the scope of reported interoperability problems, to work around or tailor responses to avoid particular user agent limitations, and for analytics regarding browser or operating system use.

Additionally the HTTP specification says that the User-Agent should be simple:

A user agent SHOULD NOT generate a User-Agent field containing needlessly fine-grained detail and SHOULD limit the addition of subproducts by third parties.

and

Likewise, implementations are encouraged not to use the product tokens of other implementations in order to declare compatibility with them, as this circumvents the purpose of the field. If a user agent masquerades as a different user agent, recipients can assume that the user intentionally desires to see responses tailored for that identified user agent, even if they might not work as well for the actual user agent being used.

The latest version of HTTP 1.1 specification acknowledges the use of user agent detection.

In addition to the HTTP user agent string, the same information is provided by JavaScript enabled user agent into the JavaScript object:

navigator.userAgent

For example, at the time of ths writing, my browser displays the following user agent string into the Web Console.

What is User Agent Detection?

User agent detection (or user agent sniffing) is a technique which is used by Web developers for adjusting the user experience depending on the capabilities of the user agent. For example, a Web developer might configure a server to guess using the user agent string if the request is done by a user with a specific device. Upon the guess, the user will be provided with a mobile, tablet or desktop version of the content. This technique may happen on the server side through the User-Agent HTTP header or on the client side through the navigator.userAgent JavaScript object. Sometimes both.

The intent of user agent detection is to provide the best possible user experience depending on the browser identification. It then relies on having big databases of known user agents strings. The method has two caveats:

1. We only know the past. The identification will be properly working if the user agent string is known in the database. New user agent strings will not be identified and then the user will be possibly given an inferior experience.
2. Some Web developers will customize (sometimes by ignorance or negligence) their detection algorithm to discriminate and not support some user agents. Instead of enhancing the user experience, it will drastically reduce or even annihilate any hope for the user to access the Web site.

The combination of these two caveats often leads to a race to the bottom for reaching Web compatibility. User agents will lie about their identity to fit the identification algorithm and provide the good experience for their users. The Web developers will try to be more precise with more chances to fail. As soon as these detection algorithms and databases are not maintained, Web sites have a higher probability to fail with new user agent strings.

Use Cases for User Agent Detection

User agent detection is used in many contexts. In November 2013, we made a call to Web developers, detection framework developers, user agent string database owners to share how and when they were using user agent detection.

Device/Software capabilities

• Send anterior version of a browser to a simpler version of the Web site. The new Web site using technologies unsupported on old browsers, the user will get a bad experience.
• Block the access to an anterior version of browser and recommend the user to download a new version of the browser.
• Redirect the browser to the (feature phone|smartphone|tablet|desktop|tv|wearable) Web site or provide directly content optimized for the device.
• Customizing content such as the choice of video formats, the UI elements size, Ads.
• Plugin and framework supports such as J2ME, DRM.
• Device own material performance

Network Performances

• Server-side optimization of media (size and formats). Certain devices type are believed to access the Web through a type of connection. The assumption is often triggered by if it's a mobile the network bandwidth and/or latency is either bad or expensive.

Technical

• Blocking some abusive bots
• A/B testing during feature deployments
• Fallback: Once features detection has failed to be able to customize the user experience: Incomplete support of features, support not optimized for a specific feature, misleading user agent with regards to support

Business

• Delivering specific content (Premium, documentation, help) for certain devices
• Native app: Encouraging to download the platform native app for capturing an audience
• Upgrading the user agent: Proposing the right software to download when upgrading
• Analytics and statistics reporting

Moving forward

More or less the list of comments which have been made on the user agents detection use case survey falls into one of these categories. They are often variation or specialization. It seems there is another possibility of reorganizing the categories for the use cases. A big effort has been made when restarting the HTML effort on defining the HTML Design Principles. Priority of constituencies is a key principle. When there is a conflict over a design decision, who should have the priority from top to bottom:

1. users (people using browsers)
2. authors (people writing/creating Web sites)
3. implementors (people implementing softwares)
4. specifiers (people writing specification)
5. theoretical purity

Web Development

This could be certainly adjusted in a way which makes sense for Web development. For example, blocking a user because of a browser choice goes against the user and should not happened. Finding the right level of recommendations, guidelines on how to use user agent detections, so that it doesn't break in the future, doesn't exclude users on their choices would be possible. Some of the user agent detection practices are completely orthogonal to the user experience such as Analytics (except when it is used to justify blocking some users).

Something to explore with the rest of the industry would be along:

When developing a Web site, give priority in that order from top to bottom:

1. Ensure that any users have access to the site whatever the device they are using.
2. Make the Web site light enough that it can be loaded on a poor performance network
3. Make the Web site flexible enough that it adjusts itself to small screens scenario first.
4. Use features detection to improve without destroying 1 to 3
5. Adjust and correct with user agent detection (enhance) without contradicting 4 to 5.

This needs refinement and iteration. We (Web development community) are currently not doing a great job at helping users without mainstream devices to access the Web.

Browsers

Ilya Grigorik at Google has started a work (exploratory) if some of the use cases described above could be resolved through HTTP headers: HTTP Client Hints. It is called client hints and has for goal to solve issues with responsive design and size of images. In this case, the client informs which type of resources it is willing to process. Note that it is interesting because it is closer of a user choice. You may be using a small screen on a good wifi and receive big high definition images.

Are there other things we could do in browsers and libraries to reduce the need for user agent detection?

Acknowledgements

Thanks a lot to each individual (private or companies) who took the time to answer the survey.

Otsukare