Many network environments have limited bandwidth between certain geographic locations, between offices and home users using VPNs, etc. Deploying large Microsoft patches (for example: the Windows XP SP2 update is 275 MB!) can easily overwhelm limited bandwidth connections and cause bandwidth problems for certain users or applications.

To avoid these types of problems, TEM provides a number of mechanisms to reduce bandwidth usage including; a properly implemented and maintained TEM Relay architecture, distributing patch deployments over time, and the use of bandwidth throttling configurations between TEM components. Through client settings, the TEM Console operator has the ability to set the maximum number of bytes per second that will be used to send files over a network connection. TEM can be configured to throttle bandwidth at the TEM Server, TEM Relay, or TEM Client component levels. Below are instructions on how to configure these TEM components to use bandwidth throttling.

Important Note: Starting in BigFix/TEM version 5.0 the bandwidth throttling settings were moved to Tasks in the BES Support external content site instead of using custom settings. Under All Content, drill down on the left hand navigation tree: Fixets and Tasks > Tasks Only > By Site > BES Support and search for the term "throttl" in the Search box:

151 - BES Server Setting: Throttle Outgoing Download Traffic

152 - BES Relay Setting: Download Throttling

163 - BES Relay Setting: Throttle Outgoing Download Traffic

167 - BES Client Setting: Download Throttling

457 - BES Client Setting: Dynamic Download

458 - BES Relay Setting: Dynamic Download Throttling

459 - BES Relay Setting: Dynamically Throttle Outgoing Traffic

462 - BES Server Setting: Dynamically Throttle Outgoing Traffic

605 - BES Client Setting: Enable/Disable Dynamic Throttling

702 - BES Relay/Server Setting: Enable/Disable Dynamic Throttling

There is also an Analysis in the BES Support site (#218 Bandwidth Throttling Status) that can be activated to display the bandwidth throttling status of TEM Client for the following settings:



BES Client Download Throttling

BES Relay Download Throttling

BES Relay Total Outbound Throttling

BES Server Total Outbound Throttling

BES Client Dynamic Download Throttling: (min - max) percentage

BES Relay Dynamic Download Throttling: (min - max) percentage

BES Relay Total Outbound Dynamic Throttling: (min - max) percentage

BES Server Total Outbound Dynamic Throttling: (min - max) percentage

 

Methods of Bandwidth Throttling



Throttling through BigFix Relays (A)



BigFix Relays downloading from the BigFix Server

BigFix Relays can be configured to throttle file downloads when downloading from the BigFix Server. When BigFix Relay throttling is enabled, the BigFix Relay will download from the BigFix Server no more than the specified number of bytes per second. This setting is especially useful for BigFix Relays that have a slow connection to the BigFix Server (such as a BigFix Relay in a remote location connected by a 56 kbps modem).

Configuration: BigFix Client throttling can be configured by using the "_BESGather_Download_LimitBytesPerSecond" setting on the BigFix Relays. Information on how to configure this setting can be found here.

Total outgoing download traffic for the BigFix Relay

The BigFix Relay in BigFix 4.0 and above can be configured to throttle the cumulative file downloads at any given time. When this throttling setting is enabled, a BigFix Relay will send out no more than the specified number of bytes per second for all file downloads (including BigFix Clients and child BigFix Relays). This setting is especially useful if there is a concern in a local area network that too much bandwidth will be used when a patch is sent simultaneously to many BigFix Clients.

Configuration: BigFix Relay cumulative download throttling can be configured by using the "_BESRelay_HTTPServer_ThrottleKBPS" setting on the BigFix Relay. Set this number to the total number of kilobytes that the BigFix Relay will give to all of the BigFix Clients combined per second.



Throttling through BigFix Clients (B)



BigFix Clients downloading files from the BigFix Server or BigFix Relay

BigFix Clients can be configured to throttle file downloads when downloading from the BigFix Server or BigFix Relays. When BigFix Client throttling is enabled, a BigFix Client will download from the BigFix Server or BigFix Relays at no more than than the specified number of bytes per second. This setting is especially useful for individual computers that are on slow connections (such as travelling sales representatives or home users on dial-ups).

Configuration: BigFix Client throttling can be configured by using the "_BESClient_Download_LimitBytesPerSecond" setting on the BigFix Clients. Information on how to configure this setting can be found here.



Throttling through the BigFix Server/Relays (C/D)



Total outgoing download traffic for the BigFix Server/Relays

The BigFix Server can be configured to throttle the cumulative file downloads at any given time. When this throttling setting is enabled, the BigFix Server/Relays will send out no more than the specified number of bytes per second for all file downloads (including BigFix Clients and child BigFix Relays). This setting is especially useful if there is a concern in a local area network that too much bandwidth will be used when a patch is sent out to many BigFix Clients simultaneously.

Configuration: BigFix Server cumulative download throttling can be configured by using the "_BESRelay_HTTPServer_ThrottleKBPS" setting on the BigFix Server. Information on how to configure this setting can be found here.

This information is current as of BES 7.2. Before using these settings, you may want to check for any changes in BES Console/Server Settings or BES Client Settings. Please update this document if appropriate. Note that BESRootServer or BESRelay require a service restart before any changes to these settings will take effect.





Overview



TEM has a great number of different settings that control how different components throttle traffic between each other. The following goal is give a comprehensive view of all the throttling options.

What can be throttled?

A common misconception is that various "bandwidth throttling" features in TEM are designed to throttle all TEM traffic between any two machines. In fact, throttling is always done for specific components, and many components have no throttling.



Throttled Components

These are any files that are downloaded directly from the wwwrootbes directory of the Relay/Server/WebReports. Files downloaded for actions (e.g. patches, service packs, etc.) are downloaded this way. The contents of Fixlet sites are also downloaded this way during the gather interaction (although the site directory listing comes in separately). Web Reports also serves up "support" files (such as SWFs) this way, although really, why are you trying to throttle Web Reports?

These are files uploaded through the "Upload Manager"

File uploads



Unthrottled Components

There are however, a number of "throttle-like" capabilities around client registration. The amount of downstream UDP traffic generated can be rate limited on the Relay/Server side, the frequency of client registration can be turned down on the client side, and the amount of ICMP traffic generated during relay selection can be rate limited on the client side. Note that client registration is not bandwidth intensive, although it can be a significant source of load.

Download request plugin, status reporting plugins, the "Web Reports" plug-in, etc. For the Client, we expect this traffic to be negligible. However, things like the download status report generate a lot of traffic between the Console and the Root Server, and they cannot be throttled.

When the Client asks the Relay "please tell me the latest contents of site X", the interaction is not throttled. The response of the Relay is typically small (anywhere from 0-~40k). If absolutely necessary, you can turn down the gather intervals on Clients to get this information less frequently, but this traffic should usually be negligible.

There are also things you can do to control the amount of traffic going upstream through the "PostResults" interface. The most basic "throttling" mechanism is simply to turn up the minimum report interval on the Client or lengthen the heartbeat interval. You can also put a crude limit on the amount of traffic a Relay can send up through a combination of the "ResultSizeLimit" and "ResultTimeLimit" Relay settings. However, you should check with Tyler or a developer before doing so: it's unlikely that you'll get the behavior you expect. Posting results takes more bandwidth than client registration, but is still much smaller than the amount of bandwidth used by the download and upload components. For most deployments, the amount of traffic should be negligible.

 

 

Throttling Minimum Transfer Rates

During throttled communication, the BES Client will send chucks of data and then wait longer than necessary before sending the next chuck. The BES Client can vary the amount of data in each chuck along with the amount of time to wait. By lowering the amount of data per chuck and maximizing the amount of data between chuncks, a minimum transfer rate will be established for throttling.

Dynamic throttling and static throttling systems share the code that creates the minimum transfer rate so these thresholds are the same. The introductions of dynamic throttling ended up raising the minimum transfer rate and then we had to lower it again due to customer feedback, so the minimum throttling rate has changed for different versions of the product.

The BES Relay component actually does the throttling work so if you need to upgrade BES to change the minimum throttling rate, it is important to upgrade the BES Relays along with the BES Clients to the newer version.

The effective minimum throttling rates by version are:

All throttling settings are given in either BPS (Bytes per Second) or KBPS (KBPS). Given the effective minimum values listed above, if you set the throttling rate below the effective minimum it doesn't go that low, it only goes to the effective minimum value. Also notice that the effective minimum is less than 1 KBPS (in most versions) but the minimum value you can set KBPS throttling setting to is only 1 so you can't reach the effective minimum. IE., in some cases the effective minimum rate is controlled by the setting and in other cases the effective minimum is controlled by the clients throttling limits.

 

Upload Throttling

The uploads generated by the "Upload Manager" can be throttled from the Client side or from the Relay/Server side. This component only supports static throttling. There is a "PostFile" setting that sets an overall throttle on incoming connections, and an "UploadManager" setting that sets the throttle on outgoing connections (there is only one outgoing connection at a time). When both are set, the child is responsible for using the lesser of the two values.

 

Download Throttling

Download throttling settings have two major axes:

When Server-side and Client-side throttling is in effect, BES components use the lower of the calculated bandwidth limits. When Dynamic and Static throttling are both in effect, the Dynamic throttling settings are used in place of the Static throttling settings.

Server-side static throttling is the only type of throttling that can affect non-BES components (such as Web Browsers).



Static Throttling

Server Side

Server-side static throttling settings control the total amount of download traffic that a server will send out to Clients using static throttling. The amount of bandwidth allocated to any given write connection is simply the "ThrottleKBPS" setting divided by the number of active write connections. Note that plug-in connections do not count as "write" connections. However, file downloads with static or dynamic throttling enabled do count as "write" connections.

If you have:

...then the Client without dynamic throttling will get 250 KBPS of bandwidth allocated. The bandwidth usage of the Client with dynamic throttling will be determined by the dynamic throttling algorithm -- it may turn out to be much less or greater than 250 KBPS, so that the total bandwidth usage of the server will not necessarily be 500 KBPS.

Server-side settings are in KBPS.

For Relay and Root Server:

For Web Reports:

Client Side

Client-side static throttling is the simplest of these settings. A "Client" (could be a Client or a Relay) simply tells its parent "please send me files at this speed" and the parent obliges. Settings are in BPS.

For a BES Client:

For a BES Relay downloading files from its parent:

Throttle Groups

"Throttle Groups" are a part of the static throttling functionality that allow a set of Clients to throttle themselves as a group instead of individually (or along with every other connection through Server-side throttling). If you use this functionality, please tell me (Chris Loer): I suspect that this complication is unnecessary and would like to remove it for sanity's sake.

When a Client identifies itself as part of a "throttle group", it sends up the name of the group it belongs to, along with the speed it would like the entire group to have. So for example a Client might say "I'm in the 'remote' group, and we'd like to be given 10000 BPS as a whole". When the Server sends data down to that Client, it throttles based on the total number of connections in the Clients group. So if there are five active connection from the 'remote' group, our Client will get 2000 BPS. Note that different Clients can send up different values for the "limit bytes per second", so another Client could say "I'm in the 'remote' group, and we'd like to be given 5000 BPS as a whole", and it would be given 1000 BPS at the same time our first Client was given 2000 BPS.

The special group "ipaddress" will cause the Server to group this connection along with other connections from the same IP address. This is the default for Relay upstream traffic. Clients default to the group "<my computer id>", so that their "LimitBytesPerSecond" setting is shared among all of their currently active file downloads.

Dynamic throttling is unaffected by throttle groups (an interesting side effect of this is that a Client set to target 20% of available bandwidth may end up using 40% if it's downloading two files simultaneously).

Dynamic Throttling

Dynamic throttling is co-operative: this means it must be turned on for both the server-side and client-side in order for any throttling to happen. Once you have turned on dynamic bandwidth throttling, BES components will try to calculate the amount of "available" bandwidth on the link they are using. They will then use a percentage of the bandwidth that you specify, with minimum and maximum bounds.

Server Side

Server-side dynamic throttling settings control the total amount of download traffic that a server will send out to Clients using dynamic throttling. The "min", "max", and "percentage" settings are all divided by the number of active write connections. Note that plug-in connections do not count as "write" connections. However, file downloads with static or dynamic throttling enabled do count as "write" connections.

If you have:

...then the Client with dynamic throttling will get 25 percent of estimated bandwidth allocated. Note that this is an estimate of the bandwidth between that specific Client and the server, so another Client with the same settings might end up with a different amount of bandwidth. There is no way to say "use 50% of all outgoing bandwidth" because the server may have different bandwidth availability for different connections. However, if 10 Clients are all connecting over the same thin pipe, and the server is set to target "20%" usage as a whole, than it will target "2%" for each of the ten Clients and should end up at roughly "20%" usage for the pipe.

Server-side settings are in KBPS.

Client Side

Settings are in BPS.

For a BES Client:

For a BES Relay downloading files from its parent:

Useful Tools for determining bandwidth usage

Netlimiter 2 Monitor is a very useful freeware tool that allows you to track bandwidth usage per process in multiple unit types.

Downloadable from http://www.netlimiter.com