BES Network Traffic Guide

BES is designed to be a real time (seconds and minutes) system capable of workng in extremely limited bandwidth environments. When everything is set up and configured properly, BES can effectively and quickly deploy files that are hundreds of MBs large to tens of thousands of computers without significantly affecting general network performance. However, because of the speed and prevalence of BES in the network, it is very important that network engineers understand the different types of traffic that BES can generate.


The following information describes the different types of network traffic and explains the purpose of the traffic as well as information about how to handle "Possible Worst Case Traffic". (NOTE: The "possible worst case traffic" are definitely not expected; however, in the interest of preparing for the worst case scenario, example situations are given as well mechanisms to recover from the situation). This information is intended to educate network engineers and provide the basis of a disaster recovery procedure related to BES.


Please refer to the following diagram in the network discussions below:

A

Network Traffic: Gather / Post / Download
Purpose: The BES Client will contact its BES Relay (or BES Server) for several reasons including: gathering the latest information about actions and Fixlet messages, sending results about the computer properties or status of actions/Fixlets, and downloading files (such as patches).
Port: 52311 (configurable by the BES Administrator at install time)
Protocol: HTTP (TCP/IP)
Origination: The BES Client initiates the request to the BES Relay (or BES Server).
Expected Traffic Frequency:
  • Gather: The BES Clients will gather the latest Fixlet content or actions once a day OR whenever a new action or new Fixlets are available from the BES Relay. Each gather is approximately 1 KB - 3 KB (only compressed differences are gathered). An active deployment will have this occur many times (up to many hundreds of times a day when the BES Console operators are very active)

  • Post: The BES Clients will send results of their actions / Fixlets (approximately 1 KB each submission) for each action. The BES Clients will also send "heartbeats" periodically (configurable by BES Administrator - default 5 minutes) to the BES Relay of about 1 KB that contain the differences in the BES Clients properties.

  • Downloads: Downloads only occur when a BES Console operator sends out an action to tell a BES Client to download and run a file. The frequency of this operation is completely dependent on the activity of the BES Console Operators.

Note:

If BES is properly configured, the traffic described is between the BES Client and the BES Relay (which is usually a high bandwidth connection) and the information between the BES Relay and the BES Server (sometimes a slow bandwidth connection) is minimal (gather/download information only sent once and posts are compressed and forwarded to the BES Server).
Expected Traffic Volume:
  • Gather: Low - Perhaps 5 KB - 50 KB per day per BES Client spread throughout the day.

  • Post: Low - Perhaps 10 KB - 50 KB per day per BES Client spread throughout the day.

  • Downloads: High - If no actions are pushed, then no downloads will occur. If a patch is downloaded, the BES Client will download the file (30 KB - 300 MB) from the BES Relay.

Note:

This traffic varies heavily depending on usage. A very active deployment (BES Console operators sending lots of actions) might see numbers that are 10x the numbers above.
Possible Worst Case Traffic:

In a properly configured BES deployments, the BES Relays always have a high-bandwidth connection to the BES Relays to minimize network traffic on the slow network connections. The worst case scenario is if the BES Relays are improperly configured or the if the BES Clients or BES Relays fail in some form while a large download is being pushed. For instance, if all the BES Relays simultaneously failed while Windows XP SP2 (250 MB) action is sent, the BES Clients will all begin to download the file from the BES Server. In such a scenario, all the slow network connections between the BES Clients and BES Server will be flooded with HTTP traffic on port 52311.


This situation can be recovered from by stopping the action immediately in the BES Console (depending on how busy the network is, this could take 2 mins - 2 hours), blocking TCP on port 52311 at the network routers, or by shutting down the BES Server/BES Relays (this can be done through BES, thus eliminating the source of the download servers).
Network Controls: BES provides the following controls in the product for this type of traffic:
  • Configurable bandwidth throttling to BES Relay or BES Clients
  • Configurable gather interval (default 1 per day per Fixlet site)
  • Configurable minimum time to wait between posts (default 15 sec)
  • Configurable temporal distribution (spread out downloads over time) per action
  • Ability to set "policy" to prevent computers from downloading files if they are not pointed at the proper BES Relay
Other Notes: One of the major focuses of deploying BES is to get the BES Relays in locations that are near the BES Clients so that WAN traffic is minimized and the vast majority of the network traffic is between the BES Clients and the BES Relays on a fast LAN connection. BES offers a number of reports to see the status of the BES Clients and which BES Relays they are pointing to and it is the job of the BES Administrators to make sure the BES Relay and BES Clients are properly configured.

B

Network Traffic: UDP "new information" message
Purpose: When there is new information available such as new Fixlet messages or new actions, a UDP message is sent to the BES Clients' last known IP addresses to tell them to gather the lastest data from the BES Server.
Port: 52311 (configurable by the BES Administrator at install time)
Protocol: UDP
Origination: The UDP messages are sent from the BES Clients' immediate "parent", which can be either a BES Relay or the BES Server.
Expected Traffic Volume: Low - The UDP messages are sent from the BES Relay to the BES Client every time a new action is sent or a new Fixlet message is gathered. The BES Relay will send 3 UDP messages to each BES Client (to help ensure delivery). Each UDP message is about 60 bytes.
Expected Traffic Frequency: In an idle deployment (no actions being sent) there will be virtually no UDP traffic. In a very active deployment where many actions are sent, there will be 3 UDP messages sent per action per BES Client (as a reference point, sending 100 actions over the course of a day is considered very high BES usage). Note that if the BES Relays "nearby" the BES Clients, the UDP traffic will be sent on the LAN and the WAN will have very little UDP traffic.
Possible Worst Case Traffic: The worst case scenario for this type of traffic is for the BES Relays to fail and send much more traffic than they are supposed. In such a scenario, the BES Relays would be repeatedly sending UDP messages to each of the BES Clients. There is a control built in for the BES Relays never to send more than 500 UDP messages per second (configurable), but if there were many BES Relays or many BES Clients, this UDP traffic could potentially cause lots of network traffic or router usage. In such a situation, you would see massive amounts of UDP traffic on port 52311 from the BES Relays.

To recover from this situation, UDP traffic on port 52311 could be blocked at the routers or you could shut off the BES Relays (can be done through BES).

Network Controls: BES provides the following controls in the product for this type of traffic:
  • Configurable limit of the number of UDP messages sent at one time from a BES Relay
  • Configurable limit of the amount of time to wait after sending UDP messages from a BES Relay
Other Notes: It is not strictly necessary for BES to use UDP traffic for the system to work properly, but the UDP messages allow the BES Clients to be notified of new information as opposed to repeated polling by the BES Clients. Thus, the UDP messages can help reduce overall network usage and increase response time.



C

Network Traffic: BES Relay Selection
Purpose: In a well configured BES network, each BES Client is configured to communicate with a local BES Relay; however, manually selecting the BES Relay for each BES Client can take a lot of time and is error prone. To solve this problem, the BES Clients can be configured to find their closest BES Relay based on the number of network hops. To find their closest BES Relay, the BES Client use the ICMP protocol (similiar to tracert).
Port: N/A (the ICMP protocol doesn't use a port)
Protocol: ICMP
Origination: Each BES Client will send progressive "rounds" of ICMP packets to each BES Relay with increasing TTLs until a BES Relay responds. For example, in a network of 2 BES Relays, one 1 hop away and one 2 hops away, the BES Client will send a ICMP message to both with TTL 1 and will receive 2 "time exceeded" messages from the local router. The BES Client will then send an ICMP message to both BES Relays with TTL 2 and will receive one "time exceeded" message and one reply message. BES Client will then choose the BES Relay that is one hop away.
Expected Traffic Volume: Low - The volume of the ICMP traffic is directly porportional to the number of BES Clients, number of BES Relays, and number of hops to each BES Relay. For example, in the example above with a BES Client and 2 BES Relays, the number of pings is 1 BES Client * 2 BES Relays * 2 hops to closest BES Relay = 4 ICMP pings. Each ICMP packet and the response is about 50-60 bytes. Note that if BES Relays are close to the BES Clients, the total network traffic is reduced considerably than if the BES Relays are far away from the BES Clients.
Expected Traffic Frequency: By default, the BES Clients attempt to find a better BES Relay once every 6 hours (configurable) or every time the BES Client is started.
Possible Worst Case Traffic:

The worst case scenario for this type of traffic is for a large number of BES Clients to have a large number of BES Relays that are very far away or for the BES Client to fail in some way that makes them repeatedly send ICMP packets. In such a scenario, there would be high amounts of ICMP packets being sent very quickly from the BES Clients to each of the BES Relays.


To recover from this situation, you can block ICMP at the routers or turn off BES Client autoselection (can be done through BES in anywhere from 2 min to 2 hours depending on how busy the network is).
Network Controls: BES provides the following controls in the product for this type of traffic:
  • Relay autoselection can be disabled
  • Configurable interval for when the BES Clients perform autoselection
  • Configurable limit on the maximum number of ICMP packets to send out in a given time interval (available in BES 5.0 and above)
  • Configurable limit on the maximum number of "rounds" to send out during relay autoselection (available in BES 5.0 and above)
Other Notes: The BES Client autoselection can be a huge savings of time and effort because it allows the network to change (new BES Relays added, BES Relays removed, routers go up or down, etc.) and the BES Clients will automatically change with the newtork. However, if necessary, manual BES Relay selection can be used too.



D

Network Traffic: BES Console traffic
Purpose: To administer BES, the BES Console operators will connect to the database on the BES Server computer.
Port:
  • 52311 for sending actions (configurable at install time)
  • Database connection port varies (usually 1433)
Protocol:
  • HTTP (TCP/IP) for sending actions
  • Database connection protocol varies (usually ODBC over TCP/IP)
Origination: The BES Console will connect to the database on the BES Server.
Expected Traffic Volume: High - There is a lot of information (computer properties, vulnerabilities, etc.) sent between the BES Console and the database. The amount of traffic is usually measured in MB per minute, but can vary considerably depending on the size of the deployment. Note: there are usually very few BES Consoles active and they are usually on a high-speed connection.
Expected Traffic Frequency: Whenever a BES Console operator is administering BES, the database will repeatedly be polled for new data.
Possible Worst Case Traffic: It would be unlikely that this type of traffic would cause a "catastrophic" situation because unlike the situations involving the BES Clients, there are very few BES Consoles. If a BES Console user connected to the BES Server and repeatedly hit "F5" to refresh the data, there would be a lot of traffic between the BES Server and BES Console.
Network Controls: BES provides the following controls in the product for this type of traffic:
  • There is a "refresh rate" for each BES Console user (default 15 seconds)
Other Notes: If the connection is slow between the BES Console and the BES Server, it is not easy to administer BES so usually the BES Console has a high-speed connection to the BES Server.



E

Network Traffic: BES Server gets new data from external Fixlet servers and download files from various download servers (such as Microsoft)
Purpose: The BES Server will periodically check to see if any new Fixlet messages are available from the Fixlet server on the public Internet. Also, when a file needs to be downloaded, the BES Server will download and cache the file from a download server (for instance, all Microsoft patches are downloaded from Microsoft's download servers)
Port: 80
Protocol: HTTP (TCP/IP) (sometimes FTP for downloads)
Origination: The BES Server will connect to the BigFix Fixlet servers or download servers on the public Internet.
Expected Traffic Volume: Low - The initial gather of all Fixlet sites will likely be less than 500 KB. Subsequent updates will usually be less than 1-2 KB of traffic. Downloads, however, can be quite large, but since the files are cached, they will only be downloaded one time.
Expected Traffic Frequency: The BES Server will check for new Fixlets once every hour by default. Files will be downloaded whenever a BES Console operator sends an action that requires a non-cached patch.
Possible Worst Case Traffic: It would be unlikely that this type of traffic would cause a "catastrophic" situation because unlike the situations involving the BES Clients, there is only one BES Server. If the BES Server was sent many many actions or somehow malfunctioned, it might download lots of files from the Internet, but it seems unlikely that this would cause a problem.
Network Controls: BES provides the following controls in the product for this type of traffic:
There is a configurable interval that the BES Server will check for new Fixlet messages.
Other Notes: The BES Server is the only component in a default BES installation that contacts the Internet directly.

More information about BES Relays can be found here and more general documentation about BES can be found here.