BES Performance

The BES Server installer includes an installer for MSDE 2000 SP3a. Due to restrictions built into MSDE 2000 by Microsoft, when too many database connections are open, MSDE will slow itself down. The BES Server components use several database connections and each BES Console operator uses a database connection. When multiple BES Console operators are using BES at the same time, you will most likely see performance degradation due to the MSDE restrictions. This can corrected by using SQL Server 2000 instead of MSDE 2000.

By default, Microsoft SQL Server 2000 uses "Full" transaction logging to allow for optimal data recovery. To get better performance from BES, you should change the transaction type to "Simple". Here are instructions on how to change the SQL Server 2000 transaction logging type. If you are using MSDE 2000, you will not need to change this setting because MSDE 2000 defaults to using "Simple" transaction logging.

By default, the BES Server downloads patches, upgrades, etc. from the Internet and then the BES Clients download the files from the BES Server. If the BES Server is in a separate geographic location from a group of BES Clients with a slower network connection, each BES Client must download the file over the slow link from the BES Server.

For example, imagine that there is a company with a main office in LA with the BES Server and a remote office in NY with 500 BES Clients connected using a T1. If Windows 2000 SP4 (120 MB download) is deployed to each of the BES Clients in NY, there will be a total of 60 GB of network traffic, which can easily overwhelm the T1 connection (this situation would be much worse if there is a slower connection of 256 kbps or even 56 kpbs), which is used for many other activities.

To avoid this unattractive use of bandwidth, a BES Relay should be installed on a computer in NY and the BES Clients in NY should be configured to download from the BES Relay. Using this configuration, the BES Relay will download and cache each file from the BES Server and the BES Clients will download directly from the BES Relay. In the previous example, the total network traffic across the link from LA to NY would be reduced from 6 GB to 120 MB. Note that the BES Relays will also group and compress communications other than downloads from the BES Clients to the BES Server providing even more bandwidth savings.

For more information about BES Relays, see http://support.bigfix.com/bes/misc/besrelays.html.

For all but the smallest BES deployments (< 500 BES Clients), a primary BES Relay should be set for each BES Client even if they are not in a remote location. The reason for this is that the BES Server performs a lot of tasks including gathering new Fixlet content from the BigFix servers, distributing new Fixlet content to the BES Clients, accepting and processing reports from the BES Clients, providing data for the BES Consoles, sending downloaded files (which can be quite large) to the BES Client, and much more. By using BES Relays, the burden of communicating directly with every BES Client is effectively moved to a different computer (the BES Relay computer), which frees up the BES Server to do other tasks. If BES Relays are not used, you may see that performance degrades significantly when an action with a download is sent to the BES Server and you may even see errors.

Setting up BES Relays in the appropriate places and correctly configuring the BES Clients to use them (either by letting the BES Client auto-select their closest BES Relay or by manually configuring the BES Clients to use a specific BES Relay) is the most important and highest impact performance change.

For more information about BES Relays, see http://support.bigfix.com/bes/misc/besrelays.html.

By default the BES Clients "check in" to the BES Server on a regular interval known as a "heartbeat". When the BES Clients send in a heartbeat to the BES Server, they will update their "Last Report Time" property along with any other properties that have changed since the last heartbeat. In medium to large BES deployments, processing the heartbeats can consume significant BES Server resources. To ensure optimal performance, the heartbeat should be raised from the default 15 minutes to 1 hour or even 2-6 hours for larger BES deployments. The heartbeat can be changed under the File > Preferences menu in the BES Console.

When the BES Console is being used, the BES Console will query the BES Server database and cache the results locally. The cache is updated according to the BES Console refresh period. The more BES Clients, the more data is transferred to the BES Console using database resources and network bandwidth. The BES Console refresh period should be raised to from its default of 15 seconds to 30 seconds, 60 seconds, or even 120 seconds for large deployments with lots of simultaneous BES Console users. The refresh rate can be changed under the File > Preferences menu in the BES Console (note that this setting is per BES Console.

Note: Starting in BES 5.0 and above, you can use the BES Administration tool to set a global minimum refresh rate for all your BES Console operators. You should set this to a higher number (30-60 seconds might be a good starting point) if you are experiencing slowness in the BES Console.

The BES Server and BES Relay's normal operations involve creating and processing a lot of temporary files. This activity is essential for good performance of BES, but can be slowed down dramatically if a virus scanner is scanning each file. To address this issue, configure your virus scanner on the BES Server and BES Relay computers to exclude the BES Server folder and all subfolders (default is "C:\program files\bigfix enterprise\bes server") or the BES Relay and its subfolders (default is "C:\program files\bigfix enterprise\bes relay"). Refer to instructions from your virus scanner for more information on how to set this exclusion rule.

The BES Server and BES Relay's normal operations involve creating and processing a lot of temporary files. This activity is essential for good performance of BES, but can be slowed down dramatically if Windows file indexing is turned on or if the drive is set to use file compression. If these computers have indexing or compression enabled, you should disable them.

The hard drive performance of the BES Server is very important to the overall system performance. If improperly configured, RAID controllers can cause severe performance problems. Specifically, some on-board RAID controllers for hard drives come configured with the hard drive cache set to 100% Read/0% Write performance. This configuration will cause very poor performance because all hard drive writes will be very slow. The optimal configuration is 50% Read/50% Write performance. Refer to your RAID instructions for how to set this. If the on-board controller cannot be configured for 50% Read/50% Write performance, then you should purchase an off-board controller.

The hard drive performance of the BES Server is very important to the overall system performance. If improperly configured, RAID controllers can cause severe performance problems. The BES Server will both read and write lots of data to disk. For this reason, RAID 0 or RAID 0+1 are all good choices for RAID configurations (note that RAID 0 offers no data redundancy).

The BES Server performance is affected heavily by the write performance of the computer. The BES Server writes many files to disk and also writes data to the database at the same time. If the BES Server's program folder is on a separate physical drive than the SQL Server database, the hard drive writes can be done in parallel, which can lead to speed increases.

The BES Server and BES Relay's performance are partially bound by bandwidth -- especially for large file downloads. By using gigabit network connections or dual network cards on the BES Server and BES Relays, you can effectively increase the bandwidth and increase the speed of the BES Clients downloading and reporting.

When the BES Console connects to the BES Server computer, all of the data about the BES Clients, the Fixlet messages, actions, etc. are sent from the BES Server to the BES Console. The amount of information transferred grows as the number of BES Clients grows and can become quite large for medium to large deployments. By using the management rights feature to restrict the number of BES Clients managed by each BES Console operator, the bandwidth between the BES Server and BES Console can be reduced significantly because the BES Console will only receive information about a subset of the BES Clients.

When the BES Console operator takes an action, there is an option under the "Execution" tab to "Distribute the execution of this action over __ minutes to reduce network load". If this setting is not used, the targeted BES Clients will all simultaneously attempt to execute the action, which usually involves downloading a file from the BES Server (or BES Relay). Not only will this put a large load on the BES Server, but you may also use a significant load on total network bandwidth (this is especially a problem if BES Relays are not used). To avoid this problem, always set the "temporal distribution" setting. You should set the number higher (120 minutes) if many BES Clients are going to download a large file and set it lower (30 minutes) when targeting only a few BES Client computers or when deploying smaller patches.

The system requirements for the BES Console computer increase substantially for medium to large deployments. If you notice that the BES Console is running very slowly, using a lot of memory, or constantly causing HD access, you should use the BES Console on a more powerful computer. In particular, adding more memory can significantly improve the performance of the BES Console. The BES Console recommendations are available here.

Actions are either in the Open, Closed, or Stopped state. Open actions are sent to BES Clients so that the BES Client can deteremine if it is supposed to run the actions. Each open action puts a small additional load on the BES Client. If there are many open actions, the BES Client will be a little slower to respond to newly relevant Fixlets/actions. It is good practice to close open actions if there is no need to have them open. As a reference point, over 10,000 open actions is a very large number of open actions; 5,000 is pretty large; 1,000 is a little high; 200 should be no problem.

The BES Console will list every action that has been taken including every result from every BES Client. As the number of actions grows large, the BES Console will use more memory to load the actions and the cache load/write times increase. You can right click and delete old actions, which will mark them as deleted in the database. Once they are marked as deleted, the BES Console will no longer load them saving memory and load time. The actions will continue to be in the database after they are deleted, but they will not be accessible to the BES Console or to web reports.