Problem Definition
An alert has been configured to fire when CPU utilization has reached more than 90 percent for a period of 5 minutes or more. An email action is also configured. Email alerts are received from the Citrix EdgeSight server similar to the following:
Note: Email alerts do not necessarily need to be configured for this issue to arise in any environment.
"Subject: Edgesight System Slow CPU > 90% for 5 minutes
Alert Summary — System slowdown alert: 1 | |||
Source |
Date |
Device\User | |
System performance |
3/4/2008 7:42:49 AM |
Server1 | |
System performance thresholds were exceeded. Conditions generating this alert are: TOTAL_PROCESSOR_TIME: 90. | |||
The alert explains that at 7:42:49 AM on 3/4/2008, the CPU utilization was equal to or greater than 90 percent for a period of 5 minutes or more for the Server1 computer.
For comparative testing, a third-party utility was also running on Server1 and it captured data showing that the CPU utilization never reached 20 percent during the specified time period.
Environment
Troubleshooting Methodology
Alerts are generated by each agent. Agent Worker 150 creates a bullet and triggers the alert status in the EdgeSight Console. All current alerts can be monitored under Real Time > Alerts > Current Alerts.
Under this node, it was found that Server1 was not listed as having any alerts. There was a similar alert for Server2 where the performance thresholds had been exceeded. However, no email alert had been sent or received concerning this server.
Using the same third-party utility mentioned earlier, we found that Server2 experienced CPU spikes around 7:42 AM.
It was necessary to determine when the last Performance Upload was performed for the servers. This can be found under Company Settings > Server > Device Management > Devices.
Last Upload for Server1 – Feb 15, 2008
Last Upload for Server2 – Mar 4, 2008
Note: Server1 had not uploaded in almost a month.
The next logical step was to manually force a Performance Upload for Server1 by navigating to Company Settings > Agent > Workers > Run Remotely. We Specified Server1 in the Device field. Running both a Configuration Check and Performance Upload were successful.
Navigating back to Real Time > Alerts > Current Alerts now displayed some interesting results: Server1’s moniker had been replaced with Server2 for the alert.
So, why was EdgeSight confusing the server names? When the EdgeSight Agent first runs, Worker 104 creates the Firebird database, and in turn, a unique identifier for that particular computer. As it turns out, the agent machines were cloned after the EdgeSight Agent had run its first time. Now, all the agents had the same Unique Identifier (UID). Hence, the payloads were being overwritten by each other and the alerts, as well as reports, and so on, were skewed.
Resolution
New unique identifiers had to be created for the two servers. To do so:
1. Stop the Citrix System Monitoring Agent and Firebird Server – CSMInstance services on both servers.
2. Delete the RSDATR.FDB firebird database located at C:\Documents and Settings\All Users\Application Data\Citrix\System Monitoring\Data.
3. Restart the services. Within a few minutes the Worker 104 kick offs, creating a new database and UID for each server. These actions must be repeated on any remaining EdgeSight Agents.
4. As best practice, when cloning servers or workstations with the EdgeSight Agent preinstalled, create the image before allowing the agent to run for the first time (or immediately after installing it, prior to the restart).
