Case Study: Users Cannot Connect to Their Published Applications (Introduction to the Citrix MedEvac Utility)

Problem Definition

Users are unable to connect to their published applications. When a user logs on and attempts to launch any published application, they receive one of the following error messages:

"Server unavailable" or "Cannot connect to the Citrix MetaFrame Server. I/O error"

This article provides an introduction to the Citrix MedEvac Troubleshooting utility.

Environment

§ Citrix Presentation Server 4.0

§ Windows Server 2003 Enterprise with Service Pack 1

§ Microsoft SQL 2000 with Service Pack 3a data store

§ 300 servers

§ 3 zones

§ Web Interface 4.0 connection method

§ Presentation Server Client version 9.0

Troubleshooting Methodology

The information known about this issue is as follows:

• Issue is affecting all users

• Issue has never occurred before in this environment

• Nothing has changed in the environment

• Users are connecting through Web Interface 4.0

• Web Interface does enumerate the published applications

• Connecting directly to a server works

• A custom ICA connection to a published application exhibits the same problem

• Not all applications are published to servers from ALL zones – some applications are published to specific zones only

• The default load evaluator is being used?

• No Event Log Errors seem suspicious on the Zone Data Collectors

• With so many servers the Admin has not isolated or knows where to begin isolating the issue

• All servers have active sessions on them

• The Zone information in the Management Console displays information correctly indicating that the Zone configuration for the farm is correct

• QFARM returns valid information

• The information in QFARM and QFARM /ZONE matches

• QFARM /OFFLINE does not return any servers

• Forcing an election of a new ZDC per zone does not help the issue

• Recreating the Local Host Cache on the ZDC’s did not help issue

Isolation Steps

Because direct Server connections seem to work we could publish an instance of Notepad.exe, with a unique name for each individual server and make custom ICA connections to each published instance of Notepad.exe. However, with 300 servers this would be virtually impossible at peak production time since this has significant business impact.
Had the customer publish an individual instance of Notepad.exe and assign it to 20 servers each per zone.

Example: Notepad1 on servers 1-20, Notepad2 on servers 21-40 etc... This way we can try to narrow the issue to a problem server or servers.
We would have to use the application user load evaluator to properly test the issue for each group...

Customer was able to reproduce the problem connecting to Notepad on two separate Server Groups in the same Zone...(zone3).
Advised customer to now make direct ICA connections to each of those servers to pinpoint the problem server(s). Customer identified three problem servers.
Recreated the local host cache on those three servers - IMA hung on stopping...

Resolution

Using PSKill from http://www.sysinternals.com we manually forced termination of the IMA service.
Recreated the local host cache.
Restarted IMA - started successfully.
Deleted the published Notepad(s) used for isolating the issue.
Users able to launch their published applications as normal now.

Additional Information

The previous troubleshooting methodology can be time consuming and ineffective based on certain circumstances. Businesses usually have major impact when users are unable to connect at peak production hours and if a problem like this does occur, any time wasted can be detrimental to productivity and costs.

Understanding the severe business impact of issues such as described above; the Citrix Escalation Team has developed a new troubleshooting utility to aid in pinpointing the problem with these types of issues, thus increasing efficiency when troubleshooting.

This utility is called Citrix MedEvac 1.0 and can be obtained from the Citrix Knowledge Base:

CTX107935 – Project MedEvac 2.11