ADC HA node fail to load Admin GUI and/or Gateway page after upgrade from 11.1/12.0 to 12.1/13.0

Pages stay stuck in blank empty screen. Developer Tools in the browser shows missing content 404 like .css, js, etc.

http.conf is being SYNCed to secondary even though HA SYNC or PROP are disabled. Right now “file sync” would still happen, even though HA SYNC is disabled. Only way to disable file sync is to comment out fsyncd from crontab.

Please follow these steps carefully to complete the upgrade procedure to 12.1 with httpd.cof customization.

1. On Secondary node, prior to upgrade, go to the Shell prompt and comment out the nsfsyncd line in the /etc/crontab


2. To persist the changes across reboots, copy the modified /etc/crontab to /var/tmp folder using the following commands:
> cp /etc/crontab /var/tmp/crontab.modified; echo cp /var/tmp/crontab.modified /etc/crontab >> /nsconfig/rc.netscaler

3. Remove the httpd.conf from /nsconfig folder if present
4. Upgrade to 12.1 on secondary
5. After reboot, 12.1 comes with default httpd.conf in /etc.
6. Copy httpd.conf from /etc to /nsconfig
7. (Optional) New edit httpd.conf in /nsconfig to change the maxclient to 128 (this is to augment Apache capacity for 1000+ concurrent users ADC environments)
8. Reboot the secondary again and once it comes up, test the GUI etc.
9. If GUI access succeeds then this completes the Upgrade of secondary successfully
10. Now, if there was any change in HA Sync, change it to normal -participate in HA-
11. Do a force failover to this node to further test.
12. On old primary with 11.1, remove the httpd.conf from /nsconfig and upgrade the other node
13. To enable the file synchronization process, remove the command added to the rc.netscaler and revert the changes to /etc/crontab
14. Make sure that httpd.conf got synced to newly upgraded node
15. Access GUI on both nodes

 

---

Problem Cause

1. From 12.0 onwards, PHP library version changed, so any upgrade from 11.1 to higher version with customized http.conf in /nsconfig would fail, in other words httpd wouldn’t run after upgrade.
2. For above problem, removing the httpd.conf from /nsconfig and upgrade would have worked, but since HA is present, file sync is happening which is syncing the httpd.conf from Primary before upgrade resulting in same issue as in 1.

When upgrading a system with these customizations, there is a message produced by “installns”:

“WARNING:A symbolic link exists for httpd.conf.  Please review it for compatibility!!!”

Unfortunately customers tend to gloss over this message which is very important and intended for their protection.

Customer did tried to remove the httpd.conf from /nsconfig before upgrade/reboot but “HA file sync” quickly got this file back from Primary which can created the problem.

Even though “config SYNC” is disabled, file sync would continue to happen between nodes.