"show cluster node"  output shows the below error
Health: NOT UP Reason(s): Cluster health is not up due to config sync is in progress The config sync operation
(full sync) is in progress

 "show arp" output shows the below error
Node Id: 2 Error connecting to remote node <IP> : Invalid username or password Done

Resolution is to reset the rpcnode passwords in cluster. Since the cluster. nodes are out of sync, this is kind of tricky to change the rpc passwords. 
Note: This procedure is applicable only two node cluster.
We can follow the below procedure to revert the rpc passwords to default so that cluster comes up fine and then you can change the passwords from CLIP.

1. Disable cluster instance on nodeB
2. From clip unset the rpc paswd of both NSIPs and CLIP
3. Disable Cluster Instance on NodeA
4. enable cluster instance on NodeB
5. After the clip is reachable again, From clip unset the rpc paswd of both NSIPs and CLIP
6. Disable cluster instance on NodeB
7. enable cluster instance on NodeA
8. Once node A is UP, enable cluster instance on NodeB
9. Once the system is stable change rpc passwords from clip.

Problem Cause

Genreally these issues were due to rpc password of node A on node B is not correct, hence its unable to connect to node A as is evident from the below logs on nodeB

 

Apr 10 18:00:09 <local0.info> hostname [36983]: nsnet_connectremoteas: user name and password combination does not match

Apr 10 18:00:09 <local0.info> hostname[36983]: cfsync_pullcfg_n_apply(): Error connecting master nsconfigd: <IP>

Apr 10 18:00:09 <local0.info> hostname [36983]: cfsync_pullcfg_n_apply(): Connection failed [Could not connect to any of master nodes provided in sync start message from clusterd]