By default Netscaler tries to do SSO to RDP server, due to this if backend RDP server is in different domain then the single sign on credentials won't work,
Now Netscaler doesn't send a prompt to user for entering the credentials due to which it fails.

From 11.0.64.34 version you can disable SSO to RDP server so that you are prompted for credentials.

You can add following policy and bind it to your Netscaler gateway. ( your netscaler gateway should be 11.0.64.34 build as below build it’s not supported)
add vpn trafficAction t_act1 http -SSO OFF
add vpn trafficPolicy t_pol1 "REQ.HTTP.URL CONTAINS rdpproxy" t_act1
bind vpn vs -policy t_pol1

This policy will disable the SSO and you should be prompted for a authentication.

---

Problem Cause

Netscaler by default tries to do SSO to the RDP Servers with the cached credentials.

While doing the RDP via Netscaler Gateway you will see an error that "Remote Desktop can't connect to remote desktop"

Customer's might run into one more issue if they disable SSO as mentioned in article above. Please refer the below article for further confirmation and troubleshooting:

http://support.citrix.com/article/CTX207025

Note: RDP Proxy with SSO is not tested / Supported on Win2008.On Windows based OS Win2008R2 / Win 7 / 8 /10 this has been Tested.