While SDX is backed up, it fails with error "Backup Operation failed"


Following logs were observed :

mps_service.log :
 

Monday, 9 Jul 18 07:42:14.792 +0000 [Debug] [TCPServerConnection (default[#773])] Request from 10.2.25.38:60001 - POST - SSLCertPresent=false - type=https - /nitro/v1/config/backup

Monday, 9 Jul 18 07:42:14.792 +0000 [Debug] [TCPServerConnection (default[#773])] Json Web Content: {"backup":{"backup_password":"***********"}}

Monday, 9 Jul 18 07:42:14.793 +0000 [Debug] [TCPServerConnection (default[#773])] [GUI] Incoming request in ServiceProcessor for type "backup"

Monday, 9 Jul 18 07:42:14.857 +0000 [Debug] [TCPServerConnection (default[#773])] Sending Message to CONFIG /tmp/mps/ipc_sockets/mps_config_sock:{ "errorcode": 0, "message": "Done", "is_user_part_of_default_group": false, "skip_auth_scope": true, "message_id": -1, "resrc_driven": true, "login_session_id": "##7E875767D046786C75A8842FC9719788233249813DCCF55880198BF4CB52", "username": "xxxx", "tenant_name": "Owner", "mps_ip_address": "xxxxx", "client_ip_address": "xxxxxx", "client_protocol": "https", "client_port": 60001, "mpsSessionId": "", "source": "service", "target": "CONFIG", "version": "v1", "messageType": "config", "client_type": "GUI", "resourceType": "backup", "orignal_resourceType": "backup", "resourceName": "", "operation": "add", "asynchronous": false, "params": { "pageno": 0, "pagesize": 0, "detailview": true, "compression": false, "count": false, "total_count": 0, "action": "", "type": "", "onerror": "EXIT", "is_db_driven": false, "order_by": "", "asc": false, "duration": "", "duration_summary": 0, "report_start_time": "0", "report_end_time": "0" }, "additionalInfo": { "Referer": "https:\/\/cs-2-cgprod2-mgmt\/admin_ui\/svm\/html\/configuration.html", "cert_present": "false", "rand_key": "f84d69cc3a32b24", "request_source": "NITRO_WEB_APPLICATION", "sessionId": "##7E875767D046786C75A8842FC9719788233249813DCCF55880198BF4CB52" }, "backup": [ { "backup_password": "***********" } ] }

Monday, 9 Jul 18 07:43:04.909 +0000 [Debug] [TCPServerConnection (default[#773])] Received Message from CONFIG:{ "errorcode": 30030, "message": "Backup Operation failed", "severity": "ERROR", "resourceName": "backup"

mps_config.log
============

Monday, 9 Jul 18 00:30:51.477 +0000 [Information] [default[#1]] Backup:ns

{ "errorcode": 0, "message": "Done", "is_user_part_of_default_group": false, "skip_auth_scope": true, "message_id": -1, "resrc_driven": true, "login_session_id": "##7E875767D046786C75A8842FC9719788233249813DCCF55880198BF4CB52", "username": "dever", "tenant_name": "Owner", "mps_ip_address": "xxxxxx", "client_ip_address": "xxxxx", "client_protocol": "https", "client_port": 60001, "mpsSessionId": "", "source": "service", "target": "CONFIG", "version": "v1", "messageType": "config", "client_type": "GUI", "resourceType": "backup", "orignal_resourceType": "backup", "resourceName": "", "operation": "add", "asynchronous": false, "params": { "pageno": 0, "pagesize": 0, "detailview": true, "compression": false, "count": false, "total_count": 0, "action": "", "type": "", "onerror": "EXIT", "is_db_driven": false, "order_by": "", "asc": false, "duration": "", "duration_summary": 0, "report_start_time": "0", "report_end_time": "0" }, "additionalInfo": { "Referer": "https:\/\/cs-2-cgprod2-mgmt\/admin_ui\/svm\/html\/configuration.html", "cert_present": "false", "rand_key": "f84d69cc3a32b24", "request_source": "NITRO_WEB_APPLICATION", "sessionId": "##7E875767D046786C75A8842FC9719788233249813DCCF55880198BF4CB52" }, "backup": [ { "backup_password": "***********" } ] }

Monday, 9 Jul 18 07:42:14.858 +0000 [Debug] [Main] Incoming request in ConfigProcessor for type "backup"

Monday, 9 Jul 18 07:42:14.858 +0000 [Debug] [Main] File to be parsed: /var/mps/policy/mps_policy_backup.xml

Monday, 9 Jul 18 07:42:14.858 +0000 [Debug] [Main] Policy Name: mps_policy_backup, Policy Type: Backup

Monday, 9 Jul 18 07:42:14.859 +0000 [Debug] [Main] exclude are: device_backup

Monday, 9 Jul 18 07:42:14.859 +0000 [Debug] [Main] Type of policy rule is: backup

Monday, 9 Jul 18 07:42:14.859 +0000 [Debug] [Main] Backups to keep: 3

Monday, 9 Jul 18 07:42:14.860 +0000 [Debug] [Main] Total Backups to keep: 3

Monday, 9 Jul 18 07:42:14.907 +0000 [Debug] [Main] Starting to take the backup of mpsdb

Monday, 9 Jul 18 07:42:14.913 +0000 [Debug] [Main] Dumping table: backup_external_storage

Monday, 9 Jul 18 07:42:14.913 +0000 [Debug] [Main] Dumping table: backup_policy

Monday, 9 Jul 18 07:42:15.441 +0000 [Debug] [Main] Backup complete. pg_dump:

Monday, 9 Jul 18 07:42:15.450 +0000 [Debug] [Main] Copy file:ns_ssl_keys: TO :/var/mps/backup/Backup_10.1.83.32_11.1_58.13_09Jul2018_07_42_14/backup//var/mps/tenants/root/

Monday, 9 Jul 18 07:42:15.456 +0000 [Debug] [Main] Copy file:ns_ssl_certs: TO :/var/mps/backup/Backup_10.1.83.32_11.1_58.13_09Jul2018_07_42_14/backup//var/mps/tenants/root/

Monday, 9 Jul 18 07:42:15.462 +0000 [Debug] [Main] Copy file:ns_ssl_csrs: TO :/var/mps/backup/Backup_10.1.83.32_11.1_58.13_09Jul2018_07_42_14/backup//var/mps/tenants/root/

Monday, 9 Jul 18 07:42:15.463 +0000 [Debug] [Main] Copy file:ns_diff_reports: TO :/var/mps/backup/Backup_10.1.83.32_11.1_58.13_09Jul2018_07_42_14/backup//var/mps/tenants/root/

Monday, 9 Jul 18 07:42:15.469 +0000 [Debug] [Main] Match failed:device_backup


 

Monday, 9 Jul 18 07:43:04.789 +0000 [Information] [Main] Backup:ns

Monday, 9 Jul 18 07:43:04.909 +0000 [Error] [Main] Exception in mps_policy_backup

{ "errorcode": 30030, "message": "Backup Operation failed", "severity": "ERROR", "resourceName": "backup" }

Monday, 9 Jul 18 08:04:25.773 +0000 [Debug] [Config[#8]] Table copied: backup_external_storage

Monday, 9 Jul 18 08:04:25.774 +0000 [Debug] [Config[#8]] Table copied: backup_policy

The var utilization was also normal :

 

bash-2.05b# df -kh

Filesystem Size Used Avail Capacity Mounted on

/dev/md0 542M 375M 156M 71% /

devfs 1.0k 1.0k 0B 100% /dev

procfs 4.0k 4.0k 0B 100% /proc

/dev/ad0s1a 1.4G 624M 707M 47% /flash

/dev/ad0s1e 110G 12G 89G 12% /var

Only NS upgrade images are expected to be present here. Due to a number of unwanted files present here, backup fails with exception.