Upgrade Precheck Hardening: Detecting Corrupted Installers, Configuration Drift, and Container Image Failures Before Deployment

Problem

During a recent SevOne upgrade, the installation failed because of multiple issues that were not detected before execution:

• Corrupted nms-installer.tar files on appliances
• Residual files from a previous 7.2.7 upgrade
• sshd_config incompatibility on the cluster leader
• Pods or container images failing to load
• Version mismatches between the cluster leader and other appliances
• Containers not starting correctly after the upgrade

The existing SevOne-validate image and upgrade prechecks did not identify these conditions before the upgrade began, resulting in service degradation and outage impact.

Proposed Improvement

Enhance the SevOne-validate image and upgrade prechecks to perform comprehensive validation before allowing the upgrade to proceed.

Recommended checks include:

Installer archive validation

• Verify file existence, size, permissions, and ownership.
• Validate archive integrity with tar -tf or equivalent.
• Compare SHA-256 checksums against the source package.
• Detect truncated or partially copied files.

Upgrade-directory validation

• Detect stale files from previous upgrade attempts.
• Validate expected directory contents.
• Flag unexpected or conflicting files before execution.
• Confirm sufficient disk space and inode availability.

SSH configuration validation

• Validate sshd_config syntax with sshd -t.
• Check required SSH settings and compatibility with the target release.
• Confirm connectivity and authentication between cluster appliances.
• Report configuration differences from the expected baseline.

Container and image validation

• Confirm all required images are present locally.
• Validate image digests and signatures where supported.
• Verify image load/import operations.
• Check that required containers can start successfully.
• Confirm container versions match the target release.

Cluster consistency checks

• Verify all appliances are reachable.
• Confirm cluster membership and roles.
• Validate leader and peer software versions.
• Detect version mismatches before the upgrade.
• Confirm all required services are healthy before proceeding.

Safe preflight execution

• Run checks across all appliances before modifying any node.
• Fail fast when a critical issue is detected.
• Clearly identify the affected appliance, file, service, or configuration.
• Provide remediation instructions.
• Prevent the upgrade from starting until critical failures are resolved.

Expected Outcome

These checks would identify corrupted packages, incompatible configuration, stale upgrade files, missing images, and unhealthy pods before the upgrade starts. This would reduce avoidable upgrade failures and provide operators with actionable remediation steps instead of discovering the issue during or after the upgrade.

Acceptance Criteria

• A corrupted installer archive is detected before upgrade execution.
• Invalid or incompatible sshd_config causes the precheck to fail.
• Missing or unloadable container images are reported.
• Stale upgrade artifacts are detected and clearly identified.
• Version mismatches across appliances are reported.
• Prechecks return a clear pass/fail summary per appliance.
• Critical precheck failures block the upgrade.
• The checks are included in future SevOne-validate image and upgrade releases.