# Enhancement Request: Support Distributed CSI Storage for Concert Operate / Cloud Pak for AIOps on Linux
## Summary
Provide a fully supported distributed CSI-backed persistent-storage option for Concert Operate / IBM Cloud Pak for AIOps deployments on Linux, including a validated Rook-Ceph implementation or another IBM-supported equivalent.
The primary goal is to enable persistent workloads to remain accessible following a worker-node drain, reboot, replacement, or single-node failure, subject to documented application-specific quorum, disruption, and data-consistency requirements.
## Problem Statement
Linux-based Concert Operate / Cloud Pak for AIOps deployments that rely on Rancher `local-path` storage bind persistent workload data to an individual worker node.
As a result, draining, rebooting, patching, replacing, or losing a worker node can make affected persistent volumes unavailable until the original node returns. Depending on the workload, recovery may require an extended maintenance window, manual intervention, application recovery procedures, or restoration from a previous backup.
This creates several customer challenges:
- Rolling Linux OS patching is difficult or impractical for stateful workloads.
- Routine worker-node maintenance can require a broad platform outage.
- Hardware replacement or unexpected node failure increases the risk of service disruption.
- Local persistent-volume loss may force recovery from backup.
- Customers cannot use a documented, supported distributed-storage design to improve availability and maintenance flexibility.
- Customers may attempt unsupported third-party CSI or storage integrations to address the limitation.
## Requested Enhancement
Add a supported distributed CSI-backed storage topology for Concert Operate / IBM Cloud Pak for AIOps on Linux.
Rook-Ceph is one possible implementation because it provides Kubernetes-native CSI provisioning and replicated block storage. However, the primary request is not limited to Rook-Ceph. IBM may provide another supported distributed-storage implementation if it delivers the required operational outcomes and has a defined compatibility, lifecycle, and support model.
The solution should support persistent workloads requiring `ReadWriteOnce` storage and should provide `ReadWriteMany` support where needed for validated backup and restore workflows.
## Required Outcomes
The supported storage solution must provide the following outcomes:
- Persistent workload data can remain available after a worker-node drain, reboot, or replacement, within documented workload-specific limits.
- Persistent volumes can be detached from an unavailable node and attached to an eligible healthy node.
- Customers can perform rolling Linux maintenance without a full-platform outage when all documented storage-health, quorum, replica, and disruption requirements are satisfied.
- Customers have a validated, documented recovery path for node, disk, storage-service, and cluster-level failure scenarios.
- IBM provides a clear support boundary for the storage platform, its integration with Concert Operate / Cloud Pak for AIOps, and relevant maintenance and recovery procedures.
## Acceptance Criteria
### Supported Architecture
Provide an official supported reference architecture for Linux/K3s-based Concert Operate / Cloud Pak for AIOps deployments using distributed CSI-backed storage.
The architecture documentation should include:
- Minimum and recommended number of storage nodes.
- Minimum and recommended number of worker nodes.
- Whether storage daemons can be colocated with application workers.
- Whether dedicated storage nodes are required or recommended for production.
- Failure-domain guidance across host, rack, availability zone, and storage-device boundaries.
- Required anti-affinity rules for storage monitors, managers, OSDs, CSI components, and stateful application workloads.
- Network requirements for storage client traffic, replication, recovery, rebalancing, and monitoring.
- Recommended bandwidth, latency, MTU, redundancy, and network isolation practices.
- Supported disk media, including NVMe, SSD, and HDD guidance.
- Restrictions for mixed-media pools or uneven storage-node configurations.
- Required operating system, kernel, filesystem, and device configuration.
- Recommendations for direct-attached disks, JBOD, RAID controller mode, HBA mode, and hardware RAID restrictions.
- Requirements for encryption at rest and key-management integration, if supported.
- Capacity planning requirements that account for replication overhead, recovery headroom, storage expansion, rebalancing, and backup staging.
### Storage Classes
Document the supported storage classes and their intended uses.
The documentation should specify:
- Supported `ReadWriteOnce` block-storage classes.
- Supported `ReadWriteMany` filesystem-storage classes, where required.
- Required CSI driver parameters.
- Reclaim-policy guidance.
- `volumeBindingMode` guidance.
- Volume-expansion support and operational procedure.
- Snapshot support and limitations.
- Encryption support and limitations.
- Topology-aware provisioning behavior.
- Storage pool selection and performance-tier guidance.
- Requirements for default versus explicitly specified storage classes.
- Supported PVC access modes and volume modes for each Concert Operate / Cloud Pak for AIOps component.
### Compatibility Matrix
Provide a tested compatibility statement covering:
- Concert Operate version.
- IBM Cloud Pak for AIOps version.
- Supported Linux distributions.
- Supported Linux kernel versions.
- K3s version.
- Kubernetes version.
- Container runtime version and configuration requirements.
- Rook version, if Rook-Ceph is selected.
- Ceph version, if Rook-Ceph is selected.
- CSI specification and CSI driver version.
- CNI compatibility and network-plugin requirements.
- Node hardware requirements.
- Storage-device and filesystem requirements.
- Supported upgrade paths.
- Supported version-skew limits.
- End-of-support and lifecycle alignment.
### Migration from Rancher local-path
Provide a documented and supported migration path from Rancher `local-path` volumes to the distributed CSI-backed storage class.
The migration guidance should cover:
- Fresh deployments using distributed storage from initial installation.
- Existing deployments using `local-path` volumes.
- Inventory and assessment of existing PVCs and StatefulSets.
- Workload-by-workload migration sequence.
- Required maintenance windows.
- Expected service disruption during migration.
- Application quiescing requirements.
- Pre-migration backup requirements.
- Logical backup and restore procedures where block-level migration is not safe.
- Use of snapshots, volume copies, or application-native replication where supported.
- Data-integrity validation before and after cutover.
- Rollback criteria and rollback procedures.
- Recovery procedures if migration is interrupted.
- Post-migration validation of data consistency, replicas, shards, broker health, and database health.
- Explicit identification of workloads that cannot be migrated in place.
The documentation should clearly state whether migration is supported by:
- In-place PVC migration.
- Application-level export and import.
- Backup and restore.
- Parallel deployment and controlled cutover.
- Data replication and switchover.
- A combination of the above methods.
### Rolling Node Maintenance
Provide tested operational guidance for cordon, drain, reboot, patch, and uncordon procedures.
The runbook should include:
- Pre-maintenance health checks.
- Storage cluster health checks.
- Application health checks.
- Capacity checks.
- Replica and quorum checks.
- Pod disruption budget validation.
- PVC attachment and mount validation.
- Node cordon procedure.
- Node drain procedure.
- Safe handling of daemonsets and storage-related pods.
- Node reboot and Linux patching procedure.
- Node return-to-service validation.
- Application-level validation after node recovery.
- Storage rebalance, recovery, and backfill validation.
- Criteria for proceeding to the next node.
- Maximum number of nodes that may be drained simultaneously.
- Conditions under which maintenance must stop.
- Escalation and recovery actions for failed drain, mount, attach, scheduling, or storage-health events.
The procedure should explicitly define supported rolling-maintenance limits, such as:
- One worker node at a time.
- One storage node at a time.
- No maintenance while the storage cluster is degraded.
- No maintenance while recovery, rebalancing, or backfill is active beyond defined thresholds.
- No maintenance when capacity is below the documented recovery reserve.
- No maintenance when an affected application does not have sufficient quorum or replicas.
### Stateful Workload Support Matrix
Provide a workload-specific support matrix for all stateful Concert Operate / Cloud Pak for AIOps services.
At minimum, the matrix should cover:
| Component | Storage mode | Distributed CSI support | Rolling maintenance support | Required validation |
|---|---|---|---|---|
| PostgreSQL | RWO block | Explicitly state support | Explicitly state support | Replication, failover, backup, recovery |
| Kafka | RWO block | Explicitly state support | Explicitly state support | ISR health, replication factor, controlled broker maintenance |
| ZooKeeper | RWO block | Explicitly state support | Explicitly state support | Quorum preservation and sequential maintenance |
| Cassandra | RWO block | Explicitly state support | Explicitly state support | Replication factor, repair, node replacement, latency |
| OpenSearch / Elasticsearch | RWO block | Explicitly state support | Explicitly state support | Cluster health, shard replicas, allocation, recovery |
| CouchDB | RWO block | Explicitly state support | Explicitly state support | Replication, compaction, data consistency |
| MinIO | Documented mode | Explicitly state support | Explicitly state support | Erasure-set health, filesystem requirements, healing |
| Redis | RWO block | Explicitly state support | Explicitly state support | Sentinel or cluster health, persistence, failover |
| Other stateful components | As required | Explicitly state support | Explicitly state support | Component-specific requirements |
For every component, identify whether it is:
- Fully supported for rolling worker-node maintenance.
- Supported only with a defined minimum replica count.
- Supported only with a defined storage class or topology.
- Supported only with a controlled outage.
- Not supported with distributed CSI storage.
- Not supported for migration from existing `local-path` storage.
- Dependent on application-specific backup, restore, replication, or validation procedures.
### Backup and Restore
Provide supported backup, restore, and disaster-recovery procedures for both the application platform and the distributed storage platform.
The documentation should cover:
- Application-level backup requirements.
- Platform configuration backup.
- Persistent-volume data backup.
- Storage-cluster configuration and metadata backup.
- Object-storage requirements for backups.
- Backup retention guidance.
- Off-cluster backup requirements.
- Backup encryption and access controls.
- Backup scheduling and performance impact.
- Restore sequencing.
- Restore validation.
- Restore to the same cluster.
- Restore to a rebuilt cluster.
- Restore to an alternate cluster.
- Recovery from accidental PVC deletion.
- Recovery from logical data corruption.
- Recovery from storage-device failure.
- Recovery from storage-node failure.
- Recovery from worker-node failure.
- Recovery from loss of multiple replicas within supported fault-tolerance limits.
- Full-cluster disaster recovery.
The documentation should clearly distinguish between:
- CSI snapshots.
- Storage-platform snapshots.
- Crash-consistent backups.
- Application-consistent backups.
- Logical database backups.
- Platform backup and restore.
- Disaster-recovery recovery points.
- Disaster-recovery rebuild procedures.
### Failure Scenarios
Provide documented support guidance for failure scenarios during normal operation and planned maintenance.
The tested scenarios should include:
- Loss of one worker node.
- Loss of one storage node.
- Loss of one disk or OSD.
- Loss of multiple disks within the documented replica-failure tolerance.
- Loss of a local boot device.
- Worker-node failure while another worker node is drained.
- Storage-node failure while another node is under maintenance.
- Disk or OSD failure during rolling OS patching.
- Storage recovery or rebalance activity during planned maintenance.
- Network failure affecting storage replication.
- Network partition affecting monitor quorum or storage-client connectivity.
- Inability to attach or mount a persistent volume after a worker-node failure.
- Persistent volume remaining attached to an unavailable node.
- Storage capacity exhaustion during recovery or rebalancing.
- Corruption or unavailability of storage-cluster metadata.
- Simultaneous application quorum degradation and storage degradation.
Each scenario should define:
- Expected platform behavior.
- Supported customer action.
- Recovery procedure.
- Required escalation point.
- Data-loss expectations.
- Service-impact expectations.
- Conditions that require restoration from backup.
- Conditions under which IBM support will assist.
### Performance and Sizing
Provide production performance characterization and sizing guidance.
The guidance should include:
- Expected PVC provisioning time.
- Expected volume attach and mount time.
- Expected failover or reattachment time after node loss.
- Read and write latency expectations.
- IOPS expectations by disk type and storage-pool configuration.
- Throughput expectations by topology.
- Network bandwidth and latency requirements.
- Impact of replication factor on usable capacity and performance.
- Impact of storage recovery, rebalancing, backfill, and scrubbing.
- Capacity headroom requirements during normal operation.
- Capacity headroom requirements during degraded operation.
- Capacity headroom requirements during OSD replacement and recovery.
- Storage sizing examples for starter, production, and larger-scale deployments.
- Performance expectations for database, search, messaging, and object-storage workloads.
- Recommended disk counts and media types by deployment size.
- Benchmark methodology and workload assumptions.
### Monitoring and Operations
Provide monitoring, alerting, runbook, and operational guidance for the supported distributed storage solution.
This should include:
- Storage-cluster health monitoring.
- Storage quorum monitoring.
- Disk and OSD health monitoring.
- Storage-pool capacity monitoring.
- Near-full, backfill-full, and full threshold guidance.
- Placement-group health monitoring.
- Slow-operation monitoring.
- Client I/O latency monitoring.
- Volume provisioning, attachment, mount, resize, and snapshot failure monitoring.
- Storage-node availability monitoring.
- Recovery, rebalancing, backfill, and scrub monitoring.
- Storage-network monitoring.
- Application storage-latency correlation guidance.
- Recommended Prometheus metrics.
- Recommended Grafana dashboards.
- Recommended alert thresholds.
- Runbook guidance for common alerts.
- Capacity forecasting guidance.
- Storage expansion procedures.
- Node replacement procedures.
- Disk replacement procedures.
- Upgrade procedures.
- Health validation before and after operational changes.
## Support Boundary Requested
IBM should explicitly define support for the validated distributed-storage topology when it is deployed according to the reference architecture.
The support statement should cover:
- Installation of Concert Operate / Cloud Pak for AIOps using the approved CSI storage classes.
- Supported storage topology and configuration.
- Supported version combinations.
- Upgrade sequencing.
- Node maintenance procedures.
- Node replacement procedures.
- Disk or OSD replacement procedures.
- Backup and restore procedures.
- Disaster-recovery procedures.
- Interaction between application workloads and the storage platform.
- Monitoring, alerting, and operational health checks.
- Escalation boundaries between IBM support and any underlying storage-provider support organization.
If Rook-Ceph cannot be supported directly, IBM should provide an alternative distributed CSI-backed storage solution for Linux/K3s deployments that delivers the same operational outcomes.
## Definition of Success
This enhancement is successful when a customer can deploy Concert Operate / Cloud Pak for AIOps on Linux with an IBM-supported distributed CSI-backed persistent-storage platform and can:
- Drain and reboot a worker node without losing persistent workload availability, subject to documented quorum and disruption constraints.
- Perform rolling Linux OS patching without requiring a full-platform maintenance outage.
- Replace a failed worker node or storage device using a documented recovery procedure.
- Recover from a supported single-node, single-disk, or single-storage-service failure without immediately requiring restoration from a previous platform backup.
- Use validated backup, restore, and disaster-recovery procedures for distributed persistent storage.
- Monitor storage health, capacity, recovery, latency, and fault conditions using documented operational guidance.
- Remain within a defined IBM support boundary throughout normal operation, planned maintenance, upgrades, and supported recovery scenarios.
