IBM Apptio Cloudability currently supports ingesting billing and usage data from the native object storage services of the major hyperscalers:

• Amazon S3 (AWS)

• Azure Blob Storage

• Google Cloud Storage

Many enterprise customers, however, utilize S3-compatible object storage platforms that implement the Amazon S3 API while operating outside of AWS. These include private cloud deployments, sovereign cloud providers, and enterprise object storage solutions.

Cloudability should provide support for configurable S3-compatible storage endpoints, allowing ingestion from any object storage platform implementing the standard S3 API.

This enhancement would significantly increase deployment flexibility while maintaining Cloudability's existing ingestion architecture.

Business Problem

Organizations increasingly operate hybrid and multi-cloud environments.

Examples include:

• On-premises object storage

• Sovereign cloud providers

• Government cloud environments

• Regional cloud providers

• Enterprise storage appliances

Many of these expose an Amazon S3-compatible API but are not hosted on AWS.

Today, Cloudability requires customers to:

• replicate data into AWS S3,

• or stage data through Azure Blob Storage,

• or stage data through Google Cloud Storage.

This creates unnecessary:

• operational complexity,

• storage costs,

• network transfer costs,

• security concerns,

• compliance challenges,

• latency.

In many regulated environments, replication into AWS is not permitted.

Business Value

Supporting configurable S3-compatible endpoints would provide:

Reduced Operational Overhead

Customers can ingest directly from existing storage without maintaining synchronization jobs.

Lower Costs

Eliminates duplicate storage.

Reduces:

• egress charges

• replication infrastructure

• lifecycle management

Improved Security

Sensitive billing data remains within the customer's existing storage platform.

No additional cloud providers are introduced.

Regulatory Compliance

Supports environments where billing exports cannot leave a sovereign or private infrastructure.

Increased Platform Adoption

Many organizations already centralize billing exports into S3-compatible object stores.

Native support removes a significant adoption blocker.

Functional Requirements

FR-1: Custom S3 Endpoint Configuration

Allow administrators to configure:

• Endpoint URL

• Bucket Name

• Region (optional or configurable)

• Access Key

• Secret Key

• Session Token (optional)

Example:

Endpoint:

https://object.company.com

FR-2: Path Style vs Virtual Host Style

Support both addressing models.

Example:

Path Style

https://endpoint/bucket/object

Virtual Host Style

https://bucket.endpoint/object

Configuration option:

Use Path Style Addressing

Yes / No

FR-3: TLS Support

Support:

• HTTPS

• Customer certificates

• Standard TLS validation

Optionally allow trusted custom CA certificates for enterprise deployments.

FR-4: Authentication

Support standard S3 authentication:

• Signature Version 4

• Access Key

• Secret Key

• Temporary Credentials

• STS-compatible tokens (future enhancement)

FR-5: Endpoint Validation

Provide a "Test Connection" feature that validates:

• endpoint reachable

• credentials valid

• bucket exists

• permissions sufficient

• object listing successful

Display meaningful error messages.

FR-6: Object Discovery

Support existing Cloudability ingestion behavior:

• prefix

• folder selection

• wildcard matching

• scheduled scans

No behavioral differences compared to AWS S3 ingestion.

FR-7: Performance

Support:

• multipart downloads

• concurrent object retrieval

• retry logic

• exponential backoff

Equivalent to existing S3 ingestion performance.

FR-8: Logging

Log:

• authentication failures

• endpoint errors

• network failures

• object parsing failures

• permission issues

Non-Functional Requirements

Availability

Must meet existing ingestion service SLA.

Security

Credentials must be encrypted using existing Cloudability secret management.

No credentials shall be written to logs.

Compatibility

Must remain fully backward compatible with existing:

• AWS S3 integrations

• Azure Blob integrations

• Google Cloud Storage integrations

No changes required for existing customers.

Scalability

Support buckets containing:

• millions of objects

• multi-terabyte exports

Equivalent to existing AWS ingestion.

Configuration Example

Supported Storage Platforms (Examples)

The feature should support any storage implementing the Amazon S3 API, including but not limited to:

• MinIO

• Stackit object storage

• Dell ECS

• NetApp StorageGRID

• Ceph Object Gateway (RGW)

• IBM Cloud Object Storage (S3 API)

• Wasabi

• Backblaze B2 (S3 Compatible)

• Cloudflare R2

• DigitalOcean Spaces

• Oracle Cloud Object Storage (S3 Compatibility)

• Linode Object Storage

• Vultr Object Storage

• OpenStack Swift with S3 compatibility

• Regional sovereign cloud object storage providers

The implementation should not whitelist providers, but instead allow any standards-compliant S3 endpoint.

User Story

As a Cloudability administrator

I want to configure a custom S3-compatible object storage endpoint

So that Cloudability can ingest billing and usage exports directly from our existing object storage platform without requiring replication into AWS, Azure, or Google Cloud.

Acceptance Criteria

The feature shall be considered complete when:

1. Administrators can configure any S3-compatible endpoint.

2. Connection validation succeeds using standard S3 APIs.

3. Cloudability ingests objects identically to AWS S3 ingestion.

4. Existing AWS/Azure/GCP integrations remain unaffected.

5. Documentation includes configuration examples and troubleshooting guidance.

6. Standard IAM/S3 authentication methods are supported.

7. Path-style and virtual-host addressing are both supported.

Expected Customer Benefits

• Simplified deployment architecture

• Reduced cloud storage costs

• Elimination of unnecessary replication workflows

• Improved compliance with data residency requirements

• Greater flexibility for hybrid, sovereign, and private cloud deployments

• Broader compatibility with enterprise object storage ecosystems

Proposed Implementation Approach

Rather than introducing provider-specific integrations, Cloudability should abstract the storage layer by allowing a configurable S3 endpoint URL alongside the existing S3 authentication settings. This approach leverages the widely adopted Amazon S3 API standard and enables compatibility with any standards-compliant object storage solution without requiring bespoke integrations for individual vendors. Existing AWS S3 functionality would remain unchanged, while customers gain the flexibility to connect to private, sovereign, or third-party object storage using the same ingestion workflow.