Working with Prometheus in container environment

The intent of prometurbo is to query for custom metrics across the scope of all metrics in a single prometheus instance and we cannot try to logically merge or de-deduplicate that queries that are set on a project or namespace if the queries are pulling the same data. There are also potential performance considerations.

Eva,

I think the idea is that I would like my customers to create the PrometheusQueryMapping with their deployments since those queries are going to be custom with their application. Ideally we would like to keep all the resources they create in a namespace that they own.

For example below is just a test manifest of a deployment to a namespace turbo-horiz-metric

Well if you look at the PrometheusQueryMapping resource it has to be in turbonomic namespace and we would really like for that to be in any namespace. This way we don't have to give our users access to the turbonomic namespace. Correct me if I am wrong but the operator looking at that kind is bound to turbonomic namespace, and what we would like to see is the operator able to pick that up from any namespace.

Happy to discuss.

---
apiVersion: v1
kind: Namespace
metadata:
  name: turbo-horiz-metric
---
apiVersion: v1
kind: Service
metadata:
  name: turbotest-horizontal-metric
  namespace: turbo-horiz-metric
  labels:
    app: turbotest-horizontal-metric
spec:
  type: ClusterIP
  ports:
  - port: 8080
    targetPort: http
    protocol: TCP
    name: http
  selector:
    app: turbotest-horizontal-metric
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: turbotest-horizontal-metric
  name: turbotest-horizontal-metric
  namespace: turbo-horiz-metric
spec:
  selector:
    matchLabels:
      app: turbotest-horizontal-metric
  template:
    metadata:
      labels:
        app: turbotest-horizontal-metric
    spec:
      containers:
      - image: artifactory-rd.netskope.io/pe-docker/turbonomic-tester:v0.1.9
        imagePullPolicy: Always
        name: turbotest-horizontal-metric
        ports:
        - name: http
          containerPort: 8080
          protocol: TCP
        resources:
          requests:
            cpu: 100m
            memory: 100Mi
          limits:
            cpu: 1
            memory: 1024Mi
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: turbotest-horizontal-metric
  namespace: turbo-horiz-metric
  labels:
    release: application-monitoring
    app: turbotest-horizontal-metric
spec:
  endpoints:
  - interval: 1m
    port: http
    metricRelabelings:
    - action: keep
      regex: turbo_test_.*
      sourceLabels: [__name__]
  selector:
    matchLabels:
      app: turbotest-horizontal-metric
---
apiVersion: metrics.turbonomic.io/v1alpha1
kind: PrometheusQueryMapping
metadata:
  name: turbotest-horizontal-metric
  # Currently PrometheusQueryMapping must be in turbonomic namespace
  # Waiting for IBM to fix that so these can live in any namespace
  namespace: turbonomic
  labels:
    release: application-monitoring
spec:
  entities:
  - type: application
    metrics:
    - type: transaction
      queries:
      - type: used
        promql: |
          turbo_test_transactions{job="turbotest-horizontal-metric"}
    - type: responseTime
      queries:
      - type: used
        promql: |
          turbo_test_response_time{job="turbotest-horizontal-metric"}
    attributes:
    - name: ip
      label: instance
      matches: \d{1,3}(?:\.\d{1,3}){3}(?::\d{1,5})??
      isIdentifier: true
    - name: namespace
      label: namespace
    - label: service
      name: service
---
apiVersion: policy.turbonomic.io/v1alpha1
kind: PolicyBinding
metadata:
  name: turbotest-horizontal-metric
  namespace: turbo-horiz-metric
spec:
  policyRef:
    kind: SLOHorizontalScale
    name: turbotest-horizontal-metric
  targets:
  - kind: Service
    name: .*turbotest-horizontal-metric.*
---
apiVersion: policy.turbonomic.io/v1alpha1
kind: SLOHorizontalScale
metadata:
  name: turbotest-horizontal-metric
  namespace: turbo-horiz-metric
spec:
  behavior:
    scaleDown: Manual
    scaleUp: Manual
  maxReplicas: 4
  minReplicas: 1
  objectives:
  - name: Transaction
    value: 100
  - name: ResponseTime
    value: 5000

@Bosheng and @jason - I do not understand the ask as written. Please clarify.

What metric needs to be in the Turbo namespace?

Or is this an enhancement on the Prometurbo configuration? If yes please specify the details.