Improve the agent deployment process : have a dev/prod or beta/stable type of release

I/we would like to have a discussion about how agents are released into the "Dynamic Agent." What we are talking about is how the .jar files are pulled from maven into the dynamic agent.

Today, you have a few options in how your agents work. If anything is incorrect here, please don't hesitate to correct me!

1. Dynamic. New agents typically come out around 4a CT. There is nothing here that externally quantifies the quality of the release. This is how my team runs today. 99% of the time, this is great. Until it is not.

2. Dynamic w/ Pinning. You have to decide which version of the Jar file are valid and then take the responsibility of rolling the versions forward. There is nothing here that externally quantifies the quality of the release. You have to pick them and move them through your environment.

3. Static. You install an agent that has a snapshot in-time of the agent jar files. It does not update automatically. There is nothing that externally quantifies the quality of this release. This means you would need to "pick one" and move it through your environment.

I would like to see another option. An option where my non-production environment can help participate in externally quantifying the quality of the release. My experience is that our environments don't look exactly like the ones that the Instana team has access to test with. This has created several opportunities for agents to go offline in production, cause issues with CPU resource usage on VMs, and in at least 2 cases caused us to have to stop/start our JVMs due to the new agents completely causing the JVMs to no longer function. These JVM restarts, or deploying new JVM instances, means that customers were negatively impacted.

For me, I would like to see a way to have the dynamic agent subscribe to an "internally tested" channel or an "externally tested channel." Typically when we see issues it is within minutes-to-hours of the 4a daily release times. If the internally tested channel was promoted to the externally tested channel after a week or two, this would give us an opportunity to prevent agent issues.

I am not sure the best way to move this suggestion forward. I realize this may be more involved than a "feature request." It is something we have been talking about for awhile and have been hoping to get some traction.