ASM ServiceNow Observer - enhance CI relationship discovery to handle larger queries / Support splitting of servicenow jobs

Based on case TS010786559 it is clear that the service now observer falls down on CI relationship discovery when a large number of CI classes are selected for discovery. This is demonstrated by the relationship query becoming too large for servicenow to handle so it throws an error. The servicenow observer stops processing relationships at this point so any resource discovery up to the point of failure doesn't get any relationships. One suggestion in the case was to split the servicenow discovery into two or more jobs and discover different CI tables within each. However this didn't work as Job2 overwrites what job1 discovered.

So we would require the servicenow observer to do either of the following :

1. Allow jobs to be split so different CI classes and relationships could be discovered in smaller amounts of time. This would ensure the relationship query was within the maximum size expected by servicenow. It would also allow us to discover tables less likely to change in a less frequent schedule and thus speed up discovery of the tables we know will change more often. The observer should allow this data to be loaded separately without overriding or removing data from a different job name.

2. Alter the relationship discovery process so that the observer works out how long a query is likely to be when sent to service now and then will split the query as many times as required to keep the servicenow query and pagination headers so that they remain within the accepted query length of servicenow. Then once it has gathered all of the relationships it would then process this within ASM.

I'm tempted more towards option 1 as it would allow me to gather static tables say once per week and tables more likely to update happen once per day. However as it stands it takes around 4-8 hours sometimes to get data from servicenow into ASM and as you only do a load job for the observer it doesn't allow us the option of only fetching resources modified recently.