Skip to Main Content
Jedox Ideas

Let us know how we can make Jedox even better!

Status Shipped
Workspace Jedox Platform
Categories Integrator
Created by Guest
Created on Mar 15, 2021

Multiple instances of Integrator

Our Partner had the following request:

We need some of the jobs to be able to run when a long job is already running but they shouldn't run when other jobs are running.
This means we cannot set them as standard and cannot set them as parallel.
The ideal solution would be if we could set some of them to run from one instance of Integrator and some of them from another instance.
Is there a possibility to have 2 instances of Integrator running on the same server?

More details regarding his case:

There are several ETLs which copy data into the Income Statement cube from different Trial Balance cubes. These can be triggered as someone uploads a TB, but we can’t allow all the different TBs to be run at once. Each of the jobs takes about 10 minutes. If we have these jobs set as parallel, then someone could potentially run all 3 at the same time which will corrupt the data.

Meanwhile, we have budgeting, weekly forecasting and annual reforecasting cubes. These have input reports on them which also call ETLs, some to do seasonalisation, some to update workflow, etc. These can be run at the same time as the income statement jobs and at the same time as each other. So they can be parallel, but if in the income statement jobs are running, they can be blocked for 10 – 30 minutes which is not ideal if someone is submitting their Budget or trying to run a seasonalisation.

Some clients have multiple TB’s from different systems for different parts of the business. Different people are in charge of these TBs and will upload them at different times so we keep them as separate jobs. We could make the job do all 3 TBs at the same time, but this means each person would need to wait longer for it to finish.

We made them parallel to stop blocking the other ETLs for the system but then there was the odd occasion when 2 people were uploading at the same time and so the jobs interfered with each other (doing sql deletes at the same time as sql inserts to the same tables).

We have managed to get around this particular one by leaving it as a parallel job and then wrapping it in a Groovy job that first checks if certain jobs are running before executing – it also checks if one of the scheduled loads are due to run and will also fail if one is due within 10minutes.

This works well in this instance, but as more jobs are added, and as we need to do this for more and more customers, it’s becoming complicated to maintain. The ideal solution would be if Jedox had a new type of job which was parallel that could run at the same time as a standard job was running. The next best solution is to have 2 instances of Integrator running whereby all the Income Statement and Balance Sheet jobs run on one instance as a standard job, and all the other jobs run on the second instance as parallel. I’ve tested this solution and it works well, but I’m trying to avoid having to have a separate machine just for this especially if I need one extra per customer.

  • ADMIN RESPONSE
    Jul 14, 2021

    We are currently working on architecural enhancements for Jedox Integrator that will leverage the capabilities of modern Cloud infrastructure to scale up the processing capabilities of Integrator. However, for on-premise environments using the Jedox setups, this feature will not available.