Calling Apex Queueable Jobs from Triggers in Salesforce

Recently on our project, one of the Apex Batch Jobs started failing with an exception of “Too many queueable jobs added to the queue: 2”; the batch job performed a DML operation on a Custom Object. This encouraged me to investigate this topic deeper.

Let’s imagine that we have an Apex Trigger on a Contact object which executes 5 different Queueable Jobs depending on different criteria:

• The first one performs callouts to external systems.
• Another one performs DML operations on setup objects; it was extracted to Queueable because of the mixed DML operations.
• The third one was created to extract part of the heavy logic from the Apex Trigger which kept failing because of the limit on the number of SOQL queries.
• The next one writes important logs to the Big Object to be analyzed by the external system.
• The last one was created to execute logic which updates a lot of related records. The logic was extracted to deal with the limit on the number of records processed as a result of the DML statements.

Also, the Apex Trigger inserts new Event records. And, the Apex Trigger on the Event object executes a Queueable Job to query new related EventRelation records just after the execution of the Apex Trigger on the Event object.

The Queueable Jobs are definitely not the only way to execute operations asynchronously. The differences between the Queueable Jobs, future methods, and Apex Batch Jobs are described in the Salesforce Stack Exchange comments and in the Apex Developer Guide.

Let’s suppose that the functionality mentioned above was deployed to a production organization. A sales team was able to create and update the Contact records from the Record Detail Page, and everybody was happy. But, the issue arises when DML operations on the Contact records are performed from an Apex Batch Job, future method, and from Queueable jobs. According to the Salesforce governor limits, it is allowed to execute no more than 1 Queueable job from the contexts. Below are possible reasons for getting the “Too many queueable jobs added to the queue: 2” exception. Let’s explore further, in case you have any of those in your code.

• Multiple Queueable Jobs are enqueued from the same Apex Trigger.
• The Apex Trigger performs a DML operation on a related Object and another Apex Trigger on the related Object enqueues one more Queueable Job.
• The asynchronous logic performs multiple DML operations and the Apex Triggers on the corresponding Objects enqueue the Queueables.
• The asynchronous logic performs a single DML operation on a list of more than 200 records. The Apex Trigger processes the records by chunks of 200 records, so it is executed multiple times and as a result enqueues multiple Queueable Jobs.
• The Apex Trigger logic can be executed multiple times in the case of workflow rule field updates.
• The Roll-Up Summary on the parent Object of Master-Detail relationship causes the execution of the Apex Trigger logic on a Parent. As a result, an additional Queueable job on a Parent object is executed in addition to the one which was executed from the Apex Trigger on the Child object.

The final solution to deal with the exception can depend on multiple factors. Governor Limits on the number of asynchronous operations, which can be executed from different contexts, is one of the most important ones. So, let’s review the limits closer.

Governors limits on asynchronous executions

Information about the limits is spread between multiple sources:

• Execution Governors and Limits – Apex Developer Guide
• getQueueableJobs(), getLimitQueueableJobs(), getFutureCalls(), getLimitFutureCalls(), getPublishImmediateDML(), and getLimitPublishImmediateDML() methods from the System.Limits class
• DailyAsyncApexExecutions and an HourlyPublishedPlatformEvents org limits which can be obtained from the System.OrgLimit class

Actual limits were not always obvious to me personally and to some of my colleagues as well, even after reading the documents. So, I have gathered them into a single table, which you can find below. It should be helpful when choosing the best approach for an asynchronous Apex execution.

The table represents the maximum number of jobs which can be executed from a single transaction. But on top of that, Apex Batch Jobs and Schedulable Jobs have another limit. The maximum number of Apex Batch jobs in the Apex flex queue that are in a Holding status can not be higher than 100. Also, the maximum number of Apex classes scheduled concurrently for the whole organization can not be higher than 100, and in the Developer Edition orgs the limit is 5.

As pointed out by Salesforce.stackexchange.com the main reason for introducing the limit to execute no more than a single Queueable Job from an asynchronous context is to prevent an explosive execution (a so-called “Rabbit Virus” effect).

Prevent the execution of the queueable job

The easiest solution to the issue is to prevent a Queueable execution, in case the limits are reached. It can work in some scenarios; e.g., when execution of the asynchronous logic is optional, it is handled by a scheduled job, or it is covered in some other way. Possible examples of the approach are highlighted below.

The methods, like System.isBatch(), System.isFuture(), and System.isQueueable(), can be used as a part of the condition to prevent the execution of the Queueable Job from asynchronous context. As a variation of the approach, the logic of the Queueable can be executed synchronously, in case it meets the above criteria. It works for Queueable Jobs which were created to avoid CPU time, heap size, or the number of SOQL queries limit.

Pros:

• Very simple from an implementation point of view

Cons:

• The approach use cases are very limited
• High risk of reusing the approach in cases where the Queueable job execution is extremely important

Similarly to the previous example, Limits.getQueueableJobs() and Limits.getLimitQueueableJobs() methods can be applied to prevent the execution of the Queueable job in cases where we are about to reach the limit. Pros and Cons are the same as for the previous approach.

A static boolean variable, shouldSkipExecutionOfMyQueueable, can be used to prevent the execution of the Queueable Job from an Apex Trigger in case it is executed from the Apex Batch Job. The Apex Batch Job, at the top of its execute() method, sets the variable to true. The Apex Trigger checks the variable and skips the execution of the Queueable. This approach can be useful in cases where the Apex Batch Job performs the same operation which is implemented in the Queueable.

We have used this approach for creating and updating User records utilizing data from an external API. The Apex Batch Job created users through an external ID and a Queueable Job, then executed them from an Apex Trigger which updated them. The static boolean variable is used to prevent updating the same User records that were just created from the Apex Batch Job.

Pros:

• Simplistic from the implementation point of view
• The same logic is not executed multiple times

Cons:

• Global static variable – Why are static variables considered evil? – Stack Overflow
• Possible use cases of the approach are very limited
• Forgetting to apply the approach for the new Apex Batch Job is highly likely

Platform event and change data capture

The next technique embraces the use of an Apex Trigger on Platform Events to execute some logic. For example, in Run more than one async jobs from Future/Quable context Pranay Jaiswal described in StackExchange applying the Apex Trigger for a Platform Event to save data to a Big Object.

Pros:

• Easy way to replace a Queueable Job in case you hit some limits related to the Queueable Jobs
• The same logic can be easily triggered from the external system by firing corresponding Platform Events

Cons:

• With some exceptions, one generally can’t make Apex callouts from the triggers
• Synchronous governor limits
• By default, the Apex Trigger for Platform Events is executed under an “Automated Process“ user with limited permissions. But starting with the Salesforce Spring ‘21 release, it is possible now to override a default running user of a platform event Apex trigger, you can find the details in Salesforce’s release notes.

Change Data Capture is one more approach. A good introduction to a Change Data Capture is described in this trailhead module. Instead of executing the Queueable Job from an Apex Trigger, you can create a new Apex Trigger for a Change Data Capture, and use it for processing your changes. The main purpose of the Change Data Capture is to be applied to a data replication from the Salesforce organization into the external system, but for sure, the possible use cases are not limited to it.

Pros:

• Standard Salesforce functionality to execute the Apex logic in a separate transaction on record modifications.

Cons:

• No Email Support from a Change Data Capture Trigger.
• Owner ID field of new records should be populated as it will contain an “Automated Process” by default.
• With some exceptions, one generally can’t make Apex callouts from the triggers.
• An Apex Trigger for a Change Data Capture is executed under the “Automated Process“ user with limited permissions. Here is a workaround demonstrating how to deal with it.
• Synchronous governor limits.
• Formula fields aren’t supported in a Change Data Capture.

Move part of the logic to schedulable jobs or apex batch jobs

Recently, as part of an integration with an external system, we had to create new Contact records from an Apex Trigger on Opportunity. To optimize performance and deal with the Salesforce limit, the logic was implemented as a Queueable Job and was executed from the trigger. But during the testing stage of the integration, we got a lot of UNABLE_TO_LOCK_ROW exceptions from the Queueable Job and from the ETL tool. The reason for the exception was that the ETL tool and the Queueable Job updated the Opportunities which are related to the same parent Account records. The solution for the task was to move the logic of the Contacts creation to an Apex Batch Job and execute it from the ETL after the completion of the Opportunities update.

Scheduling the Apex Batch Job to be executed on a periodic basis, e.g., on a daily basis, is another option. The job can process either all the records or only the records which meet some definite criteria. Also, the Apex Batch Job can be implemented as an addition to the Apex Trigger logic to cover scenarios which are not covered by the Apex Trigger.

Pros:

• Easier to implement in case numerous records need to be processed.
• Can process records which failed during a previous update; e.g., external REST API was not available during a previous execution.

Cons:

• The logic will be executed on a periodic basis instead of near real-time execution.

Chain execution of queueable jobs

A Queueable Job contains a list of items which should be processed. The first instance of the job processes the first item or first bunch of items. At the end of its execute() method, it removes the processed items from the list and enqueues itself to process the rest of them. The item could be a Queueable Job which should be enqueued, Standard or Custom Object record, or the instance of an AsyncTask interface which is defined below.

interface AsyncTask {
    public void processItem();
}

Here are a few examples of the implementation details: QueueableUtil on StackExchange, QueueableChain on StackExchange, and NSQueuebleJob on StackExchange.

Pros:

• Allows you to increase the number of a Queueable Job which can be executed from the current execution context.
• Easy to implement.

Cons:

• Does not handle use cases of Apex Trigger logic being executed multiple times from an Apex Batch Job or a future method.
• When one of the Queueable Jobs fails on governor limits, the information about all the next jobs will be lost.
• It could be tricky or even impossible to gather Queueable Jobs and enqueue them together. For example, asynchronous code performs multiple DML operations on different Objects and the corresponding Apex Triggers enqueue different Queueable Jobs.
• Similar to the System.enqueueJob() method, it is impossible to execute the Queueable in case limits on the number of Queueable Jobs that have been already used. For example, an Apex Batch Job logic has already enqueued another Queueable Job before executing the current logic.

Chaining Queueable Jobs from an Apex Trigger on a specific Object is one more approach. The Apex Trigger enqueues a new Queueable Job, in case there are no jobs of the same type in the progress of execution and there are items for processing. The Queueable Job processes records which meet some criteria until all the records are processed or the governor limits are reached. At the end of its execution, it enqueues itself in case there are additional records for processing. The criteria to specify the records for processing can be built using either the existing fields or a new custom checkbox Should_Be_Processed_By_Queueable_Job__c which is checked by the Apex Trigger.

Pros:

• Easy to implement and support.
• Usually no more than a single Queueable Job of this type is in progress of execution.

Cons:

• Possible use cases for applying the approach are limited.
• Similar to a System.enqueueJob() method, it is impossible to execute the Queueable in case the limits on the number of Queueable Jobs has already been used. For example, an Apex Batch Job logic already enqueued another Queueable Job before executing the current logic, but the records will be handled when additional records are added for processing later.
• Because of timing issues, in very rare cases, the Apex Trigger can determine that the Queueable Job is in progress and as a result doesn’t enqueue a new Queueable Job. At the same time, a current SOQL query for a number of records processing from the current Queueable Job returns a zero because the transaction for the current DML operation is not committed yet. You can get more details on the subject from this answer on StackExchange.

Use a custom object to store a list of asynchronous jobs

And, the last solution to the issue is using a Custom Object to store a list of asynchronous jobs which should be executed. Compared to all the options mentioned above, this one is the most robust and can be applied as a generic approach for all Queueable job projects of different complexities.

The first approach is described in the article Async Queue Framework by Jitendra Zaa. The framework dumps the Queueable Jobs into Custom Objects if the governor limits on the number of executed Queueable Jobs is exceeded. A Scheduler job is executed once per 10 minutes to execute Queueable jobs from the Custom Object.

Pros:

• Easy to implement.
• Allows you to enqueue an arbitrary number of Queueable Jobs from any execution context.

Cons:

• Some of the Queueable Jobs can be executed under different user than requested originally.
• Queueable Jobs should be serializable to JSON with a serialized size less than or equal to 131072 characters.
• Processing all the records from the Custom Object AsyncQueue is time-consuming, especially if many of them were created from an Apex Batch Job or during data load.

“Going Asynchronous with a Queueable Apex” by Dan Appleman regarding the Advanced Apex Programming in the Salesforce book described another solution for enqueueing Queueable Jobs. Here are some key points of the solution:

• Queueables should contain a global on/off switch in order to disable them in case of an infinite recursion.
• Create a Custom Object AsyncRequest with the fields: “Params” of a Long Text Area type, picklist “AsyncType”, checkbox “Error”, and a long text area field “Error Message”.
• Create an Apex Trigger on an AsyncRequest object to enqueue Queueable Jobs once the records are either inserted or updated.
• The next AsyncRequest record is queried for processing with the following WHERE condition: Error__c=FALSE AND CreatedById = :UserInfo.getUserId().
• The Queueable Job enqueues other Queueables from an AsyncRequest object at the end of its execution.
• As a backup mechanism, new Queueable Jobs can be enqueued using future methods or Scheduled Jobs.

Pros:

• Allows you to enqueue an arbitrary number of Queueable Jobs from any execution context.
• A Queueable Job is executed under the same user as the job was created under.

Cons:

• The article provides code samples and descriptions of the approach, but does not contain a ready to use framework.

There is an alternative approach which is based on techniques from two previous ones, tshevchuk/Async_Request__c.object. I actually implemented this small framework during my work on this article. The method QueueableManager.enqueue() enques a specified Queueable Job. It also stores information about the job in the “Async Request” object. The Queueable Job queries a single request submitted by a current user at the end of its execution and enqueues the request. In case of the failure of the Queueable Job, it stores an error message to the “Async Request’‘ record.

We have used the approach to execute a Queueable job created to extract CPU intensive logic from an Opportunity trigger. Another place for using it is to replace future method which fails from time to time because of an UNABLE_TO_LOCK_ROW exception. For the use case above, the framework should be extended to support retry logic in case of recoverable errors. There are also other ways of improving the framework described in the Github repository. The end goal is to use this approach for all new Queueable jobs in the project and to refactor all existing Queueable Jobs.

Pros:

• Allows you to enqueue an arbitrary number of Queueable Jobs from any execution context.
• The Queueable Job is executed under the same user as the job was created under.

Cons:

• Queueable Jobs should be serializable to JSON with a serialized size less than or equal to 131072 characters.
• Similar to the System.enqueueJob() method, it is impossible to execute the Queueable in case limits on the number of Queueable Jobs that have been already used. For example, an Apex Batch Job logic already enqueued another Queueable Job before executing its current logic, but the records will be handled when another Queueable Job is enqueued under the same user later.

The most complex approach is described in “Salesforce Asynchronous by Jitendra Zaa” in the SalesforceWay Podcast (part 1, part 2). The podcast is very motivational. It also contains a lot of technical details of the approach. This approach is an improved version of the first approach, that uses a Custom Object, from the same author. The main idea behind it is to increase the number of Queueable Jobs which can be enqueued during a single execution of the Schedulable Job. It sends Platform Events from the Schedulable Job to increase the limit on the number of Queueable Jobs enqueued from a single transaction. The automated process performs web api calls to change the context of the current user from an Automated Process to a specific user.

Pros:

• Effective way of enqueueing a large number of Queueable Jobs from different contexts.

Cons:

• Too complicated for small and medium projects.

Applying the solutions to the queueable jobs

At the beginning of the article, we listed Queueable Jobs on a Contact trigger. Let’s check to see if the approaches mentioned above can be applied to the Queueable Jobs to avoid the “Too many queueable jobs added to the queue: 2” exception.

The table rows contain the Queueable Jobs and the table columns contain the approaches. In the table cells I have put marks which indicate if the approach can be applied to the Queueable job:

• ‘+’ – possible to use the approach
• ‘?’ – depends on requirements and/or current architecture
• ‘-’ – not applicable

As you can see, the easiest solutions can usually be applied to very limited use cases. Also the solutions based on Custom Objects are the most generic. So, if you plan to implement some framework on your project to handle the exception, then the solutions based on Custom Objects are the best candidates for it.

Starting with the Salesforce Spring ‘21 release, it is possible to override the default by running the user of a platform event Apex trigger, so now it will be possible to use them in a wider range of use cases.

Transaction Finalizers

The Transaction Finalizers feature enables you to register the actions to the Queueable Jobs which will be executed even if the Job fails. You can find more details about it here: official documentation. The functionality has been available as a Beta since the Spring’21 release.

One of the key use cases for the Finalizers is handling errors. The Finalizer implementation can either log error messages or restart the Queueable Jobs automatically in case of concurrency issues like “System.QueryException: Record Currently Unavailable: The record you are attempting to edit, or one of its related records, is currently being modified by another user. Please try again.” or “FATAL_ERROR System.DmlException: Update failed. First exception on row 0 with id; first error: UNABLE_TO_LOCK_ROW, unable to obtain exclusive access to this record”.

Also, the Finalizers can be helpful in improving the solutions mentioned above, specifically solutions with chaining the Queueable Jobs. Enqueueing the job from a Transaction Finalizer allows for the continued execution of the chain even if one of the Queueable Jobs failed. It can handle errors like a governor limit exception or a limit on the number of Queueable Jobs that can be added to the queue per transaction that was used by other logic not related to the Queueable chaining.

Conclusion

We can receive a “Too many queueable jobs added to the queue: 2” exception during a Queueable Job execution from Apex Batch Jobs, future methods, and from Queueable Jobs in cases where more than one Queueable Job is enqueued from a single transaction. There are a lot of options to deal with the exception. In the situation where it is a single Queueable job in the project or you have just a couple of them, you can try to adopt a simple approach like using other asynchronous mechanisms instead of the Queueable, or you can try to prevent the execution of the Queueable Job for some specific use cases by covering the use case through some other techniques. If you prefer using the Queueable Jobs, you have a lot of them in your project, or you are going to implement a lot of them, then I would recommend looking closer at solutions based on a Custom Object. For example, you can take sources from the tshevchuk/Async_Request__c.object repository and start using them for queuing the Queueable Jobs.

→ Read more about Avenga Salesforce expertise