Throttling and Batching with Boomi Atom Queues
Through the use of Atom Queues and Listeners a single process can spawn many iterations of a listening process. Each process spawned from a listener will execute asynchronously, independent of any other executions. By default, an Atom Queue listener will spawn an instance of the listening process every time a document is written to the queue. While this will work fine in cases with a low throughput, larger numbers of documents being processed will cause a large number of executions to get kicked off.
In some cases, such as a large file being de-batched into many smaller documents, this can cause slowdown in Boomi as executions must be instantiated and then cleaned up after each completion. This may also cause errors in certain endpoints. For instance, an HTTP web service that can only handle a smaller number of requests in parallel will cause time-outs in Boomi, as many instances of the same process wait on the first few to finish their request. To work around the issue of large numbers of executions being spawned at once, the listener can be throttled, batched, or both.
One of the easiest ways to prevent massive amounts of listener instances being spawned at once is to change the “Maximum Concurrent Executions” in the Atom Queue operation settings. This setting limits the number of instances of this process that can be running at any given time. Once the limit is reached, any further instances will wait to execute until an existing instance has finished. By default, this value is set to zero. Any value zero or lower will allow infinite instances, so it’s good practice to make sure this number is something above zero. Preventing the possibility of infinite instances spawning will help keep the code maintainable a prevent unexpected errors.
Another way to reduce the number of listener executions, is to use a Flow Control shape to batch the documents in the sending process. A Flow Control shape can, among other things, group incoming documents in batches to be sent through the process. The other features of Flow Control aren’t directly relevant to this blog and need a more in depth post, so they won’t be discussed right now.
Place a Flow Control shape at some point before the Atom Queue connector. It is not necessary for the Flow Control to be placed before document processing, but this will ensure batched documents go through processing as a group. In the Flow Control configuration, select the option to “Run as Batches”. Then enter how many documents you want in each batch. In the example below, the Atom Queue connector will send 10 documents at once to the queue. This means one Listener process will execute with 10 documents instead of 10 processes executing with one document each.
Batching documents should not be used as a replacement for limiting the number of concurrent executions, but more as a way to augment throttling by reducing the total number of spawned executions. If a sending process with a Flow Control set up like the example below received 100 documents, then 10 batches of documents would be sent to the queue. This would in turn spawn 10 instances of the listener process.
Dell Boomi AtomSphere
The Dell Boomi AtomSphere integration platform is a shared-everything, multi-tenant platform that supports cloud-to-cloud, SaaS-to-SaaS, cloud-to-on-premises, on-premises-to-on-premises and B2B integration. Boomi AtomSphere supports real-time integration and elastically scales to meet high-volume needs in mobile, batch (ETL) and EDI environments. Easily accessed via a browser, it delivers an impressive range of integration, master data management (MDM) and platform extension capabilities.
Tallan Integration Solutions for Dell Boomi
Tallan is a certified Dell Boomi Partner specializing in iPaaS platform integrations. We specialize in Integrations using EDI and Dell Boomi as the iPaaS platform of choice. Leverage Tallan’s vast integration experience for your AtomSphere platform needs. Our certified architects and developers provide the expertise, best practices, and guidance to deliver a successful integration solution.