Spring Batch

Scaling Spring Batch – Step Partitioning

by on December 9, 2013 9:35 am

We have talked about how to get up and running with Spring Batch in a few of previous articles. Now, we’re going to start discussing some of the strategies available for scaling Spring Batch.

This article will focus on partitioning a step so that the step has several threads that are each processing a chunk of data in parallel. This is beneficial if you have a large chunk of data that can be logically split up into smaller chunks that can be processed in parallel. The way this works is that you will define a master step that is responsible for determining the basis of the chunks, and then farming all of those chunks out to a set of slave steps to process each chunk.

Partitioning

If I could go back in time to a previous experience, a perfect example of this would be processing all of the daily invoices for each company within a large procurement system. Our data that would be processed could be split up logically by each company being processed. Let’s say that there are 250 companies that participate in this procurement system and our partitioned step has been defined to have 15 threads. Our Partitioner would likely run a query to find all of the companies that have invoices waiting to be processed for that day. The Partitioner’s responsibility at that point would be to create an ExecutionContext for each of those companies and add that to a map with a unique key. That ExecutionContext should contain any information required to process invoices for that company, such as the company ID and any other relevant information. When the Partitioner returns the map of ExecutionContexts, Spring Batch will then create a new Step for each entry in the map and use the key value as part of the step name. Based upon the configuration, such as 15 threads, it will then create a pool of 15 threads and start executing the steps in parallel 15 at a time. For example, if you had 85 steps, Spring Batch would start executing 15 steps and as each step completed, the thread that had completed the step would pick up the next step and begin execution until all steps have been completed.

An Example

So now that we have a basic understanding of how partitioning works, let’s take a look at a simple example. For our use case, we’re going to be inspecting an inbound directory where incoming vendor catalog files will be dumped. So to create a spring batch partitioner, we’ll need to create a class that implements the Spring Batch’s Partitioner interface. Since this is something generic and can be made reusable, we’re going to call this class MultiFileResourcePartitioner, which is a simple POJO and has only one field name “inboundDir” representing the path to the directory that contains the files to be processed. The Partitioner interfaces specifies that the class should implement a method named “partition” that takes an int parameter representing the grid size and returns a Map that holds the ExecutionContext.

Here’s the class listing for MultiFileResourcePartitioner:

package com.keyhole.example.partition;

import java.io.File;
import java.util.HashMap;
import java.util.Map;

import org.springframework.batch.core.partition.support.Partitioner;
import org.springframework.batch.item.ExecutionContext;
import org.springframework.core.io.FileSystemResource;

public class MultiFileResourcePartitioner implements Partitioner {

	private String inboundDir;

	@Override
	public Map<String, ExecutionContext> partition(int gridSize) {

		Map<String, ExecutionContext> partitionMap = new HashMap<String, ExecutionContext>();
		File dir = new File(inboundDir);
		if (dir.isDirectory()) {
			File[] files = dir.listFiles();
			for (File file : files) {
				ExecutionContext context = new ExecutionContext();
				context.put("fileResource", file.toURI().toString());
				partitionMap.put(file.getName(), context);
			}
		}
		return partitionMap;
	}

	public String getInboundDir() {
		return inboundDir;
	}

	public void setInboundDir(String inboundDir) {
		this.inboundDir = inboundDir;
	}

}

The configuration for this bean in our application context will look like this:

<bean id="inventoryFilePartitioner"
		class="com.keyhole.example.partition.MultiFileResourcePartitioner"
		scope="step">
		<property name="inboundDir" value="/data/inbound" />
</bean>

Looking at the partition method that is implemented, we’re simply going to list all of the files that are found in the specified inbound directory and create an ExecutionContext for each file that is found add it to the map that is returned. The unique key that is used to place each ExecutionContext into the map will also become part of the step name that is created for each entry in the map. Spring Batch will use that partition map to create a slave step from each of the keys that are found in the map.

To partition a step, you need to first create the step that will be referenced by the partition configuration. This step should configured just like any other step within a Spring Batch and for this example we’ll define a FlatFileItemReader and a simple ItemWriter that will just call the toString() method and log it to the console.

Here are the configuration details for the step and its associated spring beans. The important thing to note here is that we have the ItemReader scoped at the step level so that we don’t run into any issues with multiple threads using the same beans to process the data. We also need them scoped that way so that we can use the Spring late binding to specify the value holding the file resource in the Step’s ExecutionContext.

<batch:step id="inventoryLoadStep"
	xmlns="http://www.springframework.org/schema/batch">
	<batch:tasklet transaction-manager="transactionManager">
		<batch:chunk reader="inventoryLoadReader" writer="logItemWriter"
			commit-interval="5000" />
	</batch:tasklet>
</batch:step>
<bean name="inventoryLoadReader" scope="step"
	class="org.springframework.batch.item.file.FlatFileItemReader">
	<property name="resource"
		value="#{stepExecutionContext['fileResource']}" />
	<property name="lineMapper" ref="inventoryLineMapper" />
<property name="linesToSkip" value="1" />
</bean>

<bean name="inventoryLineMapper"
	class="org.springframework.batch.item.file.mapping.DefaultLineMapper">
	<property name="fieldSetMapper" ref="inventoryFieldMapper" />
	<property name="lineTokenizer" ref="inventoryLineTokenizer" />
</bean>

<bean name="inventoryLineTokenizer" 		class="org.springframework.batch.item.file.transform.DelimitedLineTokenizer" />

Since we are reading and processing comma delimited text files in this example, we have very little code to write for this step configuration. The only code we’ll need to implement is the FieldSetMapper required to map the contents of the line to the object representing the file record. Each line in the file will contain the fields “category”,”sub category”,”description”,”catalog num”,”color”,”size”,”price” and “qty.” So our object will contain those fields and our FieldSetMapper code listing will look like this.

package com.keyhole.example.partition;

import org.springframework.batch.item.file.mapping.FieldSetMapper;
import org.springframework.batch.item.file.transform.FieldSet;
import org.springframework.stereotype.Component;
import org.springframework.validation.BindException;

@Component("inventoryFieldMapper")
public class InventoryItemFieldSetMapper implements FieldSetMapper<InventoryItem> {

	@Override
	public InventoryItem mapFieldSet(FieldSet fieldSet) throws BindException {
		InventoryItem item = new InventoryItem();
		item.setCategory(fieldSet.readString(0));
		item.setSubCategory(fieldSet.readString(1));
		item.setDescription(fieldSet.readString(2));
		item.setCatalogNum(fieldSet.readString(3));
		item.setColor(fieldSet.readString(4));
		item.setSize(fieldSet.readString(5));
		item.setPrice(fieldSet.readDouble(6));
		item.setQty(fieldSet.readInt(7));
		return item;
	}

}

Now that we have the Partitioner and Step created and configured, all that remains left to do is configure the partitioned step itself! And it’s as easy as this:

<batch:job id="InventoryLoader">
	<batch:step id="partitionedInventoryLoadStep">
		<batch:partition step="inventoryLoadStep" partitioner="inventoryFilePartitioner">
			<batch:handler grid-size="10" task-executor="inventoryLoadTaskExecutor" />
		</batch:partition>
	</batch:step>
</batch:job>

When configuring the partitioned step, you define a step just as you would any other step by giving it an ID and if required a next step value. Instead of defining the contents of a step as a normal chunk or tasklet, Spring Batch provides a partition tag that requires you to specify the job step to be partitioned and the Partitioner that will be used to determine the chunks of data. You will also need to define the partition handler that will be processing those steps, in this case we’ll be using a ThreadPoolTaskExecutor which will have a thread pool size of 10 and allow them to timeout if they aren’t being used.

<bean id="inventoryLoadTaskExecutor"
	class="org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor">
	<property name="corePoolSize" value="10" />
	<property name="maxPoolSize" value="10" />
	<property name="allowCoreThreadTimeOut" value="true" />
</bean>

So, if you have a Spring Batch process that has a step processing a large amount of records and you’re interested in improving the performance, consider giving step partitioning a try. It should be easy to implement and provide you with some additional performance to help speed up processing times.

Additional Resources

For the example code associated with this article, I have uploaded the source code to a github repository at https://github.com/jonny-hackett/batch-example. To execute the example code related to this article, there is a JUnit test name InventoryLoadTest. The data files are located under src/test/resources/data/inbound and need to be placed into a local directory that matches the Partitioner inbound directory. Also, check out http://docs.spring.io/spring-batch/reference/html/scalability.html.

— Jonny Hackett, asktheteam@keyholesoftware.com

Spring Batch Blog Series

Part One: Introducing Spring Batch

Part Two:  Getting Started With Spring Batch

Part Three: Generating Large Excel Files Using Spring Batch

Scaling Spring Batch – Step Partitioning

Spring Batch Unit Testing and Mockito

Spring Batch – Replacing XML Job Configuration With JavaConfig

  • Share:

5 Responses to “Scaling Spring Batch – Step Partitioning”

  1. […] Scaling Spring Batch – Step Partitioning […]

  2. […] Scaling Spring Batch – Step Partitioning […]

  3. […] Scaling Spring Batch – Step Partitioning […]

  4. David says:

    I have a Spring Batch which currently uses this implementation of partitioning, the corePoolSize and maxPoolSize are set in a property file. However, because I have some external dependencies, I cannot run the ideal max pool size all day. The system can handle a much higher load at night than during the day, and each day/night could have external factors changing the number of threads I would be able to call at a given time.

    My question, is there a way to control the number of threads in the thread pool at run time? I would like the pool size to change depending on the average response time, or some factor like that.

Leave a Reply

Things Twitter is Talking About
  • ICYMI: we've released a demo version of #GrokOla which is open to the public. Try out its features & capabilities - http://t.co/O4ladowmFU
    July 4, 2015 at 3:05 PM
  • Happy 4th of July from the Keyhole team! We hope that you have a happy and safe holiday with your family and friends.
    July 4, 2015 at 9:55 AM
  • Let's talk testing. Here are common challenges #Agile teams face when writing automated tests & how to overcome them: http://t.co/DrKbNZJcE0
    July 3, 2015 at 11:06 AM
  • #GrokOla users get free educational tutorials. But lucky you, we've released some to the public. #JavaScript primer - http://t.co/nIR9XiWY6O
    July 3, 2015 at 10:55 AM
  • Being able to isolate debugging techniques can help make you a better debugger. Here's Time-Oriented #Debugging http://t.co/UplJgP4VzC
    July 2, 2015 at 10:50 AM
  • RT @zachagardner: @zachagardner has declared it is @ChipotleTweets day at @KeyholeSoftware . You have been warned 🐓🐖🐄
    July 2, 2015 at 10:09 AM
  • Current state of random number generation & the differences in how #Java & #JavaScript approach it - http://t.co/5tBKNXnu8T #security
    July 1, 2015 at 2:45 PM
  • Woohoo - 600 followers! Thanks, everyone. We'd love to ask you - what type of tweets / dev content would you like to see more of from us?
    July 1, 2015 at 10:38 AM
  • We would like to welcome Dallas Monson to the team today! Dallas is a Senior Architect focused on UI/UX and #JavaScript. Welcome, Dallas!
    July 1, 2015 at 8:35 AM
  • Good introduction to TypeScript - http://t.co/0N22fVpAHt Plus, how to approach modularization in #TypeScript - http://t.co/wxRWGBj3Uh
    June 30, 2015 at 3:25 PM
  • .@mrbristopher just delivered a new S911 Night Drone to James Hayes, winner of our #kcdc15 giveaway! Congrats, James! http://t.co/RriJIxubH2
    June 30, 2015 at 11:35 AM
  • It feels like primitives could have been left out of the initial implementation of #Java. See why - http://t.co/A8ChCBHXJO
    June 29, 2015 at 4:05 PM
  • Developers in a bounce house! I repeat, developers in a bounce house! We had a blast at our 1st company picnic. Pics: http://t.co/XIqs7ECUst
    June 29, 2015 at 1:40 PM
  • New #SpringBatch tutorial from @jhackett01: Spring Batch – Replacing XML Job Configuration With JavaConfig http://t.co/PmdXnriKQu #java
    June 29, 2015 at 11:46 AM
  • We had such a fun time at the Keyhole company picnic! Pictures to come, including some of our developers in the bounce house. #loveourteam
    June 29, 2015 at 8:41 AM
  • In #JavaScript, how do we harness the power of callbacks without the confusing mess of nested functions? Promises - http://t.co/j1gAJ9hi3D
    June 29, 2015 at 8:40 AM
  • .@zachagardner We are so happy that your family attended! This will definitely need to be repeated every year!
    June 28, 2015 at 8:14 PM
  • Thank you to all on the Keyhole team who came to our first inaugural company picnic! Wonderful food, family and bounce house fun!
    June 28, 2015 at 7:50 PM
  • Debugging is a challenging part of being a programmer. We have a tutorial series to help, with a #JavaScript focus - http://t.co/rfhjJo64P7
    June 27, 2015 at 1:45 PM
  • We love #KCDC15! @PinsightMedia's James Hayes just was drawn as the winner of the @KeyholeSoftware drone giveaway. Stop on by the KHS booth.
    June 26, 2015 at 2:24 PM