Large Datasets with Spring Batch

Utilizing Spring Batch for Large Dataset Summarization

Clayton Neff Databases, Java, Spring Batch Leave a Comment

I was recently tasked with summarizing the data of a several-million-row table, and the task proved to be a bit grueling at first. Eventually, I found a way to summarize the large dataset with Spring Batch, but not without a wrong turn or two at first. In this post, I’ll walk you through my process and how I overcame this …

Transferring Spring Batch Apps to AWS Cloud

Rik Scarborough AWS, Cloud, Development Technology, Spring, Spring Batch, Tutorial 2 Comments

The last few years have seen a lot of movement to bring applications that don’t require manual intervention from the mainframe to Unix, Linux, Windows servers, or even to the desktop. This concept is commonly known as batch programming, and Spring Batch has been the tool many of us are using to accomplish this. Another trend that is gaining steam is to move from an internally-hosted server to a cloud-hosted system.

In this post, we discuss multiple ways for transferring Spring Batch applications up to the AWS Cloud, including EC2, Docker, Lambda, and others. I concentrate on AWS in this post, but, from my experience in Google Cloud, the same ideas will apply…

Spring Batch Testing & Mocking Revisited with Spring Boot

Jonny Hackett Development Technology, Java, Spring, Spring Batch, Spring Boot, Testing Leave a Comment

Several years ago, 2012 to be precise, I wrote an article on an approach to unit testing Spring Batch Jobs. My editors tell me that I still get new readers of the post every day, so it is time to revisit and update the approach to a more modern standard.

The approach used in the original post was purely testing the individual pieces containing any business logic. Back then, we didn’t have some of the mocking capabilities that we have today, so I went with an approach that made sense at the time.

However, there have been a few improvements in the past several years. One of those improvements has been the ability to Mock beans within a Spring Context. That’s where the @MockBean annotation comes to the rescue.

Using Apache POI With Protected Excel Files

Jonny Hackett Development Technology, Java, Spring Batch 1 Comment

While working on a recent project at a client, we had the opportunity to refactor some data extracts that were using a commercial Excel writing library, which we then converted to using the Apache POI Library for Excel. These data extracts were reports that included some calculated values, and depending on the client, were required to be password protected. When completed, the reports would be emailed to the recipients configured for each client.

In this post, we discuss the challenge of delivering protected Microsoft documents via email. We introduce a Java code solution for emailing password-protected Excel files when using the Apache POI Library.

Some of the required calculations we chose to implement using Excel formulas. Implementing formulas wasn’t a hard task and worked for what was needed.

Reading and Writing from Excel in Spring Batch

Rik Scarborough Development Technology, Java, Spring, Spring Batch, Tutorial Leave a Comment

We have discussed many different ways to read and write data in Spring Batch. The framework comes with quite an assortment of Readers and Writers that can be used directly or reused in some manner. Most of the time, the requirements consist of reading the data from some type of text file or database.

So what happens when the business we are supporting asks for something out of the ordinary, such as reading an Excel file and outputting the data to another Excel file? Typically the off-the-cuff response would be, “can you convert it to a CSV or other delimited text file?” Or “You know, Excel will read a CSV file just fine.” Sometimes that works, and sometimes the business requirements do not allow that type of flexibility.

Consider this scenario; in these days of Cloud and other online computing, the input file is likely created by a server that the company has no direct access to as far as programming. The file it creates is in one format, Excel. The output of your process has to go before several executives or other business clients and needs to be formatted in a professional looking manner. Adding a manual process to import a CSV and format it diminishes the value of using Spring Batch.

For the sake of the honor of the coding profession, you agree to the requirement to read and write from an Excel file directly. Now, how do you do that?…