multithreading,spring,spring-batch , Spring batch diffrence between Multithreading vs partitioning


Spring batch diffrence between Multithreading vs partitioning

Question:

Tag: multithreading,spring,spring-batch

I cannot understand the difference between multi-threading and partitioning in Spring batch. the implementation is of course different. In partitioning you need to prepare the partitions then process it. I want to know what is the difference and witch one is more efficient way to process when the bottle neck is the item-processor.


Answer:

TL;DR;
Neither approach is intended to help when the bottleneck is in the processor. You will see some gains by having multiple items going through a processor at the same time, but both of the options you point out get their full benefits when used in processes that are I/O bound. The AsyncItemProcessor/AsyncItemWriter may be a better option.

Overview of Spring Batch Scalability
There are five options for scaling Spring Batch jobs:

  1. Multithreaded step
  2. Parallel steps
  3. Partitioning
  4. Remote chunking
  5. AsyncItemProcessor/AsyncItemWriter

Each has it's own benefits and disadvantages. Let's walk through each:

Multithreaded step
A multithreaded step takes a single step and executes each chunk within that step on a separate thread. This means that the same instances of each of the batch components (readers, writers, etc) are shared across the threads. This can increase performance by adding some parallelism to the step at the cost of restartability in most cases. You sacrifice restartability because in most cases, the ability to restart is based on the state maintained within the reader/writer/etc. With multiple threads updating that state, it becomes invalid and useless for restart. Because of this, you typically need to turn save state off on individual components and set the restartable flag to false on the job.

Parallel steps
Parallel steps are achieved via a split. It allows you to execute multiple, independent steps in parallel via threads. This does not sacrifice restartability, but does not help improve the performance of a single step or piece of business logic.

Partitioning
Partitioning is the dividing of data, in advance, into smaller chunks (called partitions) by a master step and then having slaves work independently on the partitions. In Spring Batch, both the master and each slave, is an independent step so you can get the benefits of parallelism within a single step without sacrificing restartability. Partitioning also provides the ability to scale beyond a single JVM in that the slaves do not have to be local (you can use various communication mechanisms to communicate with remote slaves).

An important note about partitioning is that the only communication between the master and slave is a description of the data and not the data itself. For example, the master may tell slave1 to process records 1-100, slave2 to process records 101-200, etc. The master does not send the actual data, only the information required for the slave to obtain the data it is supposed to process. Because of this, the data must be local to the slave processes and the master can be located anywhere.

Remote chunking
Remote chunking allows you to scale the process and optionally the write logic across JVMs. In this use case, the master reads the data and then sends it over the wire to the slaves where it is processed and then either written locally to the slave or returned to the master for writing local to the master.

The important difference between partitioning and remote chunking is that instead of a description going over the wire, remote chunking sends the actual data over the wire. So instead of a single packet saying process records 1-100, remote chunking is going to send the actual records 1-100. This can have a large impact on the I/O profile of a step, but if the processor is enough of a bottleneck, this can be useful.

AsyncItemProcessor/AsyncItemWriter
The final option for scaling Spring Batch processes is the AsyncItemProcessor/AsycnItemWriter combination. In this case, the AsyncItemProcessor wraps your ItemProcessor implementation and executes the call to your implementation in a separate thread. The AsyncItemProcessor then returns a Future that is passed to the AsyncItemWriter where it is unwrapped and passed to the delegate ItemWriter implementation.

Because of the nature of how data flows through this option, certain listener scenarios are not supported (since we don't know the outcome of the ItemProcessor call until inside the ItemWriter) but overall, it can provide a useful tool for parallelizing just the ItemProcessor logic in a single JVM without sacrificing restartability.


Related:


Multiple Threads searching on same folder at same time


c#,multithreading,file-search
Currently I have a .txt file of about 170,000 jpg file names and I read them all into a List (fileNames). I want to search ONE folder (this folder has sub-folders) to check if each file in fileNames exists in this folder and if it does, copy it to a...

wait for an event regulary


multithreading,events
In a program I need to wait for an event (keypress) and get it's char. after that program will continue. this progress will Repeat several time. in my first try, codes run and any character did not save. I find out I should use threading but I am not Familiar...

Can't Override Equals method on Java when calling contains


java,spring,hibernate
I have the following funtion that checks if a List of codigos contains a single codigos object: if (!concorrente.getJcodigoses().contains(cod)) { return "redirect:" + referrer; } I read that i need to Override the equals method like so: @Override public boolean equals(Object object) { boolean isEqual= false; if (object != null...

How to apply HandlerInterceptor to Spring Boot Actuator endpoints?


java,spring,spring-mvc,spring-boot
I'm working with Java 8 and Spring 1.2.3 to build an application running inside a Tomcat 7 container. I do intercept every single call to my web application using a very simple HandlerInterceptor, that logs the overall time taken to create a response and the return code for every request....

GenericApplicationContext cannot be cast to WebApplicationContext : Spring Web Flow


java,spring,spring-mvc,classcastexception,spring-webflow-2
I'm trying to set up Spring Web Flow using only Java annotations in a Spring environment that also uses only Java annotations. However when I attempt to access my flow I get the following exception SEVERE: Servlet.service() for servlet [dispatcher] in context with path [/forms] threw exception [Request processing failed;...

Importing Spring MVC Project Manually to Eclipse


java,eclipse,spring,spring-mvc
I have a little problem. I had googled for this problem and I haven't found any solution related. My problem is, my office internet using proxy that banned me to download zip file. I need Spring MVC Project template in Spring Tool Suite (STS). So I downloaded the zip file...

Is there standard implementation for thread block/resume in java SE?


java,multithreading,wait
I need to block execution of a thread until resumed from another thread. So I wrote my own implementation using wait() method. Which seems to be working, but it is far from simple. Is there any ready to use solution? Preferably in java SE 6? Or do I have to...

Is it possible to construct an object partially with Spring DI and partially with Jersey DI?


spring,dependency-injection,jersey,jersey-2.0,hk2
I have a class JerseyWebService which uses Jersey DI to inject dependency @Path("/baskets") public class JerseyWebService { @Inject ExternalApiServiceInterface api; ... } The dependency is specified in the binder public class CustomBinder extends AbstractBinder { @Override protected void configure() { bind(ExternalApiService.class).to(ExternalApiServiceInterface.class); ... } But the problem here is that ExternalApiService...

Looking to pause a thread using thread.sleep


java,multithreading
I am using a mouse listener for mouse pressed and released. When the mouse is pressed I want to have a counter incrementing a variable, and when the mouse is released I want to decrement that variable. Right now, my code is working and does that but the increment is...

Spring Data Rest: Return Resources of User


java,spring,spring-security,spring-data-rest
I'm creating a simple CRUD-App (a shoppinglist) with Spring Boot and Spring Data Rest. I have a resource: ShoppingItem. Is there an easy way to only return the resources which belong to the user who send the request? (Multiple User support) So a User only get his own ShoppingItems and...

java multithreading start() and run() [duplicate]


java,multithreading
This question already has an answer here: Java: What's the difference between Thread start() and Runnable run() 10 answers Below is my Multithreading class: public class Multithreading extends Thread{ public void run(){ for(int i=1;i<5;i++){ try{ Thread.sleep(500); }catch(InterruptedException e){ System.out.println(e); } System.out.println(i); } } public static void main(String args[]) {...

Dynamic fields thymeleaf list iteration


spring,spring-mvc,spring-boot,thymeleaf
I'm getting a really weird error ! while iterating on list thymeleaf identified index as a property of my bean and not an index value ! <div th:each="phoneStat : *{phones}"> <select th:field="*{phones[__${phoneStat.index}__].variety}" > <option> </option> </select> <div class=" input-field col s4" > <input th:field="*{phones[__${phoneStat.index}__].number}" th:id="${'phonenumber-'+ phones[__${phoneStat.index}__]}" type="text" class="validate" /> <label...

How can implement long running process in spring hibernate?


java,spring,hibernate
as we know the struts interceptor execute and wait will take care of long running process by not getting the request to timeout and destroy it sends wait and at last the desired response i want to implement the same for long running process in spring and hibernate. Thanks....

Java 5 Multi threading, catch thread exceptions


java,multithreading,exception,concurrency,java-5
I've got a class that generates threads (file i/o). I need to catch exceptions in the thread - I don't want to do anything fancy, I want to kill the main thread, rather, stop processing altogether so it can start over. If I catch the exceptions in the thread, that's...

Spring Boot - How to set the default schema for PostgreSQL?


java,spring,hibernate,postgresql
What I am currently doing in my application.properties file is: spring.datasource.url=jdbc:postgresql://localhost:5432/myDB?currentSchema=mySchema Isn't there another property for this? Since it looks hacky and according to a post (still finding the link sorry, will update later), it is only applicable to PostgreSQL 9.4....

Java how to limit number of threads acting on method


java,multithreading,memory-management
I have java method in my web application doing heavy file operation. The thing is, if more than 5 threads come simultaneously (which will come in testing phase) it breaks down. I mean it cannot handle heavy traffic. That's why I want to handle maximum 5 requests at a time...

Any way to catch an exception occurring on a thread I don't own?


c#,multithreading,exception
I'm using a crappy third party library because unfortunately I have no choice. This library creates a bunch of threads internally when I call it, and occasionally throws a NullReferenceException on one of these threads. Is there any way for me to catch these exceptions even though I don't own...

JdbcTemplate Mockito ClassCastException


spring,mockito,classcastexception
I am trying to mock a method of the Spring framework's JdbcTemplate class. The method has the following signature: public <T> T queryForObject(String sql, Class<T> requiredType) throws DataAccessException {...} The mocking is being done as mentioned below: when(jdbcTemplate.queryForObject(anyString(), eq(String.class))).thenReturn("data"); However, this call throws the following exception java.lang.ClassCastException:...

Transaction error in Spring


java,spring,jpa
I have a project https://github.com/ivartanian/onlinecontrol in Spring MVC + JPA, initially worked with the Spring configuration through XML. Now I am trying to set up the project through JAVA Config (without XML). The project starts, but when I create a new user(entity User), an error transaction. But if I create...

How can we make a thread to sleep for a infinite time in java?


java,multithreading,thread-sleep
Pls help me to understand how can we make a thread to sleep for a infinite time period .

How to set property using “tasklet ref” tag


spring,spring-batch
I have a tasklet ValidarSituacaoTasklet that has an property situacao. This tasklet is used in 2 steps in distinct values for situacao. I declared steps as like: and the bean: <bean id="validarSituacaoTasklet" class="my.package.tasklet.ValidarSituacaoTasklet" scope="step"> </bean> I have to pass 'situacao' to tasklet . I tried: <step id="validaSituacaoStep"> <tasklet ref="validarSituacaoTasklet ">...

Images not appearing on WPF form when loading asynchronously


c#,wpf,multithreading,listbox,backgroundworker
I'm attempting to display (in a ListBox with a custom DataTemplate) a series of BitmapSource frames (thumbnails) extracted from a multi-page tiff image. When I process the tiff on the UI thread, and either directly add the images to a listbox's item collection or add them to a bound ObservableCollection,...

viewResolver with more folders inside of WEB-INF/jsp is not working in spring


java,spring,jsp,spring-mvc
<bean id="viewResolver" class="org.springframework.web.servlet.view.InternalResourceViewResolver"> <property name="viewClass" value="org.springframework.web.servlet.view.JstlView" /> <!-- avoid '???' --> <property name="prefix" value="/WEB-INF/jsp/" /> <property name="suffix" value=".jsp"/> </bean> if i create other subfolders under jsp , for instance /WEB-INF/jsp/reports , /WEB-INF/jsp/insertions how should i configure now the viewResolver to can resolve these new sub folders??...

How to implement customer subdomain in Spring framework


spring,saas,multi-tenancy
In many of the SaaS web applications (ex, Atlassian JIRA), a user can have dedicated subdomain. For example, if my user name is helloworld, then after I log in to the web application, I am redirected to helloworld.atlassian.net How to implement this in Spring Framework? Do I have to have...

Spring: @NestedConfigurationProperty List in @ConfigurationProperties


spring,properties,configuration,spring-boot
Hi I am trying to get the following configuration up and running. @ConfigurationProperties(prefix="my") public class Config { @NestedConfigurationProperty private List<ServerConfiguration> servers = new ArrayList<ServerConfiguration>(); public List<ServerConfiguration> getServers() { return this.servers; } } @ConfigurationProperties(prefix = "server") public class ServerConfiguration { private String name; private String description; } So, I want to...

Set Label From Thread


vb.net,multithreading,winforms
Form1.vb Imports System.Threading Public Class Form1 Dim demoThread As Thread Private Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click Dim Start As New Class1 Me.demoThread = New Thread( _ New ThreadStart(AddressOf Start.ThreadProcSafe)) Me.demoThread.Start() End Sub Delegate Sub SetTextCallback([text] As String) Public Sub SetText(ByVal [text] As String) ' InvokeRequired required...

Why is my Spring Boot autowired JPA Repository failing JUnit test?


java,spring,junit,spring-boot
My JUnit test is failing with the following error: "java.lang.IllegalArgumentException: Could not find field [userRepository] of type [null] on target [[email protected]]" Here is the Test class: @RunWith(SpringJUnit4ClassRunner.class) @SpringApplicationConfiguration(classes = App.class) @WebAppConfiguration @IntegrationTest public class UserResourceTest { @Autowired private UserRepository userRepository; private MockMvc restUserMockMvc; @Before public void setup() { UserResource userResource...

Default Spring bean when profile is not present


java,spring,spring-mvc,spring-profiles
I'm using Spring MVC controllers, and I'd like to make some extra request-mapped methods available during development. I can do this using Spring profiles: @Controller @Profile("!dev") public class DefaultController { } @Controller @Profile("dev") public class DevController extends DefaultController { } But ideally, I'd rather not pollute the production classes' code...

Calling dispatch_sync from a concurrent queue - does it block entirely?


ios,objective-c,multithreading,swift,grand-central-dispatch
Let's say I hypothetically call a dispatch_sync from a concurrent queue - does it block the entire queue or just that thread of execution?

shall I use Spring framework for a performance-critical proxy application? [closed]


java,spring,authentication,servlets
I've created a servlet (Tomcat) application which has these functions: It performs HTTP Basic Authentication. It connects to a user and role database. It works as "security facade" for some geodata servers behind It forwards requests after doing some authorization tests In case the response contains XML data, it performs...

Cannot create bean when start the application


java,spring,javabeans,sftp,job-scheduling
I am facing some issue while creating the bean it gives error. Can someone please assist me. I will retrieve the files from sftp resource and will read those file with a spring task scheduler job. App-Config.xml <?xml version="1.0" encoding="UTF-8"?> <beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:context="http://www.springframework.org/schema/context" xmlns:task="http://www.springframework.org/schema/task"...

Spring OAuth2 not giving refresh token


java,spring,spring-security,spring-security-oauth2
I am running a OAuth Provider using Spring and "password" grant type. Running this (provider is on port 8080): curl -u "app:appclientsecret" "http://localhost:8080/oauth/token" --data "grant_type=password&username=marissa&password=koala" returns: {"access_token":"56da4d2b-7e66-483e-b88d-c1a58ee5a453","token_type":"bearer","expires_in":43199,"scope":"read"} For some reason there is no refresh token. I know according to the spec, the refresh token is optional; is there some way...

Hibernate : Stale state exception


java,spring,hibernate,spring-mvc,transactions
I am working on a Spring-MVC application in which I am trying to delete an object from the database. Some days back, this error all of a sudden started, and now I am unable to delete. I checked on net, but I cannot find what am I doing wrong and...

Trouble with Login using Spring Boot and JDBC Security


spring,spring-security,spring-boot
My build.gradle file: task wrapper(type: Wrapper) { gradleVersion = '2.4' } buildscript { repositories { mavenCentral() maven { url "http://repo.spring.io/release" } } dependencies { classpath("org.springframework.boot:spring-boot-gradle-plugin:1.2.3.RELEASE") } } apply plugin: 'java' apply plugin: 'spring-boot' repositories { mavenCentral() maven { url "http://repo.spring.io/release" } } dependencies { compile("org.springframework.boot:spring-boot-starter-web") compile("org.springframework.boot:spring-boot-starter-jdbc")...

Web API - Set each thread with the HttpRequestMessage id?


c#,.net,multithreading,task-parallel-library,web-api
I have a web api coded in c#. The web api uses functionality which is shared with other in-house components. it depends on single threaded flows and uses thread local storage to store objects, and session information. Please don't say if it's good or bad, that's what I have to...

Spring Resttemplate login fails


java,spring,spring-mvc,spring-security,csrf
after an update to spring-4.1.6 i'am not able to login to my rest services any more. I looked on different sides, but couldn't solve the problem... so i ask for help. Here is my my web.xml: <?xml version="1.0" encoding="ISO-8859-1"?> <web-app xmlns="http://java.sun.com/xml/ns/javaee" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd" version="2.5"> <!-- The definition of the...

Error while setting targetConnectionFactory in UserCredentialsConnectionFactoryAdapter Spring 4


java,spring,jms,spring-boot,spring-jms
We are receiving a compilation error in the following code, while trying to set the targetConnectionFactory. According to the example given in Spring 4 documentation for UserCredentialsConnectionFactoryAdapter, the setTargetConnectionFactory() method takes the object of JndiObjectFactoryBean as an argument. But, looks like the JndiObjectFactoryBean hasn't implemented the 'ConnectionFactory' interface, which is...

Atomic/not-atomic mix, any guarantees?


c++,multithreading,c++11,atomic
Let's I have GUI thread with code like this: std::vector<int> vec; std::atomic<bool> data_ready{false}; std::thread th([&data_ready, &vec]() { //we get data vec.push_back(something); data_ready = true; }); draw_progress_dialog(); while (!data_ready) { process_not_user_events(); sleep_a_little(); } //is it here safe to use vec? As you see I not protect "vec" by any kind of...

Cron expression must consist of 6 fields (found 1 in “#{systemEnvironment['db_cron']}”)


spring,groovy,cron,spring-el
I'm trying to set a cron Scheduled annotation as follows: @Scheduled(cron = "#{systemEnvironment['db_cron']}") def void schedule() { } Next set the environment variable as: export db_cron="0 19 21 * * *" However, I get the runtime error: Cron expression must consist of 6 fields (found 1 in "#{systemEnvironment['db_cron']}") What can...

@RestController throws HTTP Status 406


java,spring,rest,maven
I am working on a basic Hello World program using Spring and Restful webservices. But when I try to call my service I am getting below error message: HTTP Status 406 - description - The resource identified by this request is only capable of generating responses with characteristics not acceptable...

std::condition_variable – notify once but wait thread wakened twice


c++,multithreading
Here's a simple C++ thread pool implementation. It's an altered version orginated from https://github.com/progschj/ThreadPool. #ifndef __THREAD_POOL_H__ #define __THREAD_POOL_H__ #include <vector> #include <queue> #include <memory> #include <thread> #include <chrono> #include <mutex> #include <condition_variable> #include <future> #include <functional> #include <stdexcept> namespace ThreadPool { class FixedThreadPool { public: FixedThreadPool(size_t); template<class F, class......

performance issues executing list of stored procedures


c#,multithreading,performance,loops
I'm having some performance issues when starting my windows service, the first round my lstSps is long (about 130 stored procedures). Is there anyway to speed this up (except for speeding the stored procedures up)? When the foreach is over and goes over to the second round it goes faster,...

Override Spring form error messages


java,spring,default,custom-errors
In Spring how do I override default form error massages ? I'm using a Validator and a properties file to add my own error messages, but how do I override messages that get printed on conversion/encoding error for example ? They seem to be generated automatically and I don't think...

Spring Data JPA user posts


java,spring,jpa,data
I have user login and profile view, I would like the users to have posts. Can someone guide me in the right direction? I have a user entity: @Entity @Table(name = "usr", indexes = { @Index(columnList = "email", unique = true) }) // using usr because in may conflict with...

How does the kernel separate threads from processes


linux,multithreading,linux-kernel
Suppose I have a browser process like Firefox, that has pid = 123. Firefox has 5 opened tabs each running in a separate thread, so in total it has 5 threads. So I want to know in depth, how the kernel will separate the process into the thread to execute...

Does wait() need synchronization on local variable


java,multithreading,synchronization
I had this code (which was working fine): public static void runOnUiThread(Activity c, final Runnable action) { // Check if we are on the UI Thread if (Looper.getMainLooper() == Looper.myLooper()) { // If we are, execute immediately action.run(); return; } // Else run the runnable on the UI Thread and...

How can I know the lock information in java?


java,multithreading,locking
Is there any tool or way that can get all the information about the locks in java? for example, if there is a java program, it creates two threads, and both threads require locks for some variable. Is there any tools that can output the information like which thread locks...

What happens if all node.js's worker threads are busy


javascript,node.js,multithreading
I try to understand how node.js works and although I have read this article: When is the thread pool used? I am not sure what happens if all worker threads are busy and another async I/O operation is ready to be executed. If I got this http://www.future-processing.pl/blog/on-problems-with-threads-in-node-js/ article right, the...

Logging operations in lightadmin


java,spring,logging,lightadmin
I have a Java web application which use Spring and Hibernate and I plan to use lightadmin to provide an administration interface. However, I found very little information about the logging part of lightadmin : if I have such an adminsitration interface, I would like that any operation made to...

Spring: Response time


java,spring,spring-ws
I have a SOAP webservice(spring ws)and need to save the response time of every request that I receive. I could create a servlet filter that measures time difference between HTTP Request and HTTP Response. But, I need to log the response time together with some values that I read from...