spring-batch,spring-integration , spring integration vs spring batch [on hold]


spring integration vs spring batch [on hold]

Question:

Tag: spring-batch,spring-integration

We have application where we are receiving file every day and it needs to be parsed and persist in db. File has 5000 record.

Should we use spring batch or spring integration? And why?

we do need to skip bad record and audit them.


Answer:

In most cases, a combination is appropriate; use Spring Integration to detect new files arriving and use the job launching gateway to launch the job. Especially if you need the richer job semantics (skip, retry etc).

See the Spring Batch documentation about using it with Spring Integration.


Related:


How to concatenate a string with SpEL in SpringXD Stream Definition


spring-integration,spring-el,spring-xd
I'm trying to create a stream definition that uses two payload fields joined together to create a concatenated string. stream create --name blah --definition "http | gemfire-json-server --keyExpression=payload.getField('deviceId') + payload.getField('timestamp')" --deploy` The concatenation piece in the keyExpression is incorrect, what should it be to get it to work inline?...

Maximum simultaneous connection when using ftp adapter


ftp,spring-integration
What can I tweak to solve this maximum connection problem? Any ways to specify a limit? This is from a Spring Integration application. <int-ftp:outbound-channel-adapter id="matasLiveProdSdkOutbound" channel="ftpOutboundChannel" session-factory="ftpsSessionFactory" charset="UTF-8" auto-create-directory="true" use-temporary-file-name="false" remote-file-separator="/" remote-directory-expression="${egnyte.remote.dir}" mode="IGNORE"> </int-ftp:outbound-channel-adapter> This is the error I get: Caused by:...

Spurious error “Cannot correlate response - no pending reply” using TcpOutboundGateway and CachingClientConnectionFactory


java,multithreading,spring-boot,spring-integration
I am getting spurious correlation errors using TcpOutboundGateway with CachingClientConnectionFactory in a multithreaded context. The log message is: 2015-05-26 14:50:38.406 ERROR 3320 --- [pool-2-thread-2] o.s.i.ip.tcp.TcpOutboundGateway : Cannot correlate response - no pending reply I do not get the error when sending from a single thread, and I have tested and...

Spring Integration: SMTP server


java,spring,email,smtp,spring-integration
I am using Spring Integration 4.1.2. In the program flow, at the end of a process I need to send an email. I am using the following: <int:payload-type-router input-channel="response.in"> <int:mapping type="java.lang.String" channel="response.out"/> <int:mapping type="org.springframework.mail.SimpleMailMessage" channel="mail.out"/> </int:payload-type-router> <mail:outbound-channel-adapter channel="mail.out" mail-sender="mailSender"/> This is working fine. I want to also handle the situation...

Which bean can I use as a alternative one of http-outbound-gateway?


spring-integration
I am changing spring-integration 3.x to 4.x. And I should change every xml configuration to the java file. But, I can't find the bean that can be replaced in place of the followings. <int-http:outbound-gateway url="http://www.google.com/ig/api?weather={city}" http-method="GET" expected-response-type="java.lang.String" request-factory="requestFactory" request-channel="requestChannel" reply-channel="replyChannel"> <int-http:uri-variable name="city" expression="payload"/> </int-http:outbound-gateway> <bean...

Running a specific spring batch job amongst several jobs contained withing a spring boot fat jar


java,jar,spring-boot,spring-batch
I am trying to run a spring batch job from a spring boot fat jar and I am having issues referencing the nested jars. Here is the command I use: java -cp bignibou-batch-core/build/libs/bignibou-batch-core.jar:lib/spring-batch-core-3.0.3.RELEASE.jar org.springframework.batch.core.launch.support.CommandLineJobRunner com.bignibou.batch.configuration.BatchConfiguration mailingJob Notice how I reference the nested spring batch jar using the colon. Why is...

Syntax error on tokens, delete these tokens - kafka spring integration demo application


java,eclipse,spring-integration,apache-kafka
I am trying to use the demo application for spring kafka integration and getting an "syntax error on token, delete these token" error in eclipse. looks to be simple one, but i dont know how to fix. errors at return args -> @Bean @DependsOn("kafkaOutboundChannelAdapter") CommandLineRunner kickOff(@Qualifier("inputToKafka") MessageChannel in) { return...

Serialization error using a jdbc-message-channel


serialization,spring-integration
I'm using a channel backed by a jdbc-message-store but I'm having a serialization error when storing the message. The message payload is a simple String. Here's my config and stacktrace: <bean id="auditMessageStoreQueryProviderBean" class="org.springframework.integration.jdbc.store.channel.MySqlChannelMessageStoreQueryProvider" /> <bean id="auditMessageStore" class="org.springframework.integration.jdbc.store.JdbcChannelMessageStore"> <property name="dataSource" ref="auditMessageStoreDataSource" /> <property...

jdbc backed channel on a cluster mode


jdbc,cluster-computing,spring-integration
I want to use the jdbc backed message channel. Reading the doc says that it is supported but after that I found a thread in the forum that explicitly says don't. Could you please clarify me if it's supported? In this case, could you guide me to do so? thanks...

How to set property using “tasklet ref” tag


spring,spring-batch
I have a tasklet ValidarSituacaoTasklet that has an property situacao. This tasklet is used in 2 steps in distinct values for situacao. I declared steps as like: and the bean: <bean id="validarSituacaoTasklet" class="my.package.tasklet.ValidarSituacaoTasklet" scope="step"> </bean> I have to pass 'situacao' to tasklet . I tried: <step id="validaSituacaoStep"> <tasklet ref="validarSituacaoTasklet ">...

Spring integration Splitter task execution


java,spring-integration,splitter
I'm working on a Spring Integration application. I have an inbound channel adapter that read a directory Then a splitter that split the File into lines Finally an udp outbound channel adapter that sends the lines <int-file:inbound-channel-adapter prevent-duplicates="false" id="filesIn" directory="file:input" channel="inputFiles" /> <int:splitter input-channel="inputFiles" output-channel="udpChannel_11111" expression="T(org.apache.commons.io.FileUtils).lineIterator(payload)" /> <!-- Define UDP...

Execute database operations inside a chunck orientad step


spring-batch
I have a chunk oriented processor in the form "reader / processor / writer" called Job1. I have to execute database EJB operations after this job ends, if possible, in the same transaction. I have others jobs (implemented by Tasklets) that I could do this in a simply manner. I...

Spring Batch : custom ItemReader


java,spring,spring-batch
I have a Spring Batch project with a simple custom reader and writer. When i run the code i end up with an endeless loop printing the first item "item 1". What am i doing wrong? Here is my code: Reader.java public class Reader implements ItemReader<SimpleItem> { public SimpleItem read()...

having spring integration tcpserver to manage clients and send them messages


java,tcp,spring-integration
I have already created a simple tcp server with spring integration which keeps a connection alive and responses to each request during the connection. In that requestMethod, I'm also able to read the MessageHeder to get the connectionId. Now I want to send messages from the server to the client....

Spring GS - Creating a Batch Service missing output from db query


spring-batch
I have run the complete source for Getting Started - Creating a Batch Service Knowing that the sample uses the memory-based database provided by the @EnableBatchProcessing, is the db query result expected or it will only be available if data will be persisted permanently? After adding some debug lines, it...

Spring batch diffrence between Multithreading vs partitioning


multithreading,spring,spring-batch
I cannot understand the difference between multi-threading and partitioning in Spring batch. the implementation is of course different. In partitioning you need to prepare the partitions then process it. I want to know what is the difference and witch one is more efficient way to process when the bottle neck...

Failure to remove a file using an outbound-gateway


spring-integration,jsch
Im trying to use an sftp outbound gateway to delete a specific log file on a remote sftp. <int-sftp:outbound-gateway id="removeSourceGateway" session-factory="sftpCachingSessionFactory" reply-channel="outputChannel" request-channel="afterSuccessBackupChannel" requires-reply="false" command="ls" expression="'/export/logs/'" remote-directory="${sftp.remote.dir}"> </int-sftp:outbound-gateway> If I execute a LS or PUT command in the configuration above, then I get the expected content - see below for...

Batch job initialization fails on error while creating bean with name 'batchPropertyPostProcessor'


java,spring-batch,jsr352
I'm trying to implement sample batch application using JSR-352 API and Spring Batch 3.0.4 as implementation. Batch job execution fails during initialization phase on error while creating bean with name 'batchPropertyPostProcessor': Exception in thread "main" javax.batch.operations.JobStartException: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'batchPropertyPostProcessor': Injection of autowired dependencies failed; nested exception...

Spring Integration Java DSL - @ServiceActivator method with @Header parameter annotations


java-8,spring-integration
I have a Spring Integration 4 bean method with the following signature: @Component public class AService { @ServiceActivator public Message<?> serviceMethod( Message<?> message, @Header(ServiceHeader.A_STATE) AState state, @Header(ServiceHeader.A_ID) String id) { ... } ... } At the moment, I call this service method from within a Spring Integration Java DSL (spring-integration-java-dsl:1.0.1.RELEASE)...

More than one tasklet in a step?


spring,spring-batch,tasklet
I have a tasklet setting some information into my JobContext, and another one checking some stuff to know if I can execute the next steps in my batch or not. Both tasklets work well if I use two different steps in my job flow, but I'd like to use these...

Getting Class cast exception after upgrading Spring Integration version


java,spring,spring-integration
I have a Spring Integration Application which polls for data from AS400 DB picks it and then after some modification transmit it to MQ. It was running fine until I decided to upgrade Spring Integration version. I am getting below error after upgrading from version - spring-integration-core-2.2.3.RELEASE to spring-integration-core-3.0.1.RELEASE. Caused...

spring integration java dsl: how to expand a uri-variable in a Simp destination header


spring-integration
I use a mix of spring-mvc websocket support and spring-integration java dsl. I have an integration flow responsible to send messages to websocket-stomp subscribers using AbstractBrokerMessageHandler. The destination is /topic/notifications/{tenantId} where tenantId is a member of the payload I want to send to subscribers. In spring-integration xml dsl uri-variable subElement....

Spring Batch Add Custom Fields


spring,spring-batch,batch-processing
I've never used Spring Batch before but it seems like a viable option for what I am attempting to accomplish. I have about 15 CSV files for 10 institutions that I need to process nightly. I am stashing the CSV into staging tables in an Oracle database. The CSV File...

How to send basicAck to inbound adapter after publisher confirm from outbound adapter


spring-integration,spring-amqp
We have an inbound channel adapter that receives notifications of an event. The complexity of the consumer's criteria restrict our ability to use a simple routing key to distribute the messages, so the application uses a splitter to send that message to interested subscriber's queues via a direct exchange. We...

Delete File after successful persist to MongoDB in Spring Integration


spring,mongodb,spring-integration
I have a Spring Integration flow that reads a csv file from a directory, splits the lines, then processes each line and extracts 2 objects from each line. These two objects are then send to two seperate int-mongodb:outbound-channel-adapter. I want to delete the incoming file after all of the lines...

spring-integration-dsl-groovy-http return null when i use httpGet method


spring,groovy,spring-integration
I use spring integration 4.1.4 and spring integration dsl groovy 1.1.0 I included spring integration core, http in dependency. When i am executing spring integration dsl groovy http sample, it throwing null value in console. I am not sure what i missed. Here is my code looks like IntegrationBuilder builder...

Send Mail using Spring MVC


spring,spring-mvc,spring-integration
I have a custom MailDto object in which I set the to,cc, bCc field and send it using RestTemplate to Spring MVC controller as below @RequestMapping(value = "/SendMail" ) public ResponseEntity<String> SendMail( @RequestBody MailMessageDto mailDto) throws NoSuchMethodException, IllegalAccessException, IllegalArgumentException, InvocationTargetException From the controller I invoke the Gateway (Spring integration). The...

Spring-integration scripting with Python


python,spring-integration,jython
I'm trying to use Python with spring-integration and jython-standalone-2.7.0: Here is my application context: <int:inbound-channel-adapter id="in" channel="exampleChannel" > <int:poller fixed-rate="1000" /> <int-script:script lang="python" location="script/message.py" /> </int:inbound-channel-adapter> <int:channel id="exampleChannel" /> <int-ip:udp-outbound-channel-adapter id="udpOut" channel="exampleChannel" host="192.168.0.1" port="11111" /> Here is my script in Python: print "Python"...

How to call a specific method of a tasklet


spring-batch
In a job context there are an 'method' parameter so a could call directly an tasklet method as said in the documentation "If the tasklet is specified as a bean definition, then a method can be specified and a POJO will be adapted to the Tasklet interface. The method suggested...

Spring integration FileTailingMessageProducer: Remember current line when restarting


java,spring,spring-integration
We are using the Spring integration FileTailingMessageProducer (Apache Commons) for remotely tailing files and sending messages to rabbitmq. Obviously when the java process that contains the file tailer is restarted, the information which lines have already been processed is lost. We would like to be able to restart the process...

can a jms inbound gateway reply include all messages of a message sequence


spring-integration
I have a jms-inbound-gateway that reads requests from a WebsphereMQ broker and passes them though my integration system which returns a message sequence as reply. The message sequence is obtained by applying the splitter on a collection. It generates 63 messages as shown in the log. The problem is that...

how should a message look like to be a response


java,tcp,spring-integration
I have read and followed the example in http://docs.spring.io/spring-integration/reference/html/ip.html#ip-correlation I have a spring-integration server <int-ip:tcp-connection-factory id="socketserver" type="server" port="30124" using-nio="true" mapper="mapper" deserializer="jsonSerializer" serializer="jsonSerializer" single-use="false"/> with this mapper from above link: <bean id="mapper" class="org.springframework.integration.ip.tcp.connection.MessageConvertingTcpMessageMapper"> <constructor-arg...

Overwriting spring-boot autoconfiguration


java,spring,spring-boot,spring-batch
I'm a little bit confused with the behaviour of spring-boot when overwriting specific autoconfigurations. I like to partly overwrite the BatchAutoConfiguration, but I guess, my question is not specific to BatchAutoConfiguration. Actually, I just want to "overwrite" two methods of this class: public BatchDatabaseInitializer batchDatabaseInitializer() and public ExitCodeGenerator jobExecutionExitCodeGenerator(). Therefore,...

Which Spring Integration Channel should be used for MQTT


spring-integration,mqtt
I am trying a demo app where we send commands over a UI which then goes through spring integration to deliver a command to a device and gives a command status back to UI, this is my first time using Spring Integration and am a bit unsure of the various...

Publisher should wait till broker is available


spring,jms,spring-integration,spring-jms
I have a simple publisher, which sends messages to a queue. <int:channel id="publishChannel"/> <int-jms:outbound-channel-adapter channel="publishChannel" destination="testQueue" session-transacted="true"/> @Publisher(channel = "publishChannel") public String sendMessage (String text) { return text; } If the broker crashes, the publisher throws an MessageHandlingException. Is it possible to block the publisher, till the broker is available...

Spring Integration - @Filter discardChannel and/or throwExceptionOnRejection being ignored?


java-8,spring-integration
I have a java DSL based spring integration (spring-integration-java-dsl:1.0.1.RELEASE) flow which puts messages through a Filter to filter out certain messages. The Filter component works okay in terms of filtering out unwanted messages. Now, I would like to set either a discardChannel="discard.ch" but, when I set the discard channel, the...

Spring batch generating reports


spring-batch
I would like to generate a summary report at the end of my batch execution. For ex: I have an ItemProcessor which receives accountId. for every accountId: get MarketplaceId's for every marketplaceId: call real time availability At the end of batch execution I need to provide a nice summary in...

To read Excel can we use Spring batch?


database,excel,spring-batch
I want to know if it is possible to use Spring Batch, in order to read from an file Excel and save it in Database. remark : the content of file Excel chang every 2 hours. And if it is not possible with Spring Batch, what other solution can i...

Spring Integration and returning schema validation errors


spring,spring-integration,json-schema-validator
We are using Spring Integration to process a JSON payload passed into a RESTful endpoint. As part of this flow we are using a filter to validate the JSON: .filter(schemaValidationFilter, s -> s .discardFlow(f -> f .handle(message -> { throw new SchemaValidationException(message); })) ) This works great. However, if the...

Continuing with Spring integration flow after SoapFault


web-services,spring-integration,soapfault
Our application integration flow is defined as splitter -> ws gateway -> aggregator The splitter splits request into a list of account numbers; so that for each account number a web service call is initiated and the responses from multiple web service calls are aggregated in the aggregator.The channel between...

How does Spring Batch manage transactions (with possibly multiple datasources)?


java,transactions,spring-batch
I would like some information about the data flow in a Spring Batch processing but fail to find what I am looking for on the Internet (despite some useful questions on this site). I am trying to establish standards to use Spring Batch in our company and we are wondering...

Return last value query in MongoDB in Spring XD Source module


mongodb,spring-integration,spring-xd
I would like to be able to create a query/query-expression that I can use in SprinXD MongoDB Source module to return only the last document that has been inserted. It seems that max-messages-per-poll doesn't do the trick (paginated data with the help of mongo inbound adapter in spring integration) I...

How to convert Spring Integration XML to Java DSL for errorChannel


spring-integration
I have the below xml configuration in my application and I would like to convert it to the Java DSL. So in this reference I'm explicitly defining the name for the error channel. Mostly for example reason. With this reference what I'm expecting to happen is when a downstream process...

Removing duplicate code from Spring job configuration


java,spring,spring-batch
Below is my step configuration - <beans:bean id="myInputFileReader" class="com.rbos.fm.risk.batch.spring.reader.InputFileReader" scope="step"> <beans:property name="delegate"> <beans:bean class="org.springframework.batch.item.file.FlatFileItemReader" scope="step"> <beans:property name="resource" ref="inputFileSystemResource" /> <beans:property name="linesToSkip" value="1" /> <beans:property name="lineMapper"> <beans:bean...

Spring Integration, change udp-outbound-channel port


java,spring,spring-integration
I'm using spring integration to send line of files in UDP. Here is what I'm doing: <int-file:inbound-channel-adapter prevent-duplicates="false" id="filesIn" directory="file:input" channel="inputFiles"> <int:poller default="true" fixed-rate="1000" /> </int-file:inbound-channel-adapter> <int:splitter input-channel="inputFiles" output-channel="output"> <bean class="fr.spring.demo.FileSplitter"> <property name="commentPrefix" value="#" /> </bean> </int:splitter> <int:transformer...

Spring batch FileItemWriter not creating file at correct path


java,linux,spring,websphere,spring-batch
I have a spring batch service containing a FileItemReader,FileItemProcessor and FileItemWriter.When creating the FileItemWriter I have to set the Resource that will be my output file. I am running the batch service on websphere on a Linux machine.The problem is if I set the resource as new FileSystemResource(new File("opt\temp1\myFile.txt")), the...

Is there any workaround for using channel name with dash character using spring integration 4 without xml?


spring-integration
I upgraded my spring integration 3.x to 4.x. And I removed all xml configuration and replace to java annotation in pojo files. But the problem is.. I want to use the channel name with '-' character. It is very good to recognize and distinguish each words in name. Besides, if...

spring integration vs spring batch [on hold]


spring-batch,spring-integration
We have application where we are receiving file every day and it needs to be parsed and persist in db. File has 5000 record. Should we use spring batch or spring integration? And why? we do need to skip bad record and audit them....

How to catch socket exceptions on TcpInboundGateway?


java,spring,sockets,spring-integration
I'm using TcpInboundGateway from spring-integration for a socket client. In TcpInboundGateway.doOnMessage() the stream is send via the socket connection. If an exception occurs eg because the socket is closed, there is just a spring error log "Failed to send reply". BUT I'd like to handle those socket errors explicit. How...