FAQ Database Discussion Community


Spring XD: active spring profiles for a module only

spring,spring-xd
If I am using Spring profiles in my custom Spring XD module, in order to active the profile I have to start the Spring XD container with SPRING_PROFILES_ACTIVE environment property set to the list of profiles I'd like to activate. (Is there a way to use -Dspring.profiles.active instead? don't find...

Spring-xd tap with filter?

spring-xd
I have a stream "myStream" of text records of different types, say "A", "B", "C", and so on...Each type of records needs to be processed differently. I want to create multiple taps on "myStream" for each type: tapA, tapB, tapC, etc. But it seems tap doesn't have any filtering capability....

Spring XD http-client processor sends text/plain instead of application/json

json,spring-xd
I am trying to create a stream that sends a POST request as part of a stream. stream create form-time --definition "<working source> | generate-form-transformer: transform --script=form-generation.groovy --outputType=application/json | http-client --inputType=application/json ..." --deploy It works, but it sets contentType to text/plain instead of application/json, which provokes a 400 since my...

Spring XD Rest api job launch with jobParameters responding with 'jobParameters' is not recognized as an internal or external command

spring-xd
I have Spring XD job already deployed which expects 2 jobParameters (absoluteFilePath and fileName). Actually this job is triggered by JMS stream whose output provides those 2 jobParameters in JSON format and that works fine. I want to launch the job with REST API like: curl -X POST http://localhost:9393/jobs/executions?jobname=loadData&jobParameters=%7B%22absoluteFilePath%22%3A%22C%3A%2FUB%2Fdev%2FBM.txt%22%2C%22fileName%22%3A%22BM.txt%22%7D Error...

Is Zookeeper Essential For XD Single Node?

zookeeper,spring-xd
This question relates to the XD 1.1.2.RELEASE. I'm brand new to XD, so please forgive if this is a dumb question. The documentation says only that XD does not ship with Zookeeper, and I took this to mean that it would only be needed once I'm ready to go multi-node....

How to call a rest api from spring-xd

spring-xd
I am new to spring-xd and would like to understand is it possible to invoke an external rest api from springxd. My rest api expects a request header for GET requests and request header+message body for POST requests. How do I invoke the URL ? In my understanding http-client is...

POST to /jobs/executions doesn't respond with an execution Id?

spring-xd
I've integrated my application with spring XD through the REST API. I have a job def set up and deployed for each datasource (my module is jdbc related) and then launch specific jobs using the appropriate job def as a template with jobParameters provided in the POST request, containing criteria...

Spring XD - instanceof in SpEL expression in filter

spring-xd
I'm trying to use the instanceof operator in a SpEL expression in a filter for a stream. I'm trying the following: stream create myStream --definition "tap:job:jobName > filter --expression='payload instanceof T(com.package.name.event.SomeEvent)' | log --deploy I am publishing my own event to the xd.job.aggregatedEvents channel. My intention is to log only...

Is there an API in spring-xd to write to a message bus?

spring-integration,spring-xd
Considering any message bus can be deployed with spring-xd, is there any API provided by spring-xd that can write to the message bus deployed, say, Kafka/rabbitmq based on the configurations in xd/config/servers.yml or any other such place. I am writing a processor (itemprocessor), which does some processing of the data...

Does HDFS sink support writing to sequence file in Spring XD?

spring-xd
I would like to use HDFS sink in stream processing for writing data of binary key-value pair to HDFS. As a result, a thin wrapper format like sequence file will suffice. Does the HDFS sink module support writing key-value to HDFS sequence file at all? Thanks in advance!...

Configure Oracle for Spring-xd

oracle,spring-xd
I am trying to run spring-xd in distributed mode and have to use Oracle DB. When I started xd-admin, it failed with the following error: Caused by: java.io.FileNotFoundException: class path resource [org/springframework/xd/batch/schema/registry-schema-oracle.sql] cannot be opened because it does not exist Spring XD guide says Until full schema support is added...

Spring XD project site documentation corrections

spring-xd
How can I submit documentation corrections to the Spring XD project site?Is there a GitHub repository accepting pull-requests for these corrections?

Spring XD (Spring Integration DSL based) AWS S3 sink module error

amazon-web-services,spring-integration,spring-xd
I'm trying to build a Spring XD (Spring Integration DSL based) sink module using the spring-integration-aws extension. My module looks like so: @Configuration @EnableIntegration public class S3Module { @Value("${accessKey:myAccessKey}") private String accessKey; @Value("${secretKey:mySecretKey}") private String secretKey; @Value("${bucket:myBucket}") private String bucket; @Value("${remoteDirectoryExpression:dir}") private String remoteDirectoryExpression; @Bean public AmazonS3MessageHandler handle() { AWSCredentials...

how to get kafka to consume http streaming data on spring xd runtime?

spring,spring-xd,http-streaming,kafka
Is it possible to have kafka source module work as a processor module in spring xd runtime? Any code samples? I am trying to achieve something like this: http (xd source) | kafka source (xd processor)| kafka consumer (xd sink) I am trying to do this because I have streaming...

Spring XD: separate configuration for multiple module instances

spring,spring-xd
My Spring XD stream definitions contain references to the same custom module but with different configuration parameters, e.g. stream create --name mystream1 --definition "tcp --port=4444 | mycustommod --myprop='very long property value 1'" | log" stream create --name mystream2 --definition "tcp --port=5555 | mycustommod --myprop='very long property value 2'" | log"...

Spring-XD: Kafka Connection refused

zookeeper,apache-kafka,spring-xd
I'm trying to use spring-xd to build a kafka consumer, but no matter what I provide for zkconnect, it tells me connection is confused stream create kafka-source-test --definition "kafka --groupId=xd-test --zkconnect=52.5.7.194:2181 --topic=vehicle-data | log" --deploy this yields: 2015-05-22 17:49:46,973 1.1.2.RELEASE WARN DeploymentsPathChildrenCache-0-SendThread(52.5.7.194:2181) zookeeper.ClientCnxn - Session 0x0 for server null, unexpected...

count(*) in mysql innodb table leading to inconsistent results while data is pumped via spring xd

mysql,spring,unix,spring-xd
echo -e "job create --name $completeJobName --definition \"filejdbc --resources=file:/var/lib/wwhs/eligibility/processing/complete/* --names=file_id,client_id,member_record_number,membe r_first_name,member_middle_initial,member_last_name ,group_id,member_id,date_of_birth,created_by,created_date,modified_by ,modified_date --tableName=eligibility.eligibility_file_staging_complete\" --deploy" > $completeJobCmdFile /var/lib/spring_xd/shell/bin/xd-shell script $completeJobCmdFile # Call XD to create the complete job...

Spring XD: pipe (>) from file source to batch job fails (IllegalArgumentException: Unable to convert provided JSON to Map)

spring-batch,spring-xd
I am trying to pipe the output (file contents) of a Spring XD file source to a simple batch job. This fails with the following exception. It appears that XD is trying to incorrectly use the file contents as JSON job parameters when launching my job. As you may notice,...

Spring XD source failure handling

spring,spring-integration,spring-xd
I have a stream that includes a module that behaves similarly to the source:file module, with the exception that files will only be detected once using a redisMetaDataStore and Spring Integration's FileSystemPersistentAcceptOnceFileListFilter so that the module remembers what files were already read between container restarts. file-persisting.xml <beans:bean id="redisMetadataStore" class="org.springframework.integration.redis.metadata.RedisMetadataStore" >...

How to concatenate a string with SpEL in SpringXD Stream Definition

spring-integration,spring-el,spring-xd
I'm trying to create a stream definition that uses two payload fields joined together to create a concatenated string. stream create --name blah --definition "http | gemfire-json-server --keyExpression=payload.getField('deviceId') + payload.getField('timestamp')" --deploy` The concatenation piece in the keyExpression is incorrect, what should it be to get it to work inline?...

Parallel job execution by setting job module count in Spring XD?

spring-batch,spring-xd
Since besides stream module, Spring-XD slap support to set module count number for batch jobs. For example, module.myjob.count=3. What will happen if the job is deployed in this way? Will the job be deployed on 3 different containers? Will the 3 copies of deployment execute in parallel? Does each copy...

jms-message-bus sample not working

spring-xd
I have tried to follow the steps in https://github.com/SpringOne2GX-2014/Spring-XD-Internals/tree/master/jms-message-bus to point to an external Active MQ. But when i start the XD with ./xd-singlenode --transport jms I'm getting an error, i think JMS is not getting recognized provided in the Wiki, what am i missing? Below is the error `2015-02-11...

Does Spring XD re-process the same message when one of it's container goes down while processing the message?

spring-xd
Application Data Flow: JSon Messages--> Active MQ --> Spring XD-- Business Login(Transform JSon to Java Object)--> Save Data to Target DB--> DB. Question: Sprin-Xd is running in cluster mode, configured with Radis. Spring XD picks up the message from the Active message queue(AMQ). So message is no longer in AMQ....

Return last value query in MongoDB in Spring XD Source module

mongodb,spring-integration,spring-xd
I would like to be able to create a query/query-expression that I can use in SprinXD MongoDB Source module to return only the last document that has been inserted. It seems that max-messages-per-poll doesn't do the trick (paginated data with the help of mongo inbound adapter in spring integration) I...

Filter data inside custom Processor module?

spring-xd
I am writing a custom Processor module using a transformer class, and for some input, I do not want the transformer to generate any output, how can I do that? I tried to return null in the transform() method, but that make Spring-xd runtime generate exceptions. And I do not...

Optimize gradle for spring-xd project with transient scala dependency

scala,gradle,spring-xd
I am working on a project that contains a series of module subprojects for spring XD which happen to have a transient dependency on a non-module subproject that happens to use Scala: ext { springXdVersion = '1.1.0.RELEASE' moduleProjects = subprojects.findAll { project -> project.path.startsWith(':modules.')} javaProjects = subprojects - (moduleProjects +...

Spring XD custom job execution event

spring-xd
I can receive pre-defined job Execution events for my spring xd job as follows: xd>job create --name myHttpJob --definition "httpJob" --deploy xd>stream create --name jobExecutionEvents --definition "tap:job:myHttpJob.job >log" --deploy Reference: http://docs.spring.io/spring-xd/docs/1.0.3.RELEASE/reference/html/#_retrieve_job_notifications However, I would like to fire my own custom event and be able to do something with it. So...

Payload Filters/Transforms & Rich Gauges

spring-xd
I have a JMS feed that provides messages with the following JSON structure: { String:currentSessions, Long:duration, Long:rows, Long:fill, Long:execute, String:source, String:category, String:url, String:hostname } What I would like to do is a define a stream similar to the following that would provide a moving average by hostname via a rich-gauge:...

Issue with file sink and filename expression

spring-xd
Trying to set up simple file copy processin spring-xd: stream create --name mystrea --definition "file --dir=/path/source --fixedDelay=5 | sink:file --dir=/path/dest --binary=true --name=headers['file_name']" This seems to create and append fils to the file header['file_name'].out in the dest folder Looking at sink:file definition <file:outbound-channel-adapter id="files" mode="${mode}" charset="${charset}" directory="${dir}" filename-generator-expression="'${name}' + '${extensionWithDot}'"/> I...

Spring XD container groups

containers,spring-xd
I have put the containers into three groups (group1,group2,group3) is it possible to deploy a stream to group1 as a priority and then, if all containers in group1 are down, to group2 as second perference and avoid going to group3. e.g. module.*.criteria=groups.contains('group1','group2')" this is for springXD 1.0.1...