FAQ Database Discussion Community

Fail to export from voltdb to kafka

VoltDB is enterprise 5.1.2 Kafka is 2.9.2-, also tried 2.10- VoltDB is at Kafka is at Here is my deployment configuration for VoltDB: <deployment> <cluster hostcount="1" sitesperhost="4" kfactor="0" /> <commandlog enabled="true"> <frequency time="400" transactions="1000" /> </commandlog> <export> <configuration enabled="true" type="kafka" stream="archive"> <property name="metadata.broker.list"></property> <property...

Kafka QuickStart, advertised.host.name gives kafka.common.LeaderNotAvailableException

I am able to get a simple one-node Kafka (kafka_2.11- working locally on one linux machine, but when I try to run a producer remotely I'm getting some confusing errors. I'm following the quickstart guide at http://kafka.apache.org/documentation.html#quickstart. I stopped the kafka processes and deleted all the zookeeper & karma files...

how to get kafka to consume http streaming data on spring xd runtime?

Is it possible to have kafka source module work as a processor module in spring xd runtime? Any code samples? I am trying to achieve something like this: http (xd source) | kafka source (xd processor)| kafka consumer (xd sink) I am trying to do this because I have streaming...

Apache Kafka is JMS api?

Apache Kafka is not a JMS implementation? I using JMS+AMQ in my application and emigration to Apache Kafka, i have to change all JMS codes?...

What exactly IS Kafka Rebalancing?

I am a new user to Kafka and have been playing around with it for about 2-3 weeks now. I believe at the moment I have a good understand of how Kafka works for the most part, but after attempting to play around with and design my own Kafka consumer...

Wildfly Kafka clients module javax/management/JMException not found

I'm try to create Kafka clients module for Wildfly. Here is module.xml: <?xml version="1.0" encoding="UTF-8"?> <module xmlns="urn:jboss:module:1.0" name="org.apache.kafka.clients"> <resources> <resource-root path="kafka-clients-"/> <resource-root path="lz4-1.2.0.jar"/> <resource-root path="snappy-java-"/> </resources> <dependencies> <module name="org.slf4j"/> </dependencies> </module> Wildfly (8.2.0-Final) gives this error on boot:...

Kafka partitions meaning

When we decide about partitions, should we do that on per-topic base, or it is topic-wide decision? If T1 partitioned on 3 partitions, and T2 partitioned on 2 partitions, can they both be consumed by 1 consumer? Or it is better make equal number of partitions if topics must be...

Error while deploying topology on storm cluster

I am trying to deploy a simple word count topology on storm clustre.I am using kafka as the input(kafka Spout).This is the error i am getting.I am very new to storm.Please suggest the changes.Thanks in Advance !! java.lang.NoClassDefFoundError: Could not initialize class org.apache.log4j.Log4jLoggerFactory at org.apache.log4j.Logger.getLogger(Logger.java:39) at kafka.utils.Logging$class.logger(Logging.scala:24) at kafka.consumer.SimpleConsumer.logger$lzycompute(SimpleConsumer.scala:30) at...

Should Apache Kafka and Hadoop be installed seperatedly (on a diffrent cluster)?

Should Apache Kafka and Hadoop be installed seperatedly (on a diffrent cluster) ?

scala + kafka does not send message with id

I'm trying to send a message to Kafka through my actor and it does not work. The following code works new KeyedMessage[String, Array[Byte]]("my-topic", msg.message) This one doesn't... why? new KeyedMessage[String, Array[Byte]]("my-topic", msg.id, msg.message) or even new KeyedMessage[String, Array[Byte]]("my-topic", msg.id, null, msg.message) setting partition to null, forcing it to fill only...

kafka log.retention.hours inconsistency in multiple brokers

I am trying to run multiple kafka brokers. There is a file named server.properties and there is a field "log.retention.hours" which is set to be 168 in my server.properties file. Now if i change the kafka brokers properties file and set different values of "log.retention.hours" in each properties file then...

Can I achieve ordered processing with multiple consumers in Kafka?

In Kafka, I have a producer queuing up work of clients. Each piece of work has a client ID on it. Work of different clients can be processed out of order, but work of one client must be processed in order. To do this, I intend to have (for example)...