FAQ Database Discussion Community


Serializing mutable state and sending it asynchronously over the network with nearly-zero-copy (Cap'n Proto + ZeroMQ)

c++,serialization,boost,boost-asio,zeromq
I have an application in which I'd like to send part of its mutable state over the network to another machine (there will be a cluster of those machines) to do some CPU-intensive computations on it and get back the results. Like asynchronous RPC. Such calls will happen many times...

Listening for connect/connection events within ZMQ request-reply pattern

node.js,zeromq
In the replier or 'server' of the ZMQ request-reply pattern, I would like to listen to requestors connecting to my replier/server. I have this code: var zmqConfig = {...}; var replier = zmq.socket('rep'); var address = 'tcp://'.concat(zmqConfig.host).concat(':').concat(zmqConfig.port); replier.bind(address, function (err) { if (err) { } }); replier.on('message', function () {...

How to choose zmq maximum buffer size properly?

c++,c,sockets,zeromq
According to ZMQ-manual The ZMQ_SNDBUF option shall retrieve the underlying kernel transmit buffer size for the specified socket. A value of zero means that the OS default is in effect. For details refer to your operating system documentation for the SO_SNDBUF socket option. How can I get it in a...

dockerizing an application on Mac OS X

osx,docker,zeromq,boot2docker
I installed boot2docker as explained on the docker website. Here are some command runs to show that I have things installed correctly: $$:~ kv$ boot2docker start Waiting for VM and Docker daemon to start... ...................ooo Started. Writing /Users/kvantum/.boot2docker/certs/boot2docker-vm/ca.pem Writing /Users/kvantum/.boot2docker/certs/boot2docker-vm/cert.pem Writing /Users/kvantum/.boot2docker/certs/boot2docker-vm/key.pem Your environment variables are already set correctly. $$:~...

How to setup a ZMQ PUB/SUB pattern to serve only for pre-authorized subscriber(s)

c++,c,c++11,zeromq,pyzmq
How can I implement or do kind of "hack" in PUB-SUB pattern to get an ability to publish only to authorized subscribers, disconnect unauthorized subscribers etc? I googled for this problem, but all the answers very similar to set subscribe filter in subscriber side. But I want, as I said,...

Gathering distributed data into central database

wcf,rabbitmq,msmq,zeromq,dds
I was assigned to update existing system of gathering data coming from points of sale and inserting it into central database. The one that is working now is based on FTP/SFTP transmission, where the information is sent once a day, usually at night. Unfortunately, because of unstable connection links (low...

How do I add a pipeline to a REQ-REP in ZeroMQ?

zeromq,zero
I am experimenting with ZeroMQ where I want to create a server that does : REQ-PIPELINE-REPLY I want to sequentially receives data query requests, push it through a inproc pipeline to parallelise the data query and the sink merges the data back. After the sink merges the data together, the...

How ZMQ works between 2 different machines

node.js,sockets,zeromq
I am not sure I understand fundamentally how ZMQ (or any message queue) knows how to communicate between two servers that otherwise don't know anything about each other. For example, using the request/reply pattern: the requester will bind to a host and port like so: var requester = zmq.socket("req"); requester.bind('tcp://*:5555'),...

Asynchronous Client/Server pattern in Python ZeroMQ

python,zeromq
I have 3 programs written in Python, which need to be connected. 2 programs X and Y gather some information, which are sent by them to program Z. Program Z analyzes the data and send to program X and Y some decisions. Number of programs similar to X and Y...

ZMQ: No subscription message on XPUB socket for multiple subscribers (Last Value Caching pattern)

python,sockets,proxy,zeromq,publish-subscribe
I implemented the Last Value Caching (LVC) example of ZMQ (http://zguide.zeromq.org/php:chapter5#Last-Value-Caching), but can't get a 2nd subscriber to register at the backend. The first time a subscriber comes on board, the event[0] == b'\x01' condition is met and the cached value is sent, but the second subscriber (same topic) doesn't...

ZMQ pair (for signaling) is blocking because of bad connection

python,zeromq,blocking,pyzmq
I have two threads. One is a Worker Thread, the other a Communication Thread. The Worker Thread is reading data off a serial port, doing some processing, and then enqueueing the results to be sent to a server. The Communication Tthread is reading the results off the queue, and sending...

ZMQ Pattern Dealer/Router HeartBeating

sockets,client-server,zeromq,router,heartbeat
I have a Dealer socket in client side, who is connected to Router socket in server side. I often see Heartbeating mechanism : the server regularly send message to the client in order that client knows if he is correctly connect to the server, so the client can reconnect if...

zeromq.node setsockopt throws invalid argument error

node.js,zeromq
This code... var zmq = require('zmq'); var req = zmq.socket('req'); req.setsockopt('hwm', 10); // or req.setsockopt(zmq.ZMQ_HWM, 10); ...throws this error: /Users/.../node_modules/zmq/lib/index.js:246 this._zmq.setsockopt(opts[opt] || opt, val); ^ Error: Invalid argument at Socket.setsockopt (/Users/.../node_modules/zmq/lib/index.js:246:13) ps. bonus point, is there a way to do something like this zmq.socket('req', {'hwm': 10}) ? zeromq.node 2.11.1 &...

Do you have to re-register a ZMQ socket with the poller?

sockets,zeromq
I'm opening a ZMQ socket and registering it with a poller as follows: poll = zmq.Poller() socket = context.socket(zmq.DEALER) socket.setsockopt(zmq.IDENTITY, "identity") socket.connect(url) poll.register(socket, zmq.POLLIN) Now I'd like to close and re-connect the socket to the same url. What is the best way to do this given that it is registered...

What does X mean when used with ØMQ socket types?

zeromq
ZeroMQ defines a series of socket types, commonly referred to as SUB, PUB, XSUB, XPUB, DEALER... Looking through some API code, there are methods such as XSend, XHasIn, XHiccupped. These X characters seem to be used as semantic modifiers. Is there any pattern or significance to their usage?...

What open ports are required on firewall to allow for salt-stack remote execution?

zeromq,firewall,network-protocols,salt-stack
The documentation on saltstack appears to be unclear regarding what ports are required from the salt-master -> salt-minion (apparently none are required). It suggests that ports only need to be opened from the salt-minion -> salt-master. (See: http://docs.saltstack.com/en/latest/topics/tutorials/firewall.html) If however commands are executed remotely on the salt-master targeted to a...

IObservable with NetMQ receive

c#,async-await,system.reactive,zeromq,netmq
I'm trying to write a typical stock trading program, which receives stock tickers/orders/trades from netmq, turn the streams into IObservable, and show them on a WPF frontend. I try to use async/await with NetMQ blocking ReceiveString (suppose I am expecting some string input) so that the ReceiveString loop wouldn't block...

Node.js socket.send( ) functions failing to complete before exit

javascript,node.js,zeromq
In some Node.js scripts that I have written, I notice that even if the last line is a synchronous call, sometimes it doesn't complete before Node.js exits. I have never seen a console.log statement fail to run/complete before exiting, but I have seen some other statements fail to complete before...

ZMQ wait for a message, have client wait for reply

c++,networking,zeromq
I'm trying to synchronise 4 clients to one server. I want to send a message to the server when the client is ready to move on, then the server counts how many requests it gets and sends a message back to the clients to say it's ready. What I've done...

TCP tunneling through Linux tap device

ruby,tcp,zeromq
I've created a tap0 device (IP 10.0.0.101), and am using zeromq's pgm pub/sub (e.g. pgm://192.168.100.2;234.5.6.7:5555) to transport Ethernet frames from the tap to the zmq subscribers and vice versa. The idea is to create a virtual network using pgm. I have 2 tap hosts on the network: 10.0.0.101, 10.0.0.11. They...

Akka clustering conflicts

akka,zeromq,distributed-computing
The Akka doc talks about a variety of seemingly inter-related Akka technologies without distinguishing much between them: Akka Networking Akka Remoting Akka Clustering The Akka ZeroMQ module My understanding is that "Akka Networking" is simply a module/lib that gives Akka the ability to speak to remote actor systems over TCP....

ZeroMQ choose recipient

zeromq,netmq
I'm new to ZeroMQ (and to networking in general), and have a question about using ZeroMQ in a setup where multiple clients connect to a single server. My situation is as follows: --1 server --multiple clients --Clients send messages to server: I've already figured out how to do this part....

Request Aggregator / Middle-tier design pattern for costly requests

java,c#,multithreading,design-patterns,zeromq
I'm working on a program that will have multiple threads requiring information from a web-service that can handle requests such as: "Give me [Var1, Var2, Var3] for [Object1, Object2, ... Object20]" and the resulting reply will give me a, in this case, 20-node XML (one for each object), each node...

Can you create a centralized topic in ZeroMQ?

c#,php,.net,zeromq
I would really like to try ZeroMQ and I am wondering whether my problem can be solved using it. THE PROBLEM: I have multiple subscribers and multiple publishers. In a centralized broker architecture the publishers would publish a message to a topic (kind of like multicast address) and the subscribers...

ZeroMQ word count app gives error when you compile in spark 1.2.1

scala,streaming,sbt,apache-spark,zeromq
I'm trying to setup zeromq data stream to spark. Basically I took the ZeroMQWordCount.scala app an tried to recompile it and run it. I have zeromq 2.1 installed, and spark 1.2.1 here is my scala code: package org.apache.spark.examples.streaming import akka.actor.ActorSystem import akka.actor.actorRef2Scala import akka.zeromq._ import akka.zeromq.Subscribe import akka.util.ByteString import org.apache.spark.streaming.{Seconds,...

Python GUI not capturing zeromq message

python,user-interface,zeromq,pyqt5
I have a small program where I am not receiving a response from the server. This is using python 3.4 and latest zeromq and pyqt5 on Ubuntu 14.04. The client GUI sends a message to the server which gets it and responds but the client does not see the...

Using process instead of thread with zeromq

python,multiprocessing,zeromq
I'm reading this code http://zguide.zeromq.org/py:mtserver But when I've tried to replace threading.Thread by multiprocessing.Process I got the error Assertion failed: ok (mailbox.cpp:84) Code is import time import threading import zmq def worker_routine(worker_url, context=None): """Worker routine""" context = context or zmq.Context.instance() # Socket to talk to dispatcher socket = context.socket(zmq.REP) socket.connect(worker_url)...

Serializing Protobuf Object and Sending with ØMQ/ZMQ

c#,c++,zeromq,protobuf-net
I have a protobuf object that I am sending from a C# application (using clrZmq) to a C++ service (using the zmq C++ bindings) on a local machine (for testing). I attempt to send my object from C# using the following Taurus.Odds odds = Util.GetFakeOdds(); using (var context = ZmqContext.Create())...

ZeroMQ pattern for load balancing work across workers based on idleness

zeromq,nonblocking,asyncsocket
I have a single producer and n workers that I only want to give work to when they're not already processing a unit of work and I'm struggling to find a good zeroMQ pattern. 1) REQ/REP The producer is the requestor and creates a connection to each worker. It tracks...

ZMQ pub/sub subscribe

node.js,sockets,tcp,zeromq
I am having trouble figuring out how to subscribe to a particularly "channel" with ZMQ with regard to its pub/sub functionality. Here is the publisher: var zmq = require('zmq'); var pub = zmq.socket('pub'); pub.bindSync('tcp://127.0.0.1:5555'); setInterval(function(){ pub.send('pub msg'); },500); here is the subscriber: var sub = zmq.socket('sub'); sub.connect('tcp://127.0.0.1:5555'); sub.subscribe(''); //herein lies...

Reason for zmq error code 156384763

c++,sockets,zeromq
I am using zmq req/rep pattern communication. The implementation is pretty simple, the req sends some data and waits on recv. The rep receives the data, process and reply back. //REQ zmq_connect zmq_send zmq_recv //blocking zmq_close //REP zmq_bind while(true) { while(data_received) { //miscellaneous process zmq_recv //non-blocking Print zmq_error_no if zmq_recv...

Using ZeroMQ or other MQ to send objects [closed]

json,node.js,rabbitmq,zeromq,mqtt
I am wondering how I can send objects via a message queue. My question is two-fold: (1) are there any message queues like ZeroMQ that support Java objects and JSON out of the box? (2) are there any message queues that don't require you to serialize/deserialize objects on both ends?...

How to use ProtoBuf extensions to get polymorphism in C++

c++,protocol-buffers,zeromq
I am trying to define a common basic message which defines the type of the message (for easier parsing) and is then extended with the actual message. The messages will be used in an RPC way. My .proto file syntax = "proto2"; package userapi; // wrapper for commands message Command...

0MQ req wait for rep in Node.js

javascript,node.js,zeromq
I'm new to javascript, Node.js, and 0MQ, so n00b * 3 here. I want to set up a simple request and reply, but I want the client to wait for a response before sending out the next request. The zguide goes over this, but the Node.js version does not behave...

PyZMQ and Django: connection from different view

python,django,zeromq,pyzmq
I am writing a Django web application where the user can do different operations from every view and when the user submit the form a JSON is pushed to ZeroMQ and waits for a response (REQ-REP). My problem is that the operations are too slow. I don't know if the...

Multi core ZeroMQ?

python,multithreading,multiprocessing,zeromq
ZeroMQ is used for receiving input parameters.. def server(): rep = context.socket(zmq.REP) rep.bind('tcp://*:{}'.format(PORT)) while True: data = rep.recv_json() result = calculate(data) rep.send_json(result) The calculation method is called calculate, after finished, result would be sent to client through ZMQ. Base on my test, it currently uses only 1 core of the...

VS2013 Error: LNK2019 When trying to build ZeroMQ server

c++,visual-studio-2013,zeromq
I'm trying to build this simple ZeroMQ server in C++ on Visual Studio 2013. #include "stdafx.h" #include "zmq.hpp" #include <string> #include <iostream> #include <windows.h> using namespace std; int _tmain(int argc, _TCHAR* argv[]) { // Prepare context and socket zmq::context_t ctx(1); zmq::socket_t sckt(ctx, ZMQ_REP); sckt.bind("tcp://*:5555"); while (true) { zmq::message_t request; //...

Initializing ZeroMQ 2.2 message in “almost always auto” style wants to use private constructor

c++,c++11,constructor,initialization,zeromq
I have written a C++11 program which uses ZeroMQ. In one particular line I want to create a new message as a local variable and initialize it with the size of a vector called serialized, using the "almost always auto"-style syntax: auto zm = zmq::message_t {serialized.size()}; This compiles fine on...

error installing zmq (ZeroMQ)

node.js,npm,message-queue,zeromq,pyzmq
I am trying to install node module zmq I have referred to this https://github.com/JustinTulloss/zeromq.node/wiki/Installation#installation-on-windows However i get this error while installing the module 0 info it worked if it ends with ok 1 verbose cli [ 'C:\\Program Files (x86)\\nodejs\\\\node.exe', 1 verbose cli 'C:\\Program Files (x86)\\nodejs\\node_modules\\npm\\bin\\npm-cli.js', 1 verbose cli 'install', 1...

MessagePack and datetime

python,datetime,python-3.x,zeromq,msgpack
I need a fast way to send 300 short messages a second over zeromq between the python multiprocessing processes. Each message needs to contain an ID and time.time() Msgpack seems like the best way to serialize the dict before sending it via zeromq, and conveniently, msgpack has an example of...

ZeroMQ pattern for multiple asynchronous requests to single endpoint

sockets,asynchronous,zeromq
I'm using zmq to develop a distributed application having the following network topology: a client node that initiates a request and a server node that replies to requests. Since the client is a node.js application I can't block after a send call to wait the response, so the scenario is...

Communicating with a classic TCP socket

c#,sockets,tcp,zeromq,netmq
I'm writing my first application with NetMQ (ZeroMQ implementation for .NET). I also need to listen to information sent from a client using a traditional TCP socket (a.k.a a non-0MQ socket). I've seen references to the availability of this socket type in the official ZeroMQ documentation here, (look for ZMQ_STREAM),...

Using Spark to call out-of-process dll functions

c++,scala,grid,apache-spark,zeromq
We're evaluating wether to use Spark to run our grid calculations and we're having some trouble with a particular use case. Wondering if the community has any good ideas. We have an extensive library of C++ functions in a dll that we need to use all over the grid. The...

python, zeromq, how to abort context.socket.recv() the right way?

python,zeromq,abort
I have a small software where I have a separate thread which is waiting for ZeroMQ messages. I am using the PUB/SUB communication protocol of ZeroMQ. Currently I am aborting that thread by setting a variable "cont_loop" to False. But I discovered that, when no messages arrive to the ZeroMQ...

Understanding ZeroMQ

zeromq
So as I have asked in a previous post, I want to be able to make programs or functions written in different languages to communicate between them. I have come across zeromq recently and I'm trying to figure out whether or not this is something that could help me since...

How does ZeroMQ connect and bind work internally

zeromq
I am experimenting with ZeroMQ. And I found it really interesting that in ZeroMQ, it does not matter whether either connect or bind happens first. I tried looking into the source code of ZeroMQ but it was too big to find anything. The code is as follows. # client side...

ZeroMQ: Executing request in serial fashion on REQ-Router socket

multithreading,sockets,zeromq
I have built front end with Router sockets-- that forwards the request to REQ-REP Thread like in example http://zguide.zeromq.org/py:lbbroker. But I have only one worker. So in order to process I need to wait untill REQ socket get free. But I cannot figure out a way to wait untill one...