ipc,mmap,quickfix,fix,memory-mapping , memory mapping files for high frequency trading?

memory mapping files for high frequency trading?


Tag: ipc,mmap,quickfix,fix,memory-mapping

I am working in a high frequency trading client for stocks in C++. I am using the quickfix/J library to receive information from my broker. I need these two processes to communicate (one written in Java and the other in C++) and the latency has to be minimized as much as possible.

At first I thought about using FIFO named pipes, but I was reading about memory mapping and how the performance increases by mapping files into memory. My question is... Is it really that much faster? And what's the best way to accomplish this? Should I map a regular text file? Wouldn't using a regular text file be slow even with memory mapping?

Thanks in advance.


Memory mapping is really fast and it gives you an area that bot processes can access.

The problem that arises is: How do the processes know that data has changed? You could add a kind of version number that increases as data changes, but with all additional locking across processes (semaphores), I do not believe that data transfer will be that much faster.

So I think, for data transfer the overall performance compared with named pipes or queues will not be that much.


How to write/read multiple times to a pipe

I'm trying to make two child processes communicate with each other via pipe.First child have to write data 1 and data 2, and then second child writes data 3 and data 4. Then childrens read each other data and print them. This is the code that i have so far....

Using mmap to allocate double 2D array

I am attempting to use mmap to create/access twi different (N+2)*(N+2) 2D array of doubles, so that multiple threads can look at their own portion of it and change it, applying the changes so that all others can see. Here is what I have: int main(int argc, char *argv[]) {...

Is it possible to write to a mmap'd file without getting a bus error

I'm trying to monitor a file that I create for when the contents change from 0 to 1. This code create the file and maps it fine, I then spin, waiting for the contents of *map to change from '0' to '1'. However as soon as I run echo 1...

Passing int using char pointer in C

I'm trying to figure out how to pass an int using a char pointer. It fails once the int value is too large for the char. This is what I'm trying to figure out: char *args[5]; int i = 20; /*some other code/assignments*/ args[2] = (char *)&i; execv(path, args); How...

Multiple mappings for a physical page

I want to create a copy-on-write like interface for accessing a mmap()ed file in GNU C. Here is the way it should work: I will map the file to the address space using mmap(). Doing so, I will have a pointer to a contiguous region of memory which will contain...

Implementing copy-on-write buffer with mmap on Mac OS X

I've been playing around with copy-on-write buffers on Linux and the following example seems to work as intended: #include <stdio.h> #include <stdlib.h> #include <sys/mman.h> #include <sys/types.h> #include <sys/stat.h> #include <fcntl.h> #include <unistd.h> #include <string.h> #define SIZE 4096 #define SHM_NAME "foobar" int main(void) { int fd = shm_open(SHM_NAME, O_RDWR | O_CREAT,...

unable to add elements to thread safe locking queue of shared pointers

I'm trying to create an inter thread message based communications using C++11 concurrency techniques. Anthony William's book 'Concurrency in Action' describes a thread safe locking queue which this implementation is based on. The difference between the thread safe locking queue that is described in the book and the one I...

How to keep a real-time data read-in loop separate from more processing intensive operations in Python?

So I have an OpenCV webcam feed that I'd like to read frames from as quickly as possible. Because of the Python GIL, the fastest rate at which my script could read in frames seems to be the following: #Parent or maybe client(?) script #initilize the video capture object cam...

Order of opening named pipes causing possible race condition?

I am trying to create a very basic client server communication between two processes using IPC via named pipes. I have 2 pipes, namely fifo_client and fifo_server I have the following two classes fifoclient.c and fifoserver.c that has the following lines of code to open the two pipes. fifoclient.c int...

MMAP, “TypeError: 'str' does not support the buffer interface” | Python

f = open('C:\Python33\File.doc') s = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) if (s.find("blabla")) != -1: print("True") When I run the following code, I get the error "TypeError: 'str' does not support the buffer interface", why is this, and what can I do to solve this problem? I have searched online for the answer,...

mmap2 function write in asm, call in c

I have problem with writing MMAP2 in ASM AT&T and call it in C. I wrote this but didn't know how should it works. I am knowingly that code is not good but I very need help. Can you tell me how should it looks ? Thanks for help! .data...

Issues with FIFO between a C process and a JAVA application

I have 2 applications, one written in C and the other in JAVA. Lets call these applications 'C' and 'JAVA'. C Creates a named pipe via mkfifo (have also tried mknod - no difference to the outcome) C forks off with the child process invoking a system() call to fire-up...

Does a python fifo have to be read with os.open?

I am experimenting with using fifos for IPC routines in Python, and have the following code to create the fifo, then start two threads to write to it at varying times and read continually from it to print out whatever is put on the fifo. It works when the fifo...

Bidrectional node/python communication

I'm trying to implement simple bidirectional communication between node and a spawned Python process. Python: import sys for l in sys.stdin: print "got: %s" % l Node: var spawn = require('child_process').spawn; var child = spawn('python', ['-u', 'ipc.py']); child.stdout.on('data', function(data){console.log("stdout: " + data)}); var i = 0; setInterval(function(){ console.log(i); child.stdin.write("i =...

Traversing `vm_area_struct`'s of a process

As an extension of an answer to Traversing all the physical pages of a process and data here http://www.makelinux.net/ldd3/chp-15-sect-1, I had a small question. In the first answer as suggested to traverse the physical pages of a process, struct vm_area_struct *vma = 0; unsigned long vpage; if (task->mm && task->mm->mmap)...

Sending an arrow key with fprintf

I'm testing out pipes and have hit a little road block. I want to be able to simulate pressing the right arrow key through a file pointer opened by popen. The file pointer opens a display program on a different terminal, much like a projector. I would like to send...

Driver mmap operation Page table creation

I am trying to understand mmap operation in a driver. I know why we are doing mmap. I am confused with the following 2 implementations of mmap: int my_mmap(struct file *filp, struct vm_area_struct *vma) { vma->vm_ops = &mmap_vm_ops; vma->vm_flags |= VM_RESERVED; /* assign the file private data to the vm...

Usage of spinlock functions

How do I use these functions? I have declared lock globally. pthread_spinlock_t lock; Also the spinlock is initialized locally. pthread_spin_init(&lock, 1); // non-zero as pshared for IPC But now I want to lock my critical integer and increment it. I have multiple processes running this in a loop: while(0 !=...

Can I use STDIN for IPC?

Can I use standard input for interprocess communication? I wrote the following gnu c code as an experiment, but the program hangs waiting for input after printing the character defined as val. Neither a newline nor fflush in the sending process seem to alleviate the problem. #include <unistd.h> #include <stdio.h>...

Stopping a process by sending SIGUSR1 and waking up by sending SIGUSR2

How do I stop a process by sending SIGUSR1 and wake that up by sending SIGUSR2? Is it possible? Please help me out.

Socket to communicate between two processes (originated from C and Java) on the same machine

I need to transfer data from one process to another. I'm quite familiar with the subject when the two processes were originated from C code - not once I used files, signals and pipes in C-code to accomplish it, but never have I tried to do it between two processes...

Why does using `execl` instead of `system` stops my program from working?

I'm trying to do basic IPC using pipes. I spent hours searching the internet, doing this and that, reading the API documentations, and ended up with the code below. But it does not work, as I quite expected. Just any help making my code 'work' would be many thanks. <edit>...

How to notify an application from windows service using WCF

I have a windows service that continuously read data from a serial port, from this service I'd like to be able to notify a WPF application with the updated data using WCF; I thought about using callbacks. The windows service would call a WCF service's method which in turn would...

how to use share memory with Golang? [closed]

How does golang share or read other processes shared memory? I've checked some information, but did not find relevant information. Can anyone give me an example?

Linux - c - ipc notification without context switch and low overhead?

Context : Linux 64bits. I am very well versed in shared memories, different IPCs mechanisms. Currently, I am seeking something that could be used in userspace to notify change between two polled processes, to save cpu cycles. I already know about futexes, eventfd, slow signals and even char devices. But...

mmap() and pthreads - copy VMA

I mmap() an anonymous VMA. How do pthreads handle that VMA? I want that the vma with its memory is copied for every thread. Which flags do I need for that purpose?

Preventing use of shared memory [closed]

A java application I have made is apparently using shared memory, which is interfering with the deployment of other programs in my client's unix environment. It was never the intension to use any shared memory and I need to identify where it is being consumed. Program Overview: -Generic executable jar...

shared memory programme is not working in c

There are two programs are there one is call server which put content in shared memory and other is client which received a content from shared memory both in both programs it is attached successfully with shared memory but data is not display in client side. Client.c #include<fcntl.h> #include<sys/ipc.h> #include<sys/shm.h>...

Using mmap and madvise for huge pages

I want to allocate memory on the hugepages being used by a Linux machine. I see that there are two ways to do this, using mmap and madvise. That is, using the MAP_HUGETLB flag with the mmap call - base_ptr_ = mmap(NULL, memory_size_, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS |...

Asm x86 segmentation fault in reading from file

I'm receiving segmentation fault in 5th line of loop2. This is the part of my code which is reading values from mmapped file byte by byte. Could you please tell me what I'm doing wrong? .global _start _start: mov $2,%rax mov 16(%rsp),%rdi mov $02,%rsi syscall cmp $0,%rax jl exit 1:...

malloc() - Does it use brk() or mmap()

c code: // program break mechanism // TLPI exercise 7-1 #include <stdio.h> #include <stdlib.h> void program_break_test() { printf("%10p\n", sbrk(0)); char *bl = malloc(1024 * 1024); printf("%x\n", sbrk(0)); free(bl); printf("%x\n", sbrk(0)); } int main(int argc, char **argv) { program_break_test(); return 0; } When compiling following code: printf("%10p\n", sbrk(0)); I get warning...

Bad address with mq_open

I am trying to open a simple queue using mq_open but I keep getting error: "Error while opening ... Bad address: Bad address" And I have no idea why. int main(int argc, char **argv) { struct mq_attr attr; //max size of a message attr.mq_msgsize = MSG_SIZE; attr.mq_flags = 0; //maximum...

Shared memory and POSIX semaphores

I wrote simple consumer-producer program in C. It is working fine while I have 1 producer and 1 consumer. But it is acting strange when I increase number of consumers. I start the producer process Producer is producing I start the consumer process Consumer is consuming and producer is producing...

C programming bi directional communication

I am trying to make something work here, i have c program where my parent process creates a pipe so he can listen request from children process. These children are created dynamically, it is never the same number. So far, i managed to send the requests to the parent through...

memory mapping files for high frequency trading?

I am working in a high frequency trading client for stocks in C++. I am using the quickfix/J library to receive information from my broker. I need these two processes to communicate (one written in Java and the other in C++) and the latency has to be minimized as much...

How to use munmap custom malloc

I'm trying to recode malloc and free functions but i got a problem with munmap.. With my custom malloc, I create a pool of memory with mmap (Ex : 4Kb), so I can return an index of this pool as an address when my malloc is called. This work fine,...

In the following case, which one is better ? fread() or mmap()?

I want to read two files via a process, the first file is about 2G, the second file is about 20M. They look like this: 1 1217907 1 1217908 1 1517737 1 2 1 3 1 4 1 5 Now I plan to read the whole file into memory, then...

Semaphore value initialized to -1?

I am trying to initialize a binary semaphore for some shared memory. I could not figure out why I hung on a semop to get the semaphore, and eventually found that the semaphore value was -1 and I was stuck waiting. What I don't get is why the semaphore value...

Slow memory allocation in OSX

I'm trying to trace down a memory allocation problem I have in OSX. If I compile and run the following code normally, it will run pretty fast. #include <sys/mman.h> #define SIZE 8 * 1024 * 1024 int main(int argc, char const *argv[]) { for (int i = 0; i <...

Typecasting over a message queue in c

I am currently trying to send an integer over a message queue. I am typecasting the address of this integer to a (char *) since in the future I would like to accept other data types and this data is sent in a struct designed for use in message queues....

Producer, consumer POSIX

I am trying to write simple producer consumer app using C POSIX semaphores. Consumer: int memoryID; struct wrapper *memory; int main(int argc, char **argv) { srand(time(NULL)); key_t sharedMemoryKey = ftok(".",MEMORY_KEY); if(sharedMemoryKey==-1) { perror("ftok():"); exit(1); } memoryID=shmget(sharedMemoryKey,sizeof(struct wrapper),0); if(memoryID==-1) { perror("shmget(): "); exit(1); } memory = shmat(memoryID,NULL,0); if(memory== (void*)-1) { perror("shmat():");...

Communication between RESTful API's on same server with NodeJS

I am building two sets of services on a website (all written in NodeJS on the server), both are using a RESTful approach. For the sake of modularity I decided to make both services separate entities. The first service deals with the products of the site and the second specifically...

Expected behaviour of shm_unlink?

Does shm_unlink store a reference count to determine when to delete the file? I ask because I have the following: Writer: #include <iostream> #include <sys/mman.h> #include <unistd.h> #include <fcntl.h> #include <cstdio> using namespace std; void Sleep(int ms) { usleep(ms * 1000); } int main() { int fd = shm_open("/Test", O_CREAT...

Merging a paid Android app with the free version

I have an Android app that comes in two versions: a free one and a premium one with more features that costs $0.99. I'd like to move to an In App Purchase approach where the premium features can be bought in the Free version. However, I obviously don't want to...

How to mmap a struct as shared anonymous memory?

I am trying to mmap a struct as anonymous shared memory so it can be shared across multiple child processes. However whenever I access the struct outside of the functions that create it, I get a segfault. The code is derived from trinity fuzzer, which can be found here https://github.com/kernelslacker/trinity...

get N-th byte from mmaped file

I'm new at C and now I'm learning about mmap. I want to get N-th byte from mmaped file, but I get this error Segmentation Fault (core dumped) When I test my program with gdb I get that something is wrong with this line printf("%d\n", (int) data[sk]); and then I...

Creating array with mmap function. Assembler x86_64 AT&T

I want to create an array that will contains for example 4 values Here's my code: mov $32,%rsi # 4 x 8bytes mov $9,%rax mov $0,%rdi mov $0x3,%rdx mov $0x01,%r10 mov $0,%r9 syscall Now I am having a new adress of alocated 32 bytes in rax? When I am trying...

C - Memory Mapping with Multiple Processes

I have an assignment requiring me to write a multi-processed program that works with a memory-mapped file containing a string of characters. After the parent process maps the file to memory, it spawns 2 children processes to modify the file. Child 1 outputs the contents of the file, converts the...

Portlet IPC after recieved Event

I have a Navigation portlet which schows all Students, when a Student gets clicked im sending a event with the id of the clicked student to other portlets. Sending and receiving the event is not the problem, that works fine. What i don't get, is how to proceed with the...