c,linux,memory,mmap,fread , In the following case, which one is better ? fread() or mmap()?

In the following case, which one is better ? fread() or mmap()?


Tag: c,linux,memory,mmap,fread

I want to read two files via a process, the first file is about 2G, the second file is about 20M. They look like this:

1   1217907
1   1217908
1   1517737
1   2
1   3
1   4
1   5

Now I plan to read the whole file into memory, then use strtok_r() to get the value of each num and store them in a data structure array. 4 processes may read these two files on the same computer at almost same time. The computer is 64-bit, physical memory maybe 4G or even less. My question is, which method is more efficient ? fread() or mmap() ?

Here is the key part of my program of reading the whole file(since someone wants to see it,but I don't know whether it is related to my question):

typedef struct My_Edge
int first;
int second;

Edge *myEdge;

int Read_Whole_File()

fseek(wholeFile, 0, SEEK_END);

long int fileSize=ftell(wholeFile);

char *buffer=malloc(sizeof(char)*fileSize+1);

fseek(wholeFile, 0, SEEK_SET);

fread(buffer, 1, fileSize, wholeFile);

char *string_first;
char *string_second;
char *save_ptr;

int temp_first;
int temp_second;

string_first = strtok_r(buffer, " \t\n", &save_ptr);
int i=0;

int temp_edge_num;
Edge *temp_edge;

while (string_first != NULL)
    temp_first = atoi(string_first);

    string_second = strtok_r(NULL," \t\n",&save_ptr);

    temp_second = atoi(string_second);

        temp_edge_num = i + EDGE_NUM_ADJUST;

        temp_edge = realloc(myEdge, sizeof(Edge)*temp_edge_num);
            myEdge = temp_edge;
        my_edge_num = temp_edge_num;



    string_first = strtok_r(NULL, " \t\n", &save_ptr);


return 0;

Now I am trying to use mmap(), but it has EXC_BAD_ACCESS when I use strtok_r() tp process the var which is processed by mmap():

char *buffer;

struct stat fileStat;

fstat(wholeFile, &fileStat);

buffer = mmap(NULL, fileStat.st_size, PROT_READ, MAP_SHARED, wholeFile, 0);

char *string_first;
char *string_second;
char *save_ptr;

int temp_first;
int temp_second;

string_first = strtok_r(buffer, " \t\n", &save_ptr);//EXC_BAD_ACCESS here, the content of buffer is correct.


My question is, which method is more efficient ? fread() or mmap() ?

First of all, let's look how fread and mmap works on linux:
Let's say we work with ext4 file system (without encryption), fread use some internal buffer and if no data in it,
it calls read, read execute "system call" and after some time we jump to:
fs/read_write.c::vfs_read and after more work we reach mm/filemap.c::generic_file_read_iter

And in this function we fill inode page cache and read to this page cache data.

So we do the basically the same as "mmap" does.

The difference that that in fread case we not directly works with pages, we just copy portion of data from kernel inode page cache to user space buffer,
in mmap we have page cache directly in program memory space. Plus in fread when no page in "inode page cache" we just read it, but in mmap that cause "page fault", and only after that we read it.

Both variants use "read pages ahead" strategy. The possible difference may be in "cache" policy, we can control it in "mmap" case with madvise and flags of mmap.

So I suppose the answer is "they are almost the same in terms of speed in sequence read case like yours".


Force linux to use php as php55

Helllo, I have a little problem here. I have PHP 5.3 installed and it's accessible via php command, I also have php 5.5 accessible as php55 command. Now I need to force linux to use php55 when I write php command. Simple way: How I switch the versions of php?...

VS2012 Identifer not found when part of static lib

Using VS2012 C/C++: I created and linked a static lib called "libtools" to my project. Calls to functions in the libtools lib worked as expected. I created and linked a second static lib called "shunt" to my project. But when I incorporate a call to a function in shunt, I...

Galois LFSR - how to specify the output bit number

I am trying to understand how change the galois LFSR code to be able to specify the output bit number as a parameter for the function mentioned below. I mean I need to return not the last bit of LFSR as output bit, but any bit of the LFSR (...

CallXXXMethod undefined using JNI in C

So I've tried to use the JNI interface to call Java methods from C. Calling static methods is no problem, but I get stuck when I want to call a method on an object. The code is as follows: #include <stdio.h> #include <string.h> #include <jni.h> int main() { JavaVMOption options[1];...

Using an ad-hoc libc with a tool which is an argument of another tool

I need to use a particular libc to run a tool (cp). The problem is that this tool has to be used as argument of another tool (for example timeout) and I don't want to use the modified libc with this one. I tried to do: timeout 10 LD_LIBRARY_PATH=/path/to/mod/libc/ cp...

Is there Predefined-Macros define about byte order in armcc

Is there Predefined-Macros define about byte order in armcc. I am a novice on the armcc.and sorry for my English. In gcc these are macros: __BYTE_ORDER__ __ORDER_LITTLE_ENDIAN__ __ORDER_BIG_ENDIAN__ __ORDER_PDP_ENDIAN__ ... Now I have to use armcc, Is there same like these with armcc? Thank a lot. by the way,the armcc...

C programming - Confusion regarding curly braces

The following code is for replacing multiple consecutive spaces into 1 space. Although I manage to do it, I am confused in the use of curly braces. This one is actually running fine: #include <stdio.h> #include <stdlib.h> int main() { int ch, lastch; lastch = 'a'; while((ch = getchar())!= EOF)...

How can I resolve the “Could not fix timestamps in …” “…Error: The requested feature is not implemented.”

I have been trying to build a project in F# on Linux that I have located here on github. It's a basic kata project that I am working on as a demo. However on Linux (specifically Ubuntu 14.04 LTS Desktop) I haven't been able to get it to build yet...

Reverse ^ operator for decryption

I'm trying to reverse the following code in order to provide a function which takes the buffer and decrypts it. void crypt_buffer(unsigned char *buffer, size_t size, char *key) { size_t i; int j; j = 0; for(i = 0; i < size; i++) { if(j >= KEY_SIZE) j = 0;...

While loop in bash using variable from txt file

I am new to bash and writing a script to read variables that is stored on each line of a text file (there are thousands of these variables). So I tried to write a script that would read the lines and automatically output the solution to the screen and save...

How to control C Macro Precedence

#define VAL1CHK 20 #define NUM 1 #define JOIN(A,B,C) A##B##C int x = JOIN(VAL,NUM,CHK); With above code my expectation was int x = 20; But i get compilation error as macro expands to int x = VALNUMCHK; // Which is undefined How to make it so that NUM is replaced first...

Ignore first few lines and last few lines in a file Linux

I have a file like this and would like to print $0 except the first two and last three lines in linux. Tried awk command but no luck, is there any options I am using the following command - I suppose I am doing something wrong, but not able to...

Extra backslash when storing grep in a value

In a bash script I have: Check="grep -e '"'\(-S mount\)'"' /etc/audit/audit.rules" set -x When you run it it shows it as: CHECK='grep -e '\''\(-S mount\)'\'' /etc/audit/audit.rules' Now it works exactly what I want but I want to understand it. Why is there 2 extra \'s?...

AWK write to new column base on if else of other column

I have a CSV file with columns A,B,C,D. Column D contains values on a scale of 0 to 1. I want to use AWK to write to a new column E base in values in column D. For example: if value in column D <0.7, value in column E =...

Segmentation Fault if I don't say int i=0

void removeVowels(char* array){ int i,j,v; i=0; char vowel[]={'a','e','i','o','u'}; while(array[i]!='\0') { for(v=0;v<5;v++) { if (array[i]==vowel[v]) { j=i; while(array[j]!='\0') { array[j]=array[j+1]; j++; } i--; break; } } i++; } } in function removeVowels() if I don't include i=0; and just say int i; why does it give segmentation fault? Isn't it automatically...

How to read string until two consecutive spaces?

A well known function of the scanf() functions is that you can pass a format to scan input according to this format. For my case, I cannot seem to find a solution searching this and this documentation. I have a string (sInput) as the following: #something VAR1 this is a...

Segmentation fault with generating an RSA and saving in ASN.1/DER?

#include <string.h> #include <openssl/aes.h> #include <openssl/rand.h> #include <openssl/bio.h> #include <openssl/rsa.h> #include <openssl/evp.h> #include <openssl/pem.h> #define RSA_LEN 2048 #define RSA_FACTOR 65537 int genRSA2048(unsigned char **pub,unsigned int *pub_l,unsigned char **priv,unsigned int *priv_l){ RSA *pRSA = NULL; pRSA = RSA_generate_key(RSA_LEN,RSA_FACTOR,NULL,NULL); if (pRSA){ pub_l = malloc(sizeof(pub_l)); *pub_l = i2d_RSAPublicKey(pRSA,pub); priv_l = malloc(sizeof(priv_l));...

Set precision dynamically using sprintf

Using sprintf and the general syntax "%A.B" I can do this: double a = 0.0000005l; char myNumber[50]; sprintf(myNumber,"%.2lf",a); Can I set A and B dynamically in the format string?...

C++ / C #define macro calculation

Suppose I have #define DETUNE1 sqrt(7)-sqrt(5) #define DETUNE2 sqrt(11)-sqrt(7) And I call these multiple times in my program. Are DETUNE1 and DETUNE2 calculated every time it is called? Thanks. Please don't downvote this, I really want to know and a search didn't turn up anything definite. ...

Text justification C language

I have to solve a problem that involves left justification string length and leading zeros. I have the following table : BEGIN CLOSE CONCATENATE DELETE END INITIALIZE PRINT WRITE This is produced by a simple program. My problem is to find out how to convert it like that : It...

What are correct permissions for Linux Apache2 PHP 5.3 log file?

I discovered the reason why I was not getting entries into my php_errors.log file related to permissions. Right now, I have set it to 666 (rw-rw-rw-) but surely this is a security weakness? Thus, my question. php.ini file: error_log /var/log/httpd/php_errors.log log_errors On # ls -ld /var/log /var/log/httpd /var/log/httpd/php_errors.log drwxr-xr-x 6...

What does it indicate if /proc/PID/maps shows zero for all addresses?

I'm debugging a problem with a Linux DNS server. Curiously, when I look at /proc/PID/maps for the DNS server process, this is what I get: 00000000-00000000 r-xp 00000000 00:0e 2344 /usr/sbin/unbound 00000000-00000000 rw-p 00000000 00:0e 2344 /usr/sbin/unbound 00000000-00000000 ---p 00000000 00:00 0 00000000-00000000 rw-p 00000000 00:00 0 [heap] 00000000-00000000 rw-p...

Delete some lines from text using Linux command

I know how to match text using regex patterns but not how to manipulate them. I have used grep to match and extract lines from a text file, but I want to remove those lines from the text. How can I achieve this without having to write a python or...

linux running command as root from c code that run as normal user

I have a c++ code and I need to running from it a command to adjust the system time. so I thought using system("su root -c date hh:mm") command from my c++ code. The problem is that when I write 'su root -c date hh:mm' in the terminal its requires...

C binary tree sort - extending it

I need some help in C Help me to extend the binary tree sort on C. I need to return a sorted array in sort function. here it is: #include <stdio.h> #include <stdlib.h> struct btreenode { struct btreenode *leftchild ; int data ; struct btreenode *rightchild ; } ; void...

how to modify an array value with given index?

I want to modify an array cell, which I can do when I know the cell as a number. However here my cell position is given by $i. pomme[`${i}`]="" I tried without the `` and it doesn't work either? How am I suppose to do it?...

Array breaking in Pebble C

I'm trying to create a simple dice-rolling application in Pebble using C on CloudPebble. I have an array of different die sizes you can scroll through using Up/Down, and you roll (currently just generate a random number, it'll get fancier later) using the middle button. There's also a label at...

Does realloc() invalidate all pointers?

Note, this question is not asking if realloc() invalidates pointers within the original block, but if it invalidates all the other pointers. I'm new to C, and am a bit confused about the nature of realloc(), specifically if it moves any other memory. For example: void* ptr1 = malloc(2); void*...

Does strlen() always correctly report the number of char's in a pointer initialized string?

As long as I use the char and not some wchar_t type to declare a string will strlen() correctly report the number of chars in the string or are there some very specific cases I need to be aware of? Here is an example: char *something = "Report all my...

Passing int using char pointer in C

I'm trying to figure out how to pass an int using a char pointer. It fails once the int value is too large for the char. This is what I'm trying to figure out: char *args[5]; int i = 20; /*some other code/assignments*/ args[2] = (char *)&i; execv(path, args); How...

Django MySQLClient pip compile failure on Linux

Django documentation as of v1.8 recommends using mysqlclient connector for the framework. I'm attempting to pip install the package on Ubuntu 14.04 with Python 3.4 and running into a GCC error that I'm unable to find reference to. I'm not an expert on compiling software, so was hoping somebody can...

getchar() not working in c

getchar() is not working in the below program, can anyone help me to solve this out. I tried scanf() function in place of getchar() then also it is not working. I am not able to figure out the root cause of the issue, can anyone please help me. #include<stdio.h> int...

Linux-wget command

I need a quick help on customizing my wget command in a shell script: The wget command looks something like this: wget http://infamvn:8081/nexus/content/groups/LDM_REPO_LIN64/com/infa/com.infa. products.ldm.ingestion.server.scala/ Here I'd like to add the in a variable, so I created a script something like this: n = wget...

ret_from_syscall source code and when it is called

In the below call trace we see function called ret_from_syscall. Which function is this ? When it will called during system call ? Where is the corresponding code for this ? May 7 16:40:34.322086 warn TCU-0 kernel: [cf83ddc0] [00009751] 0x9751 (unreliable) May 7 16:40:34.322086 warn TCU-0 kernel: [cf83ddd0] [c00469ac] do_syslog+0x198/0x424...

Disadvantages of calling realloc in a loop

I'm trying to implement some math algorithms in C on Windows 7, and I need to repeatedly increase size of my array. Sometimes it fails because realloc can't allocate memory. But if I allocate a lot of memory at once in the beginning it works fine. Is it a problem...

How to increment the value of an unsigned char * (C)

I have a value stored as an unsigned char * (in C). This holds the SHA1 hash of a string. My goal is to cover the SHA1 key space. Since I'm using <openssl/evp.h> to generate the hashes, I end up with an unsigned char* holding the SHA1 value. Now I...

How to append entry the end of a multi-line entry using any of stream editors like sed or awk

I am trying to use a script to append the host name at the end of a multi-line entry of a specific Host_Alias field in sudoers file. The current sudoers file has an entry similar to : Host_Alias srv_linuxestate= \ host10,host12,host13,host1,host50,\ host16,host1,host2,host11,host15,host21,\ host3,host14 My required output would be something like...

NASM: copying a pointer from a register to a buffer in .data

I am new to asm. I am trying to copy a pointer from a register to a .data variable using NASM, on linux 64-bit. Concider this program: section .data ptr: dq 0 section .text global _start _start: mov [ptr], rsp mov rax, 60 mov rdi, 0 syscall Here I try...

Loop through database table and compare user input

I am trying to loop through the rows in a MySql table and compare the data in a certain column to some user input using C. Currently my code looks like this: MYSQL *cxn = mysql_init(NULL); MYSQL_RES *result; unsigned int num_fields; unsigned int num_rows; char *query_string; MYSQL_ROW *row; if (mysql_real_connect(cxn,...

Program to reverse a string in C without declaring a char[]

I need to reverse a given string and display it without using the value At[index] notation , I tried the below program using pointers,but it does not print anything for the reverse string, Please help! int main() { char* name=malloc(256); printf("\nEnter string\n"); scanf("%s",name); printf("\nYou entered%s",name); int i,count; count=0; //find the...

How does this code print odd and even?

#define MACRO(num, str) {\ printf("%d", num);\ printf(" is");\ printf(" %s number", str);\ printf("\n");\ } int main(void) { int num; printf("Enter a number: "); scanf("%d", &num); if (num & 1) { MACRO(num, "Odd"); } else { MACRO(num, "Even"); } return 0; } Please explain the above code (if/else condition and how...

Bash modify CSV to change a field

I have a very big CSV file (aprox. 10.000 rows and 400 columns) and I need to modify certain columns (like 15, 156, 220) to change format from 20140321132233 to 2014-03-21 13:22:33. All fields that I need to modify are datetime. I saw some examples using awk but for math...

How does ((a++,b)) work? [duplicate]

This question already has an answer here: What does the comma operator `,` do in C? 8 answers In the below block of code, I am trying to understand how the line return reverse((i++, i)) is working. #include <stdio.h> void reverse(int i); int main() { reverse(1); } void reverse(int...

Is post-increment operator guaranteed to run instantly?

Let's say I have the following code: int i = 0; func(i++, i++); The increment is happening right after returning the value? Is it guaranteed that the first argument will be 0, and the second argument will be 1?...

AWK count number of times a term appear with respect to other columns

Given a CSV file: id, fruit, binary 1, apple, 1 2, orange, 0 3, pear, 1 4, apple, 0 5, peach, 0 6, apple, 1 How can i calculate for each unique values in fruit, the number of times the binary value =1 / number of occurences of that fruit...

scanf get multiple values at once

I need to get in one single shot different inputs from one single line. In particular I need to get a single char and then, depending on which char value I just read, it can be a string and an int or a string, an int and another string and...

Syncing Vagrant VMs across different physical servers

I'm using Vagrant to deploy my VMs and my current setup looks like this: server1 = VM1, VM2, VM3 ( main production server ) server2 = VM1, VM2, VM3 ( backup server ) My questions is, can I somehow sync the VMs across the different physical servers in case one...

Infinite loop with fread

I'm trying to allocate an array 64 bytes in size and then loop over the array indexes to put a read a byte each from the inputfile. but when I don't malloc() the array indexes, the loop stays in index0 (so each time it loops it replaces the content in...

Counting bytes received by posix read()

I get confused with one line of code: temp_uart_count = read(VCOM, temp_uart_data, 4096); I found more about read function at http://linux.die.net/man/3/read, but if everything is okay it returns 0, so how we can get num of bytes received from that? temp_uart_count is used to count how much bytes we received...

What does `strcpy(x+1, SEQX)` do?

I'm wondering what this syntax of strcpy() does in line 65 and 66: 24 #define SEQX "TTCATA" 25 #define SEQY "TGCTCGTA" 61 M = strlen(SEQX); 62 N = strlen(SEQY); 63 x = malloc(sizeof(char) * (M+2)); /* +2: leading blank, and trailing \0 */ 64 y = malloc(sizeof(char) * (N+2)); 65...