c++,boost,endianness , Clean way to make portable endian-correct file-reading / writing code in C++


Clean way to make portable endian-correct file-reading / writing code in C++

Question:

Tag: c++,boost,endianness

I want to write some C++ code that can read and write from files in an endian-correct way. More exactly, I want to be able to read a particular type of file, whose endianness I can easily detect (its magic number being reversed or not).

But how would I then go about cleanly reading the file correctly? I've read the following article which gives a helpful idea:

http://www.gamedev.net/page/resources/_/technical/game-programming/writing-endian-independent-code-in-c-r2091

The idea there is to make a class which has some function pointers to the endianness-correct read() function desired. But in my experience, function pointers are slow, especially when you have to call them so frequently as in this case. Another alternative is to have an

if (file_was_detected_big_endian) { read_bigendian(); } else { read_littleendian(); }

for every single read_x_bit_int() function I have but this too seems inefficient.

I'm using Boost so I have all its magnificence to help me. In particular, there's the endian sublibrary:

http://www.boost.org/doc/libs/develop/libs/endian/doc/buffers.html

Though I'm not sure how I can cleanly use this code to do what I want. I'd love to have some code where I can read say 16 bytes directly into the pointer of a struct that represents a part of the file while automatically correcting for endianness. I could of course write this code myself but I have a feeling a solid solution must already exist.

I think all the code I have will be manually padded and guarded against alignment issues.

Thanks!


Answer:

There are two approaches to this problem:

  1. Write your files in an endianness-agnostic way, and
  2. Add a marker, and read files in an endianness-aware way.

The first approach requires more work on writing, while the second approach makes writing "overhead-free".

Both approaches can be implemented without function pointers: the need for them has greatly diminished in C++ due to virtual functions*.

Implementing both approaches is similar: you need to make an abstract base class for serializing primitive data types, make an instance of that class that reads the correct endianness, and call its virtual member functions for reading and writing:

struct PrimitiveSerializer {
    virtual void serializeInt(ostream& out, const int val) = 0;
    virtual void serializeChar(ostream& out, const char val) = 0;
    virtual void serializeString(ostream& out, const std::string& val) = 0;
    ...
    virtual int deserializeInt(istream& in) = 0;
    virtual char deserializeChar(istream& in) = 0;
    virtual std::string deserializeString(istream& in) = 0;
};
struct BigEndianSerializer : public PrimitiveSerializer {
    ...
};
struct LittleEndianSerializer : public PrimitiveSerializer {
    ...
};

Depending on the approach, the decision which subclass to use is made differently. If you use the first approach (i.e. writing endianness-agnostic files) then you would instantiate the serializer that matches your system's endianness. If you take the second approach, you would read the magic number from the file, and choose the subclass that matches the endianness of your file.

In addition, the first approach can be implemented using hton / ntoh functions.

* Function pointers are not "slow" by themselves, although they make it easier to write inefficient code.


Related:


Make a triangle shape in C++


c++
I am trying to print out the shape of a triangle but I am kinda lost... this is what I have so far: #include <iostream> using namespace std; int main() { int i, k, n; cout << "Please enter number of rows you want to see: \n"; cin >> n;...

Why are shaders and programs stored as integers in OpenGL?


c++,opengl,opengl-es,integer,shader
I'm following the "OpenGL Superbible" book and I can't help but notice that when we create a shader and create the program that we attach the shaders to, we store them as GLuint which are unsigned integers. Why are they stored as numbers? What does the value of the number...

Undefined behaviour or may be something with memset


c++,undefined-behavior
I was trying to save the binary equivalent of a 32 bit number in an array A. For testing my showbits() function , I choosed 8,9 when I came across this thing: I am facing an unreasonable thing in my code when I am placing memset in the function showbits(),I...

create vector of objects on the stack ? (c++)


c++,vector,heap-memory
I am creating a temporary vector of pointers to myObject objects. But I am wondering about what happens to the objects I created... { std::vector<myObject *> myVector; myVector.reserve(5); for (int i = 0 ; i < 5 ; ++i){ myVector[i] = new myObject(); } } I assume that at the...

undefined reference to `vtable for implementation' error


c++,build,makefile
I wrote some c++ files and after compiling with out make file it works fine . But when using make file it pop out some errors . My codes are : include directory files : application.h #ifndef APPLICATION_H #define APPLICATION_H #include "employee.h" #include "employee_data.h" #include "employee.h" ...some defintions here... #endif...

Confused about returns in stack template


c++,templates,generic-programming
I'm implementing a generic stack (with an array) in C++ and am confused about what to return in this situation: template <class T> T Stack<T>::pop(void) { if (size != 0) { return items[size - 1]; size--; } else { cerr << "Cannot pop from empty stack." << endl; return ???;...

Checking value of deleted object


c++
I asked a question: Detecting if an object is still active or it has been destroyed Considering that I cannot use libraries, there are no good out of the box solutions in C++. So, is it a bad practice to check if the object has been destroyed by analyzing memory...

Copy text and placeholders, variables to the clipboard


c++,qt,clipboard
In my application I want generate random numbers or strings with a text in front of it. It is important for me that the text won't appear in my window, but instead gets copied to the clipboard. int randomnumber = rand() % 46 + 1; QClipboard *cb = QApplication::clipboard(); cb->setText("Just...

Get an ordered list of files in a folder


c++,boost,boost-filesystem
I have used boost::filesystem::directory_iterator in order to get a list of all the available files into a given folder. The problem is that I supposed this method would give me the files in alphabetical order, while the results seem pretty random. Is there any fancy way of alphabetically sorting them?...

3 X 3 magic square recursively


c++,algorithm,math,recursion
I'm trying to find all possible solutions to the 3X3 magic square. There should be exactly 8 solutions. My code gets them all but there are a lot of repeats. I'm having a hard time tracking the recursive steps to see why I'm getting all the repeats. // This program...

opencv window not refreshing at mouse callback


c++,opencv
I am trying to draw with mouse move in an opencv window. But when I draw, nothing draws on the window. When I try to close the window from the cross in the topleft(ubuntu), it opens a new window which it should be as I haven't pressed escape, and in...

C++ & Qt: Random string from an array area


c++,arrays,string,qt,random
In my small Qt application, I want to pick a random string out of an array after I clicked on a button. I've read many threads but nothing works for me. So in my slot there's an array with several strings in it. I also implemented <string>, <time.h> and srand....

Algorithm for [inclusive/exclusive]_scan in parallel proposal N3554


c++,algorithm,parallel-processing,c++14
Proposal N3554 (A Parallel Algorithms Library) for C++14, proposes (among other things), what seem to be parallel versions of the current std::partial_sum, e.g.: template< class ExecutionPolicy, class InputIterator, class OutputIterator, class BinaryOperation> OutputIterator inclusive_scan( ExecutionPolicy &&exec, InputIterator first, InputIterator last, OutputIterator result, BinaryOperation binary_op); With the explanation Effects: For each...

.cpp:23: error: cannot convert ‘std::string’ to ‘const char*’ for argument ‘1’ to ‘int atoi(const char*)’


c++,string
Here a basic code I'm trying to run But I'm having trouble with stoi (it's c++) I keep getting error: ‘stoi’ was not declared in this scope I tried atoi and strtol with this error .cpp:23: error: cannot convert ‘std::string’ to ‘const char*’ for argument ‘1’ to ‘int atoi(const char*)’...

C++ Isn't this a useless inline declaration?


c++,inline,private,member,protected
This is another question about inlining a function. But I will take possible comments and answers right away: Defining a function inside a class makes it inline automatically. The same behaviour can be achieved by marking a function with inline outside of the class. An inline function doesn't have to...

Incorrect Polar - Cartesian Coordinate Conversions. What does -0 Mean?


c++,polar-coordinates,cartesian-coordinates
I am getting incorrect conversions from polar to cartesian coordinates and vice versa. My code produces weird points like (1,-0). Im using this calculator to check my conversions. Also one of the conversions is completely wrong when I convert back to cartesian coordinates. Point b: (0,1) => (1,1.5708) => (0,0)...

Same function with and without template


c++,c++11
I am trying to understand a piece of code of C++11. A class contains 2 functions as shown below: class abc { public: void integerA(int x); template<typename typ> void integerA(typ x); }; I am unable to understand benefit of declaring 2 same functions. Why not declare only one template function?...

C++11 Allocation Requirement on Strings


c++,string,c++11,memory,standards
I had heard that C++11 was going to require strings to be allocated in contiguous memory. I even thought I saw a stack overflow question on it, but I can't seem to find it. I know that in practice both gcc and Visual Studio do allocate strings contiguously, I'm just...

Implicit use of initializer_list


c++,c++11,initializer-list
§[dcl.init.list] 8.5.4/2: The template std::initializer_list is not predefined; if the header <initializer_list> is not included prior to a use of std::initializer_list — even an implicit use in which the type is not named (7.1.6.4) — the program is ill-formed. Does that mean this program is ill-formed? #include <vector> int main()...

std::condition_variable – notify once but wait thread wakened twice


c++,multithreading
Here's a simple C++ thread pool implementation. It's an altered version orginated from https://github.com/progschj/ThreadPool. #ifndef __THREAD_POOL_H__ #define __THREAD_POOL_H__ #include <vector> #include <queue> #include <memory> #include <thread> #include <chrono> #include <mutex> #include <condition_variable> #include <future> #include <functional> #include <stdexcept> namespace ThreadPool { class FixedThreadPool { public: FixedThreadPool(size_t); template<class F, class......

MFC visual c++ LNK2019 link error


c++,mfc
I just don't understand why i can use the public variables on the class but are getting a link error when trying to use the getLicenceRefused method. I wasn't sure if the problem was because of the CString copy constructor problem I have had before so took the parameter out,...

No match for 'operator*' error


c++,c++11
Hello fellow programmers! I was going to write a small program for calculating total pay for different periods of time depending on the amount of hours and the salary that the user enters. I managed to make a small bit of the program but when I try to run it...

Passing iterator's element to a function: wrong type of pointer


c++,pointers,stl,iterator
I'm attempting to solve Project Euler's problem #3 using C++ to gain an understanding of how to use C++ iterators. According to the examples I've seen online, I can use the dereferened iterator as a parameter for cout, and it will print the elements successfully. By that same logic, I...

Parameters to use in a referenced function c++


c++,pointers,reference
I am very confused as to what kind of variables I would put into my function here: names. I am doing a practice problem in a C++ book, because I am learning C++ and am on References and pointers right now, and cannot find a solution. Just for background information,...

Strings vs binary for storing variables inside the file format


c++,file,hdf5,dataformat
We aim at using HDF5 for our data format. HDF5 has been selected because it is a hierarchical filesystem-like cross-platform data format and it supports large amounts of data. The file will contain arrays and some parameters. The question is about how to store the parameters (which are not made...

how to sort this vector including pairs


c++,vector
I want to sort in ascending order according to the first element of the inner pair, i.e. a in this case. But its not at all sorting. I am not sure if my function func logic is correct. #include<iostream> #include<algorithm> #include<vector> using namespace std; bool func(const pair<int,pair<int,int> >&i , const...

Marshal struct in struct from c# to c++


c#,c++,marshalling
I have the following structures in C# and C++. C++: struct TestA { char* iu; }; struct TestB { int cycle1; int cycle2; }; struct MainStruct { TestA test; TestB test2; }; C#: [StructLayout(LayoutKind.Sequential, CharSet=CharSet.Ansi, Pack = 1)] internal struct TestA { [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 36)] private string iu; public...

Can python script know the return value of C++ main function in the Android enviroment


python,c++
There are several ways of calling C++ executable programs. For example, we can use def run_exe_return_code(run_cmd): process=subprocess.Popen(run_cmd,stdout=subprocess.PIPE,shell=True) (output,err)=process.communicate() exit_code = process.wait() print output print err print exit_code return exit_code to process a C++ executable program: run_exe_return_code('abc') while abc is created by the following C++ codes: int main() { return 1;...

Add more features to stack container


c++,visual-c++,stl
I am using default features(push, pop, top, empty, size) of stack container of STL. If I want to add more features like access an element from middle of stack. How could I do this? Thanks...

template template class specialization


c++,templates,template-specialization
I am just learning about Template Template class specialisation. Not a big problem to explain in detail. From my understanding std::uniform_int_distribution is a template whereas std::uniform_int_distribution<Type> is the full specialisation of uniform_int_distribution giving a type. I pass this in the specialisation class template as follows below Main class template <template...

Issue when use two type-cast operators in template class


c++
I define a template class in which, I define two type-cast operator template <class base_t> struct subclass { base_t base; //any function which defined for 'base_t' can be used with 'subclass<base_t>' operator base_t&() { return base; } //I want 'subclass<base_t>' can be converted to any class which 'base_t' can //I...

Type function that returns a tuple of chosen types


c++,templates,c++11,metaprogramming
I've implemented a type function Tuple that turn a list of My_enum values into an std::tuple of corresponding types: #include <tuple> enum My_enum{ t_int, t_double }; // Bind_type is a type function that given a My_enum returns the corresponding type template<My_enum E> struct Bind_type; template<> struct Bind_type<t_int>{ using type =...

Storing columns on disk and reading rows


c++,file,matrix,io
So I have a C++ code that finds the inverse of a square matrix. The inverse is full and it would be bad to keep it all in memory because I am working with hundreds of thousands of columns. My code generates the columns one by one. After finding the...

segfault accessing qlist element through an iterator


c++,iterator,qlist
I get a segfault while iterating over a QList. I don't understand what I am doing wrong. I have a QList of Conversation. Inside a Conversation I have a QList of Msg. Below are the class description : Msg class : class Msg { public: Msg(); Msg(const Msg& other); Msg&...

pointer to pointer dynamic array in C++


c++,arrays,pointers
I've been having bad luck with dynamic pointers when I want to close it. why the application wrote to memory after end of heap buffer? how can I close my array? int main() { . . int **W; W = new int* [n]; for (int i=1; i <= n; i++)...

Translating a character array into a integer string in C++


c++,arrays,string
I was trying to achieve translating a character array into a integer string and corresponding character to their alphabetical order. For instance: A(a) = 0 , Z(z) = 25. string key_char = argv[1]; string key_num; for (int i = 0; i < key_char.length(); i++){ if (isalpha(key_char[i])){ if (islower(key_char[i])){ key_num[i] =...

C++ template template


c++,templates
I'm trying to understand C++ template templates by implementing a generic container class. Here is the code: using namespace std; template <typename T, template <typename STORETYPE> class Container> class Store { public: ~Store() {}; Store() {}; void someFunc( const T & ) {}; //... private: Container<T> storage; }; int main(int...

How can I convert an int to a string in C++11 without using to_string or stoi?


c++,string,c++11,gcc
I know it sounds stupid, but I'm using MinGW32 on Windows7, and "to_string was not declared in this scope." It's an actual GCC Bug, and I've followed these instructions and they did not work. So, how can I convert an int to a string in C++11 without using to_string or...

How can I tell clang-format to follow this convention?


c++,clang-format
I would like to have this: if (!enabled) { return; } turned to this: if (!enabled) { return; } (In other words, I want short if-statements on a single line but keep the {} around them) Currently I'm using the following configuration: AllowShortIfStatementsOnASingleLine: true AllowShortLoopsOnASingleLine: true AllowShortCaseLabelsOnASingleLine: true AllowShortFunctionsOnASingleLine: true...

Passing something as this argument discards qualifiers


c++,c++11
Using the below code, i get the following compile error: In static member function ‘static std::string ctedata::Record::getDispatcher<std::basic_string<char> >::impl(const ctedata::Record&, const string&)’: /home/jason/CrownTheEmpire/lib/ctedata/data.h:111:38: error: passing ‘const std::map<std::basic_string<char>, std::basic_string<char> >’ as ‘this’ argument discards qualifiers [-fpermissive] return rec.fieldValues_[field]; ^ In file included from /usr/include/c++/5.1.0/map:61:0, from...

ctypes error AttributeError symbol not found, OS X 10.7.5


python,c++,ctypes
I have a simple test function on C++: #include <stdio.h> #include <string.h> #include <stdlib.h> #include <locale.h> #include <wchar.h> char fun() { printf( "%i", 12 ); return 'y'; } compiling: gcc -o test.so -shared -fPIC test.cpp and using it in python with ctypes: from ctypes import cdll from ctypes import c_char_p...

dispatch response packet according to packet sequence id


c++,boost,boost-asio
I have a third-part server, and I'm writing a dll interface for it, my clients use my dll to communicate with the server. The protocol uses a long tcp connection, all traffic goes from this tcp connection. There could be sending/receiving multiple packets at the same time, like a send_msg...

How can I access the members of a subclass from a superclass with a different constructor?


c++,inheritance,constructor,subclass,superclass
I have the following class and typedef: class Object { protected: long int id; public: Object(void); ~Object(void) {}; long int get_id(void); }; typedef map<string, Object> obj_map; And then I have its child: class Image: public Object { private: path full_path; int x; int y; img image; public: Image(path p, int...

Method returning std::vector>


c++
As a continuation of a: Thread, I came across a problem with writing a method of a class which returns: std::vector<std::unique_ptr<Object>> I get compiler errors when such a return type is written. There is some problem with delete operand or something ... Generally, I've wanted to write a method which...

Validate case pattern (isupper/islower) on user input string


c++,user-input
I need to write a program that checks if the user-provided first and last names are correctly typed. The program needs to validate that only the first letter of each name part is uppercase. I managed to write code that checks the first character of the input. So I have...

Test if string represents “yyyy-mm-dd”


c++,command-line-arguments
I am working on a program that takes two command line arguments. Both arguments should be dates of the form yyyy-mm-dd. Since other folks will be using this program and it will be requesting from mysql, I want to make sure that the command line arguments are valid. My original...

OpenCV - Detection of moving object C++


c++,opencv
I am working on Traffic Surveillance System an OpenCv project, I need to detect moving cars and people. I am using background subtraction method to detect moving objects and thus drawing counters. I have a problem : When two car are moving on road closely them my system detects it...

Explicit instantiation of class template not instantiating constructor


c++,templates,constructor,explicit-instantiation
I'm working on a project in C++ and am having trouble understanding what members of a template class get explicitly instantiated when I explicitly instantiate the template class. I've written the following file, which I then compile using Visual C++ 2008 Express Edition's Release configuration and then pop into a...