c++,file,matrix,io , Storing columns on disk and reading rows


Storing columns on disk and reading rows

Question:

Tag: c++,file,matrix,io

So I have a C++ code that finds the inverse of a square matrix. The inverse is full and it would be bad to keep it all in memory because I am working with hundreds of thousands of columns. My code generates the columns one by one. After finding the inverse, the rest of my code requires the rows one by one. Ideally, I would like to proceed as follows:

Repeat: 1) find a column 2) store it on the disk 3) delete the column from memory.

Repeat: 1) read a row from the disk 2) process it (it is basically the objective function of a linear program) 3) delete the row from memory.

One way of doing that is writing the matrix in the file as it is.. Meaning, I write the first column as an "actual" column in the file, then I write the second next to it (which would be a bit less efficient), and so on. The last column will take O(n^2) to write if I'm not mistaken. Doing this makes reading the rows very easy though. The other way is to write the columns as rows then reading the columns later on.. But again, the last column will take O(n^2) to read. A third approach is to use n files, but opening and closing n files is inefficient.

Any ideas on how to approach such a problem? Maybe I have to use databases (possibly SQL) to make my reading and writing O(1) per entry?

Thank you.


Answer:

Mentioned solution with fseek is good. However, it can be very slow for large matrices (as disks don't like random access, especially very far away). To speed up things, you should use blocking.

I'll show a basic concept, and can explain it further if you need.

First, you split your matrix into blocks of equal size, like so: Split1

Block size should be calculated in such way that column or row of blocks can fit into RAM.

Next, you keep column of blocks in memory, and copy here first generated blocksize columns. Here yellow part is what you keep in memory, and orange - your next generated column: SplitColumns

After you finish with the first blocksize columns, dump them into separate files (for my image, you can name such files "A1", "A7" and "A13"), and start again with the next column of blocks.

When you need this matrix as rows later, invert this procedure. Read from prepared files (for example, "A1", "E1", "I1" and "M1") first row of blocks, and process it row by row:

SplitRows


Related:


How can I tell clang-format to follow this convention?


c++,clang-format
I would like to have this: if (!enabled) { return; } turned to this: if (!enabled) { return; } (In other words, I want short if-statements on a single line but keep the {} around them) Currently I'm using the following configuration: AllowShortIfStatementsOnASingleLine: true AllowShortLoopsOnASingleLine: true AllowShortCaseLabelsOnASingleLine: true AllowShortFunctionsOnASingleLine: true...

Undefined behaviour or may be something with memset


c++,undefined-behavior
I was trying to save the binary equivalent of a 32 bit number in an array A. For testing my showbits() function , I choosed 8,9 when I came across this thing: I am facing an unreasonable thing in my code when I am placing memset in the function showbits(),I...

Parameters to use in a referenced function c++


c++,pointers,reference
I am very confused as to what kind of variables I would put into my function here: names. I am doing a practice problem in a C++ book, because I am learning C++ and am on References and pointers right now, and cannot find a solution. Just for background information,...

Copy text and placeholders, variables to the clipboard


c++,qt,clipboard
In my application I want generate random numbers or strings with a text in front of it. It is important for me that the text won't appear in my window, but instead gets copied to the clipboard. int randomnumber = rand() % 46 + 1; QClipboard *cb = QApplication::clipboard(); cb->setText("Just...

Changing file name to the user's name PHP


php,file,upload
Instead of the file name being what the users want to name it, I would like the file name to be the user's username. As you can see in the code below, I got the user's username from the session. Now, I would just like to apply it to $_FILES,...

Passing something as this argument discards qualifiers


c++,c++11
Using the below code, i get the following compile error: In static member function ‘static std::string ctedata::Record::getDispatcher<std::basic_string<char> >::impl(const ctedata::Record&, const string&)’: /home/jason/CrownTheEmpire/lib/ctedata/data.h:111:38: error: passing ‘const std::map<std::basic_string<char>, std::basic_string<char> >’ as ‘this’ argument discards qualifiers [-fpermissive] return rec.fieldValues_[field]; ^ In file included from /usr/include/c++/5.1.0/map:61:0, from...

Confused about returns in stack template


c++,templates,generic-programming
I'm implementing a generic stack (with an array) in C++ and am confused about what to return in this situation: template <class T> T Stack<T>::pop(void) { if (size != 0) { return items[size - 1]; size--; } else { cerr << "Cannot pop from empty stack." << endl; return ???;...

Type function that returns a tuple of chosen types


c++,templates,c++11,metaprogramming
I've implemented a type function Tuple that turn a list of My_enum values into an std::tuple of corresponding types: #include <tuple> enum My_enum{ t_int, t_double }; // Bind_type is a type function that given a My_enum returns the corresponding type template<My_enum E> struct Bind_type; template<> struct Bind_type<t_int>{ using type =...

No match for 'operator*' error


c++,c++11
Hello fellow programmers! I was going to write a small program for calculating total pay for different periods of time depending on the amount of hours and the salary that the user enters. I managed to make a small bit of the program but when I try to run it...

OpenCV - Detection of moving object C++


c++,opencv
I am working on Traffic Surveillance System an OpenCv project, I need to detect moving cars and people. I am using background subtraction method to detect moving objects and thus drawing counters. I have a problem : When two car are moving on road closely them my system detects it...

IllegalStateException: Iterator already obtained [duplicate]


java,file,loops,path
This question already has an answer here: java.lang.IllegalStateException: Iterator already obtained 1 answer so I wrote a little Java program to test a little stack language I made vie various test file, but for some reason it won't work. Here is the code: import org.apache.commons.io.FilenameUtils; import java.io.IOException; import java.nio.file.*;...

dispatch response packet according to packet sequence id


c++,boost,boost-asio
I have a third-part server, and I'm writing a dll interface for it, my clients use my dll to communicate with the server. The protocol uses a long tcp connection, all traffic goes from this tcp connection. There could be sending/receiving multiple packets at the same time, like a send_msg...

Explicit instantiation of class template not instantiating constructor


c++,templates,constructor,explicit-instantiation
I'm working on a project in C++ and am having trouble understanding what members of a template class get explicitly instantiated when I explicitly instantiate the template class. I've written the following file, which I then compile using Visual C++ 2008 Express Edition's Release configuration and then pop into a...

Validate case pattern (isupper/islower) on user input string


c++,user-input
I need to write a program that checks if the user-provided first and last names are correctly typed. The program needs to validate that only the first letter of each name part is uppercase. I managed to write code that checks the first character of the input. So I have...

Translating a character array into a integer string in C++


c++,arrays,string
I was trying to achieve translating a character array into a integer string and corresponding character to their alphabetical order. For instance: A(a) = 0 , Z(z) = 25. string key_char = argv[1]; string key_num; for (int i = 0; i < key_char.length(); i++){ if (isalpha(key_char[i])){ if (islower(key_char[i])){ key_num[i] =...

C++11 Allocation Requirement on Strings


c++,string,c++11,memory,standards
I had heard that C++11 was going to require strings to be allocated in contiguous memory. I even thought I saw a stack overflow question on it, but I can't seem to find it. I know that in practice both gcc and Visual Studio do allocate strings contiguously, I'm just...

segfault accessing qlist element through an iterator


c++,iterator,qlist
I get a segfault while iterating over a QList. I don't understand what I am doing wrong. I have a QList of Conversation. Inside a Conversation I have a QList of Msg. Below are the class description : Msg class : class Msg { public: Msg(); Msg(const Msg& other); Msg&...

How can I access the members of a subclass from a superclass with a different constructor?


c++,inheritance,constructor,subclass,superclass
I have the following class and typedef: class Object { protected: long int id; public: Object(void); ~Object(void) {}; long int get_id(void); }; typedef map<string, Object> obj_map; And then I have its child: class Image: public Object { private: path full_path; int x; int y; img image; public: Image(path p, int...

C++ Isn't this a useless inline declaration?


c++,inline,private,member,protected
This is another question about inlining a function. But I will take possible comments and answers right away: Defining a function inside a class makes it inline automatically. The same behaviour can be achieved by marking a function with inline outside of the class. An inline function doesn't have to...

create vector of objects on the stack ? (c++)


c++,vector,heap-memory
I am creating a temporary vector of pointers to myObject objects. But I am wondering about what happens to the objects I created... { std::vector<myObject *> myVector; myVector.reserve(5); for (int i = 0 ; i < 5 ; ++i){ myVector[i] = new myObject(); } } I assume that at the...

std::condition_variable – notify once but wait thread wakened twice


c++,multithreading
Here's a simple C++ thread pool implementation. It's an altered version orginated from https://github.com/progschj/ThreadPool. #ifndef __THREAD_POOL_H__ #define __THREAD_POOL_H__ #include <vector> #include <queue> #include <memory> #include <thread> #include <chrono> #include <mutex> #include <condition_variable> #include <future> #include <functional> #include <stdexcept> namespace ThreadPool { class FixedThreadPool { public: FixedThreadPool(size_t); template<class F, class......

3 X 3 magic square recursively


c++,algorithm,math,recursion
I'm trying to find all possible solutions to the 3X3 magic square. There should be exactly 8 solutions. My code gets them all but there are a lot of repeats. I'm having a hard time tracking the recursive steps to see why I'm getting all the repeats. // This program...

pointer to pointer dynamic array in C++


c++,arrays,pointers
I've been having bad luck with dynamic pointers when I want to close it. why the application wrote to memory after end of heap buffer? how can I close my array? int main() { . . int **W; W = new int* [n]; for (int i=1; i <= n; i++)...

Can python script know the return value of C++ main function in the Android enviroment


python,c++
There are several ways of calling C++ executable programs. For example, we can use def run_exe_return_code(run_cmd): process=subprocess.Popen(run_cmd,stdout=subprocess.PIPE,shell=True) (output,err)=process.communicate() exit_code = process.wait() print output print err print exit_code return exit_code to process a C++ executable program: run_exe_return_code('abc') while abc is created by the following C++ codes: int main() { return 1;...

Checking value of deleted object


c++
I asked a question: Detecting if an object is still active or it has been destroyed Considering that I cannot use libraries, there are no good out of the box solutions in C++. So, is it a bad practice to check if the object has been destroyed by analyzing memory...

Delphi - Use a string variable's name in assignfile()


file,delphi,variables,assign
Is it possible to use a variable in the assignfile command? Eg. f : Textfile ; sFile : string ; {contains 'MyFile.txt' as content} ... cFileDir = 'C:\Users\User\Desktop\Data Engine\Data\Country' ; ... Assignfile(f, cFileDir + '\' + sFile) ; ... I appreciate your help very much. if it's unclear I'll edit...

Implicit use of initializer_list


c++,c++11,initializer-list
§[dcl.init.list] 8.5.4/2: The template std::initializer_list is not predefined; if the header <initializer_list> is not included prior to a use of std::initializer_list — even an implicit use in which the type is not named (7.1.6.4) — the program is ill-formed. Does that mean this program is ill-formed? #include <vector> int main()...

Test if string represents “yyyy-mm-dd”


c++,command-line-arguments
I am working on a program that takes two command line arguments. Both arguments should be dates of the form yyyy-mm-dd. Since other folks will be using this program and it will be requesting from mysql, I want to make sure that the command line arguments are valid. My original...

Add more features to stack container


c++,visual-c++,stl
I am using default features(push, pop, top, empty, size) of stack container of STL. If I want to add more features like access an element from middle of stack. How could I do this? Thanks...

Marshal struct in struct from c# to c++


c#,c++,marshalling
I have the following structures in C# and C++. C++: struct TestA { char* iu; }; struct TestB { int cycle1; int cycle2; }; struct MainStruct { TestA test; TestB test2; }; C#: [StructLayout(LayoutKind.Sequential, CharSet=CharSet.Ansi, Pack = 1)] internal struct TestA { [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 36)] private string iu; public...

C++ & Qt: Random string from an array area


c++,arrays,string,qt,random
In my small Qt application, I want to pick a random string out of an array after I clicked on a button. I've read many threads but nothing works for me. So in my slot there's an array with several strings in it. I also implemented <string>, <time.h> and srand....

Issue when use two type-cast operators in template class


c++
I define a template class in which, I define two type-cast operator template <class base_t> struct subclass { base_t base; //any function which defined for 'base_t' can be used with 'subclass<base_t>' operator base_t&() { return base; } //I want 'subclass<base_t>' can be converted to any class which 'base_t' can //I...

Why are shaders and programs stored as integers in OpenGL?


c++,opengl,opengl-es,integer,shader
I'm following the "OpenGL Superbible" book and I can't help but notice that when we create a shader and create the program that we attach the shaders to, we store them as GLuint which are unsigned integers. Why are they stored as numbers? What does the value of the number...

Make a triangle shape in C++


c++
I am trying to print out the shape of a triangle but I am kinda lost... this is what I have so far: #include <iostream> using namespace std; int main() { int i, k, n; cout << "Please enter number of rows you want to see: \n"; cin >> n;...

Passing iterator's element to a function: wrong type of pointer


c++,pointers,stl,iterator
I'm attempting to solve Project Euler's problem #3 using C++ to gain an understanding of how to use C++ iterators. According to the examples I've seen online, I can use the dereferened iterator as a parameter for cout, and it will print the elements successfully. By that same logic, I...

Incorrect Polar - Cartesian Coordinate Conversions. What does -0 Mean?


c++,polar-coordinates,cartesian-coordinates
I am getting incorrect conversions from polar to cartesian coordinates and vice versa. My code produces weird points like (1,-0). Im using this calculator to check my conversions. Also one of the conversions is completely wrong when I convert back to cartesian coordinates. Point b: (0,1) => (1,1.5708) => (0,0)...

C++ template template


c++,templates
I'm trying to understand C++ template templates by implementing a generic container class. Here is the code: using namespace std; template <typename T, template <typename STORETYPE> class Container> class Store { public: ~Store() {}; Store() {}; void someFunc( const T & ) {}; //... private: Container<T> storage; }; int main(int...

template template class specialization


c++,templates,template-specialization
I am just learning about Template Template class specialisation. Not a big problem to explain in detail. From my understanding std::uniform_int_distribution is a template whereas std::uniform_int_distribution<Type> is the full specialisation of uniform_int_distribution giving a type. I pass this in the specialisation class template as follows below Main class template <template...

Same function with and without template


c++,c++11
I am trying to understand a piece of code of C++11. A class contains 2 functions as shown below: class abc { public: void integerA(int x); template<typename typ> void integerA(typ x); }; I am unable to understand benefit of declaring 2 same functions. Why not declare only one template function?...

ctypes error AttributeError symbol not found, OS X 10.7.5


python,c++,ctypes
I have a simple test function on C++: #include <stdio.h> #include <string.h> #include <stdlib.h> #include <locale.h> #include <wchar.h> char fun() { printf( "%i", 12 ); return 'y'; } compiling: gcc -o test.so -shared -fPIC test.cpp and using it in python with ctypes: from ctypes import cdll from ctypes import c_char_p...

Strings vs binary for storing variables inside the file format


c++,file,hdf5,dataformat
We aim at using HDF5 for our data format. HDF5 has been selected because it is a hierarchical filesystem-like cross-platform data format and it supports large amounts of data. The file will contain arrays and some parameters. The question is about how to store the parameters (which are not made...

opencv window not refreshing at mouse callback


c++,opencv
I am trying to draw with mouse move in an opencv window. But when I draw, nothing draws on the window. When I try to close the window from the cross in the topleft(ubuntu), it opens a new window which it should be as I haven't pressed escape, and in...

Method returning std::vector>


c++
As a continuation of a: Thread, I came across a problem with writing a method of a class which returns: std::vector<std::unique_ptr<Object>> I get compiler errors when such a return type is written. There is some problem with delete operand or something ... Generally, I've wanted to write a method which...

MFC visual c++ LNK2019 link error


c++,mfc
I just don't understand why i can use the public variables on the class but are getting a link error when trying to use the getLicenceRefused method. I wasn't sure if the problem was because of the CString copy constructor problem I have had before so took the parameter out,...

How can I convert an int to a string in C++11 without using to_string or stoi?


c++,string,c++11,gcc
I know it sounds stupid, but I'm using MinGW32 on Windows7, and "to_string was not declared in this scope." It's an actual GCC Bug, and I've followed these instructions and they did not work. So, how can I convert an int to a string in C++11 without using to_string or...

Get an ordered list of files in a folder


c++,boost,boost-filesystem
I have used boost::filesystem::directory_iterator in order to get a list of all the available files into a given folder. The problem is that I supposed this method would give me the files in alphabetical order, while the results seem pretty random. Is there any fancy way of alphabetically sorting them?...

how to sort this vector including pairs


c++,vector
I want to sort in ascending order according to the first element of the inner pair, i.e. a in this case. But its not at all sorting. I am not sure if my function func logic is correct. #include<iostream> #include<algorithm> #include<vector> using namespace std; bool func(const pair<int,pair<int,int> >&i , const...

undefined reference to `vtable for implementation' error


c++,build,makefile
I wrote some c++ files and after compiling with out make file it works fine . But when using make file it pop out some errors . My codes are : include directory files : application.h #ifndef APPLICATION_H #define APPLICATION_H #include "employee.h" #include "employee_data.h" #include "employee.h" ...some defintions here... #endif...

.cpp:23: error: cannot convert ‘std::string’ to ‘const char*’ for argument ‘1’ to ‘int atoi(const char*)’


c++,string
Here a basic code I'm trying to run But I'm having trouble with stoi (it's c++) I keep getting error: ‘stoi’ was not declared in this scope I tried atoi and strtol with this error .cpp:23: error: cannot convert ‘std::string’ to ‘const char*’ for argument ‘1’ to ‘int atoi(const char*)’...