python,c++,boost,dictionary,destructor , Segfault from dict destructor


Segfault from dict destructor

Question:

Tag: python,c++,boost,dictionary,destructor

I have problem with boost python, base object destructor. When object boost::python::dict is created and destroyed in py_init scope then everything is fine. But in py_smth scope dict is only created success, after constructor executrion when local dict descructor is called then I have segmentation fault.

    class py_init
    {
    public:
        py_init::py_init()
        {
              Py_Initialize();
              object main_module = import("__main__");
              main_namespace = main_module.attr("__dict__");
              main_namespace["sys"] = import("sys');
              exec("sys.path.insert(0, "/foo/boo"), main_namespace, main_namespace);
        }
    object main_namespace;
    };


    class py_smth
    {
    public:
       py_smth(std::shared_ptr<py_init> py)
       {
             dict local;
       }

};

Backtrace:
Program terminated with signal SIGSEGV, Segmentation fault
#0  0x00007f501ddf9a38 in ?? () from /usr/lib/x86_64-linux-gnu/libpython3.4m.so.1.0
(gdb) bt
#0  0x00002b9545827a38 in ?? () from /usr/lib/x86_64-linux-gnu/libpython3.4m.so.1.0
#1  0x00002b95450ff12f in boost::python::api::object_base::~object_base() () from libplugin.so
#2  0x00002b95450ff060 in boost::python::api::object::~object() () from libplugin.so
#3  0x00002b9545101198 in boost::python::detail::dict_base::~dict_base() libplugin.so
#4  0x00002b9545101230 in boost::python::dict::~dict() libplugin.so

Libs version: -boost version 1.54 -python 3.4.2

And I don't know why...

Ok. this is sample code which is not working

main.cpp file:

#include <iostream>
#include <boost/python.hpp>
#include <memory>
#include "smth.h"

using namespace boost::python;

class py_init
{
public:
    py_init()
    {
             Py_Initialize();
             object main_module = import("__main__");
             main_namespace = main_module.attr("__dict__");
             main_namespace["sys"] = import("sys");
         main_namespace["time"] = import("time");
         main_namespace["threading"] = import("threading");
             exec("sys.path.insert(0, \"/foo/boo\")", main_namespace, main_namespace);

         exec("def foo():print (\"bla\"); time.sleep(1);" , main_namespace, main_namespace);
         exec("thread = threading.Thread(target=foo)" , main_namespace, main_namespace);
         exec("thread.start()" , main_namespace, main_namespace);
         state = PyEval_SaveThread();
    }
    ~py_init()
    {
          PyEval_RestoreThread(state);
          PyGILState_STATE gstate;
          gstate = PyGILState_Ensure();
          exec("thread.join()" , main_namespace, main_namespace);
          PyGILState_Release(gstate);
         std::cout<<"Py_init dest"<<std::endl;
    }

    object main_namespace;
    PyThreadState* state;
};

int main()
{
    std::shared_ptr<py_init> py(new py_init());
    smth s(py);
    s.print_foo();
    return 0;
}

smth.h

#ifndef SMTH_H_
#define SMTH_H_

#include <boost/python.hpp>
#include <memory>
using namespace boost::python;
class py_init;

class smth
{
public:
    smth(std::shared_ptr<py_init> py)
    {
        dict local;
    }
    void print_foo()
    {
        std::cout<<"FOO"<<std::endl;
    }
    ~smth()
    {
        std::cout<<"smth dest"<<std::endl;
    }
};

#endif 

Backtrace:

> Program terminated with signal SIGSEGV, Segmentation fault.
> #0  0x00007f9c5d787a38 in ?? () from /usr/lib/x86_64-linux-gnu/libpython3.4m.so.1.0 (gdb) bt
> #0  0x00007f9c5d787a38 in ?? () from /usr/lib/x86_64-linux-gnu/libpython3.4m.so.1.0
> #1  0x0000000000401a9d in boost::python::api::object_base::~object_base() ()
> #2  0x0000000000401922 in boost::python::api::object::~object() ()
> #3  0x0000000000401b50 in boost::python::detail::dict_base::~dict_base() ()
> #4  0x0000000000401bde in boost::python::dict::~dict() ()
> #5  0x0000000000401c08 in smth::smth(std::shared_ptr<py_init>) ()
> #6  0x0000000000401727 in main ()

Compilation:

g++ -std=c++0x -fPIC -I/usr/include/python3.4m -c main.cpp -o app.o g++ -std=c++0x app.o -lboost_python3 -lpython3.4m


Answer:

The program is invoking undefined behavior as the boost::python::dict object is being created and destroyed by a thread that does not hold the Global Interpreter Lock (GIL). If a thread is doing anything that affects reference counting of python managed object, then it needs to have acquired the GIL. To resolve this, acquire and release the GIL within the smth constructor.

smth::smth(std::shared_ptr<py_init> py)
{
  PyGILState_STATE gstate;
  gstate = PyGILState_Ensure(); // Acquire GIL.
  // Use scope to force destruction while GIL is held.
  {
    boost::python::dict local;
  } 
  PyGILState_Release(gstate); // Release GIL.
}

It may be worth considering using RAII classes to help manage the GIL. For example, with the following gil_lock class, when a gil_lock object is created, the calling thread will acquire the GIL. When the gil_lock object is destructed, it releases the GIL.

/// @brief RAII class used to lock and unlock the GIL.
class gil_lock
{
public:
  gil_lock()  { state_ = PyGILState_Ensure(); }
  ~gil_lock() { PyGILState_Release(state_);   }
private:
  PyGILState_STATE state_;
};

The smth constructor could then be written as:

smth::smth(std::shared_ptr<py_init> py)
{
  gil_lock lock;
  boost::python::dict local;
}

Here is an annotated version of the minimal original code. It accentuates that the caller does not hold the GIL once py_init's constructor returns. Thus, both the creation and destruction of the boost::python::dict results in undefined behavior.

class py_init
{
public:
  py_init()
  {
    Py_Initialize(); // Acquires GIL (1).
    // ... spawn Python thread that will compete for GIL.
    state = PyEval_SaveThread(); // Release GIL (0).
  }

  ~py_init()
  {
    PyEval_RestoreThread(state); // Acquire GIL (1). 
    PyGILState_STATE gstate;
    gstate = PyGILState_Ensure(); // Acquire GIL (2).
    // ...
    PyGILState_Release(gstate); // Release GIL (1).
  }

  PyThreadState* state;
};    

int main()
{
  std::shared_ptr<py_init> py(new py_init());
  // GIL not held.
  {
    // Modifying object without GIL; other thread may hold it.
    boost::python::dict local;
  }
} // ~py_init() acquires GIL.

In Python 3.4, one can also use PyGILState_Check() function to check if the calling thread holds the GIL. Per the documentation, it is mainly a helper/diagnostic function. For earlier versions, one can perform a similar check by directly accessing one of the Python globals:

_PyThreadState_Current == PyGILState_GetThisThreadState();

The PyThreadState_Get() function cannot be used for this type of diagnostics, as it issues a fatal error if no threads are holding the GIL, which is a valid state.


Related:


Add more features to stack container


c++,visual-c++,stl
I am using default features(push, pop, top, empty, size) of stack container of STL. If I want to add more features like access an element from middle of stack. How could I do this? Thanks...

represent an index inside a list as x,y in python


python,list,numpy,multidimensional-array
I have a list which contains 1000 integers. The 1000 integers represent 20X50 elements of dimensional array which I read from a file into the list. I need to walk through the list with an indicator in order to find close elements to each other. I want that my indicator...

How do I read this list and parse it?


python,list
I'm using requests and the output I get from the sites API is a list, I've been stuck trying to parse it to get the data from it. I use r = requests.get(urlas, params=params) r.json() to get the data I want. Here is a snippet of the list [{'relation_type': None,...

trying to understand LSH through the sample python code


python,similarity,locality-sensitive-hash
the concise python code i study for is here Question A @ line 8 i do not really understand the syntax meaning for "res = res << 1" for the purpose of "get_signature" Question B @ line 49 (SOLVED BY myself through another Q&A) "xor = r1^r2" does not really...

Python - Opening and changing large text files


python,replace,out-of-memory,large-files
I have a ~600MB Roblox type .mesh file, which reads like a text file in any text editor. I have the following code below: mesh = open("file.mesh", "r").read() mesh = mesh.replace("[", "{").replace("]", "}").replace("}{", "},{") mesh = "{"+mesh+"}" f = open("p2t.txt", "w") f.write(mesh) It returns: Traceback (most recent call last): File...

Type function that returns a tuple of chosen types


c++,templates,c++11,metaprogramming
I've implemented a type function Tuple that turn a list of My_enum values into an std::tuple of corresponding types: #include <tuple> enum My_enum{ t_int, t_double }; // Bind_type is a type function that given a My_enum returns the corresponding type template<My_enum E> struct Bind_type; template<> struct Bind_type<t_int>{ using type =...

How to use template within Django template?


python,html,django,templates,django-1.4
I have the django template like below: <a href="https://example.com/url{{ mylist.0.id }}" target="_blank"><h1 class="title">{{ mylist.0.title }}</h1></a> <p> {{ mylist.0.text|truncatewords:50 }}<br> ... (the actual template is quite big) It should be used 10 times on the same page, but 'external' html elements are different: <div class="row"> <div class="col-md-12 col-lg-12 block block-color-1"> *django...

Calling function and passing arguments multiple times


python,function,loops
I want to call the function multiple time and use it's returned argument everytime when it's called. For example: def myfunction(first, second, third): return (first+1,second+1,third+1) 1st call: myfunction(1,2,3) 2nd call is going to be pass returned variables: myfunction(2,3,4) and loop it until defined times. How can I do such loop?...

undefined reference to `vtable for implementation' error


c++,build,makefile
I wrote some c++ files and after compiling with out make file it works fine . But when using make file it pop out some errors . My codes are : include directory files : application.h #ifndef APPLICATION_H #define APPLICATION_H #include "employee.h" #include "employee_data.h" #include "employee.h" ...some defintions here... #endif...

How to change the IP address of Amazon EC2 instance using boto library


python,amazon-web-services,boto
How can I assign a new IP address (or Elastic IP) to an already existing AWS EC2 instance using boto library.

Displaying a 32-bit image with NaN values (ImageJ)


python,image-processing,imagej
I wrote a multilanguage 3-D image denoising ImageJ plugin that does some operations on an image and returns the denoised image as a 1-D array. The 1-D array contains NaN values (around the edges). The 1-D array is converted back into an image stack and displayed. It is simply black....

Implicit use of initializer_list


c++,c++11,initializer-list
§[dcl.init.list] 8.5.4/2: The template std::initializer_list is not predefined; if the header <initializer_list> is not included prior to a use of std::initializer_list — even an implicit use in which the type is not named (7.1.6.4) — the program is ill-formed. Does that mean this program is ill-formed? #include <vector> int main()...

How does the class_weight parameter in scikit-learn work?


python,scikit-learn
I am having a lot of trouble understanding how the class_weight parameter in scikit-learn's Logistic Regression operates. The Situation I want to use logistic regression to do binary classification on a very unbalanced data set. The classes are labelled 0 (negative) and 1 (positive) and the observed data is in...

The event loop is already running


python,python-3.x,pyqt,pyqt4
I have the following 5 files: gui.py # -*- coding: utf-8 -*- from PyQt4 import QtCore, QtGui try: _fromUtf8 = QtCore.QString.fromUtf8 except AttributeError: def _fromUtf8(s): return s try: _encoding = QtGui.QApplication.UnicodeUTF8 def _translate(context, text, disambig): return QtGui.QApplication.translate(context, text, disambig, _encoding) except AttributeError: def _translate(context, text, disambig): return QtGui.QApplication.translate(context, text, disambig)...

Sum of two variables in RobotFramework


python,automated-tests,robotframework
I have two variables: ${calculatedTotalPrice} = 42,42 ${productPrice1} = 43,15 I executed ${calculatedTotalPrice} Evaluate ${calculatedTotalPrice}+${productPrice1} I got 42,85,15 How can I resolve it?...

Count function counting only last line of my list


python,python-2.7
Count function counting only last line of my list N = int(raw_input()) cnt = [] for i in range(N): string = raw_input() for j in range(1,len(string)): if string[j] =='K': cnt.append('R') elif string[j] =='R': cnt.append('R') if string[0] == 'k': cnt.append('k') elif string[0] == 'R': cnt.append('R') print cnt.count('R') if I am giving...

Inserting a variable in MongoDB specifying _id field


python,mongodb,pymongo
I want to insert a variable, say, a = {1:2,3:4} into my database with a particular id "56". It is very clear from the docs that I can do the following: db.testcol.insert({"_id": "56", 1:2, 3:4}) However, I cannot figure out any way to insert "a" itself, specifying an id. In...

How to remove structure with python from this case?
python,python-2.7
How to remove "table" from HTML using python? I had case like this: paragraph = ''' <p>Lorem ipsum dolor sit amet, consectetur adipisicing elit. Quidem molestiae consequuntur officiis corporis sint.<br /><br /> <table> <tr> <td> text title </td> <td> text title 2 </td> </tr> </table> <p> lorem ipsum</p> ''' how...

C++11 Allocation Requirement on Strings


c++,string,c++11,memory,standards
I had heard that C++11 was going to require strings to be allocated in contiguous memory. I even thought I saw a stack overflow question on it, but I can't seem to find it. I know that in practice both gcc and Visual Studio do allocate strings contiguously, I'm just...

In sklearn, does a fitted pipeline reapply every transform?


python,scikit-learn,pipeline,feature-selection
Apologies if this is obvious but I couldn't find a clear answer to this: Say I've used a pretty typical pipeline: feat_sel = RandomizedLogisticRegression() clf = RandomForestClassifier() pl = Pipeline([ ('preprocessing', preprocessing.StandardScaler()), ('feature_selection', feat_sel), ('classification', clf)]) pl.fit(X,y) Now when I apply pl on a new set, pl.predict(X_classify); is RandomizedLogisticRegression going...

.cpp:23: error: cannot convert ‘std::string’ to ‘const char*’ for argument ‘1’ to ‘int atoi(const char*)’


c++,string
Here a basic code I'm trying to run But I'm having trouble with stoi (it's c++) I keep getting error: ‘stoi’ was not declared in this scope I tried atoi and strtol with this error .cpp:23: error: cannot convert ‘std::string’ to ‘const char*’ for argument ‘1’ to ‘int atoi(const char*)’...

C++ template template


c++,templates
I'm trying to understand C++ template templates by implementing a generic container class. Here is the code: using namespace std; template <typename T, template <typename STORETYPE> class Container> class Store { public: ~Store() {}; Store() {}; void someFunc( const T & ) {}; //... private: Container<T> storage; }; int main(int...

Identify that a string could be a datetime object


python,regex,algorithm,python-2.7,datetime
If I knew the format in which a string represents date-time information, then I can easily use datetime.datetime.strptime(s, fmt). However, without knowing the format of the string beforehand, would it be possible to determine whether a given string contains something that could be parsed as a datetime object with the...

Strange Behavior: Floating Point Error after Appending to List


python,python-2.7,behavior
I am writing a simple function to step through a range with floating step size. To keep the output neat, I wrote a function, correct, that corrects the floating point error that is common after an arithmetic operation. That is to say: correct(0.3999999999) outputs 0.4, correct(0.1000000001) outputs 0.1, etc. Here's...

Marshal struct in struct from c# to c++


c#,c++,marshalling
I have the following structures in C# and C++. C++: struct TestA { char* iu; }; struct TestB { int cycle1; int cycle2; }; struct MainStruct { TestA test; TestB test2; }; C#: [StructLayout(LayoutKind.Sequential, CharSet=CharSet.Ansi, Pack = 1)] internal struct TestA { [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 36)] private string iu; public...

Sort when values are None or empty strings python


python,list,sorting,null
I have a list with dictionaries in which I sort them on different values. I'm doing it with these lines of code: def orderBy(self, col, dir, objlist): if dir == 'asc': sorted_objects = sorted(objlist, key=lambda k: k[col]) else: sorted_objects = sorted(objlist, key=lambda k: k[col], reverse=True) return sorted_objects Now the problem...

How can I access the members of a subclass from a superclass with a different constructor?


c++,inheritance,constructor,subclass,superclass
I have the following class and typedef: class Object { protected: long int id; public: Object(void); ~Object(void) {}; long int get_id(void); }; typedef map<string, Object> obj_map; And then I have its child: class Image: public Object { private: path full_path; int x; int y; img image; public: Image(path p, int...

Replace nodejs for python?


python,node.js,webserver
i'm working in a HTML5 multiplayer game, and i need a server to sync player's movement, chat, battles, etc. So I'm looking for ways to use python instead nodejs, because i have I have more familiarity with python. The server is simple: var express = require('express'); var app = express();...

Same function with and without template


c++,c++11
I am trying to understand a piece of code of C++11. A class contains 2 functions as shown below: class abc { public: void integerA(int x); template<typename typ> void integerA(typ x); }; I am unable to understand benefit of declaring 2 same functions. Why not declare only one template function?...

Strings vs binary for storing variables inside the file format


c++,file,hdf5,dataformat
We aim at using HDF5 for our data format. HDF5 has been selected because it is a hierarchical filesystem-like cross-platform data format and it supports large amounts of data. The file will contain arrays and some parameters. The question is about how to store the parameters (which are not made...

How can I tell clang-format to follow this convention?


c++,clang-format
I would like to have this: if (!enabled) { return; } turned to this: if (!enabled) { return; } (In other words, I want short if-statements on a single line but keep the {} around them) Currently I'm using the following configuration: AllowShortIfStatementsOnASingleLine: true AllowShortLoopsOnASingleLine: true AllowShortCaseLabelsOnASingleLine: true AllowShortFunctionsOnASingleLine: true...

Find the tf-idf score of specific words in documents using sklearn


python,scikit-learn,tf-idf
I have code that runs basic TF-IDF vectorizer on a collection of documents, returning a sparse matrix of D X F where D is the number of documents and F is the number of terms. No problem. But how do I find the TF-IDF score of a specific term in...

pointer to pointer dynamic array in C++


c++,arrays,pointers
I've been having bad luck with dynamic pointers when I want to close it. why the application wrote to memory after end of heap buffer? how can I close my array? int main() { . . int **W; W = new int* [n]; for (int i=1; i <= n; i++)...

Translating a character array into a integer string in C++


c++,arrays,string
I was trying to achieve translating a character array into a integer string and corresponding character to their alphabetical order. For instance: A(a) = 0 , Z(z) = 25. string key_char = argv[1]; string key_num; for (int i = 0; i < key_char.length(); i++){ if (isalpha(key_char[i])){ if (islower(key_char[i])){ key_num[i] =...

Python recursive function not recursing


python,recursion
I'm trying to solve a puzzle, which is to reverse engineer this code, to get a list of possible passwords, and from those there should be one that 'stands out', and should work function checkPass(password) { var total = 0; var charlist = "abcdefghijklmnopqrstuvwxyz"; for (var i = 0; i...

How do variables inside python modules work?


python,module,python-module
I am coming from a Java background with Static variables, and I am trying to create a list of commonly used strings in my python application. I understand there are no static variables in python so I have written a module as follows: import os APP_NAME = 'Window Logger' APP_DATA_FOLDER_PATH...

Why are shaders and programs stored as integers in OpenGL?


c++,opengl,opengl-es,integer,shader
I'm following the "OpenGL Superbible" book and I can't help but notice that when we create a shader and create the program that we attach the shaders to, we store them as GLuint which are unsigned integers. Why are they stored as numbers? What does the value of the number...

Using counter on array for one value while keeping index of other values


python,collections
After reading the answers on this question How to count the frequency of the elements in a list? I was wondering how to count the frequency of something, and at the same time retreive some extra information, through something like an index. For example a = ['fruit','Item#001'] b = ['fruit','Item#002']...

group indices of list in list of lists


python,list
I am looking for an elegant solution for the following problem. I have a list of ints and I want to create a list of lists where the indices with the same value are grouped together in the order of the occurrences of said list. [2, 0, 1, 1, 3,...

Pandas - Dropping multiple empty columns


python,pandas
I have some tables where the first 11 columns are populated with data, but all columns after this are blank. I tried: df=df.dropna(axis=1,how='all') which didn't work. I then used: df = df.drop(df.columns[range(11,36)], axis=1) Which worked on the first few tables, but then some of the tables were longer or shorter...

Pandas Dataframe Complex Calculation


python,python-2.7,pandas,dataframes
I have the following dataframe,df: Year totalPubs ActualCitations 0 1994 71 191.002034 1 1995 77 2763.911781 2 1996 69 2022.374474 3 1997 78 3393.094951 I want to write code that would do the following: Citations of currentyear / Sum of totalPubs of the two previous years I want something to...

dispatch response packet according to packet sequence id


c++,boost,boost-asio
I have a third-part server, and I'm writing a dll interface for it, my clients use my dll to communicate with the server. The protocol uses a long tcp connection, all traffic goes from this tcp connection. There could be sending/receiving multiple packets at the same time, like a send_msg...

Python: histogram/ binning data from 2 arrays.


python,histogram,large-files
I have two arrays of data: one is a radius values and the other is a corresponding intensity reading at that intensity: e.g. a small section of the data. First column is radius and the second is the intensities. 29.77036614 0.04464427 29.70281027 0.07771409 29.63523525 0.09424901 29.3639355 1.322793 29.29596385 2.321502 29.22783249...

Matplotlib: Plot the result of an SQL query


python,sql,matplotlib,plot
from sqlalchemy import create_engine import _mssql from matplotlib import pyplot as plt engine = create_engine('mssql+pymssql://**:****@127.0.0.1:1433/AffectV_Test') connection = engine.connect() result = connection.execute('SELECT Campaign_id, SUM(Count) AS Total_Count FROM Impressions GROUP BY Campaign_id') for row in result: print row connection.close() The above code generates an array: (54ca686d0189607081dbda85', 4174469) (551c21150189601fb08b6b64', 182) (552391ee0189601fb08b6b73', 237304) (5469f3ec0189606b1b25bcc0',...

Undefined behaviour or may be something with memset


c++,undefined-behavior
I was trying to save the binary equivalent of a 32 bit number in an array A. For testing my showbits() function , I choosed 8,9 when I came across this thing: I am facing an unreasonable thing in my code when I am placing memset in the function showbits(),I...

Method returning std::vector>


c++
As a continuation of a: Thread, I came across a problem with writing a method of a class which returns: std::vector<std::unique_ptr<Object>> I get compiler errors when such a return type is written. There is some problem with delete operand or something ... Generally, I've wanted to write a method which...

How to check for multiple attributes in a list


python,python-2.7
I am making a TBRPG game using Python 2.7, and i'm currently making a quest system. I wanted to make a function that checks all of the quests in a list, in this case (quests), and tells you if any of of the quests in the list have the same...

Twilio Client Python not Working in IOS Browser


javascript,python,ios,flask,twilio
I have created a simple twilio client application to make phone calls from Web Browser to phones. I used a sample Flask app to generate a secure Capability Token and used twilio.min.js library to handle calls from my HTML. The functionality works fine in Computer Browsers ans Android Phone Browsers,...

SQLAlchemy. 2 different relationships for 1 column


python,sqlalchemy
I have a simple many-to-many relationship with associated table: with following data: matches: users: users_mathces: ONE user can play MANY matches and ONE match can involve up to TWO users I want to realize proper relationships in both "Match" and "User" classes users_matches_table = Table('users_matches', Base.metadata, Column('match_id', Integer, ForeignKey('matches.id', onupdate="CASCADE",...

How can I convert an int to a string in C++11 without using to_string or stoi?


c++,string,c++11,gcc
I know it sounds stupid, but I'm using MinGW32 on Windows7, and "to_string was not declared in this scope." It's an actual GCC Bug, and I've followed these instructions and they did not work. So, how can I convert an int to a string in C++11 without using to_string or...