multithreading,perl , problems copying shared hash in perl threads


problems copying shared hash in perl threads

Question:

Tag: multithreading,perl

I encountered what I feel is strange behavior of shared hash in perl and needed some help understanding it.

The actual problem is in a far larger code-base and I have tried reducing it to smaller reproducible script.

So essentially the problem I'm facing is I have a shared variable that looks something on these lines :

 my %headers :shared= map { lc($_) => $custom_headers->{$_} }  keys %{$custom_headers};   
 my %task1_request :shared; 
 $task1_request{count} = $count;
 $task1_request{header} = \%headers if(keys %headers);

i.e I end up passing a reference to a shared variable headers to two separate threads

Each of these threads performs "read-only" operations on the reference to hash "headers".

However it looks like while passing a copy of the shared hash to a function in the thread , as shown in example below :

iterate_header($request->{count},%{$request->{header}});

sub iterate_header
{
    my $count = shift;
    my $current_count = scalar(@_);
    if($count != $current_count) {
      print STDERR "Test failed Expected: $count, Actual : $current_count \n";
    }
    else {
      print STDERR "Test passed\n" ;
    }
}

results in copied hash being corrupt i.e The @_ in the iterate_header is corrupt .

It looks to me something on the lines of the iterator being global for a shared hash as a result the copy is not thread safe. However the above is just a reckless assumption on my part and I hope someone can help clarify as to why copying a shared hash is causing this seemingly strange behavior and if this is expected ?

The reproducer script is below :

use strict;
use warnings;
use threads;
use threads::shared;
use Thread::Queue;

#should run test_count * 2 times
sub iterate_header
{
    my $count = shift;
    my $current_count = scalar(@_);
    if($count != $current_count) {
      print STDERR "Test failed Expected: $count, Actual : $current_count \n";
    }
    else {
      print STDERR "Test passed\n" ;
    }
}

sub request_loop {
    my ($request_queue) = @_;

    # wait for the next reuest...
    while (defined(my $request = $request_queue->dequeue())) {
        my %result :shared;
        if(exists($request->{header})) {
            iterate_header($request->{count},%{$request->{header}});
        }
        last if(exists($request->{exit}));
        $result{is_success} = "200";
    }
}

# Main program
# create thread queues
my $task1_request_queue = Thread::Queue->new();   
my $task2_request_queue = Thread::Queue->new();    

# start worker threads
my $task1_worker = threads->create(\&request_loop, $task1_request_queue);
my $task2_worker = threads->create(\&request_loop, $task2_request_queue);

# a high number to ensure tests fail
 my $test_count = 100; 
 my $custom_headers = {
        "key" => "558193F28878E5FE",
        "username" => "Mastodon",
        "real_username" => "Mastodon",
        "type" => "EMPLOYEE",
        "expiration" => "1434556278",
        "env" => "save it",
        "for" => "some ip",
        "long-string" => "This islong string",
        "state" => "internal",
        "account" => "home",
        "original_account" => "home",
        "key" => "MCwCFAPOE74uvXso5alKytqjlfpdqeY4AhRpDeIMLCAk3ciBcyDXLdnyZjC/7Q==",
        "charset" => "iso-8859-1,*,utf-8",
        "agent" => "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.166 Workstation/2013.9.213.116 Safari/535.19",
        "accept" => "application/json, text/plain, */*",
        "encoding" => "gzip,deflate",
        "language" => "en-us,en",
        "cookie" => "TS01375c99=012e7f4fa1e82941689f22669e2e6403ce1c75f9f8c7cb86de86c19a887f61a1109c6e2aae",
        "created" => "1434555378",

    };


my @data = %{$custom_headers};
my $count = scalar(@data);
print STDERR  "Expected Count for all tests:$count\n";
for(my $i = 0;$i < 2; $i++) {
  my %headers :shared= map { lc($_) => $custom_headers->{$_} }  keys %{$custom_headers};   
  #add to task1 q
    {    

        my %task1_request :shared; 
        $task1_request{count} = $count;

        $task1_request{header} = \%headers if(keys %headers);

        $task1_request_queue->enqueue(\%task1_request);
    }

    # add to task2 q
    {
        my %task2_request :shared; 
        $task2_request{count} = $count;

        $task2_request{header} = \%headers if(keys %headers);
        $task2_request_queue->enqueue(\%task2_request);
    }
}

my %end_request :shared = (exit => 1);
$task1_request_queue->enqueue(\%end_request);
$task2_request_queue->enqueue(\%end_request);

$task1_worker->join();
$task2_worker->join();
print "testing done\n";

Example output of a test runs:

[]$ perl thread_shared_issue.pl
Expected Count for all tests:36
Test passed
Test passed
Test passed
Test passed
testing done
[]$ perl thread_shared_issue.pl
Expected Count for all tests:36
Test failed Expected: 36, Actual : 16
Test failed Expected: 36, Actual : 60
Test failed Expected: 36, Actual : 18
Test failed Expected: 36, Actual : 56
testing done

Perl Version Tested with

perl -version

This is perl 5, version 12, subversion 5 (v5.12.5) built for x86_64-linux-thread-multi

Answer:

Two threads are iterating over the same hash at the same time, so they are both changing its iterator. You need to make sure that no more than one thread uses the hash's iterator at a time.

I'd remove all those :shared and use Thread::Queue::Any.


Related:


Version-dependent fallback code


perl
I have a script that needs to run on multiple servers, however, each server may not have the same version of Perl available and may have differing features. Perl v5.14 introduced the /r modifier for regular expressions which returns the result of a substitution and leaves the original text alone....

Multiple Threads searching on same folder at same time


c#,multithreading,file-search
Currently I have a .txt file of about 170,000 jpg file names and I read them all into a List (fileNames). I want to search ONE folder (this folder has sub-folders) to check if each file in fileNames exists in this folder and if it does, copy it to a...

How can I know the lock information in java?


java,multithreading,locking
Is there any tool or way that can get all the information about the locks in java? for example, if there is a java program, it creates two threads, and both threads require locks for some variable. Is there any tools that can output the information like which thread locks...

Does wait() need synchronization on local variable


java,multithreading,synchronization
I had this code (which was working fine): public static void runOnUiThread(Activity c, final Runnable action) { // Check if we are on the UI Thread if (Looper.getMainLooper() == Looper.myLooper()) { // If we are, execute immediately action.run(); return; } // Else run the runnable on the UI Thread and...

How to calculate how many times each thread executed a critical section in OpenMP?


c,multithreading,openmp
I have an OpenMP code, where I need to calculate how many times each thread executes the critical section, any idea how to do it? Code samples are highly welcomed.

How to pass a hash as optional argument to -M in command line


perl,hash,package,command-line-interface
I know that when we need to pass some arguments to the use keyword after a package name we can pass them in the command line after the -M parameter. For example: use feature 'say'; say 'hello!'; can be invoked from the command line with >perl -Mfeature=say -e"say 'hello!'" But...

Looking to pause a thread using thread.sleep


java,multithreading
I am using a mouse listener for mouse pressed and released. When the mouse is pressed I want to have a counter incrementing a variable, and when the mouse is released I want to decrement that variable. Right now, my code is working and does that but the increment is...

EXC_BAD_ACCESS error occurring when running recursive merge sort method in a thread


c++,multithreading,sorting,recursion
I'm having trouble with my C++ code in xCode. I've tried debugging and I can't fix it. I know it's roughly when my mergeSort() method calls my mergeNumbers() method. I know it's not the method itself because I've run the method without threading and it works just fine. It's when...

Create unicode character with pack


perl,unicode
I am trying to understand how Perl handles unicode. use feature qw(say); use strict; use warnings; use Encode qw(encode); say unpack "H*", pack("U", 0xff); say unpack "H*", encode( 'UTF-8', chr 0xff ); Output: ff c3bf Why do I get ff and not c3bf when using pack ?...

performance issues executing list of stored procedures


c#,multithreading,performance,loops
I'm having some performance issues when starting my windows service, the first round my lstSps is long (about 130 stored procedures). Is there anyway to speed this up (except for speeding the stored procedures up)? When the foreach is over and goes over to the second round it goes faster,...

std::condition_variable – notify once but wait thread wakened twice


c++,multithreading
Here's a simple C++ thread pool implementation. It's an altered version orginated from https://github.com/progschj/ThreadPool. #ifndef __THREAD_POOL_H__ #define __THREAD_POOL_H__ #include <vector> #include <queue> #include <memory> #include <thread> #include <chrono> #include <mutex> #include <condition_variable> #include <future> #include <functional> #include <stdexcept> namespace ThreadPool { class FixedThreadPool { public: FixedThreadPool(size_t); template<class F, class......

java multithreading start() and run() [duplicate]


java,multithreading
This question already has an answer here: Java: What's the difference between Thread start() and Runnable run() 10 answers Below is my Multithreading class: public class Multithreading extends Thread{ public void run(){ for(int i=1;i<5;i++){ try{ Thread.sleep(500); }catch(InterruptedException e){ System.out.println(e); } System.out.println(i); } } public static void main(String args[]) {...

c - Parallelising a function


c,multithreading,table,rainbowtable
I'm trying to implement parallelism to this function I want it to take as many threads as possible, and write the results to a file. The results need to be written in the file in the incrementing order so the first result needs to be written first the second second...

Images not appearing on WPF form when loading asynchronously


c#,wpf,multithreading,listbox,backgroundworker
I'm attempting to display (in a ListBox with a custom DataTemplate) a series of BitmapSource frames (thumbnails) extracted from a multi-page tiff image. When I process the tiff on the UI thread, and either directly add the images to a listbox's item collection or add them to a bound ObservableCollection,...

Crystal convert the idea behind Thread pool to Fibers/spawn


multithreading,coroutine,crystal-lang
I'm having some hard time learning the idea behind Fibers\coroutines and the implementation in Crystal. I hope this is the right place to ask this, I'll totally accept a "not here" answer :) This is my usual way of handling multi-threading in Ruby: threads = [] max_threads = 10 loop...

Perl Debugging Using Flags


perl,debugging,script-debugging
So my goal is to find an easy way to turn on print statements in Perl a flag. In C/C++ you can use a #define to choose if certain code is run and it is a way to turn on and off debug print statements. Where if a #define DEBUG...

How can we make a thread to sleep for a infinite time in java?


java,multithreading,thread-sleep
Pls help me to understand how can we make a thread to sleep for a infinite time period .

Node.js and C/C++ integration: how to properly implement callbacks?


c++,node.js,multithreading
I am trying to implement a C++ extension to be integrated with node.js. This extension will internally invoke some blocking calls, so it needs to provide a non-blocking interface to the node.js world. As specified in https://nodejs.org/api/addons.html, there are two ways to implement non-blocking callbacks: a) By using a simple...

unable to understand qr interpolation


regex,perl
I was reading Programming Perl where I learned qr interpolation of strings as regex as : $re = qr/my.STRING/is; print $re; # prints (?si-xm:my.STRING) and it says The /s and /i modifiers were enabled in the pattern because they were supplied to qr//. The /x and /m, however, are disabled...

Regex in Perl Uninitialized $1


regex,perl
My string looks like this: <File `../Path/To/My_File.gif'> I want to extract just "Path/To/My_File.gif". Here is the check I have: if ($row =~ /(?<=File `..\/).*(?=')/) { print "Found it!\n"; print "$1\n"; } I see "Found it!" printed to the console but also get an error saying that $1 is uninitialized. What...

What does this horribly ugly (yet somehow secretly beautiful) Perl code do?


perl,formatting,deobfuscation
I found this code in a Powerpoint presentation about Perl scripting and this was on a page demonstrating how ugly Perl code can get if you really wanted to make it that way. The presentation says nothing about what this code does. Honestly I am just very curious to know......

Web API - Set each thread with the HttpRequestMessage id?


c#,.net,multithreading,task-parallel-library,web-api
I have a web api coded in c#. The web api uses functionality which is shared with other in-house components. it depends on single threaded flows and uses thread local storage to store objects, and session information. Please don't say if it's good or bad, that's what I have to...

Atomic/not-atomic mix, any guarantees?


c++,multithreading,c++11,atomic
Let's I have GUI thread with code like this: std::vector<int> vec; std::atomic<bool> data_ready{false}; std::thread th([&data_ready, &vec]() { //we get data vec.push_back(something); data_ready = true; }); draw_progress_dialog(); while (!data_ready) { process_not_user_events(); sleep_a_little(); } //is it here safe to use vec? As you see I not protect "vec" by any kind of...

Any way to catch an exception occurring on a thread I don't own?


c#,multithreading,exception
I'm using a crappy third party library because unfortunately I have no choice. This library creates a bunch of threads internally when I call it, and occasionally throws a NullReferenceException on one of these threads. Is there any way for me to catch these exceptions even though I don't own...

Capture tee's argument inside piped Perl execution


perl,unix
How to capture piped command's argument ? I use : perl my_script.pl -some_args | tee arg_filename How to get arg_filename 's value inside my_script.pl ? CONTEXT I need to send this filename in a mail which my_script.pl sends at the end. I need to use tee because we dump huge...

Set Label From Thread


vb.net,multithreading,winforms
Form1.vb Imports System.Threading Public Class Form1 Dim demoThread As Thread Private Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click Dim Start As New Class1 Me.demoThread = New Thread( _ New ThreadStart(AddressOf Start.ThreadProcSafe)) Me.demoThread.Start() End Sub Delegate Sub SetTextCallback([text] As String) Public Sub SetText(ByVal [text] As String) ' InvokeRequired required...

Is there standard implementation for thread block/resume in java SE?


java,multithreading,wait
I need to block execution of a thread until resumed from another thread. So I wrote my own implementation using wait() method. Which seems to be working, but it is far from simple. Is there any ready to use solution? Preferably in java SE 6? Or do I have to...

C# Threading : Threads executed on incorrect object


c#,multithreading
Probably this is a dumb mistake by me, but I can't figure it out. So the scenario is that I retrieve a record from a database every 200ms, if there is one available. On each record, I start a thread. In this case it's a mail that is sent. QMail...

Opening multiple files in perl array


arrays,perl
I have a perl script where by I assigned all the files with a .log extension to an array called @allfiles. How do I run my script for the files stored in each array? My idea is something like open(my $fn, '<', @allfiles) or die "Could not open file '@files':...

wait for an event regulary


multithreading,events
In a program I need to wait for an event (keypress) and get it's char. after that program will continue. this progress will Repeat several time. in my first try, codes run and any character did not save. I find out I should use threading but I am not Familiar...

How does the kernel separate threads from processes


linux,multithreading,linux-kernel
Suppose I have a browser process like Firefox, that has pid = 123. Firefox has 5 opened tabs each running in a separate thread, so in total it has 5 threads. So I want to know in depth, how the kernel will separate the process into the thread to execute...

Why does this code catch block not execute?


c++,multithreading,error-handling,try-catch
The catch handler is not run. But why? If the thread t is started before the try block, the catch handler runs. If the catch block's type does not match the type thrown, the program exits explaining that the thread terminated with an uncaught exception, suggesting that the exception is...

Why Filter::Indent::HereDoc complain when blank line in middle of HereDoc


perl,heredoc
I am trying Filter::Indent::HereDoc which allows one to indent the HereDocument. This is very useful, to be able to have HereDoc that flows with the code logic. From the above link When a 'here document' is used, the document text and the termination string must be flush with the left...

What happens if all node.js's worker threads are busy


javascript,node.js,multithreading
I try to understand how node.js works and although I have read this article: When is the thread pool used? I am not sure what happens if all worker threads are busy and another async I/O operation is ready to be executed. If I got this http://www.future-processing.pl/blog/on-problems-with-threads-in-node-js/ article right, the...

Command line arguments in Perl


perl
I am working on an open source project for GSoC and I have this piece of Perl code with me. I need to create another Perl file for a similar task. However, I am having trouble understanding 3 lines of this file. More specifically, I am not able to understand...

calling cgi script from other cgi script


perl,cgi
I found a puzzling behavior using perl cgi for which -- I guess -- there is a perfectly valid explanation but I couldn't find one. There is this question on stackoverflow, but what it describes seems to fail for me. The situation: I have two perl cgi scripts. One of...

Java how to limit number of threads acting on method


java,multithreading,memory-management
I have java method in my web application doing heavy file operation. The thing is, if more than 5 threads come simultaneously (which will come in testing phase) it breaks down. I mean it cannot handle heavy traffic. That's why I want to handle maximum 5 requests at a time...

Java 5 Multi threading, catch thread exceptions


java,multithreading,exception,concurrency,java-5
I've got a class that generates threads (file i/o). I need to catch exceptions in the thread - I don't want to do anything fancy, I want to kill the main thread, rather, stop processing altogether so it can start over. If I catch the exceptions in the thread, that's...

Can you call dispatch_sync from a concurrent thread to itself without deadlocking?


ios,objective-c,multithreading,swift,grand-central-dispatch
I know you would deadlock by doing this on a serial queue, but I haven't found anything that mentions deadlocking by doing it on a concurrent queue. I just wanted to verify it wont deadlock (it doesn't seem like it would, as it would only block one of the threads...

Check for decimal point and add it at the end if its not there using awk/perl


regex,perl,shell,awk
I have test.dat file with values given below: 20150202,abc,,,,3625.300000,,,,,-5,,,,,,,,,,,,,,,,,,,,,, 20150202,def,,,,32.585,,,,,0,,,,,,,,,,,,,,,,,,,,,, 20150202,xyz,,,,12,,,,,0.004167,,,,,,,,,,,,,,,,,,,,,, My expected output is shown below: 20150202,abc,,,,3625.300000,,,,,-5.,,,,,,,,,,,,,,,,,,,,,, ^. added here 20150202,def,,,,32.585,,,,,0.,,,,,,,,,,,,,,,,,,,,,, ^. added here 20150202,xyz,,,,12.,,,,,0.004167,,,,,,,,,,,,,,,,,,,,,, ^. added here So if column 6 and 11 doesn't have decimal point in it, then we should add '.' at the end of...

Multi-Threading error when binding a StringProperty


java,multithreading,javafx,javafx-8
I have a question about multi-threading and the binding of a StringProperty. I have a class CacheManager, which contains a Thread which updates my cache with the changes on the server. Now, I want to notify the user with a text and percentage of the progress (which are a Label...

Calling dispatch_sync from a concurrent queue - does it block entirely?


ios,objective-c,multithreading,swift,grand-central-dispatch
Let's say I hypothetically call a dispatch_sync from a concurrent queue - does it block the entire queue or just that thread of execution?

Perl : Display perl variable awk sed echo


perl
When I am using below command directly its working fine but when I am trying to put this in perl script its giving lots of error. my $calculate = `echo "$value" | awk -F "SP=" '{print $2}' | awk -F ";" '{print $1}' | awk -F ":" '{print $2}' |...

Looping variables


perl,scripting
I'm working with perl to make a script that will work with Dot products/assorted vector math. I've got a working script ( Still very much in progress/needs refinement ) that will do what I ask. #!/usr/bin/perl use strict; use warnings; use diagnostics; use Math::Vector::Real; use 5.010; use Math::Trig; my $source...

Find numbers in a file and change their value with perl


regex,perl
I have a file with some data in it but there are a bunch of annoying numbers that are less than one which I wanted to just change to 1 instead of manually doing it. I was wondering how you would do this in perl. I tried using something like...

Why this exclusion not working for long sentences?


text-processing,perl
Command perl -ne 'print unless /.[240,]/' input.txt > output.txt which includes some sentences which are longer than 240 letters. Why? Example data Development of World Funny Society program on young people who are working hard for the sport and social life such that they have time to go pizzeria every...

-M Script start time minus file modification time, in days


perl,perldoc
I would expect -M $_ to be negative, but it is zero for: perl -E 'qx(touch $_), sleep(5), say -M for "/tmp/file"' Does perldoc mentions such behavior?...

Invoke form showdialog is not modal


vb.net,multithreading,invoke
I have 2 forms, 1 MainForm and 1 Form2. I am trying to display Form2 as a modal form and background from MainForm. Here's what I have so far. The default MainForm appears and after 5 seconds it will show Form2 as a Modal form from a background thread. I...

Perl: Using Text::CSV to print AoH


arrays,perl,csv
I have an array of hashes (AoH) which looks like this: $VAR1 = [ { 'Unit' => 'M', 'Size' => '321', 'User' => 'test' } { 'Unit' => 'M' 'Size' => '0.24' 'User' => 'test1' } ... ]; How do I write my AoH to a CSV file with separators,...

How to match and remove the content preceding it from a file in unix [closed]


mysql,perl,sed,solaris
I have a mysql dump file, and i want to remove the content of the file after "-- Final view structure for view view_oss_user" using sed/perl. The input file is something like this : Content : rom `target` */; /*!50001 SET character_set_client = @saved_cs_client */; /*!50001 SET character_set_results = @saved_cs_results...