FAQ Database Discussion Community


Using Task with Parallel.Foreach in .NET 4.0

c#,multithreading,.net-4.0,parallel-processing
I started off trying to add a progress bar to the windows form that updates the progress of code running within a Parallel.Foreach loop. In order to do this the UI thread has to be available to update the progress bar. I used a Task to run the Parallel.Foreach loop...

Parallelization doesn't work with the foreach package

r,foreach,parallel-processing,parallel.foreach
Using the foreach package, I was expecting the following line to run in about 10 seconds system.time(foreach (i=1:5, .combine='c') %do% {Sys.sleep(2);i}) user system elapsed 0.053 0.011 10.012 and the following line to run in about 2 seconds system.time(foreach (i=1:5, .combine='c') %dopar% {Sys.sleep(2);i}) user system elapsed 0.069 0.017 10.019 but it...

Basic Java threading (4 threads) slower than non-threading

java,multithreading,concurrency,parallel-processing
I have a four core CPU. I create 4 threads and run a cpu intensive loop, and it takes > 4x longer than running it all procedurally in one thread. I created two projects to compare, one with threading and one without. I'll show the code and run times. Just...

Reduce by key on device array

cuda,parallel-processing,thrust
I am using reduce_by_key to find the number of elements in an array of type int2 which has same first values . For example Array: <1,2> <1,3> <1,4> <2,5> <2,7> so no. elements with 1 as first element are 3 and with 2 are 2. CODE: struct compare_int2 : public...

F# how to do List.map in parallel

parallel-processing,f#
What is the most lightweight, terse way to run the following code in parallel within the standard F# libs? Or failing that any widely used additional libs? let newlist = oldlist |> List.map myComplexFunction The best I could find was let newlist = oldlist |> List.map (fun x -> async...

Java 8, using .parallel in a stream causes OOM error

java,parallel-processing,java-8,java-stream
In the book Java 8 In Action, section 7.1.1, the authors state that a stream can benefit from parallel processing by adding the function .parallel(). They provide a simple method called parallelSum(int) to illustrate this. I was curious to see how well it worked so I executed this code: package...

To get debug outputs on client - spmd

matlab,parallel-processing,spmd
I am running a parallelized code courtesy the MATLAB Parallel Computing Toolbox using the spmd command. Specifically, the code is like this: spmd out = function(data,labindex); end Now the function involves a library (libsvm) which gives me a trained classifier for each iteration. During the training process, there are several...

How does MATLAB assign values to a variable and print to file in a parfor loop

arrays,matlab,file-io,parallel-processing,parfor
I have a parfor loop that loops over i = 1:250, gets three values from an array according to the index. Then a calculation is being made within the parfor loop using those 3 values. I then store the results in the variables theta1, theta2, and theta3, and immediately print...

What happens if one OpenMP thread crashes?

multithreading,parallel-processing,openmp
Consider the following case of a parallel for/do-loop: PARALLEL DO thread 1 thread 2 line 1 line 1 line k -> line k -> line l line l line n line n Now, thread 1 encounters an exception or an error (segmentation fault) on line l and terminates. What will...

C# parallelism in LINQ

c#,parallel-processing
I would like to know if this is a safe way of calculating x in the code below. public static IEnumerable<Object> Parse(Object obj, ref int x) { x += 10; return new List<Object>(); } public static void Query(List<Object> obj) { int x = 0; var result = obj .AsParallel() .Select(o...

Python's multiprocessing: speed up a for-loop for several sets of parameters, “apply” vs. “apply_async”

python,performance,parallel-processing,python-multiprocessing,odeint
I would like to integrate a system of differential equations using a lot of different parameter combinations and store the variables’ final values that belong to a certain set of parameters. Therefore, I implemented a simple for-loop in which random initial conditions and parameter combinations are created, the system is...

Multiple connections to SQLite with simultaneous async Tasks

android,sqlite,android-asynctask,parallel-processing
In my scenario it has Sync_Class that syncs, with AsyncTasks in background , from my app to my server. Every time that my app does one action that need to change data from my SQLite, as first step my app updates my local database as second step throws a AsyncTask...

Parallel processing mix up

c#,parallel-processing
I am new to C# programming. I am trying to get the number of updates fror a list of servers using background worker. Result for every server is shown in a listview at the report progress method. I am able to successfully get results using foreach loop, but while trying...

C++, Running two independent function in parallel

c++,multithreading,parallel-processing,collaborative
I have two methods that I want to execute in parallel, they are completely independent and do not share memory, the only common point between them are the parameters. As soon as one of the method has finished its job, it is not necessary for the other to continue, since...

Can OpenMPI guarantee order of received message come from the same process?

c,linux,parallel-processing,mpi,openmpi
For example, I use mpirun -n 4 to start 4 processes. Process 0 receives messages come from process1, process2 and process3. Process 1 sends messages in the order of message0, message1, message2. When process 0 receives these messages come from process1, can it guarantee that process receives these messages in...

Queue using several processes to launch bash jobs

bash,shell,parallel-processing,queue
I need to run many (hundreds) commands in shell, but I only want to have a maximum of 4 processes running (from the queue) at once. Each process will last several hours. When a process finishes I want the next command to be "popped" from the queue and executed. I...

Collect results from parallel stream

java,parallel-processing,java-8,java-stream
I have a piece of code like this: List<Egg> eggs = hens.parallelStream().map(hen -> { ArrayList<Egg> eggs = new ArrayList<>(); while (hen.hasEgg()) { eggs.add(hen.getEgg()); } return eggs; }).flatMap(Collection::stream).collect(Collectors.toList()); But in this way I have to create an ArrayList for every hen, and eggs are not collected until a hen is 100%...

MPJ Express eclipse - remove combination of letters

java,parallel-processing,mpi,mpj-express
I have to do an exercise for a parallel computing course. The task is using N parallel processes to remove all combinations of letters "RTY" from the string. Normally I'll do it with String strAfter=str1.replaceAll("[RTY]","") ; But how to make it in parallel? ...

Is dynamic scheduling better or static scheduling (Parallel Programming)?

multithreading,parallel-processing,openmp,scheduling
I understand my question title is rather broad, I am new to parallel programming and openmp. I tried to parallelize a C++ solution for the N-body problem and study it for different schedule types and granularity. I collected data by running program for different cases and plotted the data, this...

Proper way to handle a group of asynchronous calls in parallel

c#,parallel-processing,async-await,task
I have a console app that calls a web api and gets a list of services. It then loops through and makes calls to each service. I have the following: static int Main(string[] args) { ... Task.WaitAll(Process()); } private static async Task BeginProcess() { using(HttpClientHandler handler = new HttpClientHandler()) {...

Running java application parallely on multicore cluster nodes

java,multithreading,concurrency,parallel-processing
I have a Java program that calculates semantic similarity between two documents. The program retrieves documents from a specified file system and calculates similarity. There are around 2,00,000 such docs. I have created 10 threads for this task and I have assigned data chucks to each of the thread. For...

fork() understanding, about execution and children

unix,parallel-processing,fork
I'm just starting to fork() and I'm having some difficulties understanding the parallel execution. I've found this example code and I want to know if the first time it will go true or false (I know if pid1==0 it means it's a child, etc). I also want to know how...

Run Serial inside Paralell Bash

linux,bash,while-loop,parallel-processing,wait
I have added to my explanation a bit. Conceptually, I am running a script that processes in a loop, calling shells that use the line content as an input parameter.(FYI: a kicks off an execution and b monitors that execution) I am needing 1a and 1b to run first, in...

Java sequential implementation is 4 times faster than parallel implementation

java,multithreading,parallel-processing,threadpool,executorservice
I created a really simple scenario where I recognized a really weird behavior which I cant understand. Under following link I created an sequential implementation: http://ideone.com/B8JYeA Basically there are several big arrays with fixed size. The algorithm iterates through them and changes the value. for(int i = 0; i <...

Barrier after MPI non-blocking call, without bookkeeping?

c++,parallel-processing,mpi,cluster-computing,hpc
I'm doing a bunch of MPI_Iallreduce non-blocking communications. I've added these Iallreduce calls to several different places in my code. Every so often, I want to pause and wait for all the Iallreduce calls to finish. Version 1 with MPI_Request bookkeeping -- this works: MPI_Request requests[]; MPI_Iallreduce(..., requests[0]); ... MPI_Iallreduce(...,...

Is my ExpirableLazyObject not thread safe? Is there a race condition?

java,multithreading,concurrency,parallel-processing
A while back, I created a Java class that encapsulates lazy initialization into a LazyObect. It maintains concurrency and threadsafety and also provides a reset() method to clear the value. public final class LazyObject<T> { private volatile T value; private volatile boolean updated; private final Supplier<T> supplier; private LazyObject(Supplier<T> supplier)...

OpenMP over Summation

parallel-processing,fortran,openmp,fortran90,gfortran
I have been trying to apply OpenMP on a simple summation operation inside two nested loop, but it produced incorrect result so far. I have been looking around in here and here, also in here. All suggest to use reduction clause, but it does not work for my case by...

how to run two while loops at the same time in Python?

python,parallel-processing,python-multithreading
I am new to Python and using Zelle's graphics to create a game. I need the two while loops below to run at the same time, however I am running into difficulty. I tried nesting the while loops but then the horses and civilians would only move if the mouse...

Parallel in Java 7 with a result, using non-final variables

java,parallel-processing,executorservice
I have a something like: int [] array1 = new int[20]; int [] array2 = new int[20]; int total= 0; Random generator = new Random(); for(int i = 0; i < 10000; i++){ int tmp = generator.nextInt(20); boolean win = custom_function(...); array1[tmp]++; array2[tmp]++ total++; } // do something with the...

How to perform synchronous parallel functions in Clojure?

clojure,parallel-processing
I have an application which has an initial flow that can be done in parallel: Fetch two JSON documents (I use clj-http for that) Parse these documents (extract only required data) Join the results Dump them into a one file So there's something like that: some-entry-point /\ / \ /...

Multitenented Parallel Processing

java,multithreading,tomcat,parallel-processing,threadpool
I have a tomcat server with a handful of fairly long-running operations that I feel could be greatly sped up if broken up and processed in parallel. What is the best practice as far as this is concerned? I am concerned that if I just create some threadpool that is...

How to simultaneously launch a job 2 times on the same slave in Jenkins?

jenkins,parallel-processing,jenkins-plugins
I have Jenkins installed on my computer which configure to have just the master as a node (no other nodes) and number of executes is 5. I created a Job named "myJob" and I want to run it on the master 2 times at the simultaneously (meaning that if I...

Julia parallel programming - Making existing function available to all workers

for-loop,parallel-processing,julia-lang
I am faced with the following problem: I have a function called TrainModel that runs for a very long time on a single thread. When it finishes computing, it returns a function as an output argument, let's call it f. When I enquire the type of this f, Julia returns:...

Parallel.ForEach loop is performing like a serial loop

c#,tfs,parallel-processing,invalidoperationexception,parallel.foreach
I've spent about 8+ hours searching online for help and I couldn't find anything so, here goes. I'm working with Team Foundation Server and C# and I'm trying to acquire a list of work items and convert them into a generic object we made to be bound to a special...

How to reduce processing time for imread and imwrite

matlab,image-processing,parallel-processing,imread
I am working on a 3d reconstruction using parallel computing in Matlab. I profiled my code and found that imread and imwrite took most of the processing time. My goal is to significantly reduce the processing time as my simulations involve large data sets and iterations. My code: projection_length =...

Running multiple background processes in script

shell,unix,parallel-processing
I have a shell script of the following format : cd /path/a/ make cd /path/b/ make cd /path/c/ make Each of the make commands are independent of each other in different directories. I would like to speed up the script by running the different make commands in background using '&'...

Node async callback being called for operations complete

javascript,node.js,asynchronous,parallel-processing
I am trying to use the async library in order to deal with asynchronous functions, and am unable to get the functionality I need. The basic idea is that I am building a JSON object using multiple calls to a redis database. I want to only return the JSON object...

Spark: Different output with different number of cores

scala,parallel-processing,apache-spark,cpu-cores
I'm dealing with a strange behaviour when I change the number of cores in my Spark Application and here is the code: import org.apache.log4j.{Level, Logger} import org.apache.spark.{SparkContext, SparkConf} object Test extends App { Logger.getLogger("org").setLevel(Level.WARN) var listLink: List[String] = List() def addListLink(s: String) = { val list = s.split(",") for (i...

Parallel Speedup and Efficiency

multithreading,performance,parallel-processing
Just have a quick question. What is the difference between a parallel speedup and efficiency. Thanks...

Microsoft Release Management : option to switch deployment order in a tag does not exist?

parallel-processing,tags,sequence,ms-release-management
On this page it's described that "Now you can use tags with the servers in your Azure or on-premises (standard) environments. ...With tags you can also switch the deployment order from parallel to sequence." See this picture: However in our version from MRM (ReleaseManagementConsole.exe ; 12.0.31101.0 ; Nov 1 2014)...

using “foreach” for running different classifiers in R

r,foreach,parallel-processing
I am trying to use foreach to run different classifiers on my data, but it doesn't work. In fact it doesn't return me anything. my purpose is to parallelize my process. here is the simplified of my code: library(foreach) library(doParallel) no_cores <- detectCores() - 1 cl<-makeCluster(no_cores) registerDoParallel(cl) registerDoParallel(no_cores) model_list<-foreach(i =...

running a thread in parallel

c#,.net,multithreading,parallel-processing
I am inside a threat updating a graph and I go into a routine that makes a measurement for 4 seconds. The routine returns a double. What I am noticing is that my graph stops showing activity for 4 seconds until I am done collecting data. I need to start...

OpenMP specify thread number of a for loop iteration

c++,multithreading,parallel-processing,openmp
I'm using the following command to parallelize a single loop over the available threads of the program: #pragma omp parallel for num_threads(threads) for(long i = 0; i < threads; i++) { array[i] = calculateStuff(i,...); } For technical reasons, I would like to guarantee that thread number 0 executes i=0, and...

How to run DEoptim in parallel?

r,parallel-processing
have developed a hydrological model in R, which has 8 parameters in the function, and I wish to calibrate the model using DEoptim. Due to the length of time it takes to run each function I would like to parallel-ise the DEoptim function as it has an option to run...

OpenMP shared variable seems to be private

c,parallel-processing,openmp
I don't understand why in this code only the thread 0 has n = 1 while the other ones have n = 0 with shared n: int main() { int n, tid; #pragma omp parallel shared(n) private(tid) { tid = omp_get_thread_num(); n = 0; if (tid == 0) { n++;...

Rx extensions Parallel.ForEach throttling

c#,parallel-processing,system.reactive,reactiveui
I'm following the answer to this question: Rx extensions: Where is Parallel.ForEach? in order to run a number of operations in parallel using Rx. The problem I'm running into is that it seems to be allocating a new thread for every request, whereas using Parallel.ForEach did considerably fewer. The processes...

OpenCL barrier of finding max in a block

parallel-processing,max,opencl,gpgpu,barrier
I've found a piece of OpenCL kernel sample code in Nvidia's developer site The purpose function maxOneBlock is to find out the biggest value of array maxValue and store it to maxValue[0]. I was fully understand about the looping part, but confused about the unroll part: Why the unroll part...

MATLAB parfor delete values?

matlab,parallel-processing,parfor
with this code: s=struct([]); for i=1:5 s(i).m=i; s(i) end it's all ok.. the output is as expected ans = m: 1 ans = m: 2 ans = m: 3 ans = m: 4 ans = m: 5 using a parfor without accessing like this s=struct([]); parfor i=1:5 s(i).m=i; end seems...

Tricky way of doing foreach in C#

c#,foreach,parallel-processing
I will first provide the pseudocode and describe it below: public void RunUntilEmpty(List<Job> jobs) { while (jobs.Any()) // the list "jobs" will be modified during the execution { List<Job> childJobs = new List<Job>(); Parallel.ForEach(jobs, job => // this will be done in parallel { List<Job> newJobs = job.Do(); // after...

Parallel computing efficiency - Number of cores or MPI processes?

parallel-processing,processing-efficiency
I'm testing a parallel program on a 2 + 2 (hyper threading) cores machine. The parallelization is effected through MPI routines. While evaluating the efficiency of the parallel code, do I have to divide the speed up by the number of cores or the speed up by the number of...

array traversing: parallel performance is slower than non parallel

c#,performance,parallel-processing,parallel.for
In my program i want to detemine how many numbers have 9 digits how, many have 8 digits, etc. with this loop: for (int i = 0; i < 60000000; i++) { if (a[i] >= 1000000000) { p[10] += 1; } else if (a[i] >= 100000000) { p[9] += 1;...

Parallel disk writes C#

c#,parallel-processing
UPDATE I am totally aware that this is not going to add much benefit and the disk is going to be the bottleneck (although my disk can do 20k IOPS of 32K chunks) so please consider this before you decide to down vote. ============ I need to write the same...

C# not waiting for an async method to finish [duplicate]

c#,multithreading,parallel-processing
This question already has an answer here: Fire-and-forget with async vs “old async delegate” 4 answers I have a specific place in my code, where I am sending an email message. Well, it is not important at all here to wait for this method to be finished, this is...

Race condition with scipy.weave.inline

python,parallel-processing,scipy,race-condition
Recently I've begun to receive SyntaxErrors when running parallel neural-network simulations using brian2. These are being raised by calls to scipy.weave.inline when it tries to evaluate lines of code in a cache file. The full description of the problem and my guess at its cause is here. And here is...

Task.StartNew() vs Parallel.ForEach : Multiple Web Requests Scenario

c#,multithreading,parallel-processing,task-parallel-library,parallel.foreach
I have read through all the related questions in SO, but a little confused on the best approach for my scenario where multiple web service calls are fired. I have an aggregator service that takes an input, parses and translates it into multiple web requests, makes the web request calls...

Reactive pipeline - how to control parallelism?

c#,.net,parallel-processing,system.reactive
I'm building a straightforward processing pipeline where an item is fetched as an input, it is being operated by multiple processors in a sequential manner and finally it is output. Image below describes the overall architecture: The way it is currently working: Pipeline is fetching items from the provider as...

Async await usage for MongoDB repository

c#,mongodb,asynchronous,parallel-processing,async-await
I have a MongoDB repository class as you see below: public class MongoDbRepository<TEntity> : IRepository<TEntity> where TEntity : EntityBase { private IMongoClient client; private IMongoDatabase database; private IMongoCollection<TEntity> collection; public MongoDbRepository() { client = new MongoClient(); database = client.GetDatabase("Test"); collection = database.GetCollection<TEntity>(typeof(TEntity).Name); } public async Task Insert(TEntity entity) { if...

clEnqueueNDRangeKernel fills up entire memory

c++,memory,parallel-processing,opencl
I am currently trying to write an OpenCL application doing some memory intensive calculations. To track the progress of all the calculations I created a for loop which creates different kernel groups. Unfortunately, the calculation fills up my whole memory. My guess is that the kernels are not done executing...

How to use a M-.file in a parfor loop in Matlab?

matlab,parallel-processing,parfor
my code inside a parfor loop is getting longer and longer and I would like to split it up. Having parts of the code saved in different script files seems logical and very attractive and since it doesn't change anything except where the code is saved it seems like it...

How can a loop involving recursive multiplication and accumulation be vectorized?

matlab,parallel-processing,vectorization
In matlab, I have a loop of the form: a=1; for (i = 1:N) a = a * b(i) + c(i); end Can this loop be vectorized, or partially unrolled?...

Node JS Asynchronous Database Calls

javascript,mysql,node.js,asynchronous,parallel-processing
I am having issues getting node to make a database call without proceeding despite the database function has not returned a value. Here is the basic http server code: var http = require('http'); http.createServer(function (request, response) { response.writeHead(200, { 'Content-Type': 'text/plain', 'Access-Control-Allow-origin': '*' // implementation of CORS }); response.end("ok"); ;...

Java 8 parallelStream for concurrent Database / REST call

java,multithreading,concurrency,parallel-processing,java-8
Here I am using Javaparallel stream to iterate through a List and calling a REST call with each list element as input. I need to add all the results of the REST call to a collection for which I am using an ArrayList. The code given below is working fine...

MPI Send latency for different process localities

parallel-processing,mpi,multicore,supercomputers
I am currently participating in a course for efficient programming of supercomputers and multicore processors. Our recent assignment is to measure the latency for the MPI_Send command (thus the time spent sending a zero byte message). Now this alone would not be that hard, but we have to perform our...

Multiple levels of parallelization with scikit-learn

python,parallel-processing,machine-learning,scikit-learn,cross-validation
I am using scikit-learn's RandomForestClassifier on a multi-core sever to fit a large dataset so I am taking advantage of its parallelization feature by setting n_jobs = -1. Simultaneously, I want to perform cross-validation (with cross_val_score) and make use of its parallelization feature as well (again, by setting n_jobs =...

What's the difference between parallel cost and parallel work?

performance,algorithm,optimization,parallel-processing,gpgpu
I read a paper in which parallel cost for (parallel) algorithms is defined as CP(n) = p * TP(n), where p is the number of processors, T the processing time and n the input. An algorithm is cost-optimal, if CP(n) is approximately constant, i.e. if the algorithm uses two processors...

Algorithm for [inclusive/exclusive]_scan in parallel proposal N3554

c++,algorithm,parallel-processing,c++14
Proposal N3554 (A Parallel Algorithms Library) for C++14, proposes (among other things), what seem to be parallel versions of the current std::partial_sum, e.g.: template< class ExecutionPolicy, class InputIterator, class OutputIterator, class BinaryOperation> OutputIterator inclusive_scan( ExecutionPolicy &&exec, InputIterator first, InputIterator last, OutputIterator result, BinaryOperation binary_op); With the explanation Effects: For each...

Can I read 1 big CSV file in parallel in R? [duplicate]

r,multithreading,csv,parallel-processing
This question already has an answer here: R: Is it possible to parallelize / speed-up the reading in of a 20 million plus row CSV into R? 3 answers I have a big csv file and it takes ages to read. Can I read this in parallel in R...

Call methods parallel and combine results

c#,parallel-processing
I am sorry if this is a duplicate question. Please guide me to it if it is. I have this urgent scenario. I have a MainMethod which needs to call two methods Method1 and Method2 parallel. Both of them will return list of Employee but from different database. I need...

How to measure if a program was run in parallel over multiple cores in Linux?

linux,parallel-processing,benchmarking,perf
I want to know if my program was run in parallel over multiple cores. I can get the perf tool to report how many cores were used in the computation, but not if they were used at the same time (in parallel). How can this be done?...

how to generalize square matrix multiplication to handle arbitrary dimensions

c,cuda,parallel-processing,matrix-multiplication
I have written this program and I am having some trouble understanding how to use multiple blocks by using dim3 variable in the kernel call line. This code works fine when I am doing 1000*1000 matrix multiplication, but not getting correct answer for lower dimensions like 100*100 , 200*200. #include...

Concatenating parallel streams

java,parallel-processing,java-8,java-stream
Suppose that I have two int[] arrays input1 and input2. I want to take only positive numbers from the first one, take distinct numbers from the second one, merge them together, sort and store into the resulting array. This can be performed using streams: int[] result = IntStream.concat(Arrays.stream(input1).filter(x -> x...

R - Parallelization in EasyABC. Error: … could not find function

r,parallel-processing
I am trying to run the ABC_sequential() function from the package EasyABC in parallel in R. But I am getting the error: Error in checkForRemoteErrors(val) : 2 nodes produced errors; first error: could not find function "f" I think this is because ABC_sequential() is ultimately calling parLapplyLB() (https://github.com/cran/EasyABC/blob/master/R/EasyABC-internal.R) and I...

How do I throw an Exception from inside a Parallel.ForEach loop?

c#,foreach,exception-handling,parallel-processing
I have a Parallel.Foreach loop which downloads files like so: try { var parallelOptions = new ParallelOptions(); parallelOptions.MaxDegreeOfParallelism = 8; int failedFiles = 0; Parallel.ForEach(FilesToDownload, parallelOptions, tile => { bool downloaded = DownloadTile(File); if (downloaded) { //Downloaded :) } else { failedFiles++; if (failedFiles > 10) { throw new Exception("10...

How do I cancel a concurrent heavy Task?

c#,.net,parallel-processing,task-parallel-library,cancellationtokensource
I have a Task by heavy process runs in that's body. In addition, we have no access to the body of this method (heavy process) and we must wait until to completion the process. Now my question is, how do I cancel without interrupting the task so that I do...

Clever aggregating of tuples in C#

c#,parallel-processing
I am processing a complex query in parallel. From the called methods I get a lot of Tuple<IEnumerable<Object>, int> objects. I would like to aggregate them quickly, but probably .Aggregate (code below) is not the best option. What is the right way to do it? public static Tuple<IEnumerable<Object>, int> Parse(Object...

C# Parallel.For not executing every step

c#,parallel-processing,parallel.for
I have been working on a mock-up for an import service which currently runs in sequence. However my mock-up seems to exhibit a strange problem where by sometimes one or two items in the for loop aren't executed. class Service { private Thread _worker; private bool _stopping; private CancellationTokenSource _cts;...

Operating in parallel on a large constant datastructure in Julia

performance,parallel-processing,julia-lang,hpc
I have a large vector of vectors of strings: There are around 50,000 vectors of strings, each of which contains 2-15 strings of length 1-20 characters. MyScoringOperation is a function which operates on a vector of strings (the datum) and returns an array of 10100 scores (as Float64s). It takes...

Parallelize operations for each cell in a numpy array

python,numpy,parallel-processing,theano
I am trying to figure out which is the best way to parallelize the execution of a single operation for each cell in a 2D numpy array. In particular, I need to do a bitwise operation for each cell in the array. This is what I do using a single...

How to run two set of code in parallel using openmp in c++

c++,multithreading,parallel-processing,openmp
I have two function which are not related to each other for example: int add(int num) { int sum=0; for(i=0;i<num;++i) sum+=i; return sum; } int mul(int num) { int mul=1; for(int i=1;i<num;++i) mul * i; return mul; } and I am suing them as follow: auto x=add(100); auto m=mul(200); cout<<a<<...

ElasticSearch Multiple Scrolls Java API

java,scroll,elasticsearch,parallel-processing
I want to get all data from an index. Since the number of items is too large for memory I use the Scroll (nice function): client.prepareSearch(index) .setTypes(myType).setSearchType(SearchType.SCAN) .setScroll(new TimeValue(60000)) .setSize(amountPerCall) .setQuery(MatchAll()) .execute().actionGet(); Which works nice when calling: client.prepareSearchScroll(scrollId) .setScroll(new TimeValue(600000)) .execute().actionGet() But, when I call the former method multiple times,...

OpenMP Matrix-Vector Multiplication Executes on Only One Thread

c++,multithreading,parallel-processing,openmp,mex
I have this code (outlined below) for parallelizing matrix-vector multiplication. But whenever I run it, I discover that it is executing on just one thread (even though I specified 4). How can I separate parts of the parallel code to run on separate threads. Any help will be highly appreciated....

Multiprocessing a python script

python,parallel-processing,multiprocessing,python-multiprocessing
I learnt about the multiprocessing tool in python: https://docs.python.org/2/library/multiprocessing.html. Say I have a python program which is complicated and fleshed out, but it does not use up all my cores when running. So it uses 100% of one core and takes forever to complete. It is hard for me to...

Parallel.ForEach with Dictionary and multiple local variables in C#

c#,variables,dictionary,parallel-processing,local
I'm trying to convert a normal function to a parallel function. The function checks if the coordinates given are close enough to consider them duplicated, checking one by one in a full list of coordinates. I need access to 3 local variables: Coordinates to be tested (poi_data) the list of...

From Java parallelstream spawns other parallelStreams and fails seldom

java,lambda,parallel-processing,java-8,java-stream
Considering the following function: public void execute4() { File filePath = new File(filePathData); File[] files = filePath.listFiles((File filePathData) -> filePathData.getName().endsWith("CDR")); List<CDR> cdrs = new ArrayList<CDR>(); Arrays.asList(files).parallelStream().forEach(file -> readCDRP(cdrs, file)); cdrs.sort(cdrsorter); } which reads a list of Files containing CDR and executes the readCDRP() which is this: private void readCDRP(List<CDR> cdrs,...

Is Parallel.ForEach obsolete. old, out of fashion?

multithreading,design-patterns,parallel-processing,async-await,.net-4.5
Good day, Parallel execution can be achieved through multiple ways. From strictly manual "multithreading" to using various "helpers" created by Microsoft. One of those helpers is the Parallel class. One of my colleagues is insisting that Parallel.ForEach (or Parallel class overall) is "old" and shouldn't be used. Instead, he says,...

How to classify a matrix within a Matlab parfor loop?

matlab,matrix,parallel-processing,parfor
I am looking to classify the values of a matrix. The following example works outside of a parfor loop, however it does not work when used within a parfor loop. What are my options to classify matrices, following the provided example, within a parfor loop? % Sample data Imag1 =...

How does asyncio (python) work?

python,concurrency,parallel-processing,python-asyncio
Yield from coroutine vs yield from task As reported in this question(second answer) "yield from coroutine()" is like calling a function. I don't understand how something like yield from, that actually waits until the coroutine is finished should enable concurrency and non blocking operations. For example in Golang you just...

Finding reason for the inaccurate results, copying code from research paper

cuda,parallel-processing,research
I am trying to replicate the linear programming solver that this person has made http://www.idi.ntnu.no/~elster/master-studs/spampinato/spampinato-linear-prog-gpu-report.pdf. First of, the device I am using is Quadro FX 1800M with compute capability 1.2. My problem is that when I launch more than 22 threads per block then most of the time I get...

How to perform parallel processes for different groups in a folder?

bash,unix,parallel-processing,groups
I have a folder containing a lot of images. I have a code which transforms these images into black and white format and then use tesseract to convert them into text files. I have been using the following code to split these files into subgroups: i=0; for f in *;...

Scala parallel frequency calculation using aggregate doesn't work

multithreading,scala,concurrency,parallel-processing,aggregate
I'm learning Scala by working the exercises from the book "Scala for the Impatient". Please see the following question and my answer and code. I'd like to know if my answer is correct. Also the code doesn't work (all frequencies are 1). Where's the bug? Q10: Harry Hacker reads a...

Method returns before executing all threads even though executor.awaitTermination is used

java,multithreading,concurrency,parallel-processing,executorservice
I have a method which takes number of threads to work and then executes the run() method for each thread accordingly as shown below public static Map<String, Integer> execute(int thread_count) { ExecutorService executor = Executors.newFixedThreadPool(thread_count); File folder = new File("logFiles/"); Collection<File> files = FileUtils.listFiles(folder, null, true); for(File file : files){...

Spawn process that run in parallel in R

r,parallel-processing
I am writing a script that needs to be running continuously storing information on a MySQL database. However, at some point of the day I will like to produce some summary of the data being colected, but writing this in the same script will stop collecting data while doing these...

System time for parallel and serial processing

r,parallel-processing,mcmc
I'm running a Bayesian MCMC probit model, and I'm trying to implement it in parallel. I'm getting confusing results about the performance of my machine when comparing parallel to serial. I don't have a lot of experience doing parallel processing, so it is possible I'm not doing it right. I'm...

How to composite large images stored on S3 in ImageMagick from EC2 instances?

amazon-ec2,parallel-processing,imagemagick,celery
I have an ongoing list of image processing tasks to do, using ImageMagick to composite large individual graphic files (20MB each). These images are currently stored on S3 (approximately 2.5GB in total). I was thinking to use multiple EC2 instances to process the tasks, composite the images and upload the...

JFrame freezes when while(true) looping in thread

java,multithreading,parallel-processing,freeze,jinternalframe
I simplified my code to share it with you guys. Right now it has 3 classes. JFrame, JInternalFrame and a Thread. JFrame creates the JInternalFrame and the JInternalFrame creates the Thread. The constructor in my thread needs an instance of the JInternalFrame to mess around with the buttons of it....

Run function with positional and optional arguments in parallel in python (follow up)

python,python-2.7,parallel-processing,multiprocessing
This is a follow up question to: Python: How can I run python functions in parallel? Minimal Working Example: ''' Created on 06.05.2015 http://stackoverflow.com/questions/7207309/python-how-can-i-run-python-functions-in-parallel ''' from multiprocessing import Process import time def runInParallel(*fns): proc = [] for fn in fns: p = Process(target=fn) p.start() proc.append(p) for p in proc: p.join()...

OpenMP: is there a timeout for a parallel section?

c++,parallel-processing,timeout,scheduled-tasks,openmp
I'm having a problem here with OpenMP. There are two functions that shall be executed in parallel. In foo() there's a loop that shall be interrupted with stop. And as you can see it is assigned in the the other OMP-section. The code is: char stop; #pragma omp parallel {...

how to make the program pause when actor is running

multithreading,scala,parallel-processing,actor
For example import scala.actors.Actor import scala.actors.Actor._ object Main { class Pong extends Actor { def act() { var pongCount = 0 while (true) { receive { case "Ping" => if (pongCount % 1000 == 0) Console.println("Pong: ping "+pongCount) sender ! "Pong" pongCount = pongCount + 1 case "Stop" => Console.println("Pong:...

Communication cost of MPI send and recieve

c++,parallel-processing,mpi
I'm new to MPI and want to measure the communication cost of MPI_Send and MPI_Recv between two nodes. I have written the following code for this purpose: /*============================================================== * print_elapsed (prints timing statistics) *==============================================================*/ void print_elapsed(const char* desc, struct timeval* start, struct timeval* end, int numiterations) { struct timeval elapsed;...

Encounter Order wrong when sorting a parallel stream

java,parallel-processing,java-8,java-stream
I have a Record class: public class Record implements Comparable<Record> { private String myCategory1; private int myCategory2; private String myCategory3; private String myCategory4; private int myValue1; private double myValue2; public Record(String category1, int category2, String category3, String category4, int value1, double value2) { myCategory1 = category1; myCategory2 = category2; myCategory3...