node.js,multithreading,mongodb,child-process,long-running-processes , How to execute / abort long running tasks in Node JS?


How to execute / abort long running tasks in Node JS?

Question:

Tag: node.js,multithreading,mongodb,child-process,long-running-processes

NodeJS server with a Mongo DB - one feature will generate a report JSON file from the DB, which can take a while (60 seconds up - has to process hundreds of thousands of entries).

We want to run this as a background task. We need to be able to start a report build process, monitor it, and abort it if the user decides to change the params and re build it.

What is the simplest approach with node? Don't really want to get into the realms of separate worker servers processing jobs, message queues etc - we need to keep this on the same box and fairly simple implementation.

1) Start the build as a async method, and return to the user, with socket.io reporting progress?

2) Spin off a child process for the build script?

3) Use something like https://www.npmjs.com/package/webworker-threads?

With the few approaches I've looked at I get stuck on the same two areas;

1) How to monitor progress? 2) How to abort an existing build process if the user re-submits data?

Any pointers would be greatly appreciated...


Answer:

The best would be to separate this task from your main application. That said, it'd be easy to run it in the background. To run it in the background and monit without message queue etc., the easiest would be a child_process.

  1. You can launch a spawn job on an endpoint (or url) called by the user.
  2. Next, setup a socket to return live monitoring of the child process
  3. Add another endpoint to stop the job, with a unique id returned by 1. (or not, depending of your concurrency needs)

Some coding ideas:

var spawn = require('child_process').spawn

var job = null //keeping the job in memory to kill it

app.get('/save', function(req, res) {

    if(job && job.pid)
        return res.status(500).send('Job is already running').end()

    job = spawn('node', ['/path/to/save/job.js'], 
    {
        detached: false, //if not detached and your main process dies, the child will be killed too
        stdio: [process.stdin, process.stdout, process.stderr] //those can be file streams for logs or wathever
    })

    job.on('close', function(code) { 
        job = null 
        //send socket informations about the job ending
    })

    return res.status(201) //created
})

app.get('/stop', function(req, res) {
    if(!job || !job.pid)
        return res.status(404).end()

    job.kill('SIGTERM')
    //or process.kill(job.pid, 'SIGTERM')
    job = null
    return res.status(200).end()
})

app.get('/isAlive', function(req, res) {
    try {
        job.kill(0)
        return res.status(200).end()
    } catch(e) { return res.status(500).send(e).end() }
})

To monit the child process you could use pidusage, we use it in PM2 for example. Add a route to monit a job and call it every second. Don't forget to release memory when job ends.


Related:


call functions in async with node which is more recomended Q or callback


javascript,node.js,callback,promise,q
I've node app with function that inside call to to other two function,I want to use some async behavior for it,what is recommended to use in this case. example will be very helpful. function myFunction(req,res){ //from here this is the first place which I want to use warp in function...

Why is address undefined in my app?


node.js,express,jasmine,supertest
I have a simple express app: var express = require('express'); var path = require('path'); var app = express(); exports.app = app; var index = require('./routes/index'); app.use(express.static(path.join(__dirname,'client/dist/'))); app.get('/', index.get); function start(){ var port = process.env.PORT || 8080; app.listen(port, function(){ console.log('app is running on port: ' + port); }); }; exports.start =...

I'd like to count the documents with the matching “name” property AND group them by “name” at the same time


node.js,mongoose,group-by
Let's say I have a User collection: schema = mongoose.Schema({ firstName: { type: String, required: true }, ... ... }); module.exports = mongoose.model("User", schema); I would like to write a mongoose query that would count how many users go by the name Mike, Andy, Jerry... In other words, I would...

Sockets make no sense?


javascript,node.js,sockets
I'm using the 'ws' library for Node.js. I can write code that sends data from my server to my client, posting a date and time update, and closes the socket when I click a button; var wss = new WebSocketServer({server: server}); console.log("WebSocket server created"); wss.on('connection', function(socket) { // SEND DATE...

Multiple Threads searching on same folder at same time


c#,multithreading,file-search
Currently I have a .txt file of about 170,000 jpg file names and I read them all into a List (fileNames). I want to search ONE folder (this folder has sub-folders) to check if each file in fileNames exists in this folder and if it does, copy it to a...

How to use promises to do series without duplicate code


node.js,promise,bluebird
I need execute a code in series, I need execute the same function N times Example // execute asynFunc 4 times in series object.asynFunc() .then(function() { return object.asynFunc(); }) .then(function() { return object.asynFunc(); }) .then(function() { return object.asynFunc(); }) I want execute the same function 100 times...

What does this line in NodeJs mean?


node.js
I'm wondering what does these require lines in NodeJs mean. var debug = require('debug')('morgan') var deprecate = require('depd')('morgan') I'm going through the index.js of morgan package in NodeJs. Normally require only has one parameter (package). ...

websockets - reject a socket connection


node.js,sockets,websocket
I'm using ws as the socket library for my node.js library. so my question is, how can I reject a connection if the user doesn't pass the authorization process. var WebSocketServer = require('ws').Server; var wss = new WebSocketServer({port: 6969}); wss.on('connection', function(socket){ // if the socket.upgradeReq.headers.cookie doesn't exists, reject the client...

Using TypeScript type definitions with Webstorm 10 [duplicate]


node.js,typescript,webstorm
This question already has an answer here: intellisense and code complete for DefinitelyTyped (TypeScript type definitions) on WebStorm IDE 2 answers I am trying to use Webstorm 10's new built-in TypeScript compiler. When compiling a simple Node.js file such as below, it gives a TS compilation error of "Error:(1,...

Error is not thrown inside a deferred method


node.js,exception-handling,deferred
Can somebody explain to me why my error is not thrown in my first example? And why it is when I use process.nextTick() ? var deferred = require('deferred'); // This code does not work. // Error seems to never been thrown and script kind of freeze without saying anything. deferred.resolve().then(function(){...

jQuery DataTables with Node.js


javascript,jquery,node.js,datatables,jquery-datatables
So i am trying to implement a pagination table with the datatables plugin, this is my first time using this plugin. I followed the documentation on the plugin and tried to get the values from the server through the use of Ajax, as per presented in the plugins documentation. I...

How to add new items to an array in MongoDB


arrays,node.js,mongodb
I'm trying to add a new item to whichever name that was passed in under whichever id. My first problem is that it seems like its not grabbing the values from any of my variables (name, item, id), instead just using them as object keys. My next issue is that...

What does a [Function] (wrapped in square brackets) mean when inside of a javascript object?


javascript,node.js,javascript-objects
When running console.log on various functions, I'll find properties on the object that have a value of [Function: someFunctionName] in the value section. What does this mean? I want to be able to view the actual code of the function. I'm confused on what's actually being logged when I see...

Images not appearing on WPF form when loading asynchronously


c#,wpf,multithreading,listbox,backgroundworker
I'm attempting to display (in a ListBox with a custom DataTemplate) a series of BitmapSource frames (thumbnails) extracted from a multi-page tiff image. When I process the tiff on the UI thread, and either directly add the images to a listbox's item collection or add them to a bound ObservableCollection,...

React from NPM cannot be used on the client because 'development' is not defined. The bundle was generated from Webpack


javascript,node.js,npm,reactjs,webpack
I'm creating a React Node.js app and I'm trying to generate a Webpack bundle containing the React source code I loaded from NPM. However, it seems that the React code from NPM cannot be used directly in the client. It triggers this error: Uncaught ReferenceError: development is not defined The...

javascript “this” keyword works as expected in browser but not in node.js


javascript,node.js
I know I'm making a mistake here but I can't figure out what it is. The following code (non-strict mode) works as I expect in a browser and outputs "hello" to the console. function a() { console.log(this.bar); } var bar = "hello"; a(); But when I run it in node...

Replace nodejs for python?


python,node.js,webserver
i'm working in a HTML5 multiplayer game, and i need a server to sync player's movement, chat, battles, etc. So I'm looking for ways to use python instead nodejs, because i have I have more familiarity with python. The server is simple: var express = require('express'); var app = express();...

node.js winston logger no colors with nohup


node.js,logging,nohup,winston
We are using winston logger in our project with the following transport settings: file: { filename: __base + '/log/server.log', colorize : true, timestamp : true, json : false, prettyPrint : true } If the application is started with nohup, log file is not colorized. It works only without nohup. nohup...

Calling dispatch_sync from a concurrent queue - does it block entirely?


ios,objective-c,multithreading,swift,grand-central-dispatch
Let's say I hypothetically call a dispatch_sync from a concurrent queue - does it block the entire queue or just that thread of execution?

How to use a variable as an Object Key [MongoDB] [duplicate]


node.js,mongodb
This question already has an answer here: How to use a variable as a field name in mongodb-native findOne()? 1 answer I'm trying to use variable name given to me by my function, but its actually setting the object key equal to 'name' function addlist(name, item, id){ // Add...

java multithreading start() and run() [duplicate]


java,multithreading
This question already has an answer here: Java: What's the difference between Thread start() and Runnable run() 10 answers Below is my Multithreading class: public class Multithreading extends Thread{ public void run(){ for(int i=1;i<5;i++){ try{ Thread.sleep(500); }catch(InterruptedException e){ System.out.println(e); } System.out.println(i); } } public static void main(String args[]) {...

After deploying to heroky scripts and css not available


node.js,heroku
I am new in heroku. Locally project works fine but after deploying vendor scripts files are not available. Their paths are redirected to main page. Here is the deployed version with errors https://salty-woodland-8424.herokuapp.com/. In nodeJs file i use var express = require('express'); var path = require('path'); var stylus = require('stylus');...

performance issues executing list of stored procedures


c#,multithreading,performance,loops
I'm having some performance issues when starting my windows service, the first round my lstSps is long (about 130 stored procedures). Is there anyway to speed this up (except for speeding the stored procedures up)? When the foreach is over and goes over to the second round it goes faster,...

Is there a before() function in Protractor?


angularjs,node.js,automated-tests,protractor,hierarchy
I am aware there is a beforeEach() function which I am currently using. However, I would like to be able to run a before function for all my parent level 'describes' and not the 'it' functions within them. For example my test looks like this (with an x in front...

Is there standard implementation for thread block/resume in java SE?


java,multithreading,wait
I need to block execution of a thread until resumed from another thread. So I wrote my own implementation using wait() method. Which seems to be working, but it is far from simple. Is there any ready to use solution? Preferably in java SE 6? Or do I have to...

Socket.io client does not connect to server


node.js,express,socket.io
I am trying to make a Node.js app that will have an embedded chat. I am using socket.io to create that chat. I have attempted to set up my server / client, but the client does not seem to be connecting. I have my application to set log when sockets...

Looking to pause a thread using thread.sleep


java,multithreading
I am using a mouse listener for mouse pressed and released. When the mouse is pressed I want to have a counter incrementing a variable, and when the mouse is released I want to decrement that variable. Right now, my code is working and does that but the increment is...

How do I run C# within a Node.js server application?


c#,node.js,server
I have a node.js application and a C# algorithm. The algorithm puts out 15 numbers that represent symbols on a digital slot machine. The node server is posting and getting data from Firebase and the digital slot machine is doing the same on the same table. My question is how...

What type of database is the best for storing array or object like data [on hold]


database,node.js,sockets
I'm just curious what the best method would be if I'm trying to have a bot running on my Node server that I could play Blackjack against. But for multiple connected clients via sockets, each connected socket will have their own bot to play against but I need some way...

Node Server - Source Code accessible


node.js,express
I noticed that when i navigate to localhost:8080/server.js (where my server.js is the server-expressjs obviously) the code of my server is shown in the browser! Even, if i upload the application to openshift, i get the same result (you can test it): http://tickets-shkobba125.rhcloud.com/ http://tickets-shkobba125.rhcloud.com/server.js Is this a security issue? How...

nodejs head request isn't triggering events


node.js,http
Here's my code: var http = require('http'); var req = http.request( { host: 'example.com', method: 'HEAD', path: '/' }, function(res){ res.on('end',function(){ console.log('Ended'); }); res.on('finish',function(){ console.log('Finished'); }); res.on('close',function(){ console.log('Closed'); }); } ); req.end(); Strange thing about it is that it doesn't print anything. Takes longer than expected and exits. Is this...

node ssh2 shell unable to run apt-get install on remote machine


node.js
I'm doing a program in node.js to deploy my code from git to a server (Digital Ocean). I'm connecting to the server via ssh2 module and running commands via shell, like bellow: var Client = require('ssh2').Client; var conn = new Client(); conn.on('ready', function() { console.log('Client :: ready'); conn.shell(function(err, stream) {...

Create n:m objects using json and sequelize?


javascript,json,node.js,sequelize.js
I am trying to learn sequelize, but am having trouble getting a n:m object created. So far I have my 2 models that create 3 tables (Store, Product, StoreProducts) and the models below: models/Store.js module.exports = (sequelize, DataTypes) => { return Store = sequelize.define('Store', { name: { type: DataTypes.STRING, },...

Getting CROS Error even after adding header in node.js for Angular js


javascript,angularjs,node.js
I am trying to consume REST API from NODE JS for Angular js,even after adding cors header in my server code I am getting error XMLHttpRequest cannot load http://127.0.0.1:8085/issues. Request header field Access-Control-Allow-Origin is not allowed by Access-Control-Allow-Headers. I am new to both Angular JS and Node JS. Node JS...

NodeJS / ExpressJS check valid token parameter before routing


node.js,express,parameters
I have the following app code: (app.js) var express = require('express') , app = express() , port = process.env.PORT || 8082 app.use(require('./controllers')) app.use(function(req, res, next) { res.send('Test') next() }) app.listen(port, function() { console.log('Listening on port ' + port) }) and two controllers: (index.js) var express = require('express') , router =...

Access Node-Webkit App from other Application


node.js,node-webkit
Is it possible to call a function in nodewebkit from an external application? For example. I would like to decide whether the window is hidden or show through a external application or with applescript. ...

wait for an event regulary


multithreading,events
In a program I need to wait for an event (keypress) and get it's char. after that program will continue. this progress will Repeat several time. in my first try, codes run and any character did not save. I find out I should use threading but I am not Familiar...

How to get my node.js mocha test running?


javascript,node.js,mocha,supertest
I have developed a service in node.js and looking to create my first ever mocha test for this in a seperate file test.js, so I can run the test like this: mocha test I could not figure out how to get the reference to my app, routes.js: var _ =...

What are some patterns I can look at for database implementations in JavaScript?


javascript,node.js,mongodb
I'm fairly new to JavaScript, and I'm busy playing around with a node test app and MongoDB. I'm at a point where I'd like to start with the db side of the app, but I'm not sure what patterns are most commonly used in such a stack, and more importantly,...

How can I know the lock information in java?


java,multithreading,locking
Is there any tool or way that can get all the information about the locks in java? for example, if there is a java program, it creates two threads, and both threads require locks for some variable. Is there any tools that can output the information like which thread locks...

Is express similar to grunt? what is the difference? what are the advantages of express over grunt?


node.js,express,gruntjs,mean-stack
I've been working on node,grunt,bower and yeoman from couple of months. I came across MEAN stack applications, in which expressjs is providing the server environment(my understanding). Are both grunt and express similar? Requesting for some helpful link on express and MEAN stack. ...

Waiting for promises - code hangs


javascript,node.js,promise
I am using Javascript Promises for the first time and ran into something I don't understand. What I am trying to do is create a validation phase which runs around and checks things - eventually waiting for all promises to resolve. To do this, I create a validation promise: validate...

Emitting and receiving socket io within the same file


node.js,express,socket.io
I want to do something where I can set up a route to a link like /visit using app.get('/visit', function(req, res){}) etc. And inside that, I want to emit a message like socket.emit("event", "a message") and then inside my io.on("connection") function, be able to listen for event and return the...

mongodb populate method not working


node.js,mongodb,model,populate,auto-populate
Here is my code for models var postSchema = new mongoose.Schema({ created_by: {type: Schema.ObjectId, ref:'User', autopopulate: true }, //should be changed to ObjectId, ref "User" created_at: {type: Date, default: Date.now}, text: String }); var userSchema = new mongoose.Schema({ username: String, password: String, //hash created from password created_at: {type: Date, default:...

how can I import a file in node.js?


javascript,node.js
I have developed a node.js application and performing some validation with the use of a template. At the moment this 'template' is a local variable but I would like to store this in a file in my project. So this is the validation code: isvalid(req.body,template , function(err, validObj) { if...

Redis: Delete user token by email ( find Key by Value )


node.js,express,redis
I have followed tutorial on how to create token-based authentication with node from this tutorial http://www.kdelemme.com/2014/08/16/token-based-authentication-with-nodejs-redis/ I got it all worked out, but I got 1 problem. The way I store token is : KEY = TOKEN VALUE = UserData (Username, email, etc.) To protect multiple devices login, I would...

std::condition_variable – notify once but wait thread wakened twice


c++,multithreading
Here's a simple C++ thread pool implementation. It's an altered version orginated from https://github.com/progschj/ThreadPool. #ifndef __THREAD_POOL_H__ #define __THREAD_POOL_H__ #include <vector> #include <queue> #include <memory> #include <thread> #include <chrono> #include <mutex> #include <condition_variable> #include <future> #include <functional> #include <stdexcept> namespace ThreadPool { class FixedThreadPool { public: FixedThreadPool(size_t); template<class F, class......

Socket.IO message doesn't update Angular variable


javascript,angularjs,node.js,sockets
I have a socket.io client-server setup with AngularJS running on the client. // Server.js var io = require('socket.io')(server); io.on('connection', function (socket) { socket.on('message', function (msg) { //console.log(msg); console.log(msg); io.emit('message', msg); }); }); As observed, it essentially emits a message events with the data stored in the variable msg. And then...

Set Label From Thread


vb.net,multithreading,winforms
Form1.vb Imports System.Threading Public Class Form1 Dim demoThread As Thread Private Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click Dim Start As New Class1 Me.demoThread = New Thread( _ New ThreadStart(AddressOf Start.ThreadProcSafe)) Me.demoThread.Start() End Sub Delegate Sub SetTextCallback([text] As String) Public Sub SetText(ByVal [text] As String) ' InvokeRequired required...

NPM : how to just run post-install?


node.js,npm,package.json
Just a simple question : in my node.js project, how could I just run the postinstall script, without running install before ? FYI, this is my package.json : { "name": "gestionclientjs", ..., "dependencies": { ... }, "repository": {}, "devDependencies": { ... }, "engines": { "node": ">=0.10.0" }, "scripts": { "test":...