subprocess,common-lisp,sbcl , SBCL: Gather output of run-program process while running

SBCL: Gather output of run-program process while running


Tag: subprocess,common-lisp,sbcl

Dear StackExchange members,

I recently began toying around with Common Lisp and want to create a web interface for administrating a modded Minecraft server. I already tried out this solution but in this case the function just hangs and never returns.

My code looks like this:

(defvar *proc*)

(defun create-minecraft-proc ()
  (let* ((binary "/usr/bin/java")
    (jar "/home/user/temp/Vanilla 1.8/minecraft.jar")
    (dir "/home/user/temp/Vanilla 1.8")
    (args (list "-jar" jar "nogui")))
    (setf *proc* (sb-ext:run-program binary args :directory dir :wait nil :input :stream :output :stream))))

(defun copy-stream (in out)
  (loop for line = (read-line in nil nil)
     while line
       do (write-line line out)))

(defun get-minecraft-output ()
  (with-open-stream (proc-stream (process-output *proc*))
    (with-output-to-string (out)
      (copy-stream (process-output *proc*) out)))

How can I get a string containing the complete process output till that time without having to wait for it to terminate?


Experimenting with /bin/cat, I found some things that might help you.

First, of all, always use finish-output after writing to the process's input:

(format (process-input *cat*) "Hello~%")
(finish-output (process-input *cat*))

Otherwise, the input may not reach the subprocess until you close the input stream. If Minecraft requires input before it'll generate any output, then trying to read from its output without the required input would result in SBCL hanging in a way that can't be recovered with C-c C-c in SLIME.

Second, use listen to determine if any output has been generated:

(when (listen (process-output *cat*))
    (read-line (process-output *cat*)))

With finish-output and listen, I was able to avoid hanging while incrementally reading from cat's stdout.


Use NamedTemporaryFile to read from stdout via subprocess on Linux

import subprocess import tempfile fd = tempfile.NamedTemporaryFile() print(fd) print( p = subprocess.Popen("date", stdout=fd).communicate() print(p[0]) fd.close() This returns: <open file '<fdopen>', mode 'w' at 0x7fc27eb1e810> /tmp/tmp8kX9C1 None Instead, I would like it to return something like: Tue Jun 23 10:23:15 CEST 2015 I tried adding mode="w", as well as delete=False, but...

subprocess python 3 check_output not same as shell command?

I am trying to use the subprocess module in python but its a bit tricky to get working. Here's my code import sys import os import subprocess import shlex def install_module(dir_path, command): c = shlex.split(command) os.chdir(dir_path) try: p = subprocess.check_output(c, shell=True) except subprocess.CalledProcessError as e: #print('install failed for: ' +...

If strings are vectors, why are they immutable?

if strings are vectors of characters, and a vector's elements can be accessed using elt, and elt is setf-able - then why are strings immutable?

Why does ln from Python subprocess fail while it succeeds from normal command line?

As the title says: >>> from subprocess import check_output >>> check_output(['ln', '~/other_folder/src/models/', './src/models/']) Traceback (most recent call last): File "<input>", line 1, in <module> File "/usr/lib/python2.7/", line 573, in check_output raise CalledProcessError(retcode, cmd, output=output) CalledProcessError: Command '['ln', '~/other_folder/src/models/', './src/models/']' returned non-zero exit status 1 >>> exit() $ ln...

Processing file paths with parentheses with Python subprocess

The paths of the files I want to process contain parentheses in them. path = "/dir/file (with parentheses).txt" I'm trying to process them in Python as follows:"./process %s" % path, shell=True) However, I get the following error /bin/sh: 1: Syntax error: "(" unexpected How can I pass the correct...

Structuring large Lisp applications

I am currently trying to wrap my head around packages, systems & co. I now have read Packages, systems, modules, libraries - WTF? a few times, and I think I'm still having difficulties to get it right. If I simply want to split a Lisp source file into two files,...

Read input into string in Lisp reader macro

I am trying to make a reader macro that would convert @this into "this". This is what I currently have: (defun string-reader (stream char) (declare (ignore char)) (format nil "\"~a\"" (read-line stream t nil t)) ) (set-macro-character #\@ #'string-reader ) The problem is that this requires that I put a...

parameter error in python script & TOR proxy server

I am Noob in Python. My Boss instructed me to run this python script with TOR proxy server running. He told me to pass this parameter this way: python /Users/myfolder/ japan 1 222299 He configured it on MAC. I am using windows. So my parameter is this way:...

Pass Buffer to ChildProcess Node.js

Here I have on Node.Js where I want to do Image Processing in a Sub Process. As you will see I take the file image.jpg and want to write it back to hello.jpg in a subprocess: var node = require('child_process').spawn('node',['-i']); var fs = require('fs'); node.stdout.on('data',function(data) { var fs = require('fs');...

Local dynamic binding in common lisp

Honnestly, I'm not sure I fully understand what it means for a binding to be "dynamic" versus "lexical". But I understand that when I use defvar or defparameterto define a binding, 1. it declares a global variable 2. the binding is declared "special", so that it can be shadowed by...

Common Lisp Object System method execution order

I have the following two classes: (defclass person () ()) (defmethod speak ((s person) string) (format t "-A" string)) (defmethod speak :before ((s person) string) (print "Hello! ")) (defmethod speak :after ((s person) string) (print "Have a nice day!")) (defclass speaker (person) ()) (defmethod speak ((i speaker) string) (print "Bonjour!"))...

Python subprocess throws [Errno 2] No such file or directory, error generated only when it on a remote host

I'm running python 2.6. I'm getting the subprocess throws [Errno 2] No such file or directory only when I run the script via ssh. For example, if I run the script manually on the machine, there are no errors, but if I do ssh hostname --host hostname it generates...

Run a from another Python script, path issue, installing in calling script dir?

First, I've found how to call a script from within an other script in Python, the call works perfectly well, but here's the problem I'm running into : In order to easy-install my web-app (Bottle) on an another server, I packed inside a /redist rep, with mod_wsgi and PyMySQL source...

How to execute and save result of an OS command to a file [duplicate]

This question already has an answer here: Python: How to Redirect Output with Subprocess? 4 answers In python 2.7, I would like to execute an OS command (for example 'ls -l' in UNIX) and save its output to a file. I don't want the execution results to show anywhere...

PYTHON subprocess cmd.exe closes after first command

I am working on a python program which implements the cmd window. I am using subproccess with PIPE. If for example i write "dir" (by stdout), I use communicate() in order to get the response from the cmd and it does work. The problem is that in a while True...

How to collect output from a Python subprocess

I am trying to make a python process that reads some input, processes it and prints out the result. The processing is done by a subprocess (Stanford's NER), for ilustration I will use 'cat'. I don't know exactly how much output NER will give, so I use run a separate...

Backquote String Interpolation

Is it possible to use lisp's macro to do string interpolation? For instance, can I make a macro like this: (defmacro test (a) `",a") So that (test abc) returns "abc" as a string? I could probably cheat by quoting it and turning that quote into a string, but that doesn't...

Insertion into a list doesn't reflect outside function whereas deletion does?

I am new to Lisp. Deletion of an item in a list by a function gets reflected outside the function but insertion doesn't. How can I do the same for insertion? For example (defun test (a b) (delete 1 a) (delete 5 b) (append '(5) b) (member '5 b)) (setq...

What does the non-terminating-p argument of set-macro-character do?

Set-macro-character has an optional argument called non-terminating-p. It seems to be used to indicate whether another character should be read after reading the macro character, but the reader algorithm seems to ignore this argument. Is there a difference whether I set it to true or false?

Creating A Process Queue in Common Lisp

I have a server running Hunchentoot (CentOS and SBCL). When a user submits a specific type of post request, a subprocess is launched (run-program) which can take up to four minutes to complete. If five people perform that specific type of request at the same time, the server runs out...

Invalid specialized parameter in method lambda list

I am trying to write a simple coin flip program in Common Lisp. This is the code I have (defun yn (let ht (random 1) (if (eq ht 1) (princ heads) (princ tails)) ) ) It seems simple enough, but I keep getting the error: "Invalid specialized parameter in method...

Weird non-lisp errors with capturing labels with a lambda and `reduce`

I stumbled across a very weird bug (?) while I was developing an application in Allegro Common Lisp, v9.0 for Windows. I do not get the regular lisp errors, instead I get system errors encapsulated in a lisp condition. I managed to create a simple test-case to reproduce the error...

Dynamic 2d array in lisp

I want to have a 2D array in Lisp . But each row can have different number of elements(At max 5). So I thought of maintaining another single list to store the current sizes of each row, and update them whenever required. So, my code goes like this : (setq...

Using subprocess.check_output for a command with 2>/dev/null

I am on Mac OS X Yosemite 10.10 and Python 2.7. If I type the following: du -g -d1 /Users 2> /dev/null in the command line, everything works perfectly. Now, my goal is to use that command in a python script. My idea was to use the following: import subprocess...

File gets created, but subprocess call says no such file or directory

I am trying to redirect the output of a subprocess call to a file def get_arch(): global ip_vm in_cmd = "uname -p" with open('/home/thejdeep/arch_list',"w") as outfile: print outfile,stdout=outfile) print "Hello" for i in ip_vm: cmd = "ssh [email protected]"+i+" 'uname -p'" with open('/home/thejdeep/arch_list',"a") as outpfile:,stdout=outpfile) The file gets created....

Running compiled lisp program with clisp-2.49 on OS X

I have just started out programming in Common Lisp using GNU clisp-2.49 (compiled from source) as my implementation on OS X 10.10 Yosemite. I've written a simple "Hello World" program as follows: (EXT:SAVEINITMEM "test" :INIT-FUNCTION 'main :EXECUTABLE t) (defun main () (format t "Hello World!") (EXT:EXIT)) When I run: $...

How to run a line in Powershell in Python 2.7?

I have a line of Powershell script that runs just fine when I enter it in Powershell's command line. In my Python application which I run from Powershell, I am trying to send this line of script to Powershell. powershell -command ' & {. ./uploadImageToBigcommerce.ps1; Process-Image '765377' '.jpg' 'C:\Images' 'W:\product_images\import'}'...

Start a background process upon successful quickstart in cherrypy

I have a REST WebService built on top of cherrypy. The service goes online with the cherrypy.quickstart() call. I want to start a background process with subprocess.Popen() right after the service goes online. The cherrypy.quickstart() call is blocking. How can I add a callback to start the background process?...

Using dd command through Python's subprocess module

Through Python's subprocess module, I'm trying to capture the output of the dd command. Here's the snippet of code: r = subprocess.check_output(['dd', 'if=/Users/jason/Desktop/test.cpp', 'of=/Users/jason/Desktop/test.out']) however, when I do something like print r I get a blank line. Is there a way to capture the output of the dd command into...

EMACS-Live + Slime error at startup

Ok, I must be missing something obvious. I'm getting stuck since yesterday to launch Emacs-live + slime. I'm using EMACS 24.3.1, installed Emacs-live and it worked well (if I start emacs-live without Slime it works), downloaded Slime-Pack from git and added this line to .emacs-live.el (live-append-packs '(~/.live-packs/slime-pack/)) I'm on a...

Pythonic way to detach a process?

I'm running an etcd process, which stays active until you kill it. (It doesn't provide a daemon mode option.) I want to detach it so I can keep running more python. What I would do in the shell; etcd & next_cmd I'm using python's sh library, at the enthusiastic recommendation...

EVAL/APPLY: too many arguments given to F

Hello why do i get *** - EVAL/APPLY: too many arguments given to F on function call with nested lists parameter. I cannot figure it out, since I passed a simple nested list. (defun f (L) (cond ((NULL l) nil) ((listp (car L)) (append (F(car L))) (F(cdr L) (car (F...

Updating the window in response to CLIM frame commands

While trying to figure out CLIM, I ran into this example program. It's a simple maze game. The author claims to have tested it in LispWorks (and even has #+Genera in there, implying that this program would work on a real Lisp Machine), but I'm trying to get it working...

How to use FFI:def-call-in in clisp

I have figured out how to make use of shared objects created from C code into Clisp using FFI:def-call-out but I am not able to figure out how to use FFI:Def-call-in. I don't know the process and actually I am confused if clisp will also create some .so file that...

Redirect files from Popen to subdirecotry

I have a pipeline composed of a number of scripts. Almost all of the scripts read and write to files, and some of the scripts in the pipeline have conflicting file formats (they will read and attempt to perform actions on files that they should not due to the files...

How to move a file in Lisp using rename-file

What's the best way to move a file in Lisp in an implementation-independent way? For example I have an image file: (setq oldpath #P"SERVER:PICTURES;TEMP;PHOTO.PNG") and I want to move it out of the TEMP directory into the PICTURES directory. This seems to work: (setq newpath (make-pathname :host (pathname-host oldpath) :directory...

sbcl run-program hang when there is large amount of output from program

Recently I find a run-program hang issue of sbcl 1.2.7 (32bits, linux). The code as following (progn (with-open-file (s "test.out" :direction :output :if-exists :supersede) (loop repeat 900 do (write-line (make-string 76 :initial-element #\x) s))) (run-program "/bin/bash" (list "-c" "cat test.out") :output :stream)) That is when the "cat test.out" produce many...

lisp: dynamic scope vs explicit parameter passing

I see two different patterns for "output" functions in (common) lisp: (defun implicit () (format t "Life? Don't talk to me about life!")) (defun explicit (stream) (format stream "This will all end in tears.")) (defun test-im-vs-ex-plicit () (values (with-output-to-string (stream) (let ((*standard-output* stream)) (implicit))) (with-output-to-string (stream) (explicit stream)))) Is using...

ffmpeg subprocess fail to open on OS X

I've got this script: PATH = os.path.dirname(os.path.abspath(__file__)) global TEMP for video in os.listdir(VIDEOS): ffmpeg = PATH + "/ffmpeg/ffmpeg" arg1 = " -v 0 -i " arg2 = VIDEOS + "/" + video arg3 = " -r 1 -f image2 " arg4 = TEMP + "/" + os.path.splitext(video)[0] + "-%d.jpg" subprocess.Popen(ffmpeg...

where is the text printed by C printf

I happened encounter a trouble with calling C printf function from SBCL via cffi. The problem is when I call printf function, I can't find the output text, just the return value of printf function show on the REPL. But when I quit SBCL, the output text appears on the...

Python open and kill subprocess

I'm working on Widnows 7 (32 bits) and this is my code: def start_mviewer_broker(broker_path, test_name): """ The function starts the broker server""" try: print("**** start_mviewer_broker ****") p = subprocess.Popen('start python ' + broker_path + ' ' + test_name, shell=True) return p except: print("**** start_mviewer_broker - EXCEPTION ****") return 0 def...

python3 subprocess in Oracle Linux (wget -o)

I see there are several posts on python subprocess invoking bash shell commands. But I can't find an answer to my problem unless someone has a link that I'm missing. So here is a start of my code. import os; import subprocess;"wget ‐O /home/oracle/Downloads/puppet-repo.rpm"); When I do wget...

Subprocess module feeding new line character

I'm writing a script that uses subprocess module to run the net time \\server windows command, and parse the output of that using regex. import datetime import subprocess import re myDate ="%y-%m-%d") myTime ="%H:%M") myDateTime ="%y-%m-%d %H:%M") results_file = 'C:\\temp\\ServerTimes_{0}.txt'.format(myDate) servers_central = 'C:\\temp\\DMS-CENTRAL.txt' sql_central = 'gt2-dms1cst' def...

How to use a parameter in a macro call?

I have defined the following simple macro: (defmacro define-class (class-name) `(defclass ,class-name ()())) And now I want to use it in the following function: (defun create-data (mode) (define-class mode)) After compiling the last function I get the following message, the variable MODE is defined but never used. And when I...

Trying to multiply all elements in a list of sublists with elements in another list in lisp

I'm trying to multiply a list with n sublists with a list with n scalars. It's supposed to work like this: (kmult-matrix '((3 4 2 4) (2 5 6 9) (1 -2 8 10)) '(2 3 5)) => ((6 8 4 8) (6 15 18 27) (5 -10 40 50))...

Append output of multiple subprocesses to array in Python

I'm working on a script in Python which open multiple subprocesses in this way: for file in os.listdir(FOLDER): subprocess.Popen(([myprocess])) Now this processes could be 10-20 running in parallel, and each of them will output in the console a single string line. What I want to do is to append these...

python3 - subprocess with sudo to >> append to /etc/hosts

I've been wrestling with solutions from "How do I use sudo to redirect output to a location I don't have permission to write to?" and "append line to /etc/hosts file with shell script" with no luck. I want to "append puppetmaster" at the end of /etc/hosts. (Oracle/Red-Hat linux). Been...

Python subprocess Messy output

With the following code: output = subprocess.check_output(['wmic', 'PATH', 'Win32_videocontroller', 'GET', 'description']) print(output , "\n") I get the next output: b'Description \r\r\nNVIDIA GeForce 710M \r\r\nIntel(R) HD Graphics 4000 \r\r\n\r\r\n' When I use the commando wmic path win32_videocontroller get desriptionin my CMD I get only the videocard info back. Is this possible...

No-argument (and) returns t

Both CL and Scheme define (and) to return t (or #t) with no arguments. I'm trying to understand the rationale for this. My naive assumption is that an empty set is false, and passing in zero arguments feels like passing in nothing that can be true. Edit: clojure follows the...

What is a common-lisp analogue of python's argparse?

What is a common-lisp analogue of python's argparse library for parsing command-line arguments?