subprocess,common-lisp,sbcl , SBCL: Gather output of run-program process while running


SBCL: Gather output of run-program process while running

Question:

Tag: subprocess,common-lisp,sbcl

Dear StackExchange members,

I recently began toying around with Common Lisp and want to create a web interface for administrating a modded Minecraft server. I already tried out this solution but in this case the function just hangs and never returns.

My code looks like this:

(defvar *proc*)

(defun create-minecraft-proc ()
  (let* ((binary "/usr/bin/java")
    (jar "/home/user/temp/Vanilla 1.8/minecraft.jar")
    (dir "/home/user/temp/Vanilla 1.8")
    (args (list "-jar" jar "nogui")))
    (setf *proc* (sb-ext:run-program binary args :directory dir :wait nil :input :stream :output :stream))))

(defun copy-stream (in out)
  (loop for line = (read-line in nil nil)
     while line
       do (write-line line out)))

(defun get-minecraft-output ()
  (with-open-stream (proc-stream (process-output *proc*))
    (with-output-to-string (out)
      (copy-stream (process-output *proc*) out)))

How can I get a string containing the complete process output till that time without having to wait for it to terminate?


Answer:

Experimenting with /bin/cat, I found some things that might help you.

First, of all, always use finish-output after writing to the process's input:

(format (process-input *cat*) "Hello~%")
(finish-output (process-input *cat*))

Otherwise, the input may not reach the subprocess until you close the input stream. If Minecraft requires input before it'll generate any output, then trying to read from its output without the required input would result in SBCL hanging in a way that can't be recovered with C-c C-c in SLIME.

Second, use listen to determine if any output has been generated:

(when (listen (process-output *cat*))
    (read-line (process-output *cat*)))

With finish-output and listen, I was able to avoid hanging while incrementally reading from cat's stdout.


Related:


How to collect output from a Python subprocess


python,subprocess,stanford-nlp,python-multithreading
I am trying to make a python process that reads some input, processes it and prints out the result. The processing is done by a subprocess (Stanford's NER), for ilustration I will use 'cat'. I don't know exactly how much output NER will give, so I use run a separate...

EMACS-Live + Slime error at startup


emacs,common-lisp,slime,kubuntu
Ok, I must be missing something obvious. I'm getting stuck since yesterday to launch Emacs-live + slime. I'm using EMACS 24.3.1, installed Emacs-live and it worked well (if I start emacs-live without Slime it works), downloaded Slime-Pack from git and added this line to .emacs-live.el (live-append-packs '(~/.live-packs/slime-pack/)) I'm on a...

Invalid specialized parameter in method lambda list


lisp,common-lisp
I am trying to write a simple coin flip program in Common Lisp. This is the code I have (defun yn (let ht (random 1) (if (eq ht 1) (princ heads) (princ tails)) ) ) It seems simple enough, but I keep getting the error: "Invalid specialized parameter in method...

Python subprocess throws [Errno 2] No such file or directory, error generated only when it on a remote host


python,linux,ssh,subprocess,python-2.6
I'm running python 2.6. I'm getting the subprocess throws [Errno 2] No such file or directory only when I run the script via ssh. For example, if I run the script manually on the machine, there are no errors, but if I do ssh hostname script.py --host hostname it generates...

parameter error in python script & TOR proxy server


python,sockets,subprocess,tor,proxy-server
I am Noob in Python. My Boss instructed me to run this python script with TOR proxy server running. He told me to pass this parameter this way: python DownloadYP.py /Users/myfolder/ japan http://www.jpyellow.com/company 1 222299 He configured it on MAC. I am using windows. So my parameter is this way:...

How to use FFI:def-call-in in clisp


lisp,common-lisp,ffi,clisp
I have figured out how to make use of shared objects created from C code into Clisp using FFI:def-call-out but I am not able to figure out how to use FFI:Def-call-in. I don't know the process and actually I am confused if clisp will also create some .so file that...

Dynamic 2d array in lisp


common-lisp,clisp
I want to have a 2D array in Lisp . But each row can have different number of elements(At max 5). So I thought of maintaining another single list to store the current sizes of each row, and update them whenever required. So, my code goes like this : (setq...

What is a common-lisp analogue of python's argparse?


python,common-lisp,argparse
What is a common-lisp analogue of python's argparse library for parsing command-line arguments?

Subprocess module feeding new line character


python,python-3.x,subprocess
I'm writing a script that uses subprocess module to run the net time \\server windows command, and parse the output of that using regex. import datetime import subprocess import re myDate = datetime.datetime.now().strftime("%y-%m-%d") myTime = datetime.datetime.now().strftime("%H:%M") myDateTime = datetime.datetime.now().strftime("%y-%m-%d %H:%M") results_file = 'C:\\temp\\ServerTimes_{0}.txt'.format(myDate) servers_central = 'C:\\temp\\DMS-CENTRAL.txt' sql_central = 'gt2-dms1cst' def...

How to use a parameter in a macro call?


macros,lisp,common-lisp,sbcl
I have defined the following simple macro: (defmacro define-class (class-name) `(defclass ,class-name ()())) And now I want to use it in the following function: (defun create-data (mode) (define-class mode)) After compiling the last function I get the following message, the variable MODE is defined but never used. And when I...

Use NamedTemporaryFile to read from stdout via subprocess on Linux


python-2.7,subprocess,temporary-files
import subprocess import tempfile fd = tempfile.NamedTemporaryFile() print(fd) print(fd.name) p = subprocess.Popen("date", stdout=fd).communicate() print(p[0]) fd.close() This returns: <open file '<fdopen>', mode 'w' at 0x7fc27eb1e810> /tmp/tmp8kX9C1 None Instead, I would like it to return something like: Tue Jun 23 10:23:15 CEST 2015 I tried adding mode="w", as well as delete=False, but...

Weird non-lisp errors with capturing labels with a lambda and `reduce`


lambda,crash,closures,common-lisp,allegro-cl
I stumbled across a very weird bug (?) while I was developing an application in Allegro Common Lisp, v9.0 for Windows. I do not get the regular lisp errors, instead I get system errors encapsulated in a lisp condition. I managed to create a simple test-case to reproduce the error...

Insertion into a list doesn't reflect outside function whereas deletion does?


list,lisp,common-lisp
I am new to Lisp. Deletion of an item in a list by a function gets reflected outside the function but insertion doesn't. How can I do the same for insertion? For example (defun test (a b) (delete 1 a) (delete 5 b) (append '(5) b) (member '5 b)) (setq...

Python open and kill subprocess


python,subprocess
I'm working on Widnows 7 (32 bits) and this is my code: def start_mviewer_broker(broker_path, test_name): """ The function starts the broker server""" try: print("**** start_mviewer_broker ****") p = subprocess.Popen('start python ' + broker_path + ' ' + test_name, shell=True) return p except: print("**** start_mviewer_broker - EXCEPTION ****") return 0 def...

Trying to multiply all elements in a list of sublists with elements in another list in lisp


list,common-lisp
I'm trying to multiply a list with n sublists with a list with n scalars. It's supposed to work like this: (kmult-matrix '((3 4 2 4) (2 5 6 9) (1 -2 8 10)) '(2 3 5)) => ((6 8 4 8) (6 15 18 27) (5 -10 40 50))...

Backquote String Interpolation


macros,lisp,common-lisp
Is it possible to use lisp's macro to do string interpolation? For instance, can I make a macro like this: (defmacro test (a) `",a") So that (test abc) returns "abc" as a string? I could probably cheat by quoting it and turning that quote into a string, but that doesn't...

Why does ln from Python subprocess fail while it succeeds from normal command line?


python,bash,subprocess,exit-code,ln
As the title says: >>> from subprocess import check_output >>> check_output(['ln', '~/other_folder/src/models/sc_models.py', './src/models/sc_models.py']) Traceback (most recent call last): File "<input>", line 1, in <module> File "/usr/lib/python2.7/subprocess.py", line 573, in check_output raise CalledProcessError(retcode, cmd, output=output) CalledProcessError: Command '['ln', '~/other_folder/src/models/sc_models.py', './src/models/sc_models.py']' returned non-zero exit status 1 >>> exit() $ ln...

sbcl run-program hang when there is large amount of output from program


common-lisp,hang,sbcl
Recently I find a run-program hang issue of sbcl 1.2.7 (32bits, linux). The code as following (progn (with-open-file (s "test.out" :direction :output :if-exists :supersede) (loop repeat 900 do (write-line (make-string 76 :initial-element #\x) s))) (run-program "/bin/bash" (list "-c" "cat test.out") :output :stream)) That is when the "cat test.out" produce many...

Redirect files from Popen to subdirecotry


python,redirect,subprocess,stdout,popen
I have a pipeline composed of a number of scripts. Almost all of the scripts read and write to files, and some of the scripts in the pipeline have conflicting file formats (they will read and attempt to perform actions on files that they should not due to the files...

python3 subprocess in Oracle Linux (wget -o)


python-3.x,path,subprocess
I see there are several posts on python subprocess invoking bash shell commands. But I can't find an answer to my problem unless someone has a link that I'm missing. So here is a start of my code. import os; import subprocess; subprocess.call("wget ‐O /home/oracle/Downloads/puppet-repo.rpm https://yum.puppetlabs.com/puppetlabs-release-el-6.noarch.rpm"); When I do wget...

Processing file paths with parentheses with Python subprocess


python,linux,bash,subprocess
The paths of the files I want to process contain parentheses in them. path = "/dir/file (with parentheses).txt" I'm trying to process them in Python as follows: subprocess.call("./process %s" % path, shell=True) However, I get the following error /bin/sh: 1: Syntax error: "(" unexpected How can I pass the correct...

Run a setup.py from another Python script, path issue, installing in calling script dir?


python,subprocess,setup.py
First, I've found how to call a script from within an other script in Python, the call works perfectly well, but here's the problem I'm running into : In order to easy-install my web-app (Bottle) on an another server, I packed inside a /redist rep, with mod_wsgi and PyMySQL source...

where is the text printed by C printf


io,common-lisp,cffi
I happened encounter a trouble with calling C printf function from SBCL via cffi. The problem is when I call printf function, I can't find the output text, just the return value of printf function show on the REPL. But when I quit SBCL, the output text appears on the...

Pass Buffer to ChildProcess Node.js


javascript,node.js,image-processing,subprocess
Here I have on Node.Js where I want to do Image Processing in a Sub Process. As you will see I take the file image.jpg and want to write it back to hello.jpg in a subprocess: var node = require('child_process').spawn('node',['-i']); var fs = require('fs'); node.stdout.on('data',function(data) { var fs = require('fs');...

Pythonic way to detach a process?


python,subprocess,detach,etcd
I'm running an etcd process, which stays active until you kill it. (It doesn't provide a daemon mode option.) I want to detach it so I can keep running more python. What I would do in the shell; etcd & next_cmd I'm using python's sh library, at the enthusiastic recommendation...

File gets created, but subprocess call says no such file or directory


python,subprocess
I am trying to redirect the output of a subprocess call to a file def get_arch(): global ip_vm in_cmd = "uname -p" with open('/home/thejdeep/arch_list',"w") as outfile: print outfile subprocess.call(in_cmd,stdout=outfile) print "Hello" for i in ip_vm: cmd = "ssh [email protected]"+i+" 'uname -p'" with open('/home/thejdeep/arch_list',"a") as outpfile: subprocess.call(cmd,stdout=outpfile) The file gets created....

python3 - subprocess with sudo to >> append to /etc/hosts


python,subprocess
I've been wrestling with solutions from "How do I use sudo to redirect output to a location I don't have permission to write to?" and "append line to /etc/hosts file with shell script" with no luck. I want to "append 10.10.10.10 puppetmaster" at the end of /etc/hosts. (Oracle/Red-Hat linux). Been...

Append output of multiple subprocesses to array in Python


python,arrays,subprocess
I'm working on a script in Python which open multiple subprocesses in this way: for file in os.listdir(FOLDER): subprocess.Popen(([myprocess])) Now this processes could be 10-20 running in parallel, and each of them will output in the console a single string line. What I want to do is to append these...

subprocess python 3 check_output not same as shell command?


python-3.x,subprocess
I am trying to use the subprocess module in python but its a bit tricky to get working. Here's my code import sys import os import subprocess import shlex def install_module(dir_path, command): c = shlex.split(command) os.chdir(dir_path) try: p = subprocess.check_output(c, shell=True) except subprocess.CalledProcessError as e: #print('install failed for: ' +...

How to move a file in Lisp using rename-file


lisp,common-lisp,pathname
What's the best way to move a file in Lisp in an implementation-independent way? For example I have an image file: (setq oldpath #P"SERVER:PICTURES;TEMP;PHOTO.PNG") and I want to move it out of the TEMP directory into the PICTURES directory. This seems to work: (setq newpath (make-pathname :host (pathname-host oldpath) :directory...

Structuring large Lisp applications


lisp,common-lisp,quicklisp,asdf
I am currently trying to wrap my head around packages, systems & co. I now have read Packages, systems, modules, libraries - WTF? a few times, and I think I'm still having difficulties to get it right. If I simply want to split a Lisp source file into two files,...

Running compiled lisp program with clisp-2.49 on OS X


common-lisp,clisp
I have just started out programming in Common Lisp using GNU clisp-2.49 (compiled from source) as my implementation on OS X 10.10 Yosemite. I've written a simple "Hello World" program as follows: (EXT:SAVEINITMEM "test" :INIT-FUNCTION 'main :EXECUTABLE t) (defun main () (format t "Hello World!") (EXT:EXIT)) When I run: $...

Creating A Process Queue in Common Lisp


common-lisp
I have a server running Hunchentoot (CentOS and SBCL). When a user submits a specific type of post request, a subprocess is launched (run-program) which can take up to four minutes to complete. If five people perform that specific type of request at the same time, the server runs out...

How to run a line in Powershell in Python 2.7?


python-2.7,powershell,subprocess
I have a line of Powershell script that runs just fine when I enter it in Powershell's command line. In my Python application which I run from Powershell, I am trying to send this line of script to Powershell. powershell -command ' & {. ./uploadImageToBigcommerce.ps1; Process-Image '765377' '.jpg' 'C:\Images' 'W:\product_images\import'}'...

Common Lisp Object System method execution order


lisp,common-lisp,clos
I have the following two classes: (defclass person () ()) (defmethod speak ((s person) string) (format t "-A" string)) (defmethod speak :before ((s person) string) (print "Hello! ")) (defmethod speak :after ((s person) string) (print "Have a nice day!")) (defclass speaker (person) ()) (defmethod speak ((i speaker) string) (print "Bonjour!"))...

lisp: dynamic scope vs explicit parameter passing


design-patterns,coding-style,common-lisp,anti-patterns,dynamic-scope
I see two different patterns for "output" functions in (common) lisp: (defun implicit () (format t "Life? Don't talk to me about life!")) (defun explicit (stream) (format stream "This will all end in tears.")) (defun test-im-vs-ex-plicit () (values (with-output-to-string (stream) (let ((*standard-output* stream)) (implicit))) (with-output-to-string (stream) (explicit stream)))) Is using...

No-argument (and) returns t


clojure,scheme,lisp,common-lisp
Both CL and Scheme define (and) to return t (or #t) with no arguments. I'm trying to understand the rationale for this. My naive assumption is that an empty set is false, and passing in zero arguments feels like passing in nothing that can be true. Edit: clojure follows the...

If strings are vectors, why are they immutable?


lisp,common-lisp
if strings are vectors of characters, and a vector's elements can be accessed using elt, and elt is setf-able - then why are strings immutable?

Read input into string in Lisp reader macro


macros,common-lisp,reader-macro
I am trying to make a reader macro that would convert @this into "this". This is what I currently have: (defun string-reader (stream char) (declare (ignore char)) (format nil "\"~a\"" (read-line stream t nil t)) ) (set-macro-character #\@ #'string-reader ) The problem is that this requires that I put a...

PYTHON subprocess cmd.exe closes after first command


python,cmd,subprocess,stdout,stdin
I am working on a python program which implements the cmd window. I am using subproccess with PIPE. If for example i write "dir" (by stdout), I use communicate() in order to get the response from the cmd and it does work. The problem is that in a while True...

ffmpeg subprocess fail to open on OS X


python,ffmpeg,subprocess
I've got this script: PATH = os.path.dirname(os.path.abspath(__file__)) global TEMP for video in os.listdir(VIDEOS): ffmpeg = PATH + "/ffmpeg/ffmpeg" arg1 = " -v 0 -i " arg2 = VIDEOS + "/" + video arg3 = " -r 1 -f image2 " arg4 = TEMP + "/" + os.path.splitext(video)[0] + "-%d.jpg" subprocess.Popen(ffmpeg...

Start a background process upon successful quickstart in cherrypy


python,rest,subprocess,cherrypy
I have a REST WebService built on top of cherrypy. The service goes online with the cherrypy.quickstart() call. I want to start a background process with subprocess.Popen() right after the service goes online. The cherrypy.quickstart() call is blocking. How can I add a callback to start the background process?...

What does the non-terminating-p argument of set-macro-character do?


macros,common-lisp
Set-macro-character has an optional argument called non-terminating-p. It seems to be used to indicate whether another character should be read after reading the macro character, but the reader algorithm seems to ignore this argument. Is there a difference whether I set it to true or false?

Using subprocess.check_output for a command with 2>/dev/null


python,osx,subprocess
I am on Mac OS X Yosemite 10.10 and Python 2.7. If I type the following: du -g -d1 /Users 2> /dev/null in the command line, everything works perfectly. Now, my goal is to use that command in a python script. My idea was to use the following: import subprocess...

EVAL/APPLY: too many arguments given to F


lisp,common-lisp,clisp
Hello why do i get *** - EVAL/APPLY: too many arguments given to F on function call with nested lists parameter. I cannot figure it out, since I passed a simple nested list. (defun f (L) (cond ((NULL l) nil) ((listp (car L)) (append (F(car L))) (F(cdr L) (car (F...

Updating the window in response to CLIM frame commands


common-lisp,clim,mcclim
While trying to figure out CLIM, I ran into this example program. It's a simple maze game. The author claims to have tested it in LispWorks (and even has #+Genera in there, implying that this program would work on a real Lisp Machine), but I'm trying to get it working...

Python subprocess Messy output


python,subprocess
With the following code: output = subprocess.check_output(['wmic', 'PATH', 'Win32_videocontroller', 'GET', 'description']) print(output , "\n") I get the next output: b'Description \r\r\nNVIDIA GeForce 710M \r\r\nIntel(R) HD Graphics 4000 \r\r\n\r\r\n' When I use the commando wmic path win32_videocontroller get desriptionin my CMD I get only the videocard info back. Is this possible...

Local dynamic binding in common lisp


closures,common-lisp,dynamic-binding
Honnestly, I'm not sure I fully understand what it means for a binding to be "dynamic" versus "lexical". But I understand that when I use defvar or defparameterto define a binding, 1. it declares a global variable 2. the binding is declared "special", so that it can be shadowed by...

Using dd command through Python's subprocess module


python,subprocess,dd
Through Python's subprocess module, I'm trying to capture the output of the dd command. Here's the snippet of code: r = subprocess.check_output(['dd', 'if=/Users/jason/Desktop/test.cpp', 'of=/Users/jason/Desktop/test.out']) however, when I do something like print r I get a blank line. Is there a way to capture the output of the dd command into...

How to execute and save result of an OS command to a file [duplicate]


python,python-2.7,subprocess
This question already has an answer here: Python: How to Redirect Output with Subprocess? 4 answers In python 2.7, I would like to execute an OS command (for example 'ls -l' in UNIX) and save its output to a file. I don't want the execution results to show anywhere...