go,io,sync , How can I redirect, in Go, the stdout and stderr of a command to both the console and a log file while outputting in real time?


How can I redirect, in Go, the stdout and stderr of a command to both the console and a log file while outputting in real time?

Question:

Tag: go,io,sync

The following bit of code does exactly what I want, except it only prints to the console.

cmd := exec.Command("php", "randomcommand.php")

cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr

if err := cmd.Run(); err != nil {
    log.Fatal(err)
}

randomcommand.php:

// randomcommand.php simply alternates output between stdout and stderr 20 times

$stdout = fopen('php://stdout', 'w+');
$stderr = fopen('php://stderr', 'w+');
for ($i = 1;$i <= 20; $i++) {
    fwrite($stdout, "stdout $i\n");
    fwrite($stderr, "stderr $i\n");
}

Output:

stdout 1
stderr 1
stdout 2
stderr 2
stdout 3
stderr 3
stdout 4
stderr 4
stdout 5
stderr 5
stdout 6
stderr 6
stdout 7
stderr 7
stdout 8
stderr 8
stdout 9
stderr 9
stdout 10
stderr 10
stdout 11
stderr 11
stdout 12
stderr 12
stdout 13
stderr 13
stdout 14
stderr 14
stdout 15
stderr 15
stdout 16
stderr 16
stdout 17
stderr 17
stdout 18
stderr 18
stdout 19
stderr 19
stdout 20
stderr 20

What I'm trying to achieve is:

  1. Print the command's stdout and stderr to the console in real-time;
  2. Log the command's stdout and stderr to a file, exactly as it appears in the console;
  3. Respect the exact order of writes to stdout and stderr, and;
  4. Not modify the command itself

I've tried piping the command stdout and stderr into scanners or using io.Copy, each in a goroutine, but order of the output was not maintained. Probably because alternating between goroutines would have to occur at the same time as output alternates, which is just about impossible.

I also tried the using "select" as in the examples from here. But order of output was not maintained.

Same problem with Logstreamer.

The other idea I have to try is to see if I can somehow just read from the console buffer, but this just doesn't feel right.

Does anyone know if this is even possible and worth pursuing?


Answer:

As I said in the comment section, this can be achieved using MultiWriter

package main

import (
    "io"
    "log"
    "os"
    "os/exec"
)

func main() {
    // Logging capability
    f, err := os.OpenFile("log.log", os.O_RDWR|os.O_CREATE|os.O_APPEND, 0666)
    if err != nil {
        log.Fatalf("Error opening file: %v", err)
    }
    defer f.Close()
    mwriter := io.MultiWriter(f, os.Stdout)
    cmd := exec.Command("ls")
    cmd.Stderr = mwriter
    cmd.Stdout = mwriter
    err = cmd.Run() //blocks until sub process is complete
    if err != nil {
        panic(err)
    }
}

When you declare your command, and before you run it, just specify that Stdout and Stderr are using the MultiWriter defined above. This MultiWriter instance contains both a log file and the standard output.


Related:


Why doesn't “go get gopkg.in/…” work while “go get github.com/…” OK?


windows,git,powershell,github,go
I try to use go get gopkg.in/fatih/pool.v2 to install pool according to Readme.md, but can't success: C:\Users\xiaona\Documents\GitHub> go get -v gopkg.in/fatih/pool.v2 Fetching https://gopkg.in/fatih/pool.v2?go-get=1 https fetch failed. Fetching http://gopkg.in/fatih/pool.v2?go-get=1 import "gopkg.in/fatih/pool.v2": http/https fetch: Get http://gopkg.in/fatih/poo l.v2?go-get=1: dial tcp 107.178.216.236:80: ConnectEx tcp: A connection attempt failed because the connected party did not...

Golang, how to return in func FROM another func?


go,return,func
I want to end execution in parent func apiEndpoint() upon calling/exiting in a child func apiResponse() func apiEndpoint() { if false { apiResponse("error") // I want apiResponse() call to return (end execution) in parent func // so next apiResponse("all good") wont be executed } apiResponse("all good") } func apiResponse(message string)...

Why won't mgo unmarshall my struct properly?


mongodb,go,bson,mgo
Earlier I posted this question asking about writing custom BSON marshalling/unmarshalling in Go using mgo. Now I've come to test it I think I've hit on a bigger problem. All my structs unmarshal to nil values. This is my currency struct with the implementations of bson.Getter and bson.Setter: type Currency...

More strict compile options in GoLang


go
Is there an option in golang to fail build if a function returns a value and the returned value is not used? For example, I want an error / warning when we have: func abc() error and it is called without an lvalue as: abc() EDIT: This is not for...

Mixing Java NIO with IO input streams read operation


java,sockets,io,nio
I need to use NIO to allow the server side to support timeout on write operations, but handling reading operations on the socket channels complicates my program. I was wondering if it's possible to write to the sockets using NIO but to read using regular IO, something like ((SocketChannel) selectedKey.channel()).socket().getInputStream().read(buffer)...

Rust: Lifetime of String from file [duplicate]


file,io,rust
This question already has an answer here: Return local String as slice 1 answer I'm trying to read in some external GLSL code into Rust. The reading works properly, but I run into a lifetime issue in the final expression (in the Ok(_) branch) error: s does not live...

Handling Custom BSON Marshaling (Golang & mgo)


json,mongodb,go,bson
I have a number of structs that require custom marshalling. When I was testing I was using JSON and the standard JSON marshaller. As it doesn't marshal unexported fields, I needed to write a custom MarshalJSON function, which worked perfectly. When I called json.Marshal on the parent struct containing the...

How do I get the submit value from the request object


html,go
I am trying to make a form POST request with two different submit buttons. I would like to try and get the submit button value in go. How do I go about getting this from the http.Request object. HTML Code: <form action="/save" method="POST"> <div><span>Title: </span><textarea name="title" placeholder="Link">{{printf "%s" .Title}}</textarea></div> <div>...

How to call Fortran routine with unit number argument from C


c,io,fortran,shared-libraries,abi
If I have a Fortran subroutine which takes a Fortran IO Unit as one of its parameters (for printing debug information to), and this function is compiled into a shared library, how do I correctly call this function from C? ! An example subroutine that I want to call from...

Using allocatable/assumed-size arrays with namelist read write


io,fortran,intel-fortran,fortran2003
I am using VS2012 and Intel Visual Fortran 2015. According to https://software.intel.com/en-us/forums/topic/269585, it is now allowed to use allocatable and assumed-size arrays with namelist read and write; however, I am still getting the error "A namelist-group-object must not be an assumed-size array". example code: subroutine writeGrid(fname, grid) character*(*) :: fname...

Does Mgo cache connection strings?


mongodb,go,mgo,compose
My Go application has been connecting to a MongoDB (hosted on Compose.io) using MGO with no issues. Today I decided to delete this database and add a different one (again using Compose). I updated the connection string. So I connect using: db, err := mgo.Dial("mongodb://<username>:<password>@dogen.mongohq.com:10048/db-name") with the username and password...

Server initiated requests


http,go,http2
I know that http is a request-response protocol, the problem in short is a client makes a request to the server to start a long running process, and i want to inform the client over the progress with a simple json message containg progress info. in http1 i know that...

convert chan to non chan in golang


go,type-conversion
Is it possible to let function funcWithNonChanResult have the following interface: func funcWithNonChanResult() int { And if I want it to use function funcWithChanResult with the interface: func funcWithChanResult() chan int { In other words, can I somehow convert chan int to int? Or I must have chan int result...

Cannot call tests inside packages other than main


go
I am writing tests for my go code and inside sender folder, sender package I have added exposed_api_test.go ( also tried exposed_api_test.go because I have code in exposed_api.go) package sender import ( "log" "testing" "github.com/stretchr/testify/assert" ) func TestTimeConsuming(t *testing.T) { assert.Equal(t, "test", "test1") } and when I run build and...

How do I synchronize a variable number of channels in Go?


go
I am testing out concurrency in Go by writing a client (in the form of a web server) that makes many requests to a web server and then returns how long it took to make those requests. Basically a benchmark tool. Here is to code I am using currently: main.go...

how to read in files made of double values line by line using c++


c++,io
I'm trying to do something very simple, but just can't get it right... Working with C++, I want to read in "myfile.tsv" which looks like this: 2.3 3.3 3.4 3.5 5.6 \n 1.2 1.3 1.2 \n 3.4 3.5 3.5 \n 4.4 4.6 1.3 1.5 \n ... many lines of double...

Trouble retrieving Has Many using Gorm


go,go-gorm
When I try to get episodes back from a podcast I get invalid association []. Not sure what I'm doing wrong. package main import ( "log" "github.com/jinzhu/gorm" _ "github.com/mattn/go-sqlite3" ) type Podcast struct { Id int Title string RssUrl string `sql:"unique_index"` Url string Episodes []Episode } type Episode struct {...

Mail with html content shows break lines or ignores newlines


email,templates,go,mandrill
I am sending mail to users via mandrill and I using both smtp and mandrill api to send. Content of the mail is rendered go template (.tpl) When I put template like Hi {{.name}}, <br/> This is support. <br/> it sends via mandrill api ok, but is visible when I...

Golang ORDER BY issue with MySql


mysql,database,go
I can't seem to dynamically ORDER BY with db.Select(). I've Googled without any luck... WORKS rows, err := db.Query("SELECT * FROM Apps ORDER BY title DESC") DOES NOT WORK rows, err := db.Query("SELECT * FROM Apps ORDER BY ? DESC", "title") I'm not getting any errors, the query simply fails...

Golang switch between structs


struct,go,interface,switch-statement
I'm new to golang and I'm trying to create a function that, based on the struct it's used on, will return a formatted string using Sprintf type Name struct { Title string First string Last string } type Location struct { Street string City string State string Zip string }...

Pylint on StringIO or buffered text


python,io,pylint
I need to analyze some code generated "magically" in a function. The example I'm using is very simple. Here is what I intend to do: from pylint import epylint from StringIO import StringIO source = StringIO() source.write('def test():\n') source.write(' b = 5\n') source.write(' return\n') source.seek(0) epylint.py_run(source) The result I get:...

Different performances in Go slices resize


go,stack,benchmarking,slice
I'm spending some time experimenting with Go's internals and I ended up writing my own implementation of a stack using slices. As correctly pointed out by a reddit user in this post and as outlined by another user in this SO answer Go already tries to optimise slices resize. Turns...

Why does my concurrent function exit prematurely in Go?


concurrency,go
I am looking through Go Bootcamp and am reading the Go Concurrency chapter right now. I have never used concurrency before in programming and don't understand the output of this program: package main import ( "fmt" "time" ) func say(s string) { for i := 0; i < 2; i++...

Appending lines for existing file in python [duplicate]


python,file,python-2.7,io
This question already has an answer here: How do you append to a file in Python? 6 answers I want to add lines to an existing file in python. I wrote the following two files print_lines.py while True: curr_file = open('myfile',r) lines = curr_file.readlines() for line in lines: print...

Go Yaml Interpretation Example


go,yaml
This Go Yaml Interpretation Example, https://gist.github.com/suntong001/74c85d15b19ef4b14b0e, can unmarshal a simple YAML like this: foo: 1 bar: - one - two Now I want to interpretation an array of the exact same data structure, what's the correct way to do? Below is my code so far, and it is not working...

Build a go binary for both Linux and Mac [duplicate]


go
This question already has an answer here: Cross compile Go on OSX? 2 answers How can I go build a project in a way that it works for both Mac and Linux (x86 and x64)? Is that even possible? I searched and found some stuff, but cannot make them...

Getting a PHP configuration variable in Go


php,go
I want to get the output of the command php -r 'echo get_cfg_var("some_var");' to check it against a predefined value. Currently, I have the following code: variableName := "default_mimetype" cmd := exec.Command("php", "-r", "'echo get_cfg_var(\"" + variableName + "\");'") out, err := cmd.CombinedOutput() after running, err.Error() returns "exit status 254"...

How to change a float64 number to uint64 in a right way?


go,floating-point,type-conversion,floating-point-conversion
package main func main() { var n float64 = 6161047830682206209 println(uint64(n)) } The output will be: 6161047830682206208 It looks like that when float64 change to uint64, the fraction is discarded....

measure execution time and stop waiting if too long in golang


go
I am trying to measure execution time of funcWithUnpredictiveExecutionTime function. func measureTime(expectedMs float64) (ok bool) { t1 := time.Now() funcWithUnpredictiveExecutionTime() t2 := time.Now() diff := t2.Sub(t1) The measuring is fine when funcWithUnpredictiveExecutionTime works faster than I expected. But if it works slower than expectedMs the measuring will not stop right...

How can I redirect, in Go, the stdout and stderr of a command to both the console and a log file while outputting in real time?


go,io,sync
The following bit of code does exactly what I want, except it only prints to the console. cmd := exec.Command("php", "randomcommand.php") cmd.Stdout = os.Stdout cmd.Stderr = os.Stderr if err := cmd.Run(); err != nil { log.Fatal(err) } randomcommand.php: // randomcommand.php simply alternates output between stdout and stderr 20 times $stdout...

JSON decoding in golang


json,curl,go
So I tried something based on the example here in my code, and get no data, but no error either. The code is: import ( "io" "fmt" "net/http" "encoding/json" ) type Credential struct { username string `json:"username"` password string `json:"password"` } func login(res http.ResponseWriter, req *http.Request) { if req.Method ==...

Issue reading and saving lines from .txt file in Java


java,arraylist,io
Objective & Issue Objective of the task I am trying to accomplish is to locate and read a text file from a provided path then copy each line of the text file into a ArrayList<String>. This "ApplicationFileReader" object is created in some control class by calling the constructor which takes...

How to keep a strong reference in Go?


arrays,go,slice
Is there any way I can keep a strong reference in Go? Given the following convoluted piece of code: package main import ( "fmt" ) func main() { slice := make([]int, 5) slice[3] = 25 // whatever index between 0 and 4 included I don't care slicesArray := make([]*[]int, 2)...

Serving Static Files with a HTTP 500 Status


go
Is there a way to serve static files over HTTP in Go with a custom status code (without re-writing a significant amount of private code)? From what I can see: http.ServeFile calls the helper function http.serveFile It then calls http.ServeContent after determining the mod time and size of the file/dir...

panic: runtime error: invalid memory address or nil pointer dereference when running Blockchainr


go,blockchain
I run blockchainr and get the following terminal output: 17:39:57 2015-06-16 [INF] loading db leveldb panic: runtime error: invalid memory address or nil pointer dereference panic: runtime error: invalid memory address or nil pointer dereference [signal 0xb code=0x1 addr=0x0 pc=0x402cb63] goroutine 1 [running]: main.search(0x4911ef8, 0xc20806e2d0, 0x0, 0x0, 0x4911ef8) /Users/mesquka/Downloads/blockchainr-master/src/blockchainr/main.go:185 +0x3fa...

How to read a middle size-specified chunk of a file in Python


python,io
If I have a text file, what is a clean way to start from a line at near the 5th MB into the file and read the subsequent 5 MB worth of lines into a list? Ideally into a list of lines. I know f.seek(5000000) starts at the 5th MB,...

How to make this code more performant?


performance,go,hashmap
I have this snippet of code that iterate over a map and filter some fields based on relation type, I have to run two loops and have a feeling that it is going to be slow for big maps. Is there any way I can technique/refactoring to make this code...

Golang - structs, intializing and memcpy - what is the proper way?


arrays,struct,go,initialization,custom-type
I'm new to Go and I wish to translate some C/C++ code I have to Go but I haven't been able to. The problem resides in 2 places: how can I initialize a struct I've defined and how to do a "memcopy" The code I'm talking about is this: http://play.golang.org/p/e8N255qEAk...

Storing columns on disk and reading rows


c++,file,matrix,io
So I have a C++ code that finds the inverse of a square matrix. The inverse is full and it would be bad to keep it all in memory because I am working with hundreds of thousands of columns. My code generates the columns one by one. After finding the...

Sequential file IO from different threads fail with FileSystemException


java,multithreading,windows-7,io
I am trying to see how fast i can do something like: write a small file rename it delete it This basically looks like: import java.io.IOException; import java.nio.file.Files; import java.nio.file.Path; import java.nio.file.Paths; public class QuickIO { public static void main(String[] args) throws IOException { Path fileToWriteTo = Paths.get("C:\\Temp\\somefile.txt"); Path fileToMoveTo...

Unmarshal JSON into a map in Go


json,dictionary,go
I'm having trouble figuring out how to load a "subsection" of JSON file into a map element. Background: I'm trying to unmarshal a somewhat complicated configuration file which has a strict structure, so I assume it's preferable to unmarshal into a "static" structure rather than into an interface{}. Here's a...

Does exist on Go something like macros in C++ like #ifdef so I can choose what to build based on flag?


go
I need to build in go for linux and windows with different packages same file on windows I need to import github.com/hashicorp/go-syslog and on linux import log/syslog and inside code in file I have to use syslog.ALERT or gsyslog.ALERT depending on os. Does exist on Go something like macros in...

Declare variable with format specifier in Go


go
In Python, I can declare a variable as d ='/some/dir/%s' and later replace %s with any value as >>> d = '/some/dir/%s' >>> d % "hello" '/some/dir/hello' Is it possible to do the same in Go? If so, how?...

Referencing yourself inside of a struct


struct,go
Let's say I have a type that's a struct like so: type Authorization struct { Username string Password string Handler func(http.HandlerFunc) http.HandlerFunc } And I have an array of these: type Authorizations map[string]*Authorization I want to be able to do something like this: var auth = Authorizations{ "test": *Authorization{ "someusername",...

Find the shortest path sum in a matrix. Is Dijkstra not optimal for this case?


algorithm,go
I am trying to solve the following problem from project euler (please take a look at description and the example in the link, but here is the short explanation). in the matrix, find the minimal path sum from the top left to the bottom right, by moving left, right, up,...

CGO converting Xlib XEvent struct to byte array?


c,go,xlib,cgo
I am creating a simple window manager (code based of the c code in tinywm) in Golang. To use Xlib, I am using cgo, so my header is: // #cgo LDFLAGS: -lX11 // #include <X11/Xlib.h> And I have a variable declaration, like: event := C.XEvent{} And then, I use this...

Golang - using/iterating through JSON parsed map


json,parsing,go
With PHP and Javascript (and Node) parsing JSON is a very trivial operation. From the looks of it Go is rather more complicated. Consider the example below package main import ("encoding/json";"fmt") type fileData struct{ tn string size int } type jMapA map[string] string type jMapB map[string] fileData func parseMapA(){ var...

image.Decode results in “unknown format” when source is multipart.File


image,go,martini
I have a multipart.File that a user uploads to my server, and then I take that file and upload it to s3 using aws-sdk-go, but I also want to create a thumbnail of that image. The code below works fine in my tests when I file is the return value...

Marshalling empty map[string]interface{} results in “null” instead of nil


go
I have a problem where Go is inserting "null" into my PostgreSQL database's jsonb columns if I try to do the following, and the structs property (in this case of type map[string]interface{}) is empty: accessMembers, _ := json.Marshal(c.AccessMembers) Doing a test print it outputs the same as the value which...