FAQ Database Discussion Community


F# Changing parameters precedence

f#,pipeline,currying
I'm new to F# and have a question about functions pipeline. Let's say we have a function map which maps list of functions to array of values creating a list of arrays: //val map : ('a -> 'b) list -> 'a [] -> 'b [] list let map funcs vals...

Which one is right one in pipelining?

cpu,pipeline,gantt-chart
I'm studying CPU pipelining, and had a trouble. I want to know which one is right pipelining in below picture In my opinion, the first Gantt chart is kinda "structural hazard" becuase "IF" stage is partially overlapped. I think that using one stage for two instruction is not allowed. So...

What is the equivalent of BlobstoreLineInputReader for targeting Google Cloud Storage?

python,google-app-engine,mapreduce,pipeline
This is a python appengine question, mapreduce library 1.9.21 . I have code writing lines to a blob in the local blobstore, then processing that using mapreduce BlobstoreLineInputReader. Given that the files api is going away, I thought I'd retarget all my processing to cloud storage. I would expect to...

PowerShell Split-Job empty pipe element not allowed

powershell,pipeline
I would really like to use the function Split-Job to throttle or run the same script block in parallel, so it goes quicker. This is especially useful when I need to launch multiple times the same copy command on different input. The code can be found here and always spits...

How do I write a cmdlet function to allow input “file” or pipeline?

powershell,parameters,pipeline
I'm trying to write a cmdlet that mirrors what UNIX commands like cat give me (reading from a file or standard input in the absence of a file). It needs to allows for the following possibilities: cmdlet -file <inputFileName> cmdlet -object <objectName> <someObject> | cmdlet I'm stretching the definition of...

C++ Partial template specialization - design simplification

c++,templates,c++11,pipeline,partial-specialization
I am working on a pipeline/dataflow design patter for a specific application area. I have an 'algorithm data output' class (AlgorithmOutput) which serves as an interface between two connected network elements. In particular, it provides a set of templated getOutput<size_t N> methods which are used for the data output from...

Passing properties to a function pipeline output error

function,powershell,csv,export-to-csv,pipeline
I'm getting this error when I add to the return the -prefix "File_Name" to the end of Return $Results | Out-Excel -Prefix "File_Name" if I leave -prefix "File_Name" off of the script it works. What am I missing? Out-Excel : The input object cannot be bound to any parameters for...

Pipelined MIPS delayed branching example

assembly,mips,pipeline
I'm given this MIPS pseudo-assembly code: add r7,r8,r9 beq r1,r2,40 --- lw r4,O(r1) ' sub r5,r1,r4 ' and r6,r4,r2 ' <--------------- ' or r4,r2,r3 I am supposed to assume that there are no structural hazards and that delayed branching is used. The questions are: how will the instructions be executed...

Mongodb How to add addtional information when aggregating?

python,mongodb,pipeline
i am a beginner , i wrote a line for my pipeline that works, but i wanna add other information to my output, like screen name , or number of tweets.I tried to add that under $group but gave me an syntax error everytime here is my pipeline: def make_pipeline():...

Pipe complete array-objects instead of array items one at a time?

arrays,powershell,object,syntax,pipeline
How do you send the output from one CmdLet to the next one in a pipeline as a complete array-object instead of the individual items in the array one at a time? The problem - Generic description As can be seen in help for about_pipelines (help pipeline) powershell sends objects...

stream a lot of files to stdout

bash,pipeline
I have a large number of filed in directory - ~100k. I want to combine them and pipe them to standard output (I need that to upload them as one file elsewhere), but cat $(ls) complains that -bash: /bin/cat: Argument list too long. I know how to merge all those...

Organizing pipeline in MIPS

mips,pipeline
I'm unsure about how the following properties affect pipeline execution for a 5 stage MIPS design (IF, ID, EX, MEM, WB). I just need some clearing up. only 1 memory port no data fowarding. Branch stalls until end of * stage Does the 1 memory port mean we cannot fetch...

listening on netcat works but not grep'able (or several other utilities)

linux,bash,shell,pipeline,netcat
I'm testing some netcat udp shell tools and was trying to take output and send it through standard pipe stuff. In this example, I have a netcat client which sends 'foo' then 'bar' then 'foo' with newlines for each attempt reading from the listener: [[email protected] ~ 05:40:20 ]# exec 5<...

shell variable not keeping value [duplicate]

bash,shell,unix,pipeline,subshell
This question already has an answer here: Bash while read resetting variable values - using the read builtin in a pipeline 3 answers I have 3 lines in file /root/backuplist.txt. The first echo prints perfectly, but the last one prints an empty line; I'm not sure why. Somehow, the...

PowerShell pipeline Get-Content to split and then select expression

powershell,pipeline
i have a file and i want split selective lines and take selective cells. this work: Get-Content F:\dat | ? {$_ -like "1,*"} | % {$_.Split(',')} but I can not continue to concatenate a select some items: Get-Content F:\dat | ? {$_ -like "1,*"} | % {$_.Split(',')} | Select -Property...

Can one train estimators in a scikit-learn pipeline simultaneously?

python,machine-learning,scipy,scikit-learn,pipeline
Is it possible to do the following in scikit-learn? We train an estimator A using the given mapping from features to targets, then we use the same data (or mapping) to train another estimator B, then we use outputs of the two trained estimators (A and B) as inputs for...

Getting values with jedis pipeline

java,redis,pipeline,jedis
I have a list of ids that I want to use to retrieve hashes from a Redis server using the java client jedis. As mentioned in the documentation, Jedis provides a way to use the pipeline by declaring Response objects and then sync the pipeline to get values: Pipeline p...

Formal argument names for magrittr aliases

r,alias,plyr,pipeline,magrittr
I was wondering if anybody knew or could help me find the formal argument names for all of the magrittr alias functions. For example, I know that the argument for 'set_colnames' is 'value'. df <- data.frame(1:3, 4:6, 7:9) %>% set_colnames(value = c('a', 'b', 'c') Normally, I just pass the arguments...

Get matching elements in mongodb array

php,mongodb,aggregation,pipeline
I want to use aggregation to get this array only with those tickets, which have start field after 2015-06-16. Can someone help me with the pipeline? { "name" : "array", "tickets" : [ { "id" : 1, "sort" : true, "start" : ISODate("2015-06-15T22:00:00.000Z") }, { "id" : 2, "sort" :...

Files from live sources in gstreamer are corrupt

gstreamer,pipeline,java-gstreamer
I have a problem with files written from live sources (webcams) and pseuso-live sources (screencaptures) in GStreamer. The resulting files don't have a time length, and as a result don't play at all in Media Player Classic. They do play in Firefox, but without a time length and sometimes with...

How do I delimit a fetched data record on MySQL?

mysql,delimiter,pipeline,text-formatting
I have a column which returns pipeline delimited records as below: 'fromState=A|Count=5|highLimit=B|status=C|presentValue=D|alarmValue=E' Is there a way to have it as a more formatted output like: 'fromState=A Count=5 highLimit=B status=C presentValue=D alarmValue=E' i.e., pipelines are replaced each value is on a new line (optional) ...

Scrapy - Activating an Item Pipeline component - ITEM_PIPELINES setting

python,scrapy,settings,pipeline
In scrapy documentation there is this information: Activating an Item Pipeline component To activate an Item Pipeline component you must add its class to the ITEM_PIPELINES setting, like in the following example: ITEM_PIPELINES = { 'myproject.pipelines.PricePipeline': 300, 'myproject.pipelines.JsonWriterPipeline': 800, } The integer values you assign to classes in this setting...

What does the Sitecore CheckSize Pipeline do?

sitecore,pipeline,sitecore7
Looking through the web.config of a Sitecore project that we have I can see that there is a Pipeline in the <uiUpload> section of the code which is called CheckSize. I am hoping that I can use this to check the size of an item that is being uploaded to...

StackExchange.Redis: Batch access for multiple hashes

redis,pipeline,stackexchange.redis,batching
So I need to access in bulk many different hashes (in StackExchange.Redis, I have different RedisKey's). What is the best (fastest) way to do it? For example, for these two possible implementations, is either correct? Which one works better? List<Task<HashEntry[]>> list = new List<Task<HashEntry[]>>(); List<RedisKey> keys; //Previously initialized list of...

Strange syntax for a redirection

bash,pipe,named-pipes,pipeline,io-redirection
I just came across the following script while browsing this question: rm -f out mkfifo out trap "rm -f out" EXIT while true do cat out | nc -l 1500 > >( # parse the netcat output, to build the answer redirected to the pipe "out". export REQUEST= while read...

How to use a Pipe between two processes in Process.Start

vb.net,ffmpeg,pipeline,video-encoding
I have to run this command line using VB.NET: "H:\videotest\test.vpy" - -y | "H:\Release\data\bin64\ffmpeg.exe" -hwaccel auto -y -i - -map 0:v:0 -c:v libx265 -crf 20.0 -preset 5 -x265-params...

Unix - pipeline ls - la | less C executable giving double total file size vs shell

shell,unix,pipeline
*Edit - Stephen has answered this question in the comments below * so basically I have made two seperate child processes(using two seperate methods with their own fork) to execute the command ls -la | less using pipe. The first one executes ls like this: execl("/bin/ls", "ls", "-la", NULL); The...

Laravel 5 Commands - dispatch command pipeline

command,pipeline,laravel-5
Is it possible to create a sort of command pipeline when dispatching a command? For instance: $this->dispatch(new UploadFileCommand) would call ValidateFileCommand, WhateverCommand. It's like pipeThrough but triggers specific commands....

Difference between 2 ways working with pipes using ARGF?

ruby,command-line,pipeline
Using ARGF I can create Ruby programs that respect pipelines. Suppose, I to constantly read new entries: $ tail -f log/test.log | my_prog I can do this using: ARGF.each_line do |line| ... end Also, I found another way: while input = ARGF.gets input.each_line do |line| ... end end Looks like,...

Find lines by pattern, leave only pattern but leave unmatched lines as they are

regex,awk,sed,grep,pipeline
So there is an input file res.txt like this processing file 0x8001.values channel 1: 123 channel 2: 234 channel 3: 345 processing file 0x8002.values channel 1: 456 channel 2: 567 channel 3: 678 I have a pattern like this 0x[0-9a-f][0-9a-f][0-9a-f][0-9a-f] Using, for example, grep -o "0x[0-9a-f][0-9a-f][0-9a-f][0-9a-f]" res.txt I can get...

scikit-learn pipeline

python,scikit-learn,pipeline,feature-selection
Each sample in My (iid) dataset looks like this: x = [a_1,a_2...a_N,b_1,b_2...b_M] I also have the label of each sample (This is supervised learning) The a features are very sparse (namely bag-of-words representation), while the b features are dense (integers,there are ~45 of those) I am using scikit-learn, and...

Is there a way to create a Cmdlet “delegate” that supports pipeline parameter binding?

powershell,delegates,pipeline,scriptblock
In .NET if you have a subroutine whose implementation might change from one call to another, you can pass a delegate to the method that uses the subroutine. You can also do this in Powershell. You can also use scriptblocks which have been described as Powershell's equivalent of anonymous functions....

In sklearn, does a fitted pipeline reapply every transform?

python,scikit-learn,pipeline,feature-selection
Apologies if this is obvious but I couldn't find a clear answer to this: Say I've used a pretty typical pipeline: feat_sel = RandomizedLogisticRegression() clf = RandomForestClassifier() pl = Pipeline([ ('preprocessing', preprocessing.StandardScaler()), ('feature_selection', feat_sel), ('classification', clf)]) pl.fit(X,y) Now when I apply pl on a new set, pl.predict(X_classify); is RandomizedLogisticRegression going...

Ring buffer log file on unix

unix,logging,pipeline,tail
I'm trying to come up with a unix pipeline of commands that will allow me to log only the most recent n lines of a program's output to a text file. The text file should never be more than n lines long. (it may be less when it is first...

Pipeline refill cycles for instructions in arm

assembly,arm,pipeline,instruction-set
Following are the instructions of arm Cortex M4 processor with 3 stage pipeline.How do we come to know the number of pipeline refill cycle for such instructions?? Assembler Cycles 1. MOV PC, Rm 1 + P 2. ADD PC, PC, Rm 1 + P 3. B <label> 1 + P...

Struggle to build scrapy pipeline to export csv file in the right format, any help please?

python,csv,scrapy,pipeline
@alexce I made the improvement according to the your suggestion below. But what I need is like the picture below. however each row/line should be one review: with date, rating, review text and link. so basically I need to let item processor to process each review of every page. currently...

automatically remove source files after collectstatic with django-pipeline

django,pipeline,collectstatic
with django-pipeline, is it possible to automatically remove source files after collectstatic ? for example : PIPELINE_JS = { 'stats': { 'source_filenames': ( 'js/jquery.js', 'js/d3.js', 'js/application.js', ), 'output_filename': 'js/stats.js', } } collectstatic : $ python manage.py collectstatic $ ls static/js jquery.js d3.js application.js stats.js (i don't want jquery.js, d3.js, application.js)...

Call a PHP instance method with another PHP file in system memory [closed]

php,sockets,soap,pipeline
I have a PHP socket server script named server.php that binding to a certain port and keep running for a long time. (actually it start in cli (terminal) mode and not will stop until ctrl+c command pressed.) Also there is a Soap-server PHP file named service.php that manage chat requests....

Updating Kafka Event Log

pipeline,apache-kafka,amazon-redshift,data-partitioning
I am using Kafka as a pipeline to store analytics data before it gets flushed to S3 and ultimately to Redshift. I am thinking about the best architecture to store data in Kafka, so that it can easily be flushed to a data warehouse. The issue is that I get...

Bash error: Integer expression expected

bash,shell,pipeline
In the sections below, you'll see the shell script I am trying to run on a UNIX machine, along with a transcript. When I run this program, it gives the expected output but it also gives an error shown in the transcript. What could be the problem and how can...

Gstreamer pipeline multiple sink to one src

command-line,gstreamer,pipeline
Looking for explanation how to using named elements in respect with muxing two inputs in one module. For instance muxing audio and video in one mpegtsmux modle gst-launch filesrc location=surround.mp4 ! decodebin name=dmux ! queue ! audioconvert ! lamemp3enc dmux. ! queue ! x264enc ! mpegtsmux name=mux ! queue !...

Why does $_ behave differently when in a .ps1 versus .psm1 files?

powershell,module,pipeline
Suppose you define map_ps in map.ps1: function map_ps{ [CmdletBinding()] param([parameter(ValueFromPipeline=$true)]$InputObject,$sb,$ArgumentList) process{&$sb $ArgumentList} } Suppose you also define another function map_psm with identical implementation in a well-formed module called map.psm1: function map_psm{ [CmdletBinding()] param([parameter(ValueFromPipeline=$true)]$InputObject,$sb,$ArgumentList) process{&$sb $ArgumentList} } Calling each function with identical parameters does not yield the same result: PS C:\>...

How to pipe the output of a command to a file without powershell changing the encoding?

powershell,encoding,pipe,pipeline
I want to pipe the output of a command to a file: PS C:\Temp> create-png > binary.png I noticed that Powershell changes the encoding and that I can manually give an encoding: PS C:\Temp> create-png | Out-File "binary.png" -Encoding OEM However there is no RAW encoding option, even the OEM...