FAQ Database Discussion Community


My Concerns about Spring-Batch that you cant actually multi-thread/read in chunks while reading items

spring,batch-processing,spring-batch,spring-batch-admin
I was trying to batch simple file. I understand that I couldnt multi-thread it. So at least I tried to perform better while increasing the chunks param: @Bean public Step processFileStep() { return stepBuilderFactory.get("processSnidFileStep") .<MyItem, MyItem>chunk(10) .reader(reader()) .... My logic needs the processor to 'filter' our non valid records. but...

How do I remove trailing \ from %~p in a For command?

cmd,batch-processing
I have a Windows batch file that uses the following command for /r %%i in (dir) do @echo %server%%%~pi>>%filename% where %server% is network location of the folder (other users use different drive mappings) and %filename% is the text file I am storing the results in (done this way for debugging...

Need a BAT file to copy & rename all files in specific tree

windows,batch-file,cmd,batch-processing
I need to create a .bat that runs through a multilayered directory... copying certain files that contain the following suffix: '.full.jpg' to save as '.jpg' What I've tried: copy /y "C:\Users\myname\Desktop\maindir\*.full.jpg" "C:\Users\myname\Desktop\maindir\*.jpg" However, I cannot get it to work. The .bat is located in the 'maindir' directory and ran from...

Doctrine EntityManager clear method in nested entities

php,symfony2,doctrine2,batch-processing
I would like to use doctrine batch insert processing in order to optimize insert of a big amount of entities. The problem is with Clear method. It says that this method detach all entities that are managed by EntityManager. So what should I do in a situation where I have...

remove blank lines from multiple files within a specific directory

batch-file,batch-processing
I would like to use a batch file to delete all blank lines in multiple files in the directory "Data". I don't want to rename the files. I have seen this post, but it does not help: How to delete blank lines from multiple files in a directory for the...

Batch convert DDS file to DDS without mipmaps

bash,scripting,batch-processing,gimp,dds-format
I have a Mount & Blade: Warband mod called 1257 AD. The mod itself is great, but all the textures have to be resaved to remove mipmaps from dds files, to remove glitches on GNU/Linux. And of course, I could do this manually, but it will took a lot of...

Extract a number from string in batch script

regex,windows,batch-file,batch-processing
I've struggling to write a script that will find the drive index number from other properties of the drive. The script is as follows: @echo off REM batch file to load Veracrypt Set "driveIndex=" for /f "skip=1 tokens=1 delims= " %%a in ('wmic diskdrive where "model ='WD Elements 1078 USB...

bugfixer.bat batch file ask end user which option they want to run

batch-file,command-line,batch-processing
fun stuff!! Manager has just bought a humdinger of a utility. It does everything. It even has command line functionality. If you run BugFixer /a the application will automatically scan the entire drive for cooties. If you run the Bugfixer /b it will scan all of your files on the...

Overcoming the Midnight Barrier in Batch Script

performance,batch-file,time,timer,batch-processing
I've been using a fairly simple time script to track a software's run time measurement with clock time measurement to ensure they are roughly in sync. However, I've realized that this method will utterly fail if the clock goes past midnight. Right now I'm doing this: set start=!time! [Code to...

Setting EXIT_MESSAGE in batch_job_execution

spring-batch,batch-processing
One of the step in my job is having an exception and hence the job is failing with the EXIT_CODE "FAILED". Now I want to set the EXIT_MESSAGE as well, I did the following but the message is not getting set.. Any ideas?? chunkContext.getStepContext().getStepExecution().getJobExecution().setExitStatus(ExitStatus.FAILED); ExitStatus es = jobExecution.getExitStatus(); es =...

date validation in a text file using vba macros

vba,excel-vba,batch-processing
i have a flat file (text file) having continuous data. i need to write a macro to validate the date in this text (from offset 7 to 15). The Is Date() function requires the input to be in year/mm/dd format, but all i have is a flat file like above...

Batch modify text files

regex,markdown,batch-processing
I have a million markdown files spread across several sub-directories files with different titles where I want to change: # A Markdown title to --- title: "A Markdown title" --- …while keeping the rest of the file intact. Any ideas? OS X solutions prefered....

Expanding variables in a Windows batch file

batch-file,command-line-arguments,batch-processing,command-prompt
I run the following command in a Windows batch file: start "" "C:\PDF Viewer\PDFXCview.exe" /A "page=1&zoom=33.3" "G:\my pdfs\file 1.pdf" /A "page=4&zoom=55.5" "G:\my pdfs\file 2.pdf" The works perfectly fine and opens both PDF files using their respective parameters. However, to make the process cleaner, I would like to start using variables...

R Batch mode - Can't execute script

r,csv,batch-processing
To run R non-interactively I've decided to use Batch processing (using Windows 7). What I've entered the command prompt (CMD) cd c:\Program Files\R\R-3.1.3\bin\x64 #set directory "c:\Program Files\R\R-3.1.3\bin\x64\R.exe" CMD BATCH "c:\Program Files\R\R-3.1.3\bin\myscript.R" "c:\Program Files\R\R-3.1.3\bin\output.out" The output was generated. I've opened the output.out file and the following error message is visible: >...

Spring batch FlatFileItemWriter writing model containing a property of List type

spring,spring-batch,batch-processing
I am facing a problem in Spring batch. I have a model like this, and I want to generate a csv file out of this model, containing the field values. public class A { private B b; private List<C> c; // getters and setters ... } B class and C...

Batch script to create large text file

batch-file,batch-processing
I'm trying to create a text file that is around 4 gig. I have done this using a batch script but the creation took around 7 hours and so I think there must be a better way to do it. Here's the code I am using right now: echo "TOA5","CR6Series","CR6","790","CR6.Std.02.11","CPU:datatypes.CR6","26536","OneSec"...

square connect api batch processing

api,batch-processing,square,square-connect
I need assistance with batch processing, especially in adding tax codes to items. I'm experimenting with the square batch processing feature and my sample cases are create 2 items and add the tax code to them. In all 4 requests - 2 for creating item, 2 to 'put' the tax...

How to send a filename through batch & accept it in Java

java,batch-file,batch-processing
Ok so I might have worded this wrong, it's a little difficult to explain. So here goes: Main Goal: Call java triangle program through batch while also piping a list of testcase.txt files with input that must be sent to the java program to determine the triangle type. WITHOUT hard...

What is the benefit of the in-memory processing engines with a huge amount of data? [closed]

hadoop,apache-spark,bigdata,batch-processing
Spark performs the best if the dataset fits in memory, in case the dataset doesn't fit, it will use the disk and so it is as fast as hadoop. Let's assume that I m dealing with Tera/Peta bytes of data. with a small cluster. Obviously, there is no way to...

How to evaluate a variable with two words in a batch file?

batch-file,cmd,batch-processing
top set /p alt= echo %alt%>>123.txt if %alt% equ *ex* goto ex goto top :ex exit I need to echo something from a batch file to a .txt file. The problem I am facing is when I put two words or numbers or letters with spaces between them the program...

How to correctly quote a command with parameters in a Windows batch file

batch-file,command-line-arguments,batch-processing,command-prompt
I am attempting to open multiple PDF files in multiple tabs in multiple windows using PDF X-Change Viewer, which has command line options. All PDF files listed in the same line open within the same window. To allow the script to proceed without waiting for the windows created on each...

Unable to fetch logs from multiple servers using PsLogList

batch-file,batch-processing
I am using psloglist to get the eventlogs from remote computers. It works fine for one remote computer but doesn't work for multiple computers. Works fine: psloglist - \\Server1 -h 4 system -i 7036 -o "Service Control Manager" > c:\temp\logs.txt Doesn't work: psloglist - \\Server1,Server2 -h 4 system -i 7036...

Command to delete files all files NOT like a wildcard

windows,command-line,batch-processing,delete-file
I've found this link How can I delete files that don't match a wildcard? that is pretty similar to what I want to do but all my files have the same extension and I want to do delete all those that don't match a file name with a wildcard. E.g....

Streaming libpq results from PQexec

c,postgresql,batch-processing,postgresql-9.3,libpq
In attempting to download a large batch of data from Postgres 9.3 in C via libpq's PQexec, I'm wondering if there's a way to stream the result set from the server. From watching top, I can see my memory utilization increasing by roughly the expected result set size when I...

How do i handle ERRORLEVEL 9 in cmd

batch-file,batch-processing
I have developed a batch script which checks for the version of the software installed on a machine and based on the version it picks up the correct version patch and installs it (runs) on the machine. However i am facing an issue where when i check for a particular...

python corenlp batch parse

python,batch-processing,stanford-nlp
I am trying to batch parse document using corenlp python wrapper. batch_parse() gives generator, when I try to iterate over this generator it gives me following error: Invalid maximum heap size: -XmxTrue Error: Could not create the Java Virtual Machine. Here is my code : from corenlp import batch_parse corenlp_dir...

Make batch code to connect to server and stop services

batch-file,batch-processing
I wanted to know if there is a way to make a batch file to conncet to multiple servers and stop certain services from where you run the bat file? Thanks!

Mac Terminal batch file rename: flip string and digits

regex,osx,terminal,batch-processing
I read some threads about batch renames in Mac OS X Terminal and understood you can do something like this: for file in *.pdf do mv "$file" "<some magic regex>" done As you can guess, my issue is with the regex for this particular purpose: Old file Author-Year-Title.pdf New file...

Batch script to archive subfolder contents using 7zip with FOR /R

batch-file,batch-processing,7zip
I have a folder structure like this: C:\\\Logs\logs1\tracelogXXXX.log C:\\\Logs\logs2\tracelogXXXX.log C:\\\Logs\logs3\tracelogXXXX.log Each folder has a bunch of tracelogXXXX's, and I've got the pseudocode for a script that loops through each folder, archives each log into its own .zip, and then delete the tracelog left outside the archive (because 7zip doesnt have...

Domain Driven Design and batch processing

php,doctrine2,domain-driven-design,batch-processing
Has the following structure: Presentation Level: Web Interface, REST API and command prompt - all call only OrderService. Application Layer: class OrderService { private $em; private $repository; private $calculator; public function __construct( \Doctrine\ORM\EntityManagerInterface $em; ClientRepositoryInterface $repository, cumulativeDiscountCalculator $calculator } { $this->em = $em; $this->repository = $repository; $this->calculator = $calculator; }...

BigQuery - same query works when submitted from UI and reports SQL syntax error from batch

batch-processing,google-bigquery
I have a SQL query involving two joins on different fields. When I run this query interactively in the UI, I get back a result set, no problem. When I submit the exact same query in batch, I get back a SQL syntax error: Ambiguous field name 'video' in JOIN....

SQL Server Using TableDiff on large tables

sql-server,sql-server-2008-r2,batch-processing
We have a process which uses uses SQL Server's amazing tableDiff via: Microsoft SQL Server\100\COM\Tablediff.exe It's SQL Server 2008 R2. It connects from one instance to another identical instance. It works very well! I have a situation where a table which now has 10767594 records is taking 2.5 hours to...

Grails DuplicateKeyException / NonUniqueObjectException when batch loading within Async Promise

grails,asynchronous,gorm,promise,batch-processing
I have a large amount of data that I wish to load into a DB using GORM. class DbLoadingService { static transactional = false // these are used to expedite the batch loading process def sessionFactory def propertyInstanceMap = org.codehaus.groovy.grails.plugins.DomainClassGrailsPlugin.PROPERTY_INSTANCE_MAP // these are example services that will assist in the...

How to schedule Batch Execution Service of Azure Machine Learning in Azure Scheduler

azure,machine-learning,batch-processing,azure-scheduler,azure-machine-learning
In Batch execution API help page of Azure Machine Learning there are three different URI’s Submit Job (Response is Job ID) Start Job ( we need to use the above Job ID in this URI) Get Status or Result (we need to use the above Job ID in this URI)...

org.springframework.dao.DataIntegrityViolationException Caused by: java.sql.BatchUpdateException: Data truncation

java,batch-processing
I have a running java batch process with the following: ApplicationContext: <bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource" > <property name="driverClassName" value="net.sourceforge.jtds.jdbc.Driver"/> <property name="url" value="${example.connection.url}"/> <property name="username" value="${example.connection.username}"/> <property name="password" value="${example.connection.password}"/> <property name="maxActive" value="100"/> <property name="maxWait"...

Spring Batch Add Custom Fields

spring,spring-batch,batch-processing
I've never used Spring Batch before but it seems like a viable option for what I am attempting to accomplish. I have about 15 CSV files for 10 institutions that I need to process nightly. I am stashing the CSV into staging tables in an Oracle database. The CSV File...

What is Spark Job ?

apache-spark,batch-processing,job-scheduling
I have already done with spark installation and executed few testcases setting master and worker nodes. That said, I have a very fat confusion of what exactly a job is meant in Spark context(not SparkContext). I have below questions How different is job from a Driver program. Application itself is...

Jenkins can't use net start ?

jenkins,windows-services,batch-processing
We are trying to stop/start a windows/java service with Jenkins . The Jenkins server doesn't recognize the net batch command. Grasping at straws here, so any workaround is welcome.

Mule ESB - design a multi file processing flow when files are dependent on each other

java,mule,batch-processing,esb
Okie...I am aware of Datamappers, Batch and streaming support in File inbound elements. What i want to know is the design pattern for integration when: You have multiple files (csv or xml) to process eg: 1 file named products.csv contains all details about products. Another file images.csv has URLs to...

Convert a directory of Kodak PhotoCD images into TIFF format

linux,imagemagick,batch-processing,image-conversion
I have a directory full of Kodak Photo CD files (with file extension .pcd) and I was disappointed (and alarmed) to find that neither GIMP nor digiKam could open this old image format. Time to move these old files to a format with better software support. How can a TIFF...

R distinguishing between batch and interactive mode

r,batch-processing
I would like to have a condition in my code which allows me to distinguish if the code is running in Batch or interactive mode (via console). It would be something like this: if (interactive mode) {do this} else if (Batch mode) {do that} Is there a way to achieve...

Meta tables for H2 db in Spring Batch

java,spring,batch-processing,spring-batch,h2
I am facing some trouble in order to configuring spring batch with h2 db My configuration looks like this @Configuration @EnableBatchProcessing public class BatchConfiguration implements BatchConfigurer { @Bean public DataSource dataSource() { EmbeddedDatabaseBuilder().setType(EmbeddedDatabaseType.H2).build(); DriverManagerDataSource dataSource = new DriverManagerDataSource(); dataSource.setDriverClassName(Driver.class.getName()); String h2Url = MessageFormat.format("jdbc:h2:file:{0}note;MODE=Oracle",...

Batch script not running

windows,batch-file,architecture,batch-processing
can you help me please? This piece of script is not running and i can't figure why. @echo off ver | findstr /i "5\.1\." if %ERRORLEVEL% EQU 0 ( set os_ver="xp" ) ver | findstr /i "6\.1\." > nul if %ERRORLEVEL% EQU 0 ( set os_ver="7" ) if %os_ver% ==...

GPG: How to sign with multiple signatures with different passphrases?

encryption,batch-processing,gnupg,pgp
I have a job that runs periodically and signs/encrypts a file like so: $ gpg --homedir /path/to/.gnupg -r [email protected] -r [email protected] --local-user [email protected] --batch --passphrase-file /path/to/gpg-password --sign -ea myfile The command encrypts the file for two recipients: [email protected] and [email protected] It signs the file with [email protected] It runs in batch...

Java 8 Stream with batch processing

java,stream,java-8,batch-processing
I have a large file that contains a list of items. I would like to create a batch of items, make an HTTP request with this batch (all of the items are needed as parameters in the HTTP request). I can do it very easily with a for loop, but...

Custom operations using Stream API

java,java-8,batch-processing,java-stream
I am implementing a batch program in java. The flow is as follows: I fetch data from a database convert the data into custom objects and then put those objects in a queue. Then the goal is to run some profiling logic (for example nlp) afterwards. A friend of mine...

Changing image names programmatically to match their paths

windows,powershell,batch-file,command-line,batch-processing
I have a directory which contains a number of folders. Root Folder >directory >subdirectory >img image1.gif >subdirectory2 >img image1.gif >directory2 >img image1.gif ... I would like for each image to have prepended its parent folders, with an "_" delimiter - up until the Root Folder, without prepending the "img" folder....

Automated httr authentication with twitteR , provide response to interactive prompt in “batch” mode

r,twitter-oauth,batch-processing
I am using the R package twitteR to post items to Twitter. I put everything inside of a function and it works fine. However, I would like to run the function without being prompted for a response, and I haven't figured out how to do that. Any suggestions? Here are...

Batch processing in jdbc gateway

jdbc,batch-processing,spring-integration
my setup (simplified for clarity) is following: <int:inbound-channel-adapter channel="in" expression="0"> <int:poller cron="0 0 * * * *"/> <int:header name="snapshot_date" expression="new java.util.Date()"/> <int:header name="correlationId" expression="T(java.util.UUID).randomUUID()"/> <!-- more here --> </int:inbound-channel-adapter> <int:recipient-list-router input-channel="in" apply-sequence="true"> <int:recipient channel="data.source.1"/> <int:recipient...

Move a range of jobs to another queue with qmove

bash,batch-processing,hpc,torque
I have several (idle) jobs scheduled on a cluster that I want to move to another queue. I can move a single job like this (where 1234 is the job id): qmove newQueue 1234 But now I have hundreds of jobs that I want to move to newQueue. Is it...

XMLStarlet namespace issue/can't extract node value from XML

xml,batch-processing,xmlstarlet
I'm trying to assign a tag value from an XML file to a variable using a batch file and xmlstarlet (version 1.6.1). The XML file has a namespace and by defining the namespace in the xmlstarlet command I receive two errors. The XML file (myApp-app.xml) looks like this: <?xml version="1.0"...

insert to different tables batche in one loop

java,multithreading,jdbc,batch-processing
I'm parsing a JSON file that keep the user information. I get information one by one and build some classes such as User, UserSettings. Now all this information should be inserted into database. The JSON file is too big, can have more that 500000 users. To insert for example User...

Configure settings when importing a large dataset using Neo4j Batch-importer

neo4j,out-of-memory,batch-processing
I’m trying to import 20 million nodes and 250 million relationships to Neo4j using the Batch-importer . I have 8GB of RAM. Here are my current settings in batch.properties: use_memory_mapped_buffers=false neostore.nodestore.db.mapped_memory=500M neostore.relationshipstore.db.mapped_memory=2000M neostore.relationshipgroupstore.db.mapped_memory=10M neostore.propertystore.db.mapped_memory=1G neostore.propertystore.db.strings.mapped_memory=200M neostore.propertystore.db.arrays.mapped_memory=0M...

How to extract a string from the first line of a file using batch?

windows,batch-file,cmd,batch-processing
I have these files that contain a name in the first line I wish to extract. All the methods I have approach either result in error or give me nothing. An example file can be found here: https://www.dropbox.com/s/0kcgj8vr3wii33s/FORS2.2011-10-23T23_59_20.355_out.fits?dl=0 I wish to extract "PN-G013.3+01.1", which lies on the first line (column...

Merging CSV files with partly same filename

csv,batch-file,merge,batch-processing
I have a directory containing aprox. 1500 csv files with measuring data, like level, pressure, volume, etc. The measuring data is exported every hour for every measurement. So the generated filenames of Level would be Levelxx.csv, where xx ranges from 00-23 The same method applies for Volume and Pressure and...

PowerShell command - what does this output?

powershell,batch-processing
My colleague found this Batch command (Powershell Command in Windows) online and I need to know what "encoding" it is using to output to the file. I know that it is not utf-8. Here is the command: Select-String logfile.log -pattern "string pattern" | tee-object -filepath "output_file.txt" -append My Windows edition...

Checking for Not Listening Ports from Batch File

windows,batch-file,batch-processing
I am writing a batch file to automate certain redundant tasks. In the process, I need to find a port that is not listening so I can use it to install some services. The current test code I have is: @echo OFF :getPort Set /P oPort=Enter an open port (not...

Use call inside a forfiles

batch-file,batch-processing
I've a script where I want to call a function for each file in a directory. Using forfiles, I've been able to list them like: forfiles /p "%myPath%" /c "cmd /c echo @path NAME @file" Note: myPath is a path with spaces, so I had to put it inside ""....

How to exit a for /f loop in a batch ?

batch-file,batch-processing
I have a question about a batch file that I am writing. How can I define my "for /f" loop to execute only one part of my script and not the whole script ? In the script below, I would like I precise that I need to use a "call"...

Settings for the Batch import tool

neo4j,batch-processing
How often does the Neo4j 2.2 batch import tool flush the data in the cache to disk? I believe this the n for PERIODIC COMMIT n and can I change this? Also, is there any way that I can edit the default memory mappings for the import tool? I used...

SAXON Error- How to ignore/skip it?

javascript,xslt,batch-processing,saxon
Using Saxon HE (net version) wget and batch, I'm trying to transform a page I've downloaded via wget. Whenever calling the command on the page, I get the following error: SXXP0003: Error reported by XML parser: The entity name must immediately follow the '&' in the entity reference. It is...

Will batch script wait while it is executing NET STOP?

batch-file,batch-processing
I have a batch script which runs on a server and stops a service. Sometimes there is an issue whereby the service is unable stop using NET STOP and the service needs to be killed. To get around this I added an IF statement to Kill the service if not...

how to call all the files inside a folder using batchfile

windows,batch-file,cmd,batch-processing,saxon
java -Xss2048k -jar -DentityExpansionLimit=800000 "C:\testXsl\saxon9.jar" "C:\testXsl\In\" "C:\testXsl\XslFile\test.xsl" >"C:\testXsl\Out\out.xml" 2>"C:\testXsl\Logs\xsllog.xml" how can i select all .xml files inside the C:\testXsl\In\ folder thanks??...

How to implement batch processing in Grails

grails,gorm,batch-processing
I have a Grails application which implement multiple datasource and its task to process and save excel data(xlsx type) in corresponding datasource tables. But I got a problem when excel file contains huge data and it taken more time to save. So I have used SQL and insert values through...