FAQ Database Discussion Community


Linux script with curl to check webservice status

web-services,shell,unix,curl,wget
I have a license validation webservice provided at http://localhost/testing/validateLicence I want to write a shell script in such way that I have to pass license key with webservice call . If webservice returns 200 OK then further steps in shell script are executed otherwise execution will be terminated Please help...

Windows bat file: Download & Compare Content

windows,batch-file,wget
I'm trying to download a file from a webservice that either contains the string "empty" or something else. If "empty", I want to delete the file, else it should be moved and renamed. To achieve this, I'm using wget.exe to download the file, this works perfectly. This is what I...

wget not displaying all website data

html,shell,cygwin,wget
Input: wget -qO- http://runescape.com/community | grep -i playerCount Output: <li class="header-top__right-option"><strong id="playerCount">0</strong> Online</li> In browser: Using cygwin..I am trying to use wget to pull a number out of a webpage. As shown in the example above, the playerCount is 0. If you actually load the webpage up and look at...

Batch script for wget

windows,batch-file,wget
I need to do a script for download images resursive, How can i do it. I am on a Windows. I want something like this: for(int i = 0 ; i < 100; i++) { wget "http://download/img" + i + ".png } How can i do this with a Windows...

How to pass url to wget

grep,pipe,wget
If I have a document with many links and I want to download especially one picture with the name: www.website.de/picture/example_2015-06-15.jpeg. How can I write a command that downloads me automatically exactly this one I extracted out of my document. My idea would be this but I'll get a failure message...

Purge a URL from the Fastly CDN via API call with curl or wget

bash,curl,wget,cdn,fastly
I use a bash script to deploy a static webpage which uses Fastly as a CDN. When /example.html gets deployed I would like to purge /example.html via their API within the bash script. I know that there is a good to go Ruby gem but I'd like to do it...

How do I use wget in docker?

linux,ubuntu,docker,wget
I´m trying to download a debian package within docker, but it fails. When I execute sudo docker run ubuntu:14.04 wget https://downloads-packages.s3.amazonaws.com/ubuntu-14.04/gitlab_7.8.2-omnibus.1-1_amd64.deb I get exec: "wget": executable file not found in $PATH How can I simply download a file?...

Automatically Download Latest WhatsApp APK using Shell Scripting

bash,shell,wget
I'm trying to create a cron job that downloads the latest version of WhatsApp's APK from their website using a bash script and make it available through my site. So far, I'm able to obtain the version number from the site using the following (user-agent part omitted): wget -q -O...

Linux-wget command

linux,shell,wget
I need a quick help on customizing my wget command in a shell script: The wget command looks something like this: wget http://infamvn:8081/nexus/content/groups/LDM_REPO_LIN64/com/infa/com.infa. products.ldm.ingestion.server.scala/10.0.0.135.527-SNAPSHOT/com.infa.products.ldm.ingestion.server.scala-10.0.0.135.527-20150622.210643-1-sources.jar Here I'd like to add the 10.0.0.135.527 in a variable, so I created a script something like this: n = 10.0.0.135.527 wget...

Does executing wget via shell_exec on PHP allows me to close the browser readily without waiting for it to finish the download?

php,linux,wget,shell-exec
I am just wondering (since I do not have my unix platform to test) if I executed a multi command line via the shell_exec() on PHP, am I already safe to close the browser without waiting for it to finish? Because as far as I know, wget will keep downloading...

Download Website with formatting using Wget

php,batch-file,formatting,download,wget
I have a PHP script which runs on our website and echos a bunch of content to the browser, each line has a unique ID. I installed GnuWin32 and wrote a small BAT script to download this content to a TXT file which is working fine, however when viewed in...

Python: Subprocess call with wget - Scheme Missing

python,jira,wget
I'm trying to trying to run the command wget from within a python script. The follow wget works perfectly fine when just entering it on the command line, but it gives me a scheme missing error when I try to do it with subprocess.call([]) I'm not too sure how to...

Unable to establish SSL connection upon wget on Ubuntu 14.04 LTS

linux,ubuntu,ssl,wget
I tried to download an image through wget but I got an error: Unable to establish SSL connection. wget https://www.website.com/image.jpg --2015-02-26 01:30:17-- https://www.website.com/image.jpg Resolving www.website.com (www.website.com)... xx.xxx.xx.xx Connecting to www.website.com (www.website.com)|xx.xxx.xx.xx|:443... connected. Unable to establish SSL connection. My test case: Using Ubuntu 12.04.4 LTS (GNU/Linux 3.8.0-44-generic x86_64), GNU Wget 1.13.4...

AWS Elastic Beanstalk execute wget in cronjob

amazon-web-services,cron,wget,elastic-beanstalk
I'm trying to run a wget command every minute that simply hits a url. I currently have the following in a file called project.config in the .ebextensions folder: 01_test_cron: command: * * * * * /usr/bin/wget -O /dev/null http://name.of.url It currently does not execute - are there any glaring errors?...

wget not following the flashplayer download link

wget,tar,flash-player,gz
I'm trying to write a script that will download the latest version of the flashplayer tarball and unpack it. Should be simple enough I thought. But the "Download now" link on this page resolves to "https://get.adobe.com/flashplayer/download/?installer=Flash_Player_11.2_for_other_Linux_(.tar.gz)_64-bit&standalone=1 and when I execute the following line wget "http://get.adobe.com/flashplayer/download/?installer=Flash_Player_11.2_for_other_Linux_(.tar.gz)_64-bit&standalone=1" it yields --2015-05-08 08:56:49--...

How to force wget to not URL-encode?

wget,url-encoding
This seems like it should be simple, but I'm having trouble figuring it out. I want to send a wget request similar to the following, wget http://www.foo.com/bar.cgi?param=\"p\" but I don't want it to URL-encode the quotes (or anything else, for that matter). Is this possible? And just to make it...

Bash, wget remove comma from output filename

linux,bash,curl,wget
I am reading a file with URL's by line by line then I pass URL to the wget: FILE=/home/img-url.txt while read line; do url=$line wget -N -P /home/img/ $url done < $FILE This works, but some file contains comma in the filename. How I can save the file without the...

Using crontab to call wget every 5 minutes

php,symfony2,crontab,wget
I am working on a project where the wget and crontab are used to run a process in the background. I have a php file named "Hello.php" that I want to run this every 5 minutes. I have found that if I want to have a cron job run every...

Cron Job, call url with wget

crontab,wget
I have added this command line in my /etc/crontab: */1 * * * * wget http://IP:8080/auctions/updateStatus It doesn't work, what's wrong with it? The link is correct, it's a redirect to a simple java controller....

Odd file naming using wget with Nexus Rest API

java,rest,maven,wget,nexus
I have a script that pulls .war artifacts from our nexus repository using wget to pull the artifacts via the Nexus Rest API. The url that I hit with wget is similar to this: http://our.nexus.net:8081/nexus/service/local/artifact/maven/redirect?g=com.somecompany&a=SomeArtifact&v=1.0.0&r=releases&p=war It works fine in our dev environment, and pulls an artifact with the name SomeArtifact-1.0.0.war....

WGET - HTTPS vs HTTP = HTTPS Slower

linux,ssl,curl,raspberry-pi,wget
I am trying to do some tests using the wget and I noticed that HTTPS pages take significant to load in wget than http for the same server. This does not seem related to any network difference. Before the name resolution wget takes about 5 extra seconds. Can anyone help?...

wget directory-prefix not working, downloads to ftp host directory

linux,wget
I'm trying to use wget to download from a ftp host and download the files to a specified directory but this does not work. It instead downloads to a directory that has the same name as the ftp host. I used the -P and --directory-prefix option but with no success;...

wget returns with 500: Internal error, but browser works

wget
I am trying to download the contents of this link > http://goldsmr2.sci.gsfc.nasa.gov/daac-bin/OTF/HTTP_services.cgi?FILENAME=%2Fdata%2Fs4pa%2FMERRA%2FMAT1NXSLV.5.2.0%2F2014%2F01%2FMERRA300.prod.assim.tavg1_2d_slv_Nx.20140103.hdf&FORMAT=TmV0Q0RGLw&BBOX=-16%2C-44%2C-11%2C-40&LABEL=MERRA300.prod.assim.tavg1_2d_slv_Nx.20140103.SUB.nc&FLAGS=&SHORTNAME=MAT1NXSLV&SERVICE=SUBSET_LATS4D&LAYERS=&VERSION=1.02&VARIABLES=u10m%2Cu50m%2Cv10m%2Cv50m through wget but I get an internal error...

wget ssl alert handshake failure

ssl,https,wget
I am trying to download files from an https site and keep getting the following error: OpenSSL: error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure Unable to establish SSL connection. From reading blogs online I gather I have to provide the server cert and the client cert. I have found steps on how...

shell script wget to gives download links error

shell,curl,wget
I am looking for a shell script that can: Read download links from a text file using either wget or curl Fetch/download the links line by line Create log file either for error message of any link that are failed to download or successful. ...

Why is wget saving something when using parametrized url?

bash,shell,wget
I am using following command in my bash script to trigger jenkins build: wget --no-check-certificate "http://<jenkins_url>/view/some_view/job/some_prj/buildWithParameters?token=xxx" Output: HTTP request sent, awaiting response... 201 Created Length: 0 Saving to: “buildWithParameters?token=xxx” [ <=> ] 0 --.-K/s in 0s 2015-02-20 10:10:46 (0.00 B/s) - “buildWithParameters?token=xxx” saved [0/0] And then it's creates empty file:...

How to access https page via wget or curl?

unix,ssl,curl,wget
Let's say I want to save the contents of my Facebook page. Obviously fb uses https, thus ssl, how do I download the contents of a secure page using wget? I found a lot of sources online... and I modify my command, but it doesn't save the page I want....

wget Google search (300 000 lookups)

bash,curl,google-api,wget,lynx
I have data which I need to modify using the first entry of a certain Google search. This search has to be repeated for about 300 000 times (each row) with varying search keywords. I wrote a bash script for that using wget. However after about 30 (synchronous) requests, my...

ajax calls not showing any content in browser

javascript,jquery,python,ajax,wget
I am trying to crawl artist urls from this page https://myspace.com/discover/artists?genreId=1002532 But this page is making ajax call to get user deatils.I could see this url in firebug https://myspace.com/ajax/artistspage?chartType=heavyrotation&genreId=1002532&page=0 If i open this url in seperate tab there is no content displayed but if i view it response tab in...

Subprocess Popen not capturing wget --spider command result

python,subprocess,wget
My understanding of capturing the output of a subprocess command as a string was to set stdout=sucprocess.PIPE and use command.communicate() to capture result, error. For example, typing the following: command = subprocess.Popen(["nmcli", "con"], stdout=subprocess.PIPE) res, err = command.communicate() produces no output to the terminal and stores all my connection information...

Wget: Filenames without the query string

wget
I want to download a list of webpages from a file. How can I stop Wget appending the query strings on to the saved files? wget http://www.example.com/index.html?querystring I need this to be downloaded as index.html, not index.html?querystring...

How to download a web directory when authentication is required?

linux,wget,fetch,ubuntu-14.04,web-folders
For instance, I want to fetch the entire folder3 from a web site http://something.com/folder1/folder2/folder3 I saw similar posts where the solution is given by wget. But in this case the web site ask me for user name and password, which I have. I tried to download the folder with wget...

Fortran read data with different length

data,fortran,wget
i have a fortran problem. i want to read in data with different length. they begin with: <TITLE>University of Wyoming - Radiosonde Data</TITLE> <LINK REL="StyleSheet" HREF="/resources/select.css" TYPE="text/css"> <BODY BGCOLOR="white"> <H2>08190 Barcelona Observations at 12Z 11 Feb 2015</H2> <PRE> ----------------------------------------------------------------------------- PRES HGHT TEMP DWPT RELH MIXR DRCT SKNT THTA THTE THTV...

wget not downloading from authentic https url vaultpress

linux,wordpress,ubuntu,curl,wget
I'm using vaultpress to take my wordpress blog's backup https://dashboard.vaultpress.com/ after clicking of download backup button on this site send me a link from where I can download. when I click on this link , it starts downloding my backup in browser, and thats perfect. but I'm trying to download...

linux wget download tar.gz as html?

linux,file,download,wget
I know the problem may seem naive to most of you but I couldn't find a solution. I'm using a linux virtual machine and I'm trying to download the apache-drill.tar.gz from enter link description herethe link (10 minutes tutorial they provide) I did the wget "link" but when I tried...

403 forbidden error in perl wget command [closed]

perl,wget,httpforbiddenhandler
i am trying to download a file using Perl script but it is not working, $u='http://www.example.com/dir/files/myfile.csv'; `wget -O $filename $u`; i've tried this code but it gives me an error 403 forbidden....

Cron job with CodeIgniter using wget

php,codeigniter,cron,crontab,wget
I am pretty new to setting up cron jobs. What I've done so far is set one up to run every 5 minutes using the following script: */5 * * * * wget -q localhost:8888/example/index.php/controller/function When I run just the wget part from the command line, it works perfectly. But...

How to automatically move a fully downloaded file to a new directory

bash,wget,rsync
I am using wget to download files a group of files to be ingested in a program. I know I can use the "-P" to send the file to a tmp directory, but I am unsure what the best method is to move the file to the needed destination once...

Is there an option to wget to force it to write the response even when it's erroneous?

bash,soap,wget
I have written a Bash SOAP library that uses wget as the interface to HTTP servers. It's intentional to avoid curl, since that is not available or not installed by default on systems where this library is used. The basis of the library is the query the WSDL, determine the...

How to get extact page content in wget if error code is 404

python-3.x,curl,web-scraping,wget
I have two url one is working url another one is page deleted url.working url is fine but for page deleted url instead of getting the exact page content wget receives 404 Working url import os def curl(url): data = os.popen('wget -qO- %s '% url).read() print (url) print (len(data)) #print...

using wget to download a directory

bash,wget
I'm trying to download all the files in an online directory. The command I'm using is: wget -r -np -nH -R index.html http://www.oecd-nea.org/dbforms/data/eva/evatapes/mendl_2/ Using this command I get an empty directory. If I specify file names at the end I can get one at a time, but I'd like to...

Grep files in between wget recursive downloads

linux,bash,unix,grep,wget
I am trying to recursively download several files using wget -m, and I intend to grep all of the downloaded files to find specific text. Currently, I can wait for wget to fully complete, and then run grep. However, the wget process is time consuming as there are many files...