FAQ Database Discussion Community


Extract .tar.gz file in Build.scala (without external libraries)

scala,sbt,gzip,tar
There are a lot of questions about extracting .tar.gz files in Java/Scala but all of them involve using some library to extract the tar file. GZIPInputStream is part of Java SE but I don't know of a way to extract tarballs using Java SE. As far as I can tell,...

File Retrieved via FtpWebRequest Is Different Than Source File On Server

c#,ftp,gzip,ftpwebrequest
I am retrieving GZip files from an FTP server using example code exactly as provided in a Microsoft help page: https://msdn.microsoft.com/en-us/library/ms229711(v=vs.110).aspx The code works great! The connection is established, the file is retrieved and saved into my target path. The problem is that the file created by this code is...

How to enable GZIP for RestController?

java,web-services,rest,compression,gzip
I'm having a simple REST controller using spring. How could GZIP response of the returned application/xml stream be enabled? @RestController public class MyRest { @RequestMapping(method = RequestMethod.GET, produces = MediaType.APPLICATION_XML_VALUE) @ResponseBody public ComplexRsp test() { //... } } When using soap+wsdl approach, there would be the simple @GZIP annotation on...

Unable to decode CORS gzipped content from EC2 via Cloudfront

amazon-web-services,gzip,amazon-cloudfront
This is probablly related to Serving gzipped CSS and JavaScript from Amazon CloudFront via S3, but although I followed the doc I still can get this situation to work : An S3 bucket contains files that I have gzipped (with grunt-contrib-compress, if that's of any relevance.) My bucket has a...

Is it necessary to use multiple gzip members for input larger than 4GB?

compression,gzip,deflate
By stating Features: no 4GB limit ... Idzip just uses multiple gzip members to have no file size limit. the author of idzip seems to imply that multiple gzip members are necessary to support data > 4GB. But the deflate algorithm, whose output gzip members merely wrap with header and...

Decompressing a gzipped payload of a packet with Python

python,gzip,packet,scapy,decompression
I am currently working on a program that takes a .pcap file and separates all of the packets out by ip using the scapy package. I want to decompress the payloads that are compressed using the gzip package. I can tell if the payload is gzipped because it contains Content-Encoding:...

What tool lets me see gzip's Huffman table and blocks?

compression,gzip,huffman-coding
What tool lets me see the Huffman table that gzip and some other compression algorithms create? I know that programs like bzip2 and zpaq use additional compression techniques, but I believe gzip, zip, and the lz family of programs use Huffman tables, and I would like to see these. I...

Decode gzip compressed and base64 encoded data to a readable format

python,python-3.x,gzip,zlib
Trying to decode gzip compressed and base64 encoded data to a readable format in Python3.4. import base64 import zlib original_data = '...jU2X0NCQ19TSEEAAAABAAA=' #Data cut short. decoded64 = base64.b64decode(original_data) #format:b'\x16xe\x94...\xae\x9a\...' final_decoded = zlib.decompress(decoded64) print(final_decoded) Been getting: "Error -2 while preparing to decompress data: inconsistent stream state." Not sure what I'm doing...

D lang record separator is being lost after string cast

casting,gzip,d
I am new to D please forgive the terrible code below. I am opening a .gz file and reading it chunk by chunk for uncompressing it. The data in the uncompressed file is like : aRSbRScRSd, There are record separators(ASCII code 30) between each record(records in my dummy example a,b,c)....

MVC 4 - GZIP compression of JSON ajax action result

c#,json,iis,model-view-controller,gzip
The problem I have a Telerik MVC UI grid on an MVC 4 app running on IIS 7.5 that can potentially return a large amount of JSON data via AJAX, in extreme cases 800kb or more. As the payload can be large, I want to GZIP it. For the life...

Web API Gzip not being applied

c#,gzip,asp.net-web-api2
I have added the web.config entry to enable gzip compression based on this S/O answer Enable IIS7 gzip. I then checked the Chrome Developer window while loading an ASPX page and saw the header in the response: Cache-Control:private Content-Encoding:gzip Content-Length:3669 Content-Type:text/html; charset=utf-8 Date:Wed, 04 Mar 2015 00:46:05 GMT Server:Microsoft-IIS/7.5 Vary:Accept-Encoding...

Does GZIP Compression Level Have Any Impact On Decompression

compression,gzip
I understand that GZIP is a combination of LZ77 and Huffman coding and can be configured with a level between 1-9 where 1 indicates the fastest compression (less compression) and 9 indicates the slowest compression method (best compression). My question is, does the choice of level only impact the compression...

Socket HTTP request returning invalid GZIP

java,sockets,http,gzip
I am teaching myself more about HTTP requests and such, so I wrote a simple POST request using Java's HttpURLConnection class and it returns compressed data which is easily decompress. I decided to go a lower level and send the HTTP request with sockets (for practice). I figured it out...

Gzip CSS/JS Using OS X Terminal

osx,gzip
I am trying to use OSX Terminal to create gzip versions (.gz) of all css & js files in a folder. I found the following command, but when I cd to a test folder & then enter the command, it doesn't output anything & I would expect it to create...

How can I compress / gzip my mimified .js and .css files before publishing to AWS S3?

css,powershell,amazon-web-services,amazon-s3,gzip
I ran Google pagespeed and it suggests compressing my .js and .css Eliminate render-blocking JavaScript and CSS in above-the-fold content Show how to fix Enable compression Compressing resources with gzip or deflate can reduce the number of bytes sent over the network. Enable compression for the following resources to reduce...

Read contents of .gz file with python

gzip
I'm new to Python and am running into issues reading the contents of a .gz file: I've got a folder full of .gz files that I've extracted programatically using a private API. The contents of each .gz file is a .xml file so I need to iterate over the dir...

Decompressing text in C#

c#,zip,gzip,decompression
I have an issue which I am banging my head against, I am trying to decompress text in this format: eJx7v3t/QWJxcXl+UQoAJ94F3Q== The problem I am having is that it works awesome on this site: http://www.unit-conversion.info/texttools/compress/ But I can't seem to get it to work with C#, I have tried Gzip...

Android: uncompress concatenated gzip-compressed files

java,android,gzip
gzip is documented to support concatenation of compressed files: $ echo hello >hhh $ echo world >www $ cat hhh www hello world $ echo hello | gzip >hhhh $ echo world | gzip >wwww $ cat hhhh wwww | gunzip hello world I can create a concatenated file with...

gzip in cython file

gzip,cython
I am using cython to parsing large gzipped files. Is there any recommended way to use the gzip library as a file buffer to getline? The readline from gzip in Python takes well too long. I would like to do this: gzip_file = gzip.open(file_name, ¨rb¨) getline(line,n, gzip_file) ...

zlib inflate returning a buffer error

c++,zip,gzip,zlib,inflate
I'm trying to decompress a zip file using zlib (without using any extension or 3rd party). Initially, the src_len is 48756255, and the dest_len is 49209890. The first pass in the while loop is fine: err is Z_OK and the second pass through starts. On the second pass, no matter...

Download and read gzipped JSON data with Javascript

javascript,json,gzip
You can manually download the following link: http://saved-games-alpha.s3-website-us-east-1.amazonaws.com/[email protected] It's a gzip file. Unzipping it yields a file called "temp" that has the JSON data I want in it. I need a greasemonkey script to automate this. When I run const gzip = jQuery.get("http://saved-games-alpha.s3-website-us-east-1.amazonaws.com/[email protected]"); I get a successful response, but I...

Netica error or compression error? “buffer to decompress gzip 'internal' is too small”

java,bytearray,gzip,bytearrayinputstream
I'm using the Netica Java library in my code to read in a bayes net, enter findings, and save the updated bayes net in a byte array. The byte array saves perfectly fine (as far as I can tell) but I can't seem to load it back in using the...

TinyMCE, gzip and caching

asp.net,webforms,tinymce,gzip,http-caching
I am trying to get TinyMCE, gzip and caching to work correctly but are stuck with the browsers not caching requests to the gzip.ashx handler. My setup: TinyMCE (currently 4.1.7) TinyMCE Compressor (.NET 4.0.1) ASP.NET WebForm application / IIS. This is my code (pretty standard): <!DOCTYPE html> <html xmlns="http://www.w3.org/1999/xhtml"> <head>...

reading gzipped csv file in python 3

python,csv,gzip
I'm having problems reading from a gzipped csv file with the gzip and csv libs. Here's what I got: import gzip import csv import json f = gzip.open(filename) csvobj = csv.reader(f,delimiter = ',',quotechar="'") for line in csvobj: ts = line[0] data_json = json.loads(line[1]) but this throws an exception: File "C:\Users\yaronol\workspace\raw_data_from_s3\s3_data_parser.py",...

GZipStream reads beyond the end of my data parts

c#,.net,httpwebrequest,gzip,decompression
My program downloads via HttpWebRequest one file which consists of several stringed up gzip-compressed files/parts. So when I decompress my responseStream, the GZipStream closes itself after each file part. That is not a big problem, since I can create a new one for each file, but the thing is: The...

How to send string compressed with GZIP from Java App to PHP web service

java,php,android,json,gzip
I have this issue with GZIP compression: I need to send by POST method a huge JSON string, which is too big to be accept like URL (Ex: http://localhost/app/send/JSON STRING ENCODED BY BASE64), than it result in HTTP error 403 so, I need to compress my json and I found...

Receiving gzip with Flask

python,curl,flask,gzip
I'm trying to receive a gzipped JSON file from an HTTP POST to Flask (v0.10). I feel there may be some extra data posted that needs stripping out before trying to open the gzip. Here's the code I have: from flask import Flask, jsonify, request, abort import gzip, StringIO app...

gzip and pipe to output (performance consideration)

linux,unix,encryption,gzip,solaris
q1) Can i check if I do a gzip -c file | encrypt (some parameters) a) does gzip print out the output line by line and pipe it to the encrypt function or b) gzip will be perform 1st, then the output will be pipe all at once to the...

Decompressing REST response with GZIPInputStream

grails,groovy,gzip
I'm trying to decompress a gzip:ed response i receive from a REST service: Content-Encoding=[gzip], Content-Type=[application/json], Content-Length=[710] ... I'm using the Grails REST Client Builder Plugin: def response = new RestBuilder().get(HOST + "/api/..."){ contentType "application/json" accept "application/json" } The returned response is a Spring ResponseEntity. I'm trying to decompress the data...

Errors downloading Powerpoint and PDF files after nginx update

pdf,nginx,download,gzip,powerpoint
I am hosting my site on an nginx server at DigitalOcean & ran some updates. Before the updates, I was able to upload PDF and Powerpoint files to the server and download them without problems. Now, whenever I try to download a file I recieve an error "Powerpoint found a...

Golang writing to http response breaks input reading?

http,go,gzip
I'm attempting to write a small webapp in Go where the user uploads a gzipped file in a multipart form. The app unzips and parses the file and writes some output to the response. However, I keep running into an error where the input stream looks corrupted when I begin...

NodeJS: zlib.gunzip() on file contents returns Error: incorrect header check

javascript,node.js,gzip,zlib,fs
My script takes file data, interprets it, and spits it out in a different format. The user can input a URL or upload a file, and the file refence for either of these can be XML or GZIP (which I then decompress). What works: Parsing XML + GZIP from URL...

How to create zip file in stages?

c#,zip,gzip
I'm writing data into ZIP file in a following way: private static void ZipStringToFile(string value) { using (MemoryStream ms = new MemoryStream()) { using (GZipStream compressedzipStream = new GZipStream(ms, CompressionMode.Compress, true)) { byte[] queryData = _encUtf8.GetBytes(value); compressedzipStream.Write(queryData, 0, queryData.Length); compressedzipStream.Write(_dataEndOfLine, 0, _dataEndOfLine.Length); } using (FileStream outfile = new FileStream("TestZip.zip", FileMode.Append))...

NodeJS: zlib.gunzip( body ) erroneously returns `undefined`

javascript,xml,node.js,http,gzip
I have a barebones node server that parses XML files (referenced from user-defined URLs) and outputs the manipulated data as a CSV file for download. I'm trying to implement optional GZIP decompression: function loadURL(req, res) { request.get(req.query.url, function(error, response, body) { if (req.query.isGzip == 'true') { zlib.gunzip(body, function(err, dezipped) {...

Compress a file in memory, compute checksum and write it as `gzip` in python

python,gzip,checksum
I want to compress files and compute the checksum of the compressed file using python. My first naive attempt was to use 2 functions: def compress_file(input_filename, output_filename): f_in = open(input_filename, 'rb') f_out = gzip.open(output_filename, 'wb') f_out.writelines(f_in) f_out.close() f_in.close() def md5sum(filename): with open(filename) as f: md5 = hashlib.md5(f.read()).hexdigest() return md5 However,...

Fastest way to change head of a gzip file?

python,gzip
I maintain a benchmark library with gz-compressed files that contain descriptive metadata in the first few lines. By hand, I can decompress a 246MB gz-compressed file (using gunzip), change it, and compress it back (using gzip) in under 2 minutes using the linux terminal. On the same file, the following...

Enabling compression on Heroku using python

python,heroku,gzip,deflate
Google now penalizes due to not being so mobile friendly. So in an effort to make things better, it recommends I compress a lot of my Javascript with Gzip or Deflate. I've seen some old recommendations on stack overflow, but there is nothing out of the box and I tried...

Decompressing gzip packets with python

python,python-2.7,gzip,packet,scapy
I'm creating a PCAP parser and analyzer in Python with the Scapy package but I'm running into an issue where just about everything is compressed with gzip. Is there a way for me to have gzipped packet payloads decompressed as my program loops through the PCAP file? In the packet...

combine multiple gzip compressed files in a tar file

bash,shell,unix,gzip,cat
I have a really large tar file containing many gzip files. I would like to combine all the gzip files into one gzip file and place in another directory. To view the contents of the tar file I use: tar -zxvf Big.tar which gives: Big/FileA.gz Big/FileB.gz Big/FileC.gz Big/FileD.gz To normally...

HDF gzip compression vs. ASCII gzip compression

c,gzip,hdf5
I have a 2D matrix with 1100x1600 data points. Initially, I stored it in an ascii-file which I tar-zipped using the command tar -cvzf ascii_file.tar.gz ascii_file Now, I wanted to switch to hdf5 files, but they are too large, at least in the way I am using them... First, I...

Glassfish4 gzip encoding issue (corrupted responses)

java,glassfish,gzip
Glassfish4 after GZIP enabling works wrong. All the responses after ~12kb are corrupted.

Grep a gzipped file?

unix,gzip
I would like to see if there is a title url in the following gzip file: curl -O http://www.imdb.com/sitemap_US_02002.xml.gz grep 'imdb.com/title/tt' sitemap_US_02002.xml.gz Is there a way to do this 'on the fly', perhaps even in a one-liner?...

sending large JSON data from android to php server can I compress?

json,gzip,getjson,decompress,data-compression
i am sending large datasets in JSON format from my android device to my server (using PHP). I want to keep bandwidth costs down. I am wondering: should I gzip compress the JSON data server side before sending the data? is there a javascript gzip uncompression library in php side...

How to enable gzip compression (both static and dynamic) for Jetty9?

java,jetty,gzip
I use Jetty9 and I want to test if compressing text files will increase performance. At first I'm interested in serving off-line compressed files just like it is described at: http://marianoguerra.org/posts/201205enable-gzip-compression-in-jetty.html My code looks like: static private void set_jetty_handlers(Server server) { // gzip: I have added it to to serve...

Compress json string for gcm push notification message

json,compression,android-gcm,gzip,chrome-gcm
I have problem with json data and gcm push notification. I send big json data and it exceed 4kb limit, so is there any compression that i can use to compress that json(gzip or something like that) and send it?...

Decompression of GZIP:ed response (Grails / Groovy)

rest,groovy,gzip,grails-plugin,grails-2.0
I'm consuming a REST api using the RestBuilder plugin. I get a response where the body is compressed: Content-Encoding=[gzip] Does groovy/Grails provide any easy access / native methods for decoding gzip compression? The only thing I found is the native Java zip api (ex. GZIPInputStream). Does someone have a better...

GZipStream complains magic number in header is not correct

c#,.net,gzip,gzipstream
I'm attempting to use National Weather Service (U.S.) data, but something has changed recently and the GZip file no longer opens. .NET 4.5 complains that... Message=The magic number in GZip header is not correct. Make sure you are passing in a GZip stream. Source=System StackTrace: at System.IO.Compression.GZipDecoder.ReadHeader(InputBuffer input) at System.IO.Compression.Inflater.Decode()...

Perl to parse text file which may or may not be gzipped

perl,gzip
I'm using Perl to parse a text file which may or may not be gzipped: my $FILE; if ( $file =~ /\.gz$/ ) { open( $FILE, "gunzip -c $file |" ) || die $!; } else { open( $FILE, '<', $file ) || die $!; } Is there a simpler...

How to reconcile gzip compression and the flushing of the header of index.php

php,apache,gzip,flush
I have a pretty big index.php file(about 500kB) in which lots of logic and database query are present(and the index.php which goes to the client is about 200kB). What I'd like to do is first of all compress the file using gzip, which I do by simply adding SendOutputFilter in...

Firefox fails to decompress gzip files

firefox,amazon-s3,http-headers,gzip,mime-types
I have .gz files stored on AWS s3. Using the s3 REST-API, I'm generating authenticated links that point to individual files. I'm also setting the content-header options such that browsers requesting these urls will decompress and download the gzipped files as attachments. The generated s3 url looks like so: https://MY_BUCKET.s3.amazonaws.com/PATH_TO/file.ext.gz...

OkHttp, is the automatic GZIP disabled when using a custom interceptor?

android,gzip,okhttp
I am using OkHttp 2.3.0 in my Android app. My question is about the transparent GZIP feature. According to the documentation, it should be silently there. But I cannot see the ("accept-encoding", "gzip") header in the request. I am using a custom interceptor to add authentication: @Override public Response intercept(Chain...

Infinite loop when streaming a .gz file from S3 using boto

python,amazon-s3,gzip,boto
I'm attempting to stream a .gz file from S3 using boto and iterate over the lines of the unzipped text file. Mysteriously, the loop never terminates; when the entire file has been read, the iteration restarts at the beginning of the file. Let's say I create and upload an input...

How to identify the binary compression type from the content?

c#,sql-server-2005,compression,gzip,binary-data
I am trying to restore files which are stored in MS SQL database (used by third party application which has stopped their support) as an image data type(byte arrays). So what I do is to write those rows of byte arrays to file to convert for know file extensions. However...

Are there test vectors available for GZIP?

c++,boost,gzip
In our software, we have an interface for GZIP (RFC 1952) compression/decompression that is currently powered by boost::iostreams. Since we might need to change the underlying lib for linking reasons, I'd like to add more tests for GZIP compression and decompression, i.e. precise byte-per-byte comparison. Are there any official or...

HttpClient: Conditionally set AcceptEncoding compression at runtime

c#,gzip,.net-4.5,dotnet-httpclient,winrt-async
We are trying to implement user-determined (on a settings screen) optional gzip compression in our client which uses HttpClient, so we can log and compare performance across a number of different calls over a period of time. Our first attempt was to simply conditionally add the header as follows: HttpRequestMessage...

Enabling gzip on Amazon Elastic Beanstalk Windows environment

windows,amazon-web-services,gzip
I cannot create a proper deployment package for AWS Elastic Beanstalk to enable gzip compression on Windows IIS environments. I enabled in web config as described here. This worked only for static files, dynamic files are served as is. Anybody have a solution for this? Edit: There is another issue...

node remove a directory after gzipping it

node.js,delete,gzip
I got the following code from SO to gzip a directory: fstream.Reader({'path':'mydir','type':'Directory'}).pipe(tar.Pack()).pipe(zlib.Gzip()).pipe(fstream.Writer({'path': 'mygz.tar.gz'})); And to delete a directory: rm_rf('mydir',function(error){}); I need to put them together, so that I can gzip a dir and delete the original directory. To do this, I need to find a way to listen to the...

BitBucket API download src file zip issue

php,zip,gzip,bitbucket-api
I am connecting to the BitBucket API and would like to be able to download the zip file of a repository to the server. Due to the repositories being private it requires user access. I can download the file by the user login details within the link: https://{user}:{pass}@bitbucket.org/{owner_name}/{repository}/get/master.zip But I...