zlib,endianness,uncompress , Does zlib's “uncompress” preserve the data's original endianness, or does it do an endian conversion?

Does zlib's “uncompress” preserve the data's original endianness, or does it do an endian conversion?


Tag: zlib,endianness,uncompress

I am working with legacy C++ code that accesses two-byte integer data compressed in a sqlite database. The code uses zlib's uncompress function to extract the data, which comes out on my little-endian machine as little-endian values.

To allow for the possibility that this code may be ported to big-endian machines, I need to know if the data will always decompress in little-endian order, or if (instead) zlib will somehow do the conversion.

This is the only applicable tidbit I've been able to find for(from zlib's FAQ on their site):

  1. Will zlib work on a big-endian or little-endian architecture, and can I exchange compressed data between them? Yes and yes.

Doesn't really answer my question... I'm prepared to handle the endian conversion if needed. Is it safe to assume that the original input data endianness is what you get back out, regardless of the platform on which you run uncompress? (I don't have access to a big-endian machine at present on which to test this myself).


zlib compresses and decompresses a stream of bytes losslessly. So whatever endianess went in is exactly what comes out. This is entirely regardless of the endianess of the compressing and decompressing machines.

The FAQ entry refers to the fact that the code was written to be insensitive to the endianess of the architecture that the code is compiled to and run on.


How to implement zlib's crc32_combine() function in PHP?

I'm trying to create crc32_combine function in PHP but I don't fully understand it's source in zlib library. I need this: <?php $crc_a = crc32('a'); $crc_aa = crc32('aa'); $crc_aa === crc32_combine($crc_a, $crc_a, 1); // should result in TRUE What I have: <?php define('GF2_DIM', 32); function gf2_matrix_times($mat, $vec) { $sum =...

How do I do a bit-wise XOR on NSData in Objective-C?

I have two NSData objects, data1 and data2, and I'd like to do a bit-wise XOR and store the result in a third NSData object, xorData. The first thing I tried was this: *data1.bytes^*data2.bytes; but it gave an error of: Invalid operands to binary expression ('const void' and 'const void')...

Does zlib's “uncompress” preserve the data's original endianness, or does it do an endian conversion?

I am working with legacy C++ code that accesses two-byte integer data compressed in a sqlite database. The code uses zlib's uncompress function to extract the data, which comes out on my little-endian machine as little-endian values. To allow for the possibility that this code may be ported to big-endian...

Blowfish encryption and decryption across c sockets

I'm using the following implementation of blowfish (with some stuff cut out of the middle) to encrypt and decrypt messages. I am attempting to encrypt a message on a big endian machine (SunOS 5.10), and then send it over a socket to a little endian machine (linux). I can encrypt...

Link error: Installing boost zlib filters using NuGet in Visual Studio (2012)

I've installed these packages: boost boost-vc110 boost_iostreams_zlib zlib But the linker is unable to find the required lib: 2>LINK : fatal error LNK1104: cannot open file 'libboost_zlib-vc110-mt-1_58.lib' Is there a way to fix this problem manually by changing the file name or am I still missing a package?...

How to insert type 0 block along with data in zlib

I have to insert uncompressed data in between the compressed data bytes. Type 0 header in zlib allows me to do that. But How can i do that ? any clues ?

Is zlib Type0 header data should be included in adler checksum calculation?

When calculating the Adler-32 checksum of uncompressed data in zlib format, should it include Type 0 (uncompressed data) data as well?

How to use multiple threads for zlib compression

I have a large chunk of data (around 2 GB) that needs to be compressed using zlib (deflate()). I am currently reading 500 kb of data at 1 go, compressing it and writing it to my output file. With 1 thread, everything is fine. The data is compressed and I...

compressing multicast packets with zlib library

I am listening packets from a multicast ip port and trying to compress that packet using zlib library, ( want to know the compression ratio from zlib in the live packets, as this is the requirement from our client). I have implemented zlib compression code as below, but length of...

Clean way to make portable endian-correct file-reading / writing code in C++

I want to write some C++ code that can read and write from files in an endian-correct way. More exactly, I want to be able to read a particular type of file, whose endianness I can easily detect (its magic number being reversed or not). But how would I then...

How can a C++ template be specialized for all 32-bit POD types?

I've developed a simple template function for swapping the byte order of a single field: template <typename T> inline void SwapEndian(T& ptr) { char *bytes = reinterpret_cast<char*>(&ptr); int a = sizeof(T) / 2; while (a--) { char tmp = bytes[a]; int b = sizeof(T) - 1 - a; bytes[a] =...

Decode gzip compressed and base64 encoded data to a readable format

Trying to decode gzip compressed and base64 encoded data to a readable format in Python3.4. import base64 import zlib original_data = '...jU2X0NCQ19TSEEAAAABAAA=' #Data cut short. decoded64 = base64.b64decode(original_data) #format:b'\x16xe\x94...\xae\x9a\...' final_decoded = zlib.decompress(decoded64) print(final_decoded) Been getting: "Error -2 while preparing to decompress data: inconsistent stream state." Not sure what I'm doing...

zlib inflate returning a buffer error

I'm trying to decompress a zip file using zlib (without using any extension or 3rd party). Initially, the src_len is 48756255, and the dest_len is 49209890. The first pass in the while loop is fine: err is Z_OK and the second pass through starts. On the second pass, no matter...

Android Java - NumberFormatException while parsing “76”

I'm working on an Android app using zbar to decode QR. The structure of the QR is: "SpecificPrefix/[Number]". String d = sym.getData(); // I get the QR text from the symbol if(d.startsWith(QRUri)) { // QRUri is a string const containig the prefix int id; try { Log.d("NOT", "QR Subst \""...

Exception during integration test in new jhipster app with mongodb

When i create new jhipster app and run integration tests i have such exception: Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.mongodb.Mongo]: Factory method 'mongo' threw exception; nested exception is de.flapdoodle.embed.process.exceptions.DistributionException: java.io.EOFException: Unexpected end of ZLIB input stream at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:189) ~[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE]...

How can I deflate zlib data and find out how many input bytes there were?

I'm building a git clone implementation in Rust. I've gotten to the part where I need to parse the packfile to create the index, and I'm almost done parsing it. Each object in the packfile consists of a header (Which I'm already parsing correctly) followed by the contents which are...

How to swap endianness of Int16 without BitConverter

I need to read binary files containing millions of Int16 stored as big endian. My first method was to use BitConverter and Array.Reverse() but that appears too slow for my purpose. Is there a way to do it with bitwise arithmetic instead ?...

uncompress error when using zlib

It could be a silly question. When I tried to uncompress a compressed data in memory, got error. Here is the code. #include <zlib.h> #include <stdio.h> #include <stdlib.h> int readFile(char *fname, char buf[]) { FILE *fp = fopen(fname, "rb"); if (fp == NULL) { printf("Failed to open %s\n", fname); exit(0);}...

Store typed array in ArrayBuffer

I have an ArrayBuffer of data which is composed of Uint8s. Each is in network byte order (big-endian). My goal is to convert each Uint8 into little-endian order, but then put them back in the ArrayBuffer. I know I can easily separate the individual Uints by using a typed array,...

Write zlib compressed utf8 data to a file

I have a file with data encoded in utf-8. I would like to read the data, remove whitespaces, separate words with a newline, compress the entire content and write them to a file. This is what I am trying to do : with codecs.open('1020104_4.utf8', encoding='utf8', mode='r') as fr : data...

gethostbyname and endianness - how are the bytes returned?

On my (Intel) x86 machine, I've noticed that if I printf the results of gethostbyname for localhost, I get 100007F, even though the MSDN documentation states it should return the IP in network byte order, aka big endian. I searched a bit and found this topic. Based on the answers...

Explanation of HEX value representation and Endianess

I was working on a script to basically output some sample data as a binary blob. I'm a new intern in the software field and vaguely remember the idea of endianness. I realize that the most significant bits for big-endian starts at the top and works down the memory block....

zlib uncompress a .zip file [duplicate]

This question already has an answer here: Simple way to unzip a .zip file using zlib 2 answers So I have an array of bytes that I would like to decompress using the uncompress function in zlib but it always returns Z_DATA_ERROR. So my question is, does zlib support...

Java: Efficiently converting an array of longs to an array of bytes

I have an array of longs I want to write to disk. The most efficient disk I/O functions take in byte arrays, for example: FileOutputStream.write(byte[] b, int offset, int length) ...so I want to begin by converting my long[] to byte[] (8 bytes for each long). I'm struggling to find...

Why conditional byte order is bad?

Linus claims[12] that conditional byte order is worse than silly. The first thing comes to my mind is ZFS but surely there must be other examples. He wrote: The only sane model is to specify one fixed byte order. Seriously. It's equally portable, it generates better code - even on...

Read libcap file with specific endianness

I wrote a c-lang program to read a .pcap file.What fogs me is that the data I read was with a different endianness as to WireShark. I'm working on X86 ach, as I can see, it's LittleEndian. So, can I read the .pcap file with BigEndian? How? Code fragments: /*...

Zip file encryption readable by some Zip clients, not others

I'm investigating a bug report on my open source UnzipKit project. Basically, when writing files encrypted with a password to a Zip file, the resulting archives are readable by some Zip clients, and not others. UnzipKit writes the password as a UTF-8 string, using the MiniZip wrapper around zlib, which...

Porting C endianness & pointers black magic to Swift

I'm trying to translate this snippet : ntohs(*(UInt16*)VALUE) / 4.0 and some other ones, looking alike, from C to Swift. Problem is, I have very few knowledge of Swift and I just can't understand what this snippet does... Here's all I know : ntohs swap endianness to host endianness VALUE...

How to use deflate/inflate SetDictionary with raw deflate/inflate?

I'm trying to understand the functionality of raw deflate when we use dictionary. I'm aware of the following. 1. When we use a dictionary, the application should provide the same dictionary for deflate() and inflate(). 2. When doing raw deflate, this function must be called either before any call of...

NodeJS: zlib.gunzip() on file contents returns Error: incorrect header check

My script takes file data, interprets it, and spits it out in a different format. The user can input a URL or upload a file, and the file refence for either of these can be XML or GZIP (which I then decompress). What works: Parsing XML + GZIP from URL...

Python: unpack android backup?

I'd like to unpack an Android backup file using python. According to http://nelenkov.blogspot.com/2012/06/unpacking-android-backups.html an unencryted adb file can be decompressed using dd if=mybackup.ab bs=24 skip=1|openssl zlib -d > mybackup.tar and tar xvf mybackup.tar Can these can be done in python? Python has zlib, gzip and tarfile, which seem as if...

Compress data to chunks using Zlib

I want to compress a file with size > 50mb with zlib. But I want the compressed file to be in chunks (for example, 32Kb chunks). I don't want to read 32Kb of uncompressed file and compress it. The compressed file should contain 32kb compressed chunks. Compression should be reset...

What's wrong with this use of java Deflater/Inflater with a dictionary

I'm trying to use Inflater and Deflater with a dictionary but it's not working. When I run this simple test program: import java.io.*; import java.util.*; import java.util.zip.*; public class DictTest { public static void main(String[] args) throws Exception { final int level = 9; final boolean nowrap = true; //...

Trouble capturing IP packets with libpcap

First the structs: /* Ethernet addresses are 6 bytes */ #define ETHER_ADDR_LEN 6 /* Ethernet header */ struct sniff_ethernet { u_char ether_dhost[ETHER_ADDR_LEN]; /* Destination host address */ u_char ether_shost[ETHER_ADDR_LEN]; /* Source host address */ u_short ether_type; /* IP? ARP? RARP? etc */ }; #define ETHERTYPE_IP 0x0800 /* IP */ /*...

Convert a char array into an integer (big and little endian)

I am trying to convert a char array into integer, then I have to increment that integer (both in little and big endian). Example: char ary[6 ] = { 01,02,03,04,05,06}; long int b=0; // 64 bits this char will be stored in memory address 0 1 2 3 4 5...

Do word size and endianness interplay when writing cross platform bit level code?

I was just looking at this answer which gives the following sample code to convert an int to an array of bytes: int intValue; byte[] intBytes = BitConverter.GetBytes(intValue); if (BitConverter.IsLittleEndian) Array.Reverse(intBytes); byte[] result = intBytes; I looked up Endianness and found that the reversal of bytes (or lack thereof) is...

Implicit conversion loses integer precision in ZLib compression method

Implicit conversion loses integer precision: 'unsigned long' to 'uInt' (aka 'unsigned int') I am trying to update my application from 32bit to 64bit, since changing the architecture I have recived an error as decribed above. I have a method that takes NSData that is zlib compressed and returns the...

Node: zlib.gunzip works, but not zlib.inflate?

Why is it that GZIP files I upload can be decompressed fine with zlib.gunzip(), but ZIP files uploaded result in an [Error: incorrect header check] errno: -3, code: 'Z_DATA_ERROR' error from zlib.inflate(), using the following code? if (req.body.compression == 'GZIP') { zlib.gunzip(body, function(err, dezipped) { if (typeof dezipped != 'undefined')...

Deterministic hashing of Python 3 strings with adler32

I read here the following: Note: To generate the same numeric value across all Python versions and platforms use adler32(data) & 0xffffffff. I am hoping to apply this to a string of the form: "S89234IX", but when I do so, I get: > zlib.adler32("S89234IX") --------------------------------------------------------------------------- TypeError Traceback (most recent call...

How can I simulate an environment where BitConverter.IsLittleEndian is the opposite for my unit tests?

I am using the two methods BitConverter.GetBytes and Array.Reverse to read and write binary data from a file due the endianness. My unit tests are passing and the implementation seems fine. How can I simulate an environment where BitConverter.IsLittleEndian is the opposite for my unit tests?...