google-cloud-storage,gsutil , gsutil - no locks available

gsutil - no locks available


Tag: google-cloud-storage,gsutil

Has anyone seen this error from gsutil or know how to fix it? I get it when I try to run any gsutil command, but here is an example trying to use ls on a bucket in my google cloud project.

$ gsutil ls gs://BUCKET/FOLDER
 Traceback (most recent call last):
  File "/home/gmcinnes/bin/google-cloud-sdk/bin/bootstrapping/", line 68, in <module>
  File "/home/gmcinnes/bin/google-cloud-sdk/bin/bootstrapping/", line 279, in PrerunChecks
  File "/home/gmcinnes/bin/google-cloud-sdk/bin/bootstrapping/", line 167, in CheckCredOrExit
cred = c_store.Load()
  File "/home/gmcinnes/bin/google-cloud-sdk/bin/bootstrapping/../../lib/googlecloudsdk/core/credentials/", line 206, in Load
cred = store.get()
  File "/home/gmcinnes/bin/google-cloud-sdk/bin/bootstrapping/../../lib/oauth2client/", line 350, in get
  File "/home/gmcinnes/bin/google-cloud-sdk/bin/bootstrapping/../../lib/oauth2client/", line 222, in acquire_lock
  File "/home/gmcinnes/bin/google-cloud-sdk/bin/bootstrapping/../../lib/oauth2client/", line 281, in _lock
  File "/home/gmcinnes/bin/google-cloud-sdk/bin/bootstrapping/../../lib/oauth2client/", line 370, in open_and_lock
self._opener.open_and_lock(timeout, delay)
  File "/home/gmcinnes/bin/google-cloud-sdk/bin/bootstrapping/../../lib/oauth2client/", line 211, in open_and_lock
raise e
IOError: [Errno 37] No locks available



Figured it out. The filesystem on that machine was full. I cleaned it up and it works now.

$ df 
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda1 10079084 9678804 0 100% /


Asynchronous requests from App Engine to Cloud Storage

In my App Engine app I'm using Python Client for Google App Engine to read data from Cloud Storage. When I'm reading multiple files from Cloud Storage, is there an option to make this process asynchronous?...

Choosing the correct google cloud data storage strategy for images [closed]

I'm writing my first google cloud backend for an image sharing mobile app and I'm having difficulties understanding which data storage option to go for. The clients will be mobile apps and probably also a web frontend. From what I've read so far the AppEngine seems to be a nice...

NoSuchKey when getting a signed url for a cloudstorage object with a space in the name

The app engine code below uses app_identity.sign_blob() to request a signed url. This code works fine, when there is no space in the GCS filename. A space is allowed in object names. For testing I used the SDK. I've seen a lot of questions about this issue, but I could...

How to get the response data from cloud to string

I want to get the response to a string variable from the data from the cloud. ClientResource cr = new ClientResource("http://localhost:8888/users"); cr.setRequestEntityBuffering(true); try { try { cr.get(MediaType.APPLICATION_JSON).write(System.out); } catch (IOException e) { e.printStackTrace(); } } catch (ResourceException e) { // TODO Auto-generated catch block e.printStackTrace(); } I have response as...

How to make gsutil rsync skip symlinks and return error code 0?

I have noticed that when gsutil rsync is working, it will return a non-zero error code out if it encounters a symlink which it can not resolve: $ gsutil -m rsync -r -C /my_folder/ gs://my_bucket/ CommandException: Error opening file "file:////my_folder/my_symlink": . CommandException: 1 files/objects could not be copied/removed. Is there...

error: Bad content type. Please use multipart

I'm totally new to file uploads....I'm using angular-file-upload which creates an XHR outside of angularjs to upload files to google cloud storage. When I try to upload I keep getting the error below. How can I resolve this? 400 Bad content type. Please use multipart. Here's my controller setup: var...

.\key.p12: open failed: ENOENT (No such file or directory)

In my android application I'm accessing the Google cloud storage . I have generated the private key xxxxxxxkey.p12 .I have put my key file in assets folder . But while running the project it is not opening the key.p12 file . I have tried putting it outside the assets folder...

Upload Json Array from GAE to Cloud Storage in GO

I trying to upload to google cloud storage a json array which is posted by an app engine application using the following code: saveData : function saveData() { var _this = this, save = this.shadowRoot.querySelector('#save-data'), subData = JSON.stringify(; save.url="url"; save.body = subData; save.go(); } The posted message is handled in...

can't find import: “”

I want to work with the cloud api gcloud-golang so i did go get in the src folder goapp get I can "see" the library in th liteIDE but i cant goapp serve Im getting this exception /home/mik/go-programs/src/ can't find import: "" 2015/03/14 23:13:37 Can't find package "" in...

Changing Permissions of Google Container Engine Cluster

I have been able to successfully create a Google Container Cluster in the developers console and have deployed my app to it. This all starts up fine, however I find that I can't connect to Cloud SQL, I get; "Error: Handshake inactivity timeout" After a bit of digging, I hadn't...

Subdomain for access buckets files

I am trying to create subdomain for my cloud storage. Now, in my site all files hosted in GCS and have url like:[email protected]/file.jpg I want create subdomain with name like @[email protected] For this, I created the name record in domain zone: @[email protected] IN CNAME [email protected]@ IN CNAME

Can AppEngine *pull* in data over 32MB?

I'm writing an API that needs to ingest HD videos (at least 100MB). I only have access to the videos through an HTTP XML feed, so I can only pull the videos (with a GET) once I have the video's URL. The plan is to store the videos in GCS....

Getting the public link of the uploaded file on Google Cloud Storage from Go Runtime

Im able to create and write into file on Google Cloud from the Go Runtime , but not able to figure out a way to make the file public and get the public link of the file . Any idea on how to get the public link of the file...

android acces google storage Version v1beta2 of this API is no longer available. Please try again using JSON API v1

When i try to list files on google storage i get this error "code" : 400, Version v1beta2 of this API is no longer available. Please try again using JSON API v1. To request temporary reinstatement for your project, please visit how can i use JSON API v1 in...

List objects in Google Cloud Storage using the S3 interop API

We're currently looking into migrating some objects that we store in S3 to Google Cloud Storage. According to their documentation their XML API should be S3 compatible. Pretty much the only thing we have to do is to change the hostname and access keys. However, when I look at the...

Exclude “.html” from URL's in Google Cloud Storage

Is it possible to some how get to serve using Google Cloud Storage? With htaccess I use: RewriteCond %{SCRIPT_FILENAME} !-d RewriteRule ^([^\.]+)$ $1.html [NC,L] ...

GAE Python unable to write file properly to Cloud Storage

I am unable to figure out what I am doing wrong while uploading file using web form (python, webapp2, gae) and writing to cloud storage (GCS) jinja2 template <form action="/upload_data" method="post"> <input type="file" name="add_scanned_data_file" enctype="multipart/form-data"> <input type="image" src="stylesheets/add_data.png" alt="submit" align="left"></form> webapp2 code fx = self.request.body_file fx_name = "some_name" gcs_file =...

How do I create a Google Cloud Storage resumable upload URL with Google Client Library for Java on App Engine?

I found the follow note, which describes exactly what I'd like to do: Note: If your users are only uploading resources (writing) to an access-controlled bucket, you can use the resumable uploads functionality of Google Cloud Storage, and avoid signing URLs or requiring a Google account. In a resumable upload...

Track progress of file upload to google cloud storage via js?

I'm wondering if it is possible to track the progress of a file upload to google cloud storage when using the javascript client? As I have a progress bar that I would like to show to the user on how much of the file has been uploaded so far. I'm...

storing thousands of files on google cloud storage

I am making an app on google app engine where users will upload files. So total files may reach thousands. Will the cloud storage be able to efficiently retrive those files on demand? I am using only a single bucket.

Connect hadoop cluster to mutiple Google Cloud Storage backets in multiple Google Projects

It is possible, to connect my Hadoop cluster to multiple Google Cloud Projects at once ? I can easly use any Google Storage bucket in single Google Project via Google Cloud Storage Connector as explained in this thread Migrating 50TB data from local Hadoop cluster to Google Cloud Storage. But...

Meteor slingshot file upload to Google Cloud Storage internal server error

I am trying to upload files using edgee:slingshot, but I have several errors. I have did everything as described in github page. This is my settings on server: Slingshot.GoogleCloud.directiveDefault.GoogleSecretKey = Assets.getText('google-cloud-service-key.pem'); Slingshot.createDirective("myFileUploads", Slingshot.GoogleCloud, { bucket: 'dossum-app', GoogleAccessId: "GOOGXXXX", GoogleSecretKey: "qZEsLZ/NiXXXXXXXXXXXXUW8NVjSvRb8SgdxXXXXX2", acl: 'bucket-owner-full-control', authorize: function() { if (!this.userId) { var message...

Data backup by Google Cloud data store Vs google drive Vs other storage service

I need some basic understanding of google cloud as a whole.Supposing, I have an android app that will store user's contact,photo etc to cloud,in this case- 1.Which service should I use - google cloud storage or google drive ? if I use google cloud storage, how do I do it?...

Cloud storage and secure download strategy on app engine. GCS acl or blobstore

My appengine app creates cloudstorage files. The files will be downloaded by a third party. The files contain personal medical information. What would be the preferred way of downloading: Using a direct GCS download link with a user READER acl. Or using a blobstore download handler in an appengine app....

Replacing the FileService Api to create a Blob file in server side

I am using the FileService Api to save files to the Blobstore from server side like this: FileService fileService = FileServiceFactory.getFileService(); AppEngineFile file = null; file = fileService.createNewBlobFile(contentType, fileName); boolean lock = true; FileWriteChannel writeChannel = null; writeChannel = fileService.openWriteChannel(file, lock); ByteBuffer bb = ByteBuffer.wrap(FILE_BYTES); writeChannel.write(bb); writeChannel.closeFinally(); BlobKey blobKey =...

What are the accepted MIME content types for Google Cloud Storage (GCS)?

I have read the online documentation that I could find relating to uploading files to Google's Cloud Storage service (specifically the JSON api): And on that page it references another page/doc to "See the API reference for each method's list of accepted MIME types and size limits for uploaded...

Merging files in Google Cloud Storage using Google Cloud Dataflow

Nathan Marz in his book "Big Data" describes how to maintain files of data in HDFS and how to optimize files' sizes to be as near native HDFS block size as possible using his Pail library running on top of Map Reduce. Is it possible to achieve the same result...

Remove folder & all files in directory without glob()

I am trying to delete all files in a Cloud Storage directory using the functions that are available to me in Google App Engine. Normally, I could do something like this but glob() is not allowed: $files = glob('gs://bucket/folder/*'); // get all file names foreach($files as $file){ // iterate files...

Is it possible to upload files directly to the Google Cloud Storage?

I have written a file upload for larger files that is uploading the files to the Cloud Storage. Unfortunately this takes a while, because the Files are uploaded to the web server first, and then again from the web server to the Google Cloud Storage. Is there a way to...

Same Google Cloud Storage upload script works from one PC, but not the other, Why?

I'm trying to upload a file to Google Cloud Storage from Node.JS (using gcloud package) with Service Account credentials and I get "invalid_grant" error (probably authorization error). When I try to do this from another computer - it work fine, the error only occurs on my PC. var gcloud =...

Can I change the Google Cloud Storage bucket class after the bucket has been created?

I can't find anything in the documentation on if it is possible to change the current class of a bucket. Is this possible? If this is not possible, what would be the most cost efficient way to move contents (10TB, hundreds of thousands of files) from a DRA bucket to...

Alternative to `x-amz-website-redirect-location` in Google Cloud Storage

I would like to host a website in Google Cloud Storage, where the root points to a certain folder (basically it redirects to the /en/ folder, the default language). I can accomplish this in Amazon S3 using the x-amz-website-redirect-location header. But according to the documentation, that header is not available...

Moving multiple files with gsutil

Let's say I've got the following files in a Google Cloud Storage bucket: file_A1.csv file_B2.csv file_C3.csv Now I want to move a subset of these files, lets say file_A1.csv and file_B2.csv. Currently I do this like that: gsutil mv gs://bucket/file_A1.csv gs://bucket/file_A11.csv gsutil mv gs://bucket/file_B2.csv gs://bucket/file_B22.csv This approach requires two call...

Moving Google Cloud Storage bucket to another project

What is the best way to move an existing Google Cloud Storage bucket to another project? I don't want to copy it outside Google Cloud Storage for the transfer, have two copies of the data or use another bucket name. How close can I get to these requirements?...

Google cloud storage - Tez output files

When I run a query using tez , the number of output files are very huge. I have some 4-5 GB of data each having 46 MB or 16 MB. I want to have only 2-3 files as output files. My output files location will be google cloud storage. How...

Import CSV file from Google Cloud Storage to Google Cloud SQL

I am trying to import a csv file directly into Google Cloud SQL using the command below. LOAD DATA LOCAL INFILE "gs:/zuds/stg/ems/product_offering_fact/000002_0" INTO TABLE buydoc CHARACTER SET "utf8" FIELDS TERMINATED BY "^"; For some reason, it is returning an error like below. I have confirmed that the file exists. However,...

Store a box or dropbox file

I am writing an app engine app which has access both to box and dropbox api. I want to save the file into the google cloud storage. Both of them have the FileOutPutStream as the output stream for the file to download. DropBoxAPI : FileOutputStream outputStream = new FileOutputStream("magnum-opus.txt"); }try...

Creating a scalable database for android app | cloud hosted

I am working to launch an app which in future will store huge number of users and there profile information. But I Wanted to start small to pre-test my app idea , so what will be the options for hosting database which are cost effective initially and scalable in future...

Upload image with google app engine fails

I am trying to develop an image upload in my PHP google app engine app. First i am fetching the URL to use in the form. $uploadUrl = CloudStorageTools::createUploadUrl('/upload/blob', [ 'max_bytes_per_blob' => 10000000, // 10 mb 'max_bytes_total' => 10000000, // 10 mb ]); Then in the upload handler i just...

Upload to Google Cloud Storage via signed URL - object not publicly readable

I followed up this tutorial to allow upload of files from GWT frontend directly to Google Cloud Storage using signed URLs. I've extended the Java example by specifying content type which worked just fine. Then, I saw that files uploaded this way weren't publicly readable. To get this working I've...

404 error when uploading file to GCS with query string on Production

I am using flash to upload a file to GCS. I create a upload url using the following successUrl "/cms/requestManager?ps={"1":{"action":"Upload Audio Recording","sendId":1,"data":{"fileName":"testQuestionAudioRecording","category":"testQuestionRecording"}},"jsInstanceID":"ahdzfmRldi1keW5hY3RpdmVzb2Z0d2FyZXITCxIKSlNJbnN0YW5jZRjKm-kBDKIBBmRzLmNtcw","userActive":true}" When I send the file upload it comes back with a 404 error. I check in the appengine instance logs for 404 errors and there are none. The...

Where is the retrying logic in the Google Cloud Storage go client?

Some Google documentation for the go client for Cloud Storage says: This library supports reading and writing large amounts of data to Cloud Storage, with internal error handling and retries, so you don't have to write your own code to do this. However, when using the below snippet of code...

Error to create a disk in google compute (The resource '… diskTypes/{pd-ssd}' was not found

I try to create a disk in Google Compute. This is the process and error: gcloud compute disks create test --type={pd-ssd} For the following disks: - [test] choose a zone: [1] asia-east1-a [2] asia-east1-c [3] asia-east1-b [4] europe-west1-c [5] europe-west1-b [6] europe-west1-d [7] us-central1-a [8] us-central1-b [9] us-central1-c [10] us-central1-f...

Sync github repository with google cloud storage bucket

Is it possible to sync a GitHub repository with a google cloud storage bucket, so that I can edit the repository on GitHub and it automatically updates the bucket with the changes. I have already tried the google cloud platform source code tools, but I couldn't find any way to...

fetching just a byte range for an object (byte-offset + len) rather than full file

I am storing large files using Google Cloud Storage. Sometimes I want to retrieve the full file, but often I want to retrieve just a specific range of bytes from the file. If I know the byte offset and length that I need to retrieve, is there any way to...

Google cloud storage with app engine

Its driving me nuts,somebody pls help me out here.This google cloud stuff is confusing me.I m little bit off here,something is missing in my understanding.I want to use cloud storage.Now I have a default Android Studio Project which has an android client,an app engine backend,consisting of entity,endpoint,library for client etc...

Authenticating using .boto google cloud storage api

Having issues with the google cloud storage api. followed the setup from here: Here is my code, it should just print the files in the bucket. This structure worked with an earlier version of the oauth2plugin but cannot get it to work with the new gcs modules. from gcs_oauth2_boto_plugin...

What is the correct flow to upload an image and use getServingUrl() in app engine?

I have an android app that wants to include image sending in the instant messaging client. I'm terribly confused on the way to do this using the blobstore or google cloud storage. Can someone please outline the steps to achieve this from the backend perspective? My goal is to end...