google-cloud-storage,gsutil , gsutil - no locks available


gsutil - no locks available

Question:

Tag: google-cloud-storage,gsutil

Has anyone seen this error from gsutil or know how to fix it? I get it when I try to run any gsutil command, but here is an example trying to use ls on a bucket in my google cloud project.

$ gsutil ls gs://BUCKET/FOLDER
 Traceback (most recent call last):
  File "/home/gmcinnes/bin/google-cloud-sdk/bin/bootstrapping/gsutil.py", line 68, in <module>
bootstrapping.PrerunChecks(can_be_gce=True)
  File "/home/gmcinnes/bin/google-cloud-sdk/bin/bootstrapping/bootstrapping.py", line 279, in PrerunChecks
CheckCredOrExit(can_be_gce=can_be_gce)
  File "/home/gmcinnes/bin/google-cloud-sdk/bin/bootstrapping/bootstrapping.py", line 167, in CheckCredOrExit
cred = c_store.Load()
  File "/home/gmcinnes/bin/google-cloud-sdk/bin/bootstrapping/../../lib/googlecloudsdk/core/credentials/store.py", line 206, in Load
cred = store.get()
  File "/home/gmcinnes/bin/google-cloud-sdk/bin/bootstrapping/../../lib/oauth2client/client.py", line 350, in get
self.acquire_lock()
  File "/home/gmcinnes/bin/google-cloud-sdk/bin/bootstrapping/../../lib/oauth2client/multistore_file.py", line 222, in acquire_lock
self._multistore._lock()
  File "/home/gmcinnes/bin/google-cloud-sdk/bin/bootstrapping/../../lib/oauth2client/multistore_file.py", line 281, in _lock
self._file.open_and_lock()
  File "/home/gmcinnes/bin/google-cloud-sdk/bin/bootstrapping/../../lib/oauth2client/locked_file.py", line 370, in open_and_lock
self._opener.open_and_lock(timeout, delay)
  File "/home/gmcinnes/bin/google-cloud-sdk/bin/bootstrapping/../../lib/oauth2client/locked_file.py", line 211, in open_and_lock
raise e
IOError: [Errno 37] No locks available

Thanks


Answer:

Figured it out. The filesystem on that machine was full. I cleaned it up and it works now.

$ df 
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda1 10079084 9678804 0 100% /

Related:


Upload Json Array from GAE to Cloud Storage in GO


javascript,json,google-app-engine,go,google-cloud-storage
I trying to upload to google cloud storage a json array which is posted by an app engine application using the following code: saveData : function saveData() { var _this = this, save = this.shadowRoot.querySelector('#save-data'), subData = JSON.stringify(_this.app.userSession); save.url="url"; save.body = subData; save.go(); } The posted message is handled in...

Changing Permissions of Google Container Engine Cluster


node.js,google-cloud-storage,google-cloud-sql,google-cloud-platform,google-container-engine
I have been able to successfully create a Google Container Cluster in the developers console and have deployed my app to it. This all starts up fine, however I find that I can't connect to Cloud SQL, I get; "Error: Handshake inactivity timeout" After a bit of digging, I hadn't...

Google cloud storage - Tez output files


hadoop,hive,google-cloud-storage
When I run a query using tez , the number of output files are very huge. I have some 4-5 GB of data each having 46 MB or 16 MB. I want to have only 2-3 files as output files. My output files location will be google cloud storage. How...

Moving multiple files with gsutil


google-cloud-storage,move,gsutil
Let's say I've got the following files in a Google Cloud Storage bucket: file_A1.csv file_B2.csv file_C3.csv Now I want to move a subset of these files, lets say file_A1.csv and file_B2.csv. Currently I do this like that: gsutil mv gs://bucket/file_A1.csv gs://bucket/file_A11.csv gsutil mv gs://bucket/file_B2.csv gs://bucket/file_B22.csv This approach requires two call...

Upload to Google Cloud Storage via signed URL - object not publicly readable


google-cloud-storage,acl,pre-signed-url
I followed up this tutorial to allow upload of files from GWT frontend directly to Google Cloud Storage using signed URLs. I've extended the Java example by specifying content type which worked just fine. Then, I saw that files uploaded this way weren't publicly readable. To get this working I've...

Store a box or dropbox file


java,google-app-engine,google-cloud-storage,dropbox-api,box-api
I am writing an app engine app which has access both to box and dropbox api. I want to save the file into the google cloud storage. Both of them have the FileOutPutStream as the output stream for the file to download. DropBoxAPI : FileOutputStream outputStream = new FileOutputStream("magnum-opus.txt"); }try...

How to get the response data from cloud to string


java,json,gson,google-cloud-storage,restlet
I want to get the response to a string variable from the data from the cloud. ClientResource cr = new ClientResource("http://localhost:8888/users"); cr.setRequestEntityBuffering(true); try { try { cr.get(MediaType.APPLICATION_JSON).write(System.out); } catch (IOException e) { e.printStackTrace(); } } catch (ResourceException e) { // TODO Auto-generated catch block e.printStackTrace(); } I have response as...

Asynchronous requests from App Engine to Cloud Storage


google-app-engine,google-cloud-storage,google-cloud-platform
In my App Engine app I'm using Python Client for Google App Engine to read data from Cloud Storage. When I'm reading multiple files from Cloud Storage, is there an option to make this process asynchronous?...

Can AppEngine *pull* in data over 32MB?


python,google-app-engine,google-cloud-storage
I'm writing an API that needs to ingest HD videos (at least 100MB). I only have access to the videos through an HTTP XML feed, so I can only pull the videos (with a GET) once I have the video's URL. The plan is to store the videos in GCS....

Moving Google Cloud Storage bucket to another project


bigdata,google-cloud-storage
What is the best way to move an existing Google Cloud Storage bucket to another project? I don't want to copy it outside Google Cloud Storage for the transfer, have two copies of the data or use another bucket name. How close can I get to these requirements?...

storing thousands of files on google cloud storage


google-app-engine,google-cloud-storage
I am making an app on google app engine where users will upload files. So total files may reach thousands. Will the cloud storage be able to efficiently retrive those files on demand? I am using only a single bucket.

GAE Python unable to write file properly to Cloud Storage


python,google-app-engine,google-cloud-storage,webapp2
I am unable to figure out what I am doing wrong while uploading file using web form (python, webapp2, gae) and writing to cloud storage (GCS) jinja2 template <form action="/upload_data" method="post"> <input type="file" name="add_scanned_data_file" enctype="multipart/form-data"> <input type="image" src="stylesheets/add_data.png" alt="submit" align="left"></form> webapp2 code fx = self.request.body_file fx_name = "some_name" gcs_file =...

Subdomain for access buckets files


google-cloud-storage
I am trying to create subdomain for my cloud storage. Now, in my site all files hosted in GCS and have url like: http://commondatastorage.googleapis.com/@[email protected]/file.jpg I want create subdomain with name like @[email protected] For this, I created the name record in domain zone: @[email protected] IN CNAME c.storage.googleapis.com. [email protected]@ IN CNAME c.storage.googleapis.com....

Meteor slingshot file upload to Google Cloud Storage internal server error


meteor,google-cloud-storage,meteor-slingshot
I am trying to upload files using edgee:slingshot, but I have several errors. I have did everything as described in github page. This is my settings on server: Slingshot.GoogleCloud.directiveDefault.GoogleSecretKey = Assets.getText('google-cloud-service-key.pem'); Slingshot.createDirective("myFileUploads", Slingshot.GoogleCloud, { bucket: 'dossum-app', GoogleAccessId: "GOOGXXXX", GoogleSecretKey: "qZEsLZ/NiXXXXXXXXXXXXUW8NVjSvRb8SgdxXXXXX2", acl: 'bucket-owner-full-control', authorize: function() { if (!this.userId) { var message...

Replacing the FileService Api to create a Blob file in server side


java,google-app-engine,gwt,google-cloud-storage,blobstore
I am using the FileService Api to save files to the Blobstore from server side like this: FileService fileService = FileServiceFactory.getFileService(); AppEngineFile file = null; file = fileService.createNewBlobFile(contentType, fileName); boolean lock = true; FileWriteChannel writeChannel = null; writeChannel = fileService.openWriteChannel(file, lock); ByteBuffer bb = ByteBuffer.wrap(FILE_BYTES); writeChannel.write(bb); writeChannel.closeFinally(); BlobKey blobKey =...

Import CSV file from Google Cloud Storage to Google Cloud SQL


csv,import,google-cloud-storage,google-cloud-sql
I am trying to import a csv file directly into Google Cloud SQL using the command below. LOAD DATA LOCAL INFILE "gs:/zuds/stg/ems/product_offering_fact/000002_0" INTO TABLE buydoc CHARACTER SET "utf8" FIELDS TERMINATED BY "^"; For some reason, it is returning an error like below. I have confirmed that the file exists. However,...

How do I create a Google Cloud Storage resumable upload URL with Google Client Library for Java on App Engine?


java,google-app-engine,google-cloud-storage
I found the follow note, which describes exactly what I'd like to do: Note: If your users are only uploading resources (writing) to an access-controlled bucket, you can use the resumable uploads functionality of Google Cloud Storage, and avoid signing URLs or requiring a Google account. In a resumable upload...

Choosing the correct google cloud data storage strategy for images [closed]


google-app-engine,google-cloud-storage,google-cloud-sql,google-cloud-platform,google-cloud-datastore
I'm writing my first google cloud backend for an image sharing mobile app and I'm having difficulties understanding which data storage option to go for. The clients will be mobile apps and probably also a web frontend. From what I've read so far the AppEngine seems to be a nice...

Error to create a disk in google compute (The resource '… diskTypes/{pd-ssd}' was not found


google-cloud-storage,google-compute-engine
I try to create a disk in Google Compute. This is the process and error: gcloud compute disks create test --type={pd-ssd} For the following disks: - [test] choose a zone: [1] asia-east1-a [2] asia-east1-c [3] asia-east1-b [4] europe-west1-c [5] europe-west1-b [6] europe-west1-d [7] us-central1-a [8] us-central1-b [9] us-central1-c [10] us-central1-f...

404 error when uploading file to GCS with query string on Production


java,google-app-engine,google-cloud-storage,blobstore
I am using flash to upload a file to GCS. I create a upload url using the following successUrl "/cms/requestManager?ps={"1":{"action":"Upload Audio Recording","sendId":1,"data":{"fileName":"testQuestionAudioRecording","category":"testQuestionRecording"}},"jsInstanceID":"ahdzfmRldi1keW5hY3RpdmVzb2Z0d2FyZXITCxIKSlNJbnN0YW5jZRjKm-kBDKIBBmRzLmNtcw","userActive":true}" When I send the file upload it comes back with a 404 error. I check in the appengine instance logs for 404 errors and there are none. The...

Can I change the Google Cloud Storage bucket class after the bucket has been created?


google-cloud-storage,gsutil
I can't find anything in the documentation on if it is possible to change the current class of a bucket. Is this possible? If this is not possible, what would be the most cost efficient way to move contents (10TB, hundreds of thousands of files) from a DRA bucket to...

Data backup by Google Cloud data store Vs google drive Vs other storage service


android,google-app-engine,google-drive-sdk,google-cloud-storage,google-cloud-endpoints
I need some basic understanding of google cloud as a whole.Supposing, I have an android app that will store user's contact,photo etc to cloud,in this case- 1.Which service should I use - google cloud storage or google drive ? if I use google cloud storage, how do I do it?...

Creating a scalable database for android app | cloud hosted


android,cloud,google-cloud-storage,cloud-hosting,android-database
I am working to launch an app which in future will store huge number of users and there profile information. But I Wanted to start small to pre-test my app idea , so what will be the options for hosting database which are cost effective initially and scalable in future...

Alternative to `x-amz-website-redirect-location` in Google Cloud Storage


google-cloud-storage
I would like to host a website in Google Cloud Storage, where the root points to a certain folder (basically it redirects to the /en/ folder, the default language). I can accomplish this in Amazon S3 using the x-amz-website-redirect-location header. But according to the documentation, that header is not available...

NoSuchKey when getting a signed url for a cloudstorage object with a space in the name


python,python-2.7,google-app-engine,google-cloud-storage
The app engine code below uses app_identity.sign_blob() to request a signed url. This code works fine, when there is no space in the GCS filename. A space is allowed in object names. For testing I used the SDK. I've seen a lot of questions about this issue, but I could...

Cloud storage and secure download strategy on app engine. GCS acl or blobstore


google-app-engine,authentication,google-cloud-storage,acl,blobstorage
My appengine app creates cloudstorage files. The files will be downloaded by a third party. The files contain personal medical information. What would be the preferred way of downloading: Using a direct GCS download link with a user READER acl. Or using a blobstore download handler in an appengine app....

error: Bad content type. Please use multipart


javascript,angularjs,file-upload,xmlhttprequest,google-cloud-storage
I'm totally new to file uploads....I'm using angular-file-upload which creates an XHR outside of angularjs to upload files to google cloud storage. When I try to upload I keep getting the error below. How can I resolve this? 400 Bad content type. Please use multipart. Here's my controller setup: var...

Is it possible to upload files directly to the Google Cloud Storage?


php,google-cloud-storage,google-api-php-client
I have written a file upload for larger files that is uploading the files to the Cloud Storage. Unfortunately this takes a while, because the Files are uploaded to the web server first, and then again from the web server to the Google Cloud Storage. Is there a way to...

What are the accepted MIME content types for Google Cloud Storage (GCS)?


json,documentation,mime-types,google-cloud-storage,content-type
I have read the online documentation that I could find relating to uploading files to Google's Cloud Storage service (specifically the JSON api): https://cloud.google.com/storage/docs/json_api/v1/how-tos/upload And on that page it references another page/doc to "See the API reference for each method's list of accepted MIME types and size limits for uploaded...

Track progress of file upload to google cloud storage via js?


javascript,google-cloud-storage
I'm wondering if it is possible to track the progress of a file upload to google cloud storage when using the javascript client? As I have a progress bar that I would like to show to the user on how much of the file has been uploaded so far. I'm...

fetching just a byte range for an object (byte-offset + len) rather than full file


google-cloud-storage
I am storing large files using Google Cloud Storage. Sometimes I want to retrieve the full file, but often I want to retrieve just a specific range of bytes from the file. If I know the byte offset and length that I need to retrieve, is there any way to...

.\key.p12: open failed: ENOENT (No such file or directory)


java,android,google-cloud-storage,assets,google-cloud-platform
In my android application I'm accessing the Google cloud storage . I have generated the private key xxxxxxxkey.p12 .I have put my key file in assets folder . But while running the project it is not opening the key.p12 file . I have tried putting it outside the assets folder...

List objects in Google Cloud Storage using the S3 interop API


amazon-s3,google-cloud-storage
We're currently looking into migrating some objects that we store in S3 to Google Cloud Storage. According to their documentation their XML API should be S3 compatible. Pretty much the only thing we have to do is to change the hostname and access keys. However, when I look at the...

Google cloud storage with app engine


android,google-app-engine,google-cloud-storage
Its driving me nuts,somebody pls help me out here.This google cloud stuff is confusing me.I m little bit off here,something is missing in my understanding.I want to use cloud storage.Now I have a default Android Studio Project which has an android client,an app engine backend,consisting of entity,endpoint,library for client etc...

Getting the public link of the uploaded file on Google Cloud Storage from Go Runtime


google-app-engine,google-cloud-storage
Im able to create and write into file on Google Cloud from the Go Runtime , but not able to figure out a way to make the file public and get the public link of the file . Any idea on how to get the public link of the file...

What is the correct flow to upload an image and use getServingUrl() in app engine?


java,google-app-engine,google-cloud-storage,google-cloud-endpoints,blobstore
I have an android app that wants to include image sending in the instant messaging client. I'm terribly confused on the way to do this using the blobstore or google cloud storage. Can someone please outline the steps to achieve this from the backend perspective? My goal is to end...

Exclude “.html” from URL's in Google Cloud Storage


google-cloud-storage
Is it possible to some how get www.mydomain.com/testing to serve www.mydomain.com/testing.html using Google Cloud Storage? With htaccess I use: RewriteCond %{SCRIPT_FILENAME} !-d RewriteRule ^([^\.]+)$ $1.html [NC,L] ...

Remove folder & all files in directory without glob()


php,google-app-engine,google-cloud-storage
I am trying to delete all files in a Cloud Storage directory using the functions that are available to me in Google App Engine. Normally, I could do something like this but glob() is not allowed: $files = glob('gs://bucket/folder/*'); // get all file names foreach($files as $file){ // iterate files...

Same Google Cloud Storage upload script works from one PC, but not the other, Why?


node.js,google-cloud-storage,google-authentication,gcloud-node
I'm trying to upload a file to Google Cloud Storage from Node.JS (using gcloud package) with Service Account credentials and I get "invalid_grant" error (probably authorization error). When I try to do this from another computer - it work fine, the error only occurs on my PC. var gcloud =...

Connect hadoop cluster to mutiple Google Cloud Storage backets in multiple Google Projects


hadoop,google-cloud-storage,google-hadoop
It is possible, to connect my Hadoop cluster to multiple Google Cloud Projects at once ? I can easly use any Google Storage bucket in single Google Project via Google Cloud Storage Connector as explained in this thread Migrating 50TB data from local Hadoop cluster to Google Cloud Storage. But...

Upload image with google app engine fails


php,image,google-app-engine,google-cloud-storage
I am trying to develop an image upload in my PHP google app engine app. First i am fetching the URL to use in the form. $uploadUrl = CloudStorageTools::createUploadUrl('/upload/blob', [ 'max_bytes_per_blob' => 10000000, // 10 mb 'max_bytes_total' => 10000000, // 10 mb ]); Then in the upload handler i just...

android acces google storage Version v1beta2 of this API is no longer available. Please try again using JSON API v1


android,google-cloud-storage,json-api
When i try to list files on google storage i get this error "code" : 400, Version v1beta2 of this API is no longer available. Please try again using JSON API v1. To request temporary reinstatement for your project, please visit https://docs.google.com/forms/d/1isIxBZg3rsQbDN_TOalZaz1WT_ebJchsrlv-Qr_r9mY/viewform?entry.244568692=773473680319&entry.176324201=v1beta2&entry.1071661541-Qr_r9mY/prefill how can i use JSON API v1 in...

Authenticating using .boto google cloud storage api


python-2.7,google-cloud-storage,boto
Having issues with the google cloud storage api. followed the setup from here: https://cloud.google.com/storage/docs/gsutil_install Here is my code, it should just print the files in the bucket. This structure worked with an earlier version of the oauth2plugin but cannot get it to work with the new gcs modules. from gcs_oauth2_boto_plugin...

Where is the retrying logic in the Google Cloud Storage go client?


go,google-cloud-storage
Some Google documentation for the go client for Cloud Storage says: This library supports reading and writing large amounts of data to Cloud Storage, with internal error handling and retries, so you don't have to write your own code to do this. However, when using the below snippet of code...

Sync github repository with google cloud storage bucket


github,google-cloud-storage,google-cloud-platform
Is it possible to sync a GitHub repository with a google cloud storage bucket, so that I can edit the repository on GitHub and it automatically updates the bucket with the changes. I have already tried the google cloud platform source code tools, but I couldn't find any way to...

can't find import: “google.golang.org/cloud/storage”


go,google-cloud-storage
I want to work with the cloud api gcloud-golang so i did go get in the src folder goapp get google.golang.org/cloud I can "see" the library in th liteIDE but i cant goapp serve Im getting this exception /home/mik/go-programs/src/google.golang.org/cloud/storage/acl.go:21: can't find import: "google.golang.org/api/storage/v1" 2015/03/14 23:13:37 Can't find package "google.golang.org/api/googleapi" in...

Merging files in Google Cloud Storage using Google Cloud Dataflow


hadoop,hdfs,google-cloud-storage,google-cloud-dataflow,lambda-architecture
Nathan Marz in his book "Big Data" describes how to maintain files of data in HDFS and how to optimize files' sizes to be as near native HDFS block size as possible using his Pail library running on top of Map Reduce. Is it possible to achieve the same result...

How to make gsutil rsync skip symlinks and return error code 0?


google-compute-engine,gsutil
I have noticed that when gsutil rsync is working, it will return a non-zero error code out if it encounters a symlink which it can not resolve: $ gsutil -m rsync -r -C /my_folder/ gs://my_bucket/ CommandException: Error opening file "file:////my_folder/my_symlink": . CommandException: 1 files/objects could not be copied/removed. Is there...