FAQ Database Discussion Community


Upload zip files to service for user download

node.js,file-upload,express,amazon-s3,zip
I have an express (node.js) application that creates mp3 files for a user and stores them in a folder on the server. The server's file structure looks like this: data/ id#/ song1.mp3 song2.mp3 song3.mp3 id#/ song1.mp3 song2.mp3 song3.mp3 ... I want to create zip files and download links for the...

Reading many small files from S3 very slow

amazon-web-services,amazon-s3,hive,apache-pig,elastic-map-reduce
Loading many small files (>200000, 4kbyte) from a S3 Bucket into HDFS via Hive or Pig on AWS EMR is extremely slow. It seems that only one mapper is used to get the data, though I cannot exactly figure out where the bottleneck is. Pig Code Sample data = load...

How to upload S3 files in KeystoneJS

node.js,express,amazon-s3,keystone.js
I have an item called style which has 2 attributes, one which has raw css text and another which has an S3File. Style.add({ ... css: { type: Types.Code, language: 'css' }, cssFile: { type: Types.S3File, s3path: 'uploads/assets', }, ... }); I want to update the S3File with the contents of...

How to install SSL on CloudFront correctly?

ssl,amazon-s3,ssl-certificate,amazon-cloudfront
Hi I've just purchased an SSL for my domain. After contacting with my SSL provision I have 5 files from my provider. my_domain.crt AddTrustExternalCARoot.crt TrustedSecureCertificateAuthority5.crt USERTrustRSAAddTrustCA.rt my_domain.private_key My provider said that I have to install all of them. I used AWS CLI to install with this command. aws iam upload-server-certificate...

Writing an IAM policy and CORS configuration for Amazon S3

ruby-on-rails,amazon-web-services,amazon-s3,cloud,amazon-iam
I am very new to all this but have been able to get an avatar/image uploader to work in my Rails application. A user can upload a new avatar to my S3 bucket and the avatar shows within the web application. To this end, I've had to grant "AmazonS3FullAccess" policy...

Storing user submitted images

node.js,file-upload,amazon-s3,amazon-cloudfront
I'm building a node application in which users can submit images to customize their profile. I'm wondering what the best way would be to store these images? Is something like Amazon S3 the way to go? What about CloudFront, can this accept user submitted images? Sorry if this question is...

Weighted round robin dns between 2 Cloudfront distributions

amazon-web-services,amazon-s3,amazon-cloudfront,amazon-route53
We are trying to use aws to do some a gradual deployment test with our javascript code, but it seems to fail us we created 2 S3 buckets with CF distributions : a.example.net -> aaa.cloudfront.net b.example.net -> bbb.cloudfront.net than we created a weighted round robin DNS entry in route53 test.example.net...

s3cmd not working as cron-task when echos/dates are added

amazon-web-services,amazon-s3,cron,cron-task,s3cmd
I have a Digital Ocean Droplet (VPS) running Ubuntu 14.04. I have installed s3cmd and am able to run a sync successfully with this command: s3cmd sync --recursive --preserve /srv s3://MY-BUCKET-NAME And if I put that same command in to a .sh to run as a cron-task every minute, it...

How do I use a v2 authentication header with Amazon PHP SDK v3?

php,amazon-web-services,amazon-s3
Here's my code: $client = new S3Client([ 'credentials' => [ 'key' => 'xxx', 'secret' => 'xxx', ], 'version' => 'latest', 'region' => 'US', 'endpoint' => 'https://storage.googleapis.com', 'signature_version' => 'v2', ]); And I get this error: UnresolvedSignatureException in SignatureProvider.php line 61: Unable to resolve a signature for v2/s3/US. Valid signature versions...

mkey of a File on AWS

android,amazon-web-services,amazon-s3
I am trying to download a file from AWS using TransferManager. Can someone please suggest how can one find the mkey(required as one of the parameters in the download() function) of a File. In the AWS S3 Document(http://awsdocs.s3.amazonaws.com/mobile/sdkforandroid-dev.pdf), it says mkey is the key of the File. But I am...

How to upload a photo in Meteor to S3 and have it sync to database item?

mongodb,file-upload,meteor,amazon-s3
I am creating a cookbook web application and would like a picture to be uploaded with each new recipe. I am using this package: Amazon S3 File Uploader(https://github.com/Lepozepo/S3). I can successfully do two things upon clicking submit for a new appetizer: Add a new appetizer name and description to the...

How to put object to S3 via CloudFront

java,amazon-s3,amazon-cloudfront
I'd like to upload image to S3 via CloudFront. If you see the document about CloudFront, you can find that cloud front offers put method for uploading to cloudFront There could be someone to ask me why i use the cloud front for uploading to S3 If you search out...

Using s3 in a healthcare application, private links

ruby-on-rails,security,amazon-s3,privacy
We develop a rails-based healthcare application. What is the best way to configure our s3 implementation so that only the authenticated user has access to the image?

Node application and file storage

node.js,express,amazon-s3,passport.js,web-frameworks
Here is my Issue: I have a Node web application that is responsible for user authentication. Once the user has successfully logged in they are presented with a personalised page which contains secure links to files. Here is my file structure. secure/ is where my files are hosted The application...

Conditional Resizing using cfs:graphicsmagick

javascript,node.js,amazon-web-services,meteor,amazon-s3
Currently, I'm storing images (profile images) on Amazon S3, which is working perfectly. I'm resizing images to be 300px wide using cfs:graphicsmagick, but I only want to be doing this if they're wider than 300px. If someone's uploading something smaller, I don't want to be scaling it up, which would...

Infinite loop when streaming a .gz file from S3 using boto

python,amazon-s3,gzip,boto
I'm attempting to stream a .gz file from S3 using boto and iterate over the lines of the unzipped text file. Mysteriously, the loop never terminates; when the entire file has been read, the iteration restarts at the beginning of the file. Let's say I create and upload an input...

Amazon S3 browser direct upload unique file name

angularjs,node.js,amazon-s3,loopbackjs
I'm using Node.js and the Amazon aws sdk for signing an upload request so I can do direct uploads from an Angular application. It works really well except I can't get unique file names. I have tried with creating unique buckets instead but then I can't get the CORS settings...

ArgumentError - unknown SSL method `TLSv1_2'

ssl,amazon-s3,carrierwave,fog
I am trying to move my AWS integration over TLS instead of SSLv3, but I'm receiving an error when trying to set the config.fog_credentials as another SO post has suggested, but I am receiving the ArgumentError above (unknown SSL method 'TLSv1_2'. I am open to a different solution to move...

Route53 to S3 static website redirect from https to http

amazon-web-services,amazon-s3,amazon-route53
I have setup my domain in route53 with 2 different weighted A entries: "live": points to the live application (load balancer with ec2 instances) "maintenance": points to a static S3 website (html page telling site is down) So I have scripts that make the live application go in maintenance mode...

How to route traffic by proximity from Route 53 to closest NGINX server?

amazon-web-services,amazon-ec2,amazon-s3,cloudflare
I'm trying to set up a web server stack in the following way: Use Route 53 for my DNS Serve static content with Cloudflare from S3 buckets Route API calls to nearest NGINX server that sits in front of some Nodejs servers. So all static content is done easily enough...

Update AWSS3 with Cocoapods

ios,amazon-web-services,amazon-s3
In project I have two frameworks: AWSRuntime & AWSS3 and I use it to upload images. - (void)updateAWSCredentials:(NSDictionary *)AWSObject { if(AWSObject && [AWSObject isKindOfClass:[NSDictionary class]]) { self.key = AWSObject[@"AccessKeyId"]; self.secret = AWSObject[@"SecretAccessKey"]; self.token = AWSObject[@"SessionToken"]; self.profileImagePath = AWSObject[@"FilePrefixProfile"]; self.postImagePath = AWSObject[@"FilePrefixPost"]; self.bucket = AWSObject[@"BucketName"]; } } - (NSString...

EC2 can't access S3 file

django,amazon-ec2,amazon-s3,access-denied
I have a very strange error on my website. I keep getting this Django error: Attempted access to 'http:/***.s3.amazonaws.com/media/banderas/Cambodia-flag.png' denied. But when I access that file directly from my admin panel, it works fine. It fails only when the main page wants to access that file. Any ideas what can...

Why does my image url from Amazon S3 have AWSaccesskey and expiration even though I made the bucket public?

ruby-on-rails,amazon-s3,carrierwave
Here is the policy I added: { "Version": "2012-10-17", "Statement": [ { "Sid": "MakeItPublic", "Effect": "Allow", "Principal": "*", "Action": "s3:GetObject", "Resource": "arn:aws:s3:::bucketname/*" } ] } I created an IAM and attached the AmazonS3FullAccess to that IAM. In my rails app, I display the pictures like this: - @pictures.each do |picture|...

An illustration of AWS hosted zones and buckets

redirect,amazon-web-services,amazon-s3,amazon-route53
After more than a few experiments with hosted zones and buckets, it seems I'm getting failures due to unexpired caches even though I may well have set up everything correctly. Before proceeding, I'd like to summarize using two sketches the lengthy (and frankly, tedious) AWS documentation, and ask whether these...

Use TfileUnarchive on Amazon S3

amazon-web-services,amazon-s3,talend
I have a talend job which is simple like below: ts3Connection -> ts3Get -> tfileinputDelimeted -> tmap -> tamazonmysqloutput. Now the scenario here is that some times I get the file in .txt format and sometimes I get it in a zip file. So I want to use tFileUnarchive to...

Call to S3Client::setRegion() fails

php,amazon-web-services,amazon-s3,aws-php-sdk
I'm using "aws/aws-sdk-php": "3.0.3" via composer to access some S3 buckets in different regions, but I can't seem to get S3Client to change regions using the setRegion() function without it generating the error: PHP Catchable fatal error: Argument 2 passed to Aws\AwsClient::getCommand() must be of the type array, string given,...

Failed to Create preset with Amazon Elastic Transcoder

amazon-web-services,amazon-ec2,amazon-s3,amazon,boto
I am using Amazon Elastic transcoder and boto library to create a preset. This code works for me without any problem: preset1=transcode.create_preset('preset', 'preset', 'mp4', {"Codec":"H.264", "CodecOptions":{"Profile":"baseline", "Level":"3", "MaxReferenceFrames":"3"}, "KeyframesMaxDist":"200", "FixedGOP":"false", "BitRate":"600", "FrameRate":"10", "Resolution":"640x480", "AspectRatio":"4:3" }, {"Codec":"AAC","CodecOptions":{"Profile":"AAC-LC"}, "SampleRate":"22050", "BitRate":"32", "Channels":"1" }, {"Format":"png",...

Laravel 5.1 AWS S3 Flysytem: AWS HTTP error: cURL error 6: Couldn't resolve host name

curl,file-upload,amazon-s3,laravel-5,host
I'm on my local environment and are about to enable file uploading to AWS s3, using Laravel 5.1 Flysystem/Filesystem. All setup has been done. I have created a test form, and I'm trying to upload a file. When I push the submit button, I get this absolutely, only too lovable...

Unauthenticated bucket listing possible with boto?

amazon-s3,boto
I am using boto to interact with S3 buckets, and in some cases I just want to be able to list a publicly-readable bucket without passing my credentials. If I pass credentials, boto actually does not let me list the bucket even though it is publicly visible. Is it possible...

AmazonS3 - connecting with Python Boto according specific permissions

python,amazon-s3,boto
I am trying to connect Amazon S3 via Boto 2.38.0 and python 3.4.3. The S3 account is owned by another company and they grants just these permissions : "Statement": [ { "Effect": "Allow", "Action": "s3:ListBucket", "Resource": "arn:axs:s3:::GA-Exports", "Condition":{ "StringLike": { "s3.prefix": "Events_3112/*" } } },{ "Effect": "Allow", "Action": [ "s3:GetObject",...

Updating AWS S3 expiration time

amazon-web-services,amazon-s3
I would like to know how to EXTEND a key's expiration. I am not referring to the signed URL that accesses the object but the key itself. I setup AWS to have a rule that all objects within the bucket should expire in 90 days. This is what I want...

IOS with Rails Backend Amazon S3 direct upload

ios,ruby-on-rails,amazon-web-services,amazon-s3
I am currently trying to use amazon s3 for uploading images from IOS app with a Rails Back-end. I currently don't understand how security works. In their docs http://docs.aws.amazon.com/mobile/sdkforios/developerguide/s3transfermanager.html it's not clear to me how to upload/destroy safely objects in sync with the back-end. For instance, in order to post/destroy,...

Get Request Fulfilled Response from Amazon S3

java,amazon-web-services,amazon-s3
This is the code example of uploading an object using Amazon S3 . public class UploadObjectSingleOperation { private static String bucketName = "*** Provide bucket name ***"; private static String keyName = "*** Provide key ***"; private static String uploadFileName = "*** Provide file name ***"; public static void main(String[]...

Why are the object values getting pushed into the array 3 times?

javascript,jquery,arrays,amazon-s3
I have a simple object array into which I am pushing an object with 2 fields: bucketName and Date. The problem is that the values are getting pushed thrice into the array. Please help me. JS: sortBucket: function(bucketList) { var counter, j = 0; var str = "aws-billing-csv"; console.log("Bucket List...

AWS: How do I support more than 2 policies in IAM?

amazon-web-services,amazon-s3
In my project, I'm using EC2, SQS, EC3. But in IAM, I see that it can only support up to 2 policies. How do I use all three services with the same IAM key and secret? ...

S3 bucket policy, how to ALLOW a IAM group from another account?

amazon-web-services,amazon-s3,policy,bucket
I have one S3 bucket in one AWS account (say arn:aws:s3:::my-test-bucket), that needs to be accessed by a IAM group that is defined in another AWS account (say arn:aws:iam::1111222333444:group/mygroup). The following access policy refuses to save, and tells that arn:aws:s3:::my-test-bucket is an invalid principal. { "Statement": [ { "Action": [...

Predetermining number of partitions of RDD

amazon-s3,apache-spark,hdfs
1)How to Pre-Determine the number of RDD partitions that will be created? 2)what all factors does partitioning of data depend on? Is it only the size of the data and way it is stored(compressed, sequence etc..) For Simplicity, assume i have a 6GB file in HDFS stored as plain...

Upload to S3 return Forbidden

javascript,node.js,amazon-s3,sails.js
I have a form to upload file to my bucket in aws S3 and this is the form <form action="/user/fileUpload" enctype="multipart/form-data" class="ui form" method="post"> <div class="field"> <select class="ui dropdown"> <option value="">Print Agency</option> <option value="fastprint">Fast Print</option> <option value="printpress">Print Press</option> </select> </div> <div class="field"> <input type="file" name="uploadFile" /> </div> <div class="field">...

Add IP Address to Amazon Web Services (AWS) PEM Key

amazon-web-services,amazon-s3,ip,ip-address,pem
I have a website hosted using AWS and I can log in at the office but when I get home, it doesn't let me using the same credentials. How can I add my home's IP address to the account? I'm not sure if I have to whitelist it from somewhere...

Upload file amazon s3 pre signed post

ruby-on-rails,amazon-web-services,amazon-s3
Im developing a web app from which Id like to upload files to Amazon S3, using a pre-signed post. Following the instructions on this link: http://aws.amazon.com/articles/1434 I have the following form. <form action=<%= @aws_s3_url %> method="post" enctype="multipart/form-data"> <input type="hidden" name="key" value= <%= @base_path + "${filename}" %> > <input type="hidden" name="AWSAccessKeyId"...

How to create Datasource through AWS Machine Learning SDK

java,amazon-web-services,amazon-s3,aws-sdk
I am trying to create a Datasource using AWS Machine learning SDK. I have a data set and its corresponding schema.Both are stored in S3.Initially i thought that i just need to provide column names in schema. so initially my schema looked like this - ColName1,ColName2,....,ColNameN Later i realized that...

Delete an S3 Bucket that has some data archived in Glacier

amazon-s3,amazon-glacier
We have a huge bucket for which we have setup lifecycle rules to archive data to Glacier. Now we have decided that we do not need the data in that bucket and hence want to delete all the data stored in Glacier as well as s3. If i delete the...

what is the nodejs package for s3 image upload

node.js,image,amazon-s3
I'm looking to upload my assets to s3. Is there any package in Nodejs like carrierwave in rails I want to resize images with versions. I have come across papercut. What is the best node module for s3 image upload....

Update S3 file remove meta data

amazon-s3
We hosted our script files in S3 with cloudfront. Whenever i update older js file with a new js file, it removes the metadata of a file. I have to reenter them again. Is there any way to fix it?...

Lifecycle policy on S3 bucket

amazon-web-services,amazon-s3
I have an S3 bucket on which I've configured a Lifecycle policy which says to archive all objects in the bucket after 1 day(s) (since I want to keep the files in there temporarily but if there are no issues then it is fine to archive them and not have...

Secure file upload directly to s3 or server to s3 (from iOS app) [closed]

ios,node.js,amazon-web-services,express,amazon-s3
I need to upload sensitive images to s3 from an iOS app. I'm wondering which option is better: Upload to my server first, then upload to s3. Upload to s3 directly, then upload metadata to my server. ...

jets3t cannot upload file to s3

hadoop,amazon-s3,jets3t
I'm trying to upload files from local to s3 using hadoop fs and jets3t, but I'm getting the following error Caused by: java.util.concurrent.ExecutionException: org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException: Request Error. HEAD '/project%2Ftest%2Fsome_event%2Fdt%3D2015-06-17%2FsomeFile' on Host 'host.s3.amazonaws.com' @ 'Thu, 18 Jun 2015 23:33:01 GMT' -- ResponseCode: 404, ResponseStatus: Not Found, RequestId: AVDFJKLDFJ3242, HostId: D+sdfjlakdsadf\asdfkpagjafdjsafdj I'm...

Should I instantiate an object every request or once upon app launch?

ruby,amazon-web-services,amazon-s3,rack,aws-sdk
In my Rack app, I need to use an AWS Cognito Client for signup & login requests. Should I initialize: a separate client (assigned to a local variable inside the call method) for each request? one client (assigned to a global variable outside the call method) for all requests? Concerns:...

Keep getting function error not sure why

android,node.js,amazon-s3,knox-amazon-s3-client
Not sure what I am doing incorrectly here... I keep getting this error when trying to upload a video (multipart) from android to AmazonS3 using knox library on nodejs... I am using the async library as well to wait for the video to finish uploading before i call the callback...

How to find Unused Security Groups of all AWS Security Groups?

python-2.7,amazon-web-services,amazon-ec2,amazon-s3,boto
How to find all the used security groups attached with all the aws resources using Boto? Currently the following script which is giving only ec2 instances- sec_grps = ec2_conn.get_all_security_groups() for group in sec_grps: print group, " Instances attached ", group.instances() Is there any way to get all security groups which...

S3Client PHP SDK: Object of class could not be converted to string

php,amazon-web-services,amazon-s3
I get the following errors when I try to use the AWS PHP SDK: PHP Warning: Illegal string offset 'client.backoff' in C:\xampp\htdocs\aws_test_local\vendor\aws\aws-sdk-php\src\Aws\S3\S3Client.php on line 172 PHP Catchable fatal error: Object of class Guzzle\Plugin\Backoff\BackoffPlugin could not be converted to string in C:\xampp\htdocs\aws_test_local\vendor\aws\aws-sdk-php\src\Aws\S3\S3Client.php on line 172 PHP Warning: Illegal string offset 'signature'...

Why is this python boto S3 multipart upload code not working?

python,amazon-web-services,amazon-s3,multiprocessing,boto
I am trying to upload a 10 GB file to AWS S3, and someone said to use S3 Multipart Upload, so I stumbled upon someone's github gist: import os import sys import glob import subprocess import contextlib import functools import multiprocessing from multiprocessing.pool import IMapIterator from optparse import OptionParser from...

“remoteContext object has no attribute”

amazon-s3,apache-spark,pyspark
I'm running Spark 1.4 in Databrick's Cloud. I loaded a file into my S3 instance and mounted it. Mounting worked. But I'm having trouble creating an RDD: dbutils.fs.mount("s3n://%s:%[email protected]%s" % (ACCESS_KEY, SECRET_KEY, AWS_BUCKET_NAME), "/mnt/%s" % MOUNT_NAME) Any ideas? sc.parallelize([1,2,3]) rdd = sc.textFiles("/mnt/GDELT_2014_EVENTS/GDELT_2014.csv") ...

Image Upload Strategy with Clusters And Amazon S3

php,image,amazon-s3
Trying to sort out a strategy to deal with uploaded images whose endpoint is Amazon S3. The goal is, upon upload, that the image is immediately visible. However, the current way of handling the situation is that the end-user uploads the image and then has to wait for it to...

s3cmd put failing with access denied

amazon-ec2,amazon-s3
I am trying to copy some files from my EC2 instance to S3 and using the following command s3cmd put datafile s3://mybucket/datafile and get the following error ERROR: S3 error: Access Denied I have the following IAM policy { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "ec2:*", "s3:ListAllMyBuckets",...

AWS S3 Upload - Using AccessKey, SecretKey and SessionToken - iOS SDK

ios,amazon-web-services,amazon-s3
I am working on iOS Application, where i need to upload the images in to AWS S3Bucket. I have got the temporary credential(accessKey,secretKey, sessionToken and bucketName) from the server to upload the images. I am finding it difficult to identify the correct API to use this credential to upload the...

how to conditionally change commas into semicolon with spark-scala map / split

regex,scala,amazon-s3,apache-spark
I'm new to spark / scala and have encountered a problem I need to have some data manipulations on a full month of data. to achieve this I defined a class scala> case class zahiro(request_datetime: String, ip: String, host: String, request_uri: String, referer: String, useragent: String, uuid: String, country: String)...

How to configure aws CLI to s3 cp with anonymous user

amazon-web-services,amazon-s3,aws-sdk
I need to download files recursively from a s3 bucket. The s3 bucket lets anonymous access. How to list files and download them without providing AWS Access Key using an anonymous user? My command is: aws s3 cp s3://[email protected]/pavlo/text/tiny/rankings/uservisits uservisit --region us-east --recursive The aws compains that: Unable to locate...

React-native upload image to amazons s3

javascript,file-upload,amazon-s3,fetch,react-native
I want to upload images from my app to S3 server. I have all data and codes calculated (tested using curl on computer) but I can't figure out how to call 'fetch' correctly. I'm getting response: 'At least one of the pre-conditions you specified did not hold. Condition: Bucket POST...

MalformedXML when tagging an S3 bucket

java,amazon-s3
I'm trying to add a new set of tags to an S3 bucket using the AWS Java API, but I'm getting errors like this: com.amazonaws.services.s3.model.AmazonS3Exception: The XML you provided was not well-formed or did not validate against our published schema (Service: Amazon S3; Status Code: 400; Error Code: MalformedXML; Request...

Move files from EC2 to S3 and then delete from EC2

amazon-web-services,amazon-ec2,amazon-s3,aws-php-sdk
I'm migrating files from one remote server to S3. There are about 10k files (all accessible via http URLs from the remote server). The total size is about 300GB (no individual file is more than 1GB). I'm trying to figure out the most efficient way to make this migration. So...

List objects in Google Cloud Storage using the S3 interop API

amazon-s3,google-cloud-storage
We're currently looking into migrating some objects that we store in S3 to Google Cloud Storage. According to their documentation their XML API should be S3 compatible. Pretty much the only thing we have to do is to change the hostname and access keys. However, when I look at the...

Serve private mapping from S3 tiles by proxying data or signing urls through heroku?

heroku,amazon-s3,mapping,leaflet,cesium
I want to store mapping tiles in a private S3 bucket. Each tile has its own URL and each set of tiles could potentially have GBs of tiles. I then want to visualise these tiles through a front end mapping client (e.g leaflet). This client pulls tiles as it needs...

AWS::S3::Errors::NoSuchKey: No Such Key error

ruby-on-rails,ruby,amazon-web-services,amazon-s3
I'm trying to create a method that deletes files on an S3 instance, but I am getting a AWS::S3::Errors::NoSuchKey: No Such Key error when I try to call .head or .read on an object. app/models/file_item.rb def thumbnail { exists: thumbnailable?, small: "http://#{bucket}.s3.amazonaws.com/images/#{id}/small_thumb.png", large: "http://#{bucket}.s3.amazonaws.com/images/#{id}/large_thumb.png" } end lib/adapters/amazons3/accessor.rb module Adapters module...

Get files from amazon s3 buckets sub folder .Net

.net,amazon-web-services,amazon-s3
I am downloading some files from amazon s3 like this private static readonly IAmazonS3 _amazonS3Client = AWSClientFactory.CreateAmazonS3Client( AccessKey, SecretKey, Config ); var request = new ListObjectsRequest {BucketName = BucketName}; ListObjectsResponse response = _amazonS3Client.ListObjects(request); foreach (S3Object o in response.S3Objects) { var objRequest = new GetObjectRequest { BucketName = BucketName, Key =...

Creating amazon aws s3 pre signed url PHP

php,amazon-s3,pre-signed-url
According to this link http://docs.aws.amazon.com/aws-sdk-php/v2/guide/service-s3.html, I can easily create a presigned link just adding the life span to getObjectUrl $signedUrl = $client->getObjectUrl($bucket, 'data.txt', '+10 minutes'); // > https://my-bucket.s3.amazonaws.com/data.txt?AWSAccessKeyId=[...]&Expires=[...]&Signature=[...] But I get a plain url, you know, without the awsaccesskeyid and expires parameters, Here's my code: $bucket = 'imagenesfc'; $keyname =...

Accessing a s3 bucket from Node

node.js,amazon-s3,elastic-beanstalk,passport.js
I have a basic node application where a user logs in and is able to view links to files stored in a directory within the app. The file structure looks something like this. ├── app ├── public ├── node_modules ├── secure (files stored here) ├── .ebextensions ├── .elasticbeanstalk ├── package.json...

How do I lock down outbound firewall access to AWS S3

amazon-web-services,amazon-s3,firewall
I'm running a node app which is using the AWS-SDK module to talk to AWS S3. The infrastructure team want to lock down the firewall but it's not clear what IPs we'll need to lock it down to. Is there a guide anywhere to how to do this or does...

Creating xsd document from file download

python,amazon-s3,xsd,lxml
I am trying to load an xsd document that is stored on s3. It gives me the following err >>> from lxml import etree >>> xsd_url = 'https://s3-us-west-1.amazonaws.com/premiere-avails/movie.xsd.xml' >>> node=etree.fromstring(requests.get(xsd_url).text) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "lxml.etree.pyx", line 3092, in lxml.etree.fromstring (src/lxml/lxml.etree.c:70473) File "parser.pxi",...

AWS S3 object listing

javascript,node.js,amazon-web-services,amazon-s3
I am using aws-sdk using node.js. I want to list images in specified folder e.g. I want to list all files and folder in this location but not folder (images) content. There is list Object function in aws-sdk but it is listing all the nested files also. Here is the...

Is there a way to use http link instead of https in S3 getSignedUrl?

node.js,amazon-s3
I am currently using aws-sdk to download file from S3 using node.js. var s3 = new AWS.S3(); var params = { Bucket: bucket, Key: directory + file, Expires: 300 }; var url = s3.getSignedUrl('getObject', params); res.redirect(url); And it is giving me a link with https. https://mybucket.s3.amazonaws.com/.......qNBK%2BzP48%3D Is there a way...

AWS S3 The security of a signed URL as a hyperlink

security,hyperlink,amazon-s3,download
Is this safe? Maintaining security using a pre-signed url with AWS S3 Bucket object? <a href="https://mywebsite.s3.amazonaws.com/40.pdf?AWSAccessKeyId=[my access key]&Expires=1433297453&Signature=[this random set of numbers]">my link</a> Another words - part 1... say I'm storing a bunch of separate individual's files in a bucket. I want to provide a link to a file for...

Access file on S3 with GoodData File-Download?

amazon-s3,gooddata
Probably a very silly question - but I'm trying to have a GoodData project download a file from S3 for import into GoodData. The file is uploaded to S3, and the console there gives the link in format: https://s3.amazonaws.com/bucketname/subFolder/subfolder/fileName.csv.gz. However when I try and access this, GoodData gives an error...

How to turn an s3 object string into something useful when using laravel 5.1 filesystem

php,amazon-s3,laravel-5,file-conversion,flysystem
I'm at a loss. I'm trying to display an object (image.jpg) I successfully have uploaded to my s3 bucket. I have made sure the file is set to public. I use the Storage::get(); method which the doc says "returns a string of the object". See here: The get method may...

The Mystical Ephemeral File System of Heroku is Not Letting Me Get Files from S3

ruby-on-rails,ruby,amazon-web-services,heroku,amazon-s3
So, I've been wrestling with this issue for days...I need to get a file from S3 and write it to a directory in my Rails app on Heroku. I must have a misunderstanding of the ephemeral file system on Heroku because I can't figure out why it's not working. I...

S3 allow only encrypted policy doesn't work

amazon-s3
I'm using the example provided by Amazon to disallow uploads to S3 that don't have encryption turned on, like so: { "Version": "2012-10-17", "Id": "PutObjPolicy", "Statement": [ { "Sid": "DenyUnEncryptedObjectUploads", "Effect": "Deny", "Principal": "*", "Action": "s3:PutObject", "Resource": "arn:aws:s3:::mybucket/*", "Condition": { "StringNotEquals": { "s3:x-amz-server-side-encryption": "AES256" } } } ] } When...

Amazon CloudFront: How to get monthly cost breakdown per distribution?

amazon-web-services,amazon-s3,amazon-cloudfront
I'd like to manage the hosting for multiple clients, using my own Amazon CloudFront account. Each client has its own CloudFront distribution. But the AWS monthly bill doesn't break down by distribution. How can I find out what the cost was for each distribution? Ideally, the solution will also work...

How to download a json file from S3 via d3.json

javascript,json,d3.js,amazon-s3
I have setup a S3 bucket and upload there some json files. Each file has Content-type: application/json. The files accessible via URL, if I type it in the browser or use wget, I can download it without any problem. But if I try to download this file via d3.json or...

Uploading image from canvas to S3 in Meteor

canvas,file-upload,meteor,amazon-s3
I have no experiene with file uploads in any programming language, so this is a bit of hassle for me. I have a canvas that needs to be saved in a bucket in Amazon S3 (or any other good service if it's simpler). I know about toDataURL() and that it...

Which is a better way: retrieve images from AWS S3 or download it and store locally in a temp folder to be displayed?

objective-c,core-data,amazon-web-services,amazon-s3,awss3transfermanager
Problem: Retrieve image from S3 and load into UIButton. I'm currently doing my research on this issue and can't seem to make up my mind. Which is a better way to do it in terms of performance and security issue? Also, do I need to do caching or store these...

Naked Root Domain Hosting

redirect,amazon-web-services,amazon-s3,amazon-route53
The usual approach calls for pointing mysite.com to www.mysite.com. But suppose that I would like not to redirect. In other words, I would like to set up a naked root domain, without setting up any sub-domains, not even www.mysite.com. I am using Route53 and S3. Presumably I would need just...

Django ImageField url slow when using Amazon s3

python,django,heroku,amazon-s3,boto
In my Django Application currently running on Heroku I noticed that retrieving url's from ImageFields takes forever when hosting on s3. The application is written using Django Rest Framework. When I try to retrieve a list of objects that have image fields the response is so slow Heroku throws a...

How to transfer files from iPhone to EC2 instance or EBS?

ios,iphone,amazon-ec2,amazon-s3,amazon-ebs
I am trying to create an iOS app, which will transfer the files from an iPhone to a server, process them there, and return the result to the app instantly. I have noticed that AWS offers an SDK to transfer files from iOS app to S3, but not to EC2...

Throttling S3 commands with aws cli

amazon-web-services,amazon-s3,aws-cli
I'm running a backup script using AWS CLI to perform an S3 sync command every night on my MediaTemple server. This has run without fail for months, but I updated my Plesk installation and now every night, when the backup script runs, MediaTemple disables my server due to excessive usage....

Copy error Amazon Redshift loading from S3

amazon-web-services,amazon-s3,amazon-redshift
Very new to AWS and RedShift, so please go slow. I'm trying to figure out how to load data from a csv file I uploaded to S3 into a Redshift cluster. I managed to create the required table already, but when I try to run the Copy command to load...

Amazon Redshift: Copying Data Between Databases

postgresql,amazon-web-services,amazon-s3,amazon-redshift,amazon-data-pipeline
I am looking to Copy data within databases on Amazon Redshift. Before this, I was copying data from a Redshift database to a PostgreSQL hosted on an EC2 instance for analytical purpose. I had ruby script that would do it using dblink EXTENSION. But now since the data is un-managable...

AmazonS3Exception: x-amz-website-redirect-location header is not supported for this operation

android,amazon-web-services,amazon-s3
I have got an interesting exception when trying to upload data from Android client to AWS: com.amazonaws.services.s3.model.AmazonS3Exception: x-amz-website-redirect-location header is not supported for this operation. (Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument; Request ID: E3900749ACF1D979), S3 Extended Request ID: kFjMM7JVFSOxvaKlHgM0bVM5zKZAR/0K8qeMyt44vjvtMFcGk8CxY9gDBDs0sqWmr8r2jcCyENo= The user located in China region and data...

Can't access s3 buckets after creating instance using IAM profile

ruby,amazon-web-services,amazon-s3,boto,iam
I am trying to access some s3 buckets from an instance created applying the IAM profile that allows full access to those s3 buckets. I am able to carry out the required operations using aws cli. However, my application is written in Ruby and makes use of fog gem. Using...

Set S3 Buckets defaults before upload

amazon-web-services,amazon-s3,cache-control
I want to set S3 bucket defaults, particularly the cache control. What I need is to set defaults in order to set cache control for all object that will be uploaded to this bucket. I need a bucket configuration, not a file configuration. I have seen http://www.bucketexplorer.com/ but this is...

How can I compress / gzip my mimified .js and .css files before publishing to AWS S3?

css,powershell,amazon-web-services,amazon-s3,gzip
I ran Google pagespeed and it suggests compressing my .js and .css Eliminate render-blocking JavaScript and CSS in above-the-fold content Show how to fix Enable compression Compressing resources with gzip or deflate can reduce the number of bytes sent over the network. Enable compression for the following resources to reduce...

Understanding server architecture: Delivering content from AWS S3 using Nginx reverse-proxy or Apache server

apache,amazon-web-services,nginx,amazon-s3
The purpose of this question is to understand the strategy while designing server side architecture. Use case: I want to build a http server for an app which allow users to upload and download multimedia content (images, videos etc.) Large number of concurrent users (say, around 50k) are expected to...

Distributing installer of Windows desktop application

windows,deployment,amazon-s3,installer,hosting
I have an installer for my Windows app and it is quite big (>100 Mb). I am also using ClickOnce deployment framework, so each time I issue an update all my users have to download the installer. We tried to use Amazon S3 to store the setup file, but it...

AWS S3 utilizing for website static images

php,amazon-web-services,amazon-ec2,amazon-s3
We currently are moving our website to EC2 utilizing OpsWorks. Right now we reference all media files for the site with relative URL's in the coding... "/images/image1.jpg" and so on. Is there a way to alias the /images directory in our coding to utilize an S3 bucket? What we'd like...

Django and S3 - static URL won't change

python,django,amazon-s3,boto,django-storage
I managed to set my S3 as a subdomain static.domain.com (using CNAME and bucket name the same as subdomain).Now I'd like to amend my django settings to use this URL. I use django-storages and can't make it work. I set STATIC_URL to http://static.domain.com. When I use {{ STATIC_URL }} in...

Amazon S3 Download: Direct iOS or Web Service Node Js.?

ios,mysql,node.js,amazon-s3
I'm creating an ios app that connects to a backend Node js web service. The applications allows users to upload pics which is handled via AFnetworking and my node js backend up to mysql(filepath) and amazon S3(actual image). However, for downloading the pics into a uitableview, i'm want to know...

Regular expression to match any hyphenated word in a url in PHP

php,regex,amazon-s3
I'm trying to replace a URL with CloudFront URL. For example. the URL could be https://s3.amazonaws.com/bucket/folder/d05c229a73b5c6f169d599652015b1.png https://s3.amazonaws.com/bucket-with-hyphen/folder/folder1/d05c229a73b5c6f169d599652015b1.png https://s3.amazonaws.com/bucket-with-hyphen/folder-with-hyphen/folder1/d05c229a73b5c6f169d599652015b1.png And then I'll use: preg_replace($pattern, $replacement, $string) My $replacement would be $1cloudfront.net/$4'; Here's my $pattern so far...

Is it impossible to use AWS CloudFront for downloading my private image on S3?

ios,amazon-web-services,amazon-s3,amazon-cloudfront,aws-sdk
I'm making a global iOS App with AWS SDK. And there is a function for users to get image files; only if the requester is a friend of the image files' uploader. For this function, I use AWS S3 for storage service with private access. And I want to use...

jQuery file upload to S3 (and rails) with CORS headers

jquery,ruby-on-rails,amazon-s3,jquery-file-upload
Im trying to upload a file directly to S3, and displaying a progress bar while it uploads. But when I submit the form, I get the following error message: OPTIONS https://s3-eu-west-1.amazonaws.com/my-bucket 403 (Forbidden) 9:1 XMLHttpRequest cannot load https://s3-eu-west-1.amazonaws.com/my-bucket. No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'https://localhost:3000' is...

How to limit access in Amazon S3 files to specific people?

ruby-on-rails-4,amazon-s3
I work on a SaaS application where Creators can create Groups and invite others to their Group to share files, chat and so on. Only people within specific group should have access to this group's files. People from other group must not have access to not their group's files. And...

Unable to access files from public s3 bucket with boto

python,amazon-web-services,amazon-s3,boto
I recently created a new AWS account (let's call it "Account A") and created an S3 bucket in this account (let's call it "bucketa"), uploading a file foo.txt. Following advice from the internet, I set up what I believe to be the most permissive bucket policy possible (it should allow...