apache,amazon-ec2,bots,cloudflare,ddos , How to handle bots attack on Amazon EC2


How to handle bots attack on Amazon EC2

Question:

Tag: apache,amazon-ec2,bots,cloudflare,ddos

I'm experiencing the following problem: we host an ecommerce on an Amazon EC2 medium instance with an RDS db instance, that normally run great, but because we work with product timed "releases" some users are using bots to automatic add items to basket and checkout them, causing the website to run really slow and then crash, basically like a ddos attack. Initially there were few users doing this and so I found their ips in the access log and blocked them. Now the word is spreading and I can't add ips manually to a blacklist but I need a "professional" way of doing this. Some friend suggested me to use Cloudflare but I'm asking if there is a way to do this internally in AWS or with Apache directly. Thank in advance


Answer:

maestroosram,

This sounds like a problem you can't solve with the usual anti-scraping methods, such as blacklisting and rate limit.

Why not:

Black-listing : Depending on what kind of IP they are (hosting, open proxy) you could use one of the blacklist you can easily find on internet (these are pretty good https://www.iblocklist.com/).
BUT, Once they get a block, will switch to other solutions, until they found an IP address (or more thousands) that is not black-listed.

Rate-Limit: You could also try to block IP addresses that performs more than x requests per hour or minute. But since they are distributed over a large number of IP addresses in order to avoid detection, limiting them is not very helpful.

What you can do is implement a good Captcha system, and see what happens. This can stop these scrapers, but please consider that there are plenty of Captcha solver out there.(http://www.scrapesentry.com/scraping-wiki/common-methods-tools-break-captcha/)

Also, another way is by blocking ip addresses that share the same session_id. This is very risky, since there are ISP that balance the traffic via several gateways.


Related:


How to route traffic by proximity from Route 53 to closest NGINX server?


amazon-web-services,amazon-ec2,amazon-s3,cloudflare
I'm trying to set up a web server stack in the following way: Use Route 53 for my DNS Serve static content with Cloudflare from S3 buckets Route API calls to nearest NGINX server that sits in front of some Nodejs servers. So all static content is done easily enough...

Xampp Starting properly but in browser localhost is not working


php,apache,xampp
I am using windows 7 and XAMPP Starting properly but in browser(ALL borwsers) localhost is not working incase of that some folders are displaying with the title searchbase. Please refer the screenshot Help me to debug this problem, I checked recently installed programs also browsers windows settings every thing looks...

Codeigniter URL routing issues in .htaccess


php,apache,.htaccess,codeigniter-2
Before you mark it as duplicate FIY I have tried all solutions I could find on SO. The url is www.deltadigital.ca config file (if I use $config['base_url'] = 'http://www.deltadigital.ca' - it doesnt work at all) //$config['base_url'] = 'http://www.deltadigital.ca'; $root=(isset($_SERVER['HTTPS']) ? "https://" : "http://").$_SERVER['HTTP_HOST']; $root.= str_replace(basename($_SERVER['SCRIPT_NAME']), '', $_SERVER['SCRIPT_NAME']); $config['base_url'] = $root;...

How to rewrite example.com/catelogue.php?page=3&cat1=fruit&cat2=apple with .htaccess


php,apache,.htaccess,mod-rewrite,url-rewriting
Can anyone help me with .htaccess rewrite url, I am new to it and stuck with this below problem i have this urls http://example.com/catelogue.php?page=3 http://example.com/catelogue.php?page=3&cat1=fruit&cat2=apple and i want this to be http://example.com/catelogue http://example.com/catelogue/fruit/apple Below is my .htaccess file, but here the problem is I have achieved this (example.com/catelogue/fruit/apple) but I...

How to transfer files from iPhone to EC2 instance or EBS?


ios,iphone,amazon-ec2,amazon-s3,amazon-ebs
I am trying to create an iOS app, which will transfer the files from an iPhone to a server, process them there, and return the result to the app instantly. I have noticed that AWS offers an SDK to transfer files from iOS app to S3, but not to EC2...

Django file upload took forever with Phusion Passenger


django,apache,file-upload,passenger
I have a Django app deployed using Passenger (I did not choose mod_wsgi because mod_passenger is already there and being used). When I uploaded an MP3 file (900 kB), Google Chrome displays upload % which reached 100% pretty fast but then it took forever for the resulting page. The database...

Why same .htaccess commands working differently in localhost and server?


php,apache,.htaccess,url-rewriting
I have the following .htaccess command for redirection of php pages RewriteRule ^(.*)$ index.php/$1 [L] This code is working in localhost.It is not working in web server. But when code is slightly modified it works in server. RewriteRule ^(.*)$ index.php?/$1 [L] Why is this happening?...

How to enable mod_rewrite on Apache 2.4?


php,wordpress,apache,mod-rewrite,centos7
I'm using Wordpress and wanted to enable pretty urls feature, but they don't work, i'm thinking that it's because of mod_rewrite, but i don't know how to enable it on Apache 2.4 under CentOS 7... I've already try this: grep -i LoadModule /etc/httpd/conf/httpd.conf | grep rewrite but nothing... Also would...

Trouble Understanding URL Rewrite to Proxy to Another Server


regex,apache,rewrite
I am sorry if this is really simple and I am not paying attention. I need to redirect a url query (https://1.2.3.4:19999/abc/start?userid={digit string}) to be processed by another server (https://5.6.7.8/operate?service=checkuser&userid={digit string}). So far I've failed to do this correctly (this is my latest attempt): RewriteCond %{REQUEST_URI} ^/abc/ [NC] RewriteCond %{QUERY_STRING}...

Best way to block countries [on hold]


php,apache,.htaccess
I was requested to prevent access to a website. There are two ways I know I can choose - 1. At apache level (a tool like this: http://www.ip2location.com/blockvisitorsbycountry.aspx) which generates a long htaccess file with list of IP ranges. The other way I know of is using PHP (be it...

URL Rewriting in php CodeIgniter using .htaccess


php,apache,.htaccess,codeigniter,mod-rewrite
I am working on a project in which url Rewriting was done. Here is the link of the old uri http://www.mysite.pk/jobs/search/faisalabad/all-cats But now in the new url I have changed it to http://www.mysite.pk/jobs/pakistan/faisalabad/all-cats what I want to do is to change the redirect all the url's whcih have http://www.mysite.pk/jobs/search/ search...

Rewrite non-dynamic pages using htaccess


apache,.htaccess,mod-rewrite
I've been banging my head since many hours on this and I am unable to come up with a solution. What I need to get done. domain.com/page.php => domain.com/page/ domain.com/category1-name1.php => domain.com/category1/name1/ domain.com/category2-name1.php => domain.com/category2/name1/ Details: I have couple of files with the following names & I want them to...

htaccess errors after upgrading Apache from 2.2 to 2.4


apache,.htaccess
I use 3 hosting for my websites, all of them have the same rules in htaccess. One of the hosting was upgraded from Apache 2.2 to 2.4 and now I am getting errors spam and I don't know how to fix it without lossing the rule. [Mon Jun 15 19:38:33.189697...

Amazon EC2 Storage lacks


amazon-web-services,amazon-ec2
I have launched Amazon EC2 instance of "m3.large" type. According to this page, m3.large should have 2vCPUs, 7.5GiB Memory and 1x32GB SSD Storage. But df -ah returns following results. It seems that the instance lacks the volume. Filesystem Size Used Avail Use% Mounted on /dev/xvda1 7.9G 797M 6.7G 11% /...

access tomcat deployed application (aws) by domain name (www.mydomain.com)


tomcat,amazon-ec2,dns,ip
I am new to aws hosting, please help me out in pointing the domain name to tomcat hosted application. During configuration following steps are taken- 1- I have created ec2 ubuntu instance. Configured with java, tomcat. Deployed my war files on tomcat. I am able to access the application successfully....

Virtual host returns the default host on another pc


apache
Hello I am trying to set up an apache server on a PC.(PC A) I created the folder /var/www/test. In this folder I have my index.html. I created a conf file test.conf in the /etc/apache2/sites-available folder with the following entries: <VirtualHost www.test.tld:80> ServerName www.test.tld DocumentRoot /var/www/test ErrorLog ${APACHE_LOG_DIR}/error.log CustomLog ${APACHE_LOG_DIR}/access.log...

PHP script doesn't execute on Apache server


php,apache
I have a very simple PHP script on a remote Apache web server. On my PC, it functions perfectly fine with php web server running but nothing appears to be executing in the web server and there appears to be no errors. I have a good suspicion that this is...

.htaccess | Too many redirects


apache,.htaccess,mod-rewrite,redirect
I want to add a IP filter into my .htaccess file, but after I applied it, a too many redirects error will show. Content of .htaccess: RewriteEngine On RewriteCond %{HTTPS} off RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI} RewriteCond %{HTTPS} on RewriteCond %{HTTP_HOST} ^[^.]+\.[^.]+$ RewriteRule ^(.*)$ https://www.%{HTTP_HOST}/$1 [L,R=301] RewriteCond %{HTTPS} on RewriteCond %{REMOTE_ADDR} xxx.xxx.xxx.xxx...

Apache httpd connection refused during stress test using jmeter


apache,jmeter,connection-refused
Now, I do stress test using jemter in windows(jmeter.bat). Number of Threads = 100 Ramp-up period = 0 Loop count = 1000 But raise connection refused error in jmeter, after 20~30 seconds. I can not find connection refused error in /usr/local/apache/logs/error.log or /usr/local/apache/logs/access.log. Test is HTTP GET, index.html("It's works") Apache...

htaccess redirect to subdomain based on query string


php,apache,.htaccess,redirect,joomla
I searched for similar questions but none of them was applicable to my situation. What I need to do is redirect from old url's that have specific query string: option=com_virtuemart to a subdomain : http://parts.domain.com Tried bunch of rules and conds but haven't achieved right redirection. It was always redirecting...

Unable to increase PHP Memory Limit


php,apache,memory,memory-management,out-of-memory
I am running into an issue with a project I am working with on my localhost, where I am unable to increase the PHP memory_limit setting. I've tried increasing it directly in the php.ini config: memory_limit = 1024M I've tried increasing it in the projects .htaccess file: php_value memory_limit 1024M...

mod_rewrite - force redirecting to rewritten URL


apache,.htaccess,mod-rewrite,redirect
I have following URL [1]: www.domain.com/?search=somequery which i want to redirect to [2] www.domain.com/search/somequery I am using following code in my .htaccess: Options +FollowSymLinks -MultiViews RewriteEngine On RewriteBase / RewriteCond %{REQUEST_FILENAME} !-d RewriteCond %{REQUEST_FILENAME} !-f RewriteRule ^search/(.*)$ /?search=$1 [L] Code above works kind of fine since server understands both URLs:...

deploy django under apache mod_wsgi waiting for localhost


python,django,apache,mod-wsgi
I am deploying a Django project on Apache. and after configuration, I open the "localhost" in the browser, and nothing showed up and the status bar just keep saying "Waiting for localhost". Here is some info. Environment: OS: ubuntu Python: 2.7.3 Django: 1.8.2 Apache: 2.2.21 Django project: /var/www/ocr_service Apache virtualhost:...

.htaccess Mod_Rewrite inconsistent


php,apache,.htaccess,mod-rewrite
Here is my .htaccess file: RewriteEngine On #Working rules: RewriteRule ^privacy$ privacy.php [L] RewriteRule ^terms$ terms.php [L] RewriteRule ^team$ team.php [L] RewriteRule ^candidates$ candidates.php [L] RewriteRule ^clients$ clients.php [L] #Not working rules: RewriteRule ^article/([^/]*).*$ article.php?id=$1 [L] RewriteRule ^job/([^/]*).*$ job.php?id=$1 [L] RewriteRule ^contact/([^/]*).*$ contact.php?location=$1 [L] The top rules are all working...

Exporting Data from Cassandra to CSV file


apache,csv,cassandra,export,export-to-csv
Table Name : Product uid | productcount | term | timestamp 304ad5ac-4b6d-4025-b4ea-8b7991a3fe72 | 26 | dress | 1433110980000 6097e226-35b5-4f71-b158-a1fe39a430c1 | 0 | #751104 | 1433861040000 Command : COPY product (uid, productcount, term, timestamp) TO 'temp.csv'; Error: Improper COPY command. Am I missing something? ...

Apache - finding configuration file path


windows,apache,server
I have to migrate another apache instance from 2.2 to 2.4 but I have one problem. In Apache monitor I see multiple services running but I don't really know where apache .conf files are stored. In previous migrations from 2.2 to 2.4 I had them somewhere within conf folder. Apache...

Allow space and other special characters in htaccess url rewrite


php,regex,apache,.htaccess,mod-rewrite
Hi everyone i have this URL: orderprod_cntr/147/ok2%20-%20val1%20val2%20val3%20val4 When ever i open this URL i get 404 error. The htaccess rule that i am using at present is: RewriteRule orderprod_cntr/([-\w]+)/([-\w]+)/?$ controller/orderprod_cntr.php?id1=$1&id2=$2 [L] The error is occurring due to %20 in the URL, without it works fine. Please tell me the correct...

Htaccess rewrite URL with virtual directory and 2 variables


regex,apache,.htaccess,url,rewriting
its my URL... result.php?team=arsenal&player=ospina I want to like this mysite.com/virtualdirectory/arsenal/ospina.html I tried this code.. cant work.. Not Found RewriteRule ^(.*)/(.*)/(.*).html$ result.php?team=$2&player=$3 [L] The requested URL /subfolder/arsenal/ospina.html was not found on this server. Apache/2.2.8 (Win32) PHP/5.2.6 Server at localhost Port 80 Thanks for helps, Best regards!!...

Loop through list of AWS-instances shows only first item


ruby-on-rails,amazon-web-services,amazon-ec2,each,aws-sdk
I am working on a simple customer frontend for AWS. I want a list of all the users machines for start/stopping the EC2s. While the logic works I can only show the first of the machines in my view. I guess it's related to the AWS APIs pageable response format,...

Apache Regex How To Get This Value


php,regex,apache,.htaccess
I have a URI like this: http://example.com/demo/makemoney?matchtype=chancee&network=agagag&ifmobile=1&placement=nyc&adposition=111&copy=25&pid=123&zid=231&mid=234&cid=342&param1=zaleemm&param2=vatansvr I want to get makemoney and after "?" character values. In my .htaccess file write this rule: RewriteRule ^demo/(.*)?(.*)$ demo.php?utm_campaign=$1&$2 [L,NC] But only utm_campaign I get in $_GET super globals. What's true regex?...

Using Java web service on Amazon cloud


java,web-services,amazon-web-services,amazon-ec2
I want to make a web service in java which will take arguments and do processing and return a json response. I am not been able to figure out how to deploy this service on amazon ie (on ec2 or some where else) . what will be the url to...

Redirection loop (URL rewriting)


regex,apache,.htaccess,mod-rewrite,rewriting
I'm trying to rewrite my website URLs. I have: http:// website.com/v2/message.php?ID2=123 I want: http:// website.com/v2/message-123.php I tried something in my htaccess but I've a redirection loop :'( Here is my .htaccess: RewriteCond %{QUERY_STRING} ^ID2=([0-9]+)$ RewriteRule message.php? http://website.com/v2/message-%1.php? [L,R=301] Can someone help me with this?...

How to set default virtual server (sites-available) in Apache?


apache
I have 2 sites at my linode VPS. I configured ourdomain.com and mydomain.com in sites-available directory (/etc/apache2/sites-available/ourdomain.com.conf // mydomain.com.conf) and activated and restarted Apache. There is a way to specify ourdomain.com as the main/default site? Because when I access through the ip, Apache shows mydomain.com :(...

Make dynamic links (with ?) return error 404 not found


apache,.htaccess,dynamic,hyperlink,http-status-code-404
Using .htaccess (are there any other solutions?) is there a way to make all dynamic links (links containing the sign ?) of a site return a 404 not found response header? For example: http://www.example.com/?bla_bla - will return 404 http://www.example.com/test/index.html?no_redirect=true - will return 404...

Pattern matching in htaccess rewrite condition


apache,.htaccess,mod-rewrite,https
I have a rewrite condition in an .htaccess file which is used to 'not' force specific URLs to use a HTTPS connection but it's matching erroneously on other URLs at present, here's the rewrite rule... RewriteCond %{HTTPS} off RewriteCond !/go/ [NC] RewriteCond %{HTTP_HOST} ^app\.ihasco\.co\.uk [NC] RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI} [L,R=301] And...

.htacces rewrite by cookie value


apache,.htaccess,mod-rewrite,cookies,url-rewriting
I have a website in 2 languages. I'm inserting the text for each language through php constants and I define the user language through a cookie (lang=es or lang=en). Now the base url in each language is www.webname.com/ but i want to rewrite it depending on the language. If the...

How to find Unused Security Groups of all AWS Security Groups?


python-2.7,amazon-web-services,amazon-ec2,amazon-s3,boto
How to find all the used security groups attached with all the aws resources using Boto? Currently the following script which is giving only ec2 instances- sec_grps = ec2_conn.get_all_security_groups() for group in sec_grps: print group, " Instances attached ", group.instances() Is there any way to get all security groups which...

Multiple subdomains to same address


apache,subdomain,vhosts
I want to redirect multiple subdomain to same address. Let's say I want www.sitename.com and sitename.com to redirect same address. Do I have to say; <VirtualHost www.sitename.com:81> . . . </VirtualHost> <VirtualHost sitename.com:81> . . . </VirtualHost> or is there a simple solution....

Remove plus sign from url using .htaccess


php,regex,apache,.htaccess,mod-rewrite
My current url is: http://localhost/mywbsite/search/one+piece/149 I want to change link to like this http://localhost/mywbsite/search/one-piece/149 I have already .htaccess file with this code Options -MultiViews RewriteEngine On RewriteBase /indianrealitybytes/ RewriteCond %{THE_REQUEST} /search_advance\.php\?keywords=([^&]+)&f=([^\s&]+) [NC] RewriteRule ^ search/%1/%2? [R=301,L] RewriteRule ^search/([^/]+)/([^/]+)/?$ search_advance.php?keywords=$1&f=$2 [QSA,L,NC] I don't know where to change to remove plus(+) sign...

What are correct permissions for Linux Apache2 PHP 5.3 log file?


php,linux,apache,logging,permissions
I discovered the reason why I was not getting entries into my php_errors.log file related to permissions. Right now, I have set it to 666 (rw-rw-rw-) but surely this is a security weakness? Thus, my question. php.ini file: error_log /var/log/httpd/php_errors.log log_errors On # ls -ld /var/log /var/log/httpd /var/log/httpd/php_errors.log drwxr-xr-x 6...

What exactly IS Kafka Rebalancing?


apache,apache-kafka,kafka-consumer-api,kafka
I am a new user to Kafka and have been playing around with it for about 2-3 weeks now. I believe at the moment I have a good understand of how Kafka works for the most part, but after attempting to play around with and design my own Kafka consumer...

.htaccess allow image to display directly


php,regex,apache,.htaccess,redirect
I have many files including images, php& js scripts and subfolders. I need to redirect to index.php for any uri using .htaccess. for example: www.example.com/test should be redirect to index.php But images should not be redirected to index.php. Instead it should be printed directly in browser. For example: www.example.com/sample.jpg should...

mod-rewrite to remove a folder


apache,.htaccess,mod-rewrite
I have the following URL http://www.example.com/folder1/folder2/file.php I want to redirect it to http://www.example.com/demo/folder1/file.php I tried this, but it does not work: RewriteCond %{HTTP_HOST} www.example.com RewriteCond %{REQUEST_URI} ^folder RewriteRule /demo/folder(.*) /folder1/folder2/$1 ...

Htaccess regex: exclude REQUEST_URI


regex,apache,.htaccess,mod-rewrite,redirect
I really tried everything, the tool htaccess tester says that it's ok, but it's not really working. I want to redirect a subdomain to the main domain only if the REQUEST_URI doesn't start with 3 specific strings. I tried without success with: RewriteCond %{HTTP_HOST} ^sub.example.com$ [NC] RewriteCond %{REQUEST_URI} !(aaa|bbb|ccc) RewriteCond...

Disable direct access to application via Tomcat


apache,tomcat
I have a small local (internal to organisation) browser application running on Tomcat. I have inherited an application (also for internal organisation use only) running under PHP server which I have been asked to get running on the same box. I decided the best thing was to get both Tomcat...

ssl certificate with and without www


apache,ssl
I have a website that installed a ssl certificate for the name of www.example.com. It works fine for https://www.example.com. But it doesn't work for https://example.com. The browser gave me Error code: ssl_error_bad_cert_domain. I am using Apache 2. I tried to rewrite the url to add www in httpd-ssl.conf, see the...

guidance needed on using apache commons math


java,eclipse,apache
I am trying to utilize the Apache Commons Math (3.5) library in Eclipse IDE. I have downloaded the JAR file and have added the library to my project correctly. However my problem lies when I actually try using classes from the Apache Commons library. I try to create a new...

Rewrite url not working in htaccess


php,apache,.htaccess,mod-rewrite,url-rewriting
I have website. I just want to rewrite url using .htaccess Here is the code which I want to rewrite: RewriteEngine On RewriteBase / RewriteCond %{QUERY_STRING} /search_data.php\?keywords=([^&]+)&f=([^\s&]+) [NC] RewriteRule ^/search_data.php/?$ /search/%1/%2? [R=301,L,NC] this the current url http://localhost/mywbsite/search_data.php?keywords=one+piece&f=149 I want to convert this to this http://localhost/mywbsite/search/one-piece/149 I tried above code but...

redirect certain pages to its equivalent in another domain, otherwise redirect to its homepage


apache,.htaccess,rewrite,url-redirection
I want to use htaccess to redirect certain pages to its equivalent in another domain, other pages redirect to homepage. Like that: http://old-domain.com/certain-page redirected to: http://new-domain.com/certain-page http://old-domain.com/another-certain-page redirected to: http://new-domain.com/another-certain-page Only these pages will be redirected, otherwise, pages have to be redirected to the new domain home page. http://old-domain.com/non-certain-page redirected...

How to create a private certificate for connecting to a website


apache,ssl,privatekey,digital-certificate,self-signed
My apologies if this is a duplicate, I may just not be using the correct terminology in my queries to find what I am looking for. I have a vendor that sent me a certificate to install in my browser so that we can access their website. We cannot get...