http,elasticsearch,docker , Docker container http requests limit


Docker container http requests limit

Question:

Tag: http,elasticsearch,docker

I'm new to Docker so, most likely, I'm missing something.

I'm running a container with Elasticsearch, using this image.

I'm able to setup everyhing correctly. After that I was a using a script developed by a collegue in order to insert some data, basically querying a MySQL database and making HTTP requests .

Problem is, many of those requests get stuck until it fails. If I do netstat -tn | grep 9200 I get:

tcp6       0      0 ::1:58436               ::1:9200                TIME_WAIT  
tcp6       0      0 ::1:59274               ::1:9200                TIME_WAIT 

...
tcp6       0      0 ::1:58436               ::1:9200                TIME_WAIT  
tcp6       0      0 ::1:59274               ::1:9200                TIME_WAIT 

with a lot of requests. At this point I'm not sure if it's something related to elastic search or docker. This does not happen if Elasticsearch is instaleld on my machine.

Some info:

$ docker version
Client version: 1.6.2
Client API version: 1.18
Go version (client): go1.4.2
Git commit (client): 7c8fca2
OS/Arch (client): linux/amd64
Server version: 1.6.2
Server API version: 1.18
Go version (server): go1.4.2
Git commit (server): 7c8fca2
OS/Arch (server): linux/amd64

$ docker info
Containers: 6
Images: 103
Storage Driver: devicemapper
 Pool Name: docker-252:1-9188072-pool
 Pool Blocksize: 65.54 kB
 Backing Filesystem: extfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 4.255 GB
 Data Space Total: 107.4 GB
 Data Space Available: 103.1 GB
 Metadata Space Used: 6.758 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.141 GB
 Udev Sync Supported: false
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.82-git (2013-10-04)
Execution Driver: native-0.2
Kernel Version: 3.14.22-031422-generic
Operating System: Ubuntu 14.04.2 LTS
CPUs: 4
Total Memory: 15.37 GiB

$ docker logs elasticsearch
[2015-06-15 09:10:33,761][INFO ][node                     ] [Energizer] version[1.6.0], pid[1], build[cdd3ac4/2015-06-09T13:36:34Z]
[2015-06-15 09:10:33,762][INFO ][node                     ] [Energizer] initializing ...
[2015-06-15 09:10:33,766][INFO ][plugins                  ] [Energizer] loaded [], sites []
[2015-06-15 09:10:33,792][INFO ][env                      ] [Energizer] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/mapper/ubuntu--vg-root)]], net usable_space [145.3gb], net total_space [204.3gb], types [ext4]
[2015-06-15 09:10:35,516][INFO ][node                     ] [Energizer] initialized
[2015-06-15 09:10:35,516][INFO ][node                     ] [Energizer] starting ...
[2015-06-15 09:10:35,642][INFO ][transport                ] [Energizer] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/172.17.0.5:9300]}
[2015-06-15 09:10:35,657][INFO ][discovery                ] [Energizer] elasticsearch/Y1zfiri4QO21zRhcI-bTXA
[2015-06-15 09:10:39,426][INFO ][cluster.service          ] [Energizer] new_master [Energizer][Y1zfiri4QO21zRhcI-bTXA][76dea3e6d424][inet[/172.17.0.5:9300]], reason: zen-disco-join (elected_as_master)
[2015-06-15 09:10:39,446][INFO ][http                     ] [Energizer] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/172.17.0.5:9200]}
[2015-06-15 09:10:39,446][INFO ][node                     ] [Energizer] started
[2015-06-15 09:10:39,479][INFO ][gateway                  ] [Energizer] recovered [0] indices into cluster_state

The important part of the script:

for package in c.fetchall():
    id_package, tracking_number, order_number, payment_info, shipment_provider_name, package_status_name=package
    el['tracking_number'] = tracking_number
    el['order_number'] = order_number
    el['payment_info'] = payment_info
    el['shipment_provider_name'] = shipment_provider_name
    el['package_status_name'] = package_status_name

    requests.put("http://localhost:9200/packages/package/%s/_create"%(id_package), json=el)

Answer:

So, it wasn't a problem with either Docker or Elastic. Just to recap, the same script throwning PUT requests at a Elasticsearch setup locally worked, but when throwning at a container with Elasticsearch failed after a few thousand documents (20k). To note that the overal number of documents was roughtly 800k.

So, what happend? When you setup somethig running on localhost and make a request to it (in this case a PUT request) that request goes through the loopback interface. In pratice ths means that no TCP connection gets created making a lot faster.

When the docker container was setup, ports were bound to the host. Although the script still makes requests to localhost on the desired port, a TCP connection gets created between the host and the docker container through the docker0 interface. This comes at the expense of 2 things:

This is actually a more realistic scenario. We setup Elasticsearch on another machine and did the exact same test and got, as expected, the same result.

The problem was that we were sending to requests and for each of them creating a new connection. Due to the way TCP works, connections cannot be closed immediately. Which meant that we were using all available connections until we got none to use because the rate of creation was higher the actual close rate.

Three suggestions to fix this:

  1. Pause requests every once in a while. Maybe put a sleep at every X requests making possible for the TIME_WAIT to pass and the connection closing
  2. Send the the Connection: close header: option for the sender to signal that the connection will be closed after completion of the response.
  3. Reuse connection(s).

I ended up going with option 3) and rewrote my collegue's script and reusing the same TCP connection.


Related:


Bad scoring due to different maxDocs of IDF


elasticsearch
I have two documents with a field title of: News New Website If I search for the term new website the score for the News document is much higher than the other one which is obviously not what I want. I wrapped an explain around it and got: 'hits': [{'_explanation':...

elasticsearch aggregation group by null key


elasticsearch
here is the data in my elasticsearch server: {"system": "aaa"}, {"system": "bbb"}, {"system": null} I want to get the statistics for system. then I did the query: { "aggs" : { "myAggrs" : { "terms" : { "field" : "system" } } } it gives me the result: { "key":...

Elasticsearch and C# - query to find exact matches over strings


c#,.net,database,elasticsearch,nest
I need a way to search documents using a plain exact match over two or multiple fields which are of type "string" and "integer". I'd like to avoid standard query as I don't care about scoring or best match, just a yes/no outcome if both the fields match or not....

Operator '??' cannot be applied to operands of type IQueryContainer and lambda expression


c#,elasticsearch,nest
I am trying to create a method to process a certain query. I follow an example posted on the Nest repository (line 60), but still the MatchAll is not recognized by the compiler and if I try to build the solution, the error that shows is: Operator '??' cannot be...

How to write search queries in kibana using Query DSL for Elasticsearch aggregation


elasticsearch,querydsl,kibana-4
I am working on ELK stack to process Apache access logs. Spent a lot of time understanding Query DSL format so that more complex queries can be written. Currently am facing issues with running the queries in kibana interface but the same queries work just fine when posted using curl...

AngularJS $http request using forEach inside of Controller


angularjs,http,foreach
http://codepen.io/rcidaleassumpo/pen/eNEjGY?editors=001 That's the link for the code. Below I first used the forEach to combine the nicknames with the base link, so I could get the fullLink to make the call for with the $http. But then when I tried to make the call, and post into a empty array,...

Javascript: Altering an object where dot notation is used [duplicate]


javascript,jquery,elasticsearch
This question already has an answer here: How to access object properties containing special characters? 1 answer I'm building an Elasticsearch search interface. My method is to build the initial query object, and then alter it depending on the user input. In the filter part of my object, I...

Elasticsearch NumberFormatException when running two consecutive java tests


java,date,elasticsearch,numberformatexception,spring-data-elasticsearch
I have two test in a class, each of them containing the following query: SearchQuery searchQuery = new NativeSearchQueryBuilder().withQuery(matchAllQuery()).withFilter(rangeFilter("publishDate").lt(date)).build(); In one of the tests, the number of the results elasticsearchTemplate.count(searchQuery, Article.class), in the other one the returned values are verified elasticsearchTemplate.queryForPage(searchQuery,Article.class) If I run any of these two tests separately,...

Why I got so many connection events on Node's HTTP server?


javascript,node.js,http,websocket,socket.io
I have a web application using socket.io. It runs on localhost:8000 on my local machine. In order to find an annoying resource leak in my application, I tried to log out the connection events on my HTTP server: server.on('connection', function(conn) { //log out connection details }); When I open the...

How to have multiple regex based on or condition in elasticsearch?


elasticsearch
I want to get all 000ANT and 0BBNTA from id, is there something similar to terms which works with regexp or is there any other way? Otherwise I will have to query elasticsearch for each item say 000ANT and 0BBNTA. Please help. Below is something that I am trying out...

NEST ElasticSearch.NET Escape Special Characters


c#,elasticsearch,nest
I have been experimenting with the use of the NEST client for Elastic Search, but seem to have hit a barrier when filtering on a term which contains special/reserved characters such as '/' Below is a JSON representation of my model.. "categories": { "count": 1, "default": "root/Hello/World/Category", } When submitting...

ElasticSearch asynchronous post


database,post,asynchronous,elasticsearch,get
I'm posting data on my ElasticSearch database. I've noticed that data is not immediately available, it requires some milliseconds to show up in a GET request. I can live with that (after all, the calls are asynchronous so this behavior is expected) but in my test code I need to...

How to create the get method for the call


php,arrays,json,http,quickblox
I am studying and implementing an api and while following the examples, Here is the example 1 : curl -X POST \ -H "Content-Type: application/json" \ -H "QuickBlox-REST-API-Version: 0.1.0" \ -d '{"application_id": "2", "auth_key": "DtF9cZPqTF8Wy9Q", "timestamp": "1333630580", "nonce": "1340569516", "signature": "13293a5bd2026b957ebbb36c89d9649aae9e5503", "user": {"login": "injoit", "password": "injoit"}}' \ https://api.quickblox.com/session.json For the...

nodejs head request isn't triggering events


node.js,http
Here's my code: var http = require('http'); var req = http.request( { host: 'example.com', method: 'HEAD', path: '/' }, function(res){ res.on('end',function(){ console.log('Ended'); }); res.on('finish',function(){ console.log('Finished'); }); res.on('close',function(){ console.log('Closed'); }); } ); req.end(); Strange thing about it is that it doesn't print anything. Takes longer than expected and exits. Is this...

Get elasticsearch result based on two keys


elasticsearch,elastic
I want to get all docs who's "PayerAccountId" should equal to "123" and "UsageStartDate" should be in range [2015-05-01 TO 2015-05-10] I am expecting something to run like this, curl -X GET http://192.168.1.3:9200/_all/_search -d '{"query" : {"match" : { "PayerAccountId:\"156023466485\" AND UsageStartDate:[2015-01-01 TO 2015-01-10]" }}}' Obviously it's not working any...

Elasticsearch geospatial search, problems with index setup


elasticsearch,geospatial
I'm trying to search for documents previously added to an index, which has been configured to allow geospatial queries (or so I think). My elasticsearch instance is hosted on qbox.io. This is the code I wrote to create an index from the command line curl -XPOST username:[email protected]/events -d '{ "settings"...

Elasticsearch boost per field with function score


elasticsearch,lucene,solr-boost
I have a query with different query data for different fields and ORed results. I also want to favor hits with certain fields. Ideally this would only increase ranking but would not cause results that did not contain some of the terms in the other fields. This would skew results...

In simple RESTful design, does PATCH imply mapping to CRUD's (ORM's) “update” and PUT to “destroy”+“create” (to replace a resource)?


database,rest,http,orm,crud
I'm trying to create a simple REST API and map it to CRUD. I have an ORM (DataMapper) which has methods like create, update and destroy. If I get it right, given a resource {a:'foo',b:'bar',c:'baz'}, performing a PUT {b:'qux'} is supposed to replace the resource and result in the same...

Does Ruby support nonblocking HTTP requests without a third-party library?


ruby,http,asynchronous,gem,eventmachine
I am writing an API to access resources on one of my servers. Part of this API will make HTTP requests. In the name of good software design, I don't want my API to be blocking. I am fairly new to Ruby, but in Java I would provide an Asynchronous...

Re-index object with new fields


elasticsearch,nest
It seems like as long as the id field is maintained, its super easy to re-index a document by simply calling Index(), but is there a way to given an object was updated and new fields were added, to have it include these new fields in the index? I'm not...

MultiMatch query with Nest and Field Suffix


c#,elasticsearch,nest
Using Elasticsearch I have a field with a suffix - string field with a .english suffix with an english analyser on it as shown in the following mapping ... "valueString": { "type": "string", "fields": { "english": { "type": "string", "analyzer": "english" } } } ... The following query snippet won't...

Can I submit a form with google's recaptcha in it from my app?


http,request,authorization,captcha,recaptcha
I'm writing an app which involves letting users to share comments on a website, which has a comment form with Google's reCAPTCHA embeded. I would like to load this page via HTTP and display CAPTCHA within my app, so that user can post comments from my app. Is it easy...

How to return value in Meteor.JS from HTTP.call “GET”


javascript,ajax,http,meteor,facebook-javascript-sdk
I'm writing a meteor method, which should return a Facebook response for HTTP.call on graph api, but HTTP.call has only a callback function to show error/response, so I can't take this data outside, and Method can not return any value. Here's my method code: loadUserFBEvents: function () { var accessToken...

ElasticSearch REST - insert JSON string without using class


elasticsearch,elastic,elasticsearch-net
I am looking for an example where we can push below sample JSON string to ElasticSearch without using classes in REST api. { "UserID":1, "Username": "Test", "EmailID": "[email protected]" } We get the input as xml and we convert it to JSON string using NewtonSoft.JSON dll. I know REST api is...

Which Http error/status code is the appropriate?


javascript,http
I've carefully read all HTTP error/status codes, still not clear which is the appropriate to return in the following scenario: The request is an Ajax request, so the handling of the error situation depends on the client javascript code, preferably it will give a notification. At server side an unexpected...

How to get duplicate field values in elastic search by field name without knowing its value


elasticsearch
I have a field "EmployeeName" in an elastic search index - and I would like to execute a query that will return me all the cases where there are duplicate values of "EmployeeName". Can this be done? I found more_like_this but this requires field value for "like_text". But my requirement...

ASP.net MVC HTTP Request error with payload


asp.net,json,asp.net-mvc,http
I am creating an ASP.net MVC website with a RESTful API to a SQL database. I have implemented a controller which holds the HTTP commands. One of the commands is a POSTcommand: // POST: api/PoolTests [ResponseType(typeof(PoolTest))] public IHttpActionResult PostPoolTest(PoolTest poolTest) { if (!ModelState.IsValid) { return BadRequest(ModelState); } db.PoolTests.Add(poolTest); db.SaveChanges(); return...

Elasticsearch - Query document missing an array value


elasticsearch
I would like to query my elasticsearch index in order to retrieve the documents that don't contain a specific value in an array. For instance, if my query is : { "query": { "bool": { "must": [ { "match_all": {} } ], "must_not": [], "should": [] } }, "from": 0,...

Get document on some condition in elastic search java API


java,elasticsearch,elasticsearch-plugin
As I know we can parse document in elastic search, And when we search for a keyword, It will return the document using this code of java API:- org.elasticsearch.action.search.SearchResponse searchHits = node.client() .prepareSearch() .setIndices("indices") .setQuery(qb) .setFrom(0).setSize(1000) .addHighlightedField("file.filename") .addHighlightedField("content") .addHighlightedField("meta.title") .setHighlighterPreTags("<span class='badge badge-info'>") .setHighlighterPostTags("</span>") .addFields("*", "_source")...

Which LiftResponse class is appropriate for returning an XML based file format?


xml,http,lift
I have a Lift application which generates a Gephi file on the fly. The Gehpi native format is XML based. The current plan is that the file gets downloaded and opened in Gephi on the client, without being processed/displayed in the browser. I looked through the LiftResponse subclasses and it...

Query returns both documents instead of just one


c#,.net,elasticsearch,nest
var res = esclient.Search<MyClass>(q => q .Query(fq => fq .Filtered(fqq => fqq .Query(qq => qq.MatchAll()) .Filter(ff => ff .Bool(b => b .Must(m1 => m1.Term("macaddress", "mac")) .Must(m2 => m2.Term("another_field", 123)) ) ) ) ) ); As far as I can understand the bool and must together are the equivalent of the...

$http headers is not a function - angularjs


php,angularjs,http
I am posting data to Dynamics CRM via SOAP on my PHP server with cURL. After this is done it is giving the entity GUID in the form of a HTTP Response header. When attempting to access this via my angular factory and $http. My header is exposed and is...

Elasticsearch standard analyser stopwords


elasticsearch
I am trying to guess what is the default stopwords list in standard analyzer in elasticsearch. I run version 1.3.1, and it seems to me that the English list is used, because running a wildcard query like this { "wildcard" : { "name" : { "wildcard" : "*in*" } }...

Elasticsearch: How to query using partial phrases in quotation marks


elasticsearch
I am trying to implement a search behavior that supports partial phrases. A possible search input could look like this: example "hello world" elasticsearch Now I want to get all documents, that contain the words example and elasticsearch as well as the phrase hello world. As this is a very...

Custom HTTP Header or cookies? how custom authentication/authorization helps in CSRF?


javascript,http,cors,csrf
If someone can help me understand how a custom HTTP authorization header helps protect CSRF attack. Also correct me if i'm wrong does it prevent replay attacks using fiddler also? Thanks for your help in advance...

Parsing Google Custom Search API for Elasticsearch Documents


json,python-2.7,elasticsearch,google-search-api
After retrieving results from the Google Custom Search API and writing it to JSON, I want to parse that JSON to make valid Elasticsearch documents. You can configure a parent - child relationship for nested results. However, this relationship seems to not be inferred by the data structure itself. I've...

Server initiated requests


http,go,http2
I know that http is a request-response protocol, the problem in short is a client makes a request to the server to start a long running process, and i want to inform the client over the progress with a simple json message containg progress info. in http1 i know that...

How to read data in logs using logstash?


elasticsearch,logstash
I have just started log stash, i have log files in that log file whole object is printed in the logs, Since my object is huge i cant write the grok patterns to the whole object and also i expecting only two values out of those object. Can you please...

ElasticSearch - Configuration to Analyse a document on Indexing


elasticsearch
In a single request, I want to retrieve documents from a SOR, store them in ElasticSearch, and then search those documents using the ES search API. There seems to be some lag from the time the document is indexed and the time it is analyzed and ready to be searched....

Docker container http requests limit


http,elasticsearch,docker
I'm new to Docker so, most likely, I'm missing something. I'm running a container with Elasticsearch, using this image. I'm able to setup everyhing correctly. After that I was a using a script developed by a collegue in order to insert some data, basically querying a MySQL database and making...

elastic search sort in aggs by column


sorting,elasticsearch,group-by,order
I am trying to sort in elastic search in aggs, equivalent in mysql "ORDER BY Title ASC/DESC". Here is the index structure: 'body' => array( 'mappings' => array( 'test_type' => array( '_source' => array( 'enabled' => true ), 'properties' => array( 'ProductId' => array( 'type' => 'integer', 'index' => 'not_analyzed'...

ElasticSearch- “No query registered for…”


search,indexing,elasticsearch
ElasticSearch returns me "No query registered for [likes_count]" error when trying to look up entries using the following query. The field likes_count is a new field of documents and does not exist in every document. The same query works without the sort part. Why does this error appear? Thanks {...

How to compute the scores based on field data in elasticsearch


elasticsearch
I have the following fields in documents { name: "Pearl", age : 43, weight: 54, bodyWeight : 103, height : 1.8 } Now i want to get scores for the documents based on the bodyWeight to height ratio of the documents. How to do that?...

ASP.net HTTP Post when auto-incrementing row Id


sql,asp.net,asp.net-mvc,http
I am working in ASP.net with a Web API allowing HTTP Posts to update a mapped SQL database. I want the Id to be automatically increment by one every time a new row is inserted. The table definition is this: CREATE TABLE [dbo].[PoolTests] ( [Id] INT IDENTITY (1, 1), [SiteID]...

Not able to access Kibana running in a Docker container on port 5601


elasticsearch,docker,dockerfile,kibana-4
I have built a docker image with the following Docker file. # gunicorn-flask FROM devdb/kibana MAINTAINER John Doe <[email protected]> ENV DEBIAN_FRONTEND noninteractive RUN apt-get update RUN apt-get install -y python python-pip python-virtualenv gunicorn # Setup flask application RUN mkdir -p /deploy/app COPY gunicorn_config.py /deploy/gunicorn_config.py COPY app /deploy/app RUN pip install...

Python exception for HTTP response codes


python,http,exception
I'd like to raise a Python-standard exception when an HTTP response code from querying an API is not 200, but what specific exception should I use? For now I raise an OSError: if response.status_code != 200: raise OSError("Response " + str(response.status_code) + ": " + response.content) I'm aware of the...

NEST - Using GET instead of POST/PUT for searching


c#,elasticsearch,nest
Is there a way to tell NEST to use GET instead of POST when performing searches? Similar to how the ElasticSearch documentation shows CURL using GET I'd like to use GET when using NEST instead of using POST as it currently does.

Integrating PHP Curl


php,mysql,xml,http,php-curl
I have the code below which calls up an MySQLi and presents it in XML form in my browser. The next stage is that instead of presenting it in my browser I want to send it to another IP address using PHP Curl. Please can someone help me with the...

How to use arrays in lambda expressions?


c#,elasticsearch,nest
I am writing a program with NEST library of ElasticSearch. I want to write a lambda expression for a function with this argument: HighlighDescriptor<parentdocument> HighlighDescriptor.onFields (param Action<HighlightFieldDescriptor<ParentDocument>>[] fieldHighlighters) I don't know what is the array in the function argument?...