FAQ Database Discussion Community


Elastic(search): Query result date format differs from stored format

date,elasticsearch,format,elastic
got a problem with the elastic date format conversion when I parse the results from a query. So i have a default mapping on a date field as following: "timestamp" : { "type" : "date", "format" : "dateOptionalTime" } and it is stored as "timestamp":"2015-05-06T08:52:56.387Z" if I execute a max...

Predefined Term Mapping

mongodb,elasticsearch,full-text-search,elastic
I have not yet used Elasticsearch so please excuse the bad description. I would like to know if it is possible to configure Elasticsearch to do the following - I had some issues in MongoDB with this as the full text search functionalities seem to be a little limiting. Here's...

ElasticSearch REST - insert JSON string without using class

elasticsearch,elastic,elasticsearch-net
I am looking for an example where we can push below sample JSON string to ElasticSearch without using classes in REST api. { "UserID":1, "Username": "Test", "EmailID": "[email protected]" } We get the input as xml and we convert it to JSON string using NewtonSoft.JSON dll. I know REST api is...

ElasticSearch Separate Instances Running on Same Computer

elasticsearch,elastic
I have an elasticsearch instance running on the default port (9200) that is used to power the backend of my application. I tried to run another instance (separate directory) on 9201 which had the same index but different data and it seemed to overwrite my first instance. A little more...

get buckets count in elasticsearch aggregations

elasticsearch,elastic
I am using elasticsearch to search a database with a lot of duplicates. I am using field colapse and it works, however it returns the amount of hits (including duplicates) and not the amount of buckets. "aggs": { "uniques": { "terms": { "field": "guid" }, "aggs": { "jobs": { "top_hits":...

How to use Elasticsearch bulk index with single JSON file in Python

python,json,elasticsearch,elasticsearch-plugin,elastic
I have a single JSON file formatted like: {"index":{"_type": "summary", "_id": "0001-1200-2015-12-31-0007", "_index": "transaction"}} {"data_load_user_id": "nathan", "tender_time": 5, "ring_time": 8, "operator_id": "0111"} for about 12,000 rows. As far as I know, this is the correct way to format a JSON file for the bulk API in elasticsearch. I'm still confused...

Elasticsearch Date Range Aggregation with Sum

elasticsearch,kibana,elastic
I've followed Elastic's docs and successfully queried my index to return data for two date ranges. What I can't get right, is adding a sum of units to each range. I've managed to add a unit sum (see query below) but this only returns a sum for the entire range....

Elastic query DSL: Wildcards in terms filter?

elasticsearch,querydsl,elastic
I am trying to filter the documents using terms filter. I am not sure how to introduce wildcards in filter. I tried something like this: "filter":{ "bool":{ "must":{ "terms":{ "wildcard" : { "aircraft":[ "a380*" ] } } } } } But I get SearchParseException with this. Is there no way...

Two JSON douments linked by a key

python,json,elasticsearch,elastic
I have a python server listening to POST from an external server.I expect two JSON documents for every incident happening on the external server. One of the fields in the JSON documents is a unique_key which can be used to identify that these two documents belong together. Upon recieving the...

Elasticsearch - script_fields error

elasticsearch,elastic
I am using elastic 1.5.0. When I run this block in the query: "script_fields" : { "test1" : { "script" : "doc['actual_air_time'].value * 2", "lang":"groovy" } } I get an error: nested: ScriptException[dynamic scripting for [groovy] disabled] How can I enable groovy script for elastic? Is there any other way...

Elastic Search Sum aggregation with group by and where condition

elasticsearch,elastic
I am newbie in ElasticSearch. We are currently moving our code from relational DB to ElasticSearch. So we are converting our queries in ElasticSearch query format. I am looking for ElasticSearch equivalent of below query - SELECT Color, SUM(ListPrice), SUM(StandardCost) FROM Production.Product WHERE Color IS NOT NULL AND ListPrice !=...

Elasticsearch Completion Suggester field contains comma separated values

search,elasticsearch,lucene.net,nest,elastic
I have a field that contains comma separated values which I want to perform suggestion on. { "description" : "Breakfast,Sandwich,Maker" } Is it possible to get only applicable token while performing suggest as you type?? For ex: When I say break, how can I get only Breakfast and not get...

elasticsearch can't install head plugin

java,elasticsearch,elasticsearch-plugin,elastic,elasticsearch-head
I am planing to install head pluging for elasticsearch. in both these two official documentation, http://mobz.github.io/elasticsearch-head/ and http://docs.couchbase.com/admin/elastic/install-plugin.html they said to use this: bin/plugin -install mobz/elasticsearch-head I did, but i got the following error: PS C:\elasticsearch-1.3.9\elasticsearch-1.3.9> bin/plugin -install mobz/elasticsearch-head Exception in thread "main" org.elasticsearch.common.settings.SettingsException: Failed to load settings from [file:/C:/...

How to use ElasticSearch MinHash plugin in NEST

nest,elastic
I want to use MinHash elastic search plugin in NEST Elasticsearch.net. How can I use minhash plugin in nest?

Elasticsearch DSL: Aggregation

elasticsearch,elastic
Shown below is the structure of type (kind of a table) of my data. Aircraft | Duration A320 | 0.95 A320 | 0.55 A321 | 16.50 A321 | 3.9 In this data, I want to perform a ceil() on duration, followed by a groupBy operation to get following output: Aircraft...

Elastic DSL: Aggregate and order by keys as numeric

elasticsearch,querydsl,elastic
In my dataset, where duration field is string, I am trying to find aggregation count on duration in the way below: "aggs": { "duration": { "terms": { "script": "Math.ceil(doc[\"duration\"].value as double)", "order" : { "_term" : "asc"} } } } This returns result as : "buckets" : [ { "key"...

Getting Parse error for elasticsearch-py

python,elasticsearch,elastic,elasticsearch-py
I am trying to search my entire elasticsearch data for a certain word "tsbu" within a time range. When I try running this, I get a SearchParseException and Parse Failure. es = Elasticsearch() doc = { "query": { "match" : { "message" : "tsbu" } }, "range" : { "@timestamp"...

Elasticsearch: hyphen in PrefixQuery on Keyword-analyzed field

elasticsearch,lucene,elastic
I have a situation where I'm putting metadata for invoices into an Elasticsearch 1.5.2 index, running on Ubuntu Linux 15.04 with Oracle JDK 8u45. One of the fields is poNumber, which often has values that look like "123-R45678" or "123-4Q5678". I'm trying to use a PrefixQuery (via the query parser)...

How to use special document fields in scripts in elastic?

elasticsearch,elastic
I'm trying to write query with custom script in elasticsearch: https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-script-filter.html#query-dsl-script-filter https://www.elastic.co/guide/en/elasticsearch/reference/master/modules-scripting.html. This is useful when you need to compare two document fields. Everything worked fine, until I decide to use special document field (ex: _id, _uid, etc). The query always returns empty results and there is no errors if...

Elasticsearch DSL : Multiple aggregations

elasticsearch,querydsl,elastic
Shown below is the structure of type of my data. ArrivalAirport | DepartureAirport | ArrivalDelay | DepartureDelay A | B | 2 | 5 B | C | 3 | 6 C | A | 4 | 7 A | B | 5 | 8 I want to perform together...

ElasticSearch not showing correct results for exactmatch

elasticsearch,elastic
I am trying to search for an exact match string but ES does not give me expected results. Below is the example - Here is index creation - POST /spacetesting1 { "settings" : { "number_of_shards" : 1 }, "mappings" : { "couchbaseDocument" : { "_source" : { "enabled" : false...

Active Directory AuthorizationException in elastic search

elasticsearch,nest,elasticsearch-plugin,elastic,elasticsearch-shield
I am trying to use AD authentication. I am able to login successfully but i am not authorized to perform any query in marvel. Everytime i execute query in marvel, i get the following error.Below are the details { "error": "AuthorizationException[action [indices:data/read/search] is unauthorized for user [shivang.Mittal]]", "status": 403 }...

In elastic search is a collection queryable while its being added to the index

elasticsearch,nest,elastic
I'm using Nest to insert a list of 60k+ objects into elastic search, specifically calling client.IndexMany(list, indexName). As the list is inserted, is it query-able? or is it only query-able after the complete list is indexed? If its the former, is there a way to force it to only be...

Email issues with Wordpress running on AWS Elastic Beanstalk

wordpress,email,amazon-web-services,beanstalk,elastic
I'm running Wordpress 4.2 on AWS Elastic Beanstalk. I'm using a plugin that sends emails using the WP email API, which uses the server's (i.e. Elastic Beanstalk's) internal mail system. The problem: emails that the app attempts to send to my university email (let's call it .uni.edu) fail, and they...

How to create an alias on two indexes with logstash?

elasticsearch,alias,logstash,logstash-grok,elastic
In the cluster that I am working on there are two main indexes, let's say indexA and indexB but these two indexes are indexed each day so normaly I have indexA-{+YYYY.MM.dd} and indexB-{+YYYY.MM.dd}. What I want is to have one alias that gathers indexA-{+YYYY.MM.dd} and indexB-{+YYYY.MM.dd} together and named alias-{+YYYY.MM.dd}....

How to add 2 values in elasticsearch script?

elasticsearch,elastic
I am trying to create a rows_processed field by adding 2 fields src_s_rows and tgt_s_rows, but some how it is not working, it always gives me 0. Even when I give "script": "(doc['src_s_rows'].value)" instead of "script": "(doc['src_s_rows'].value+doc['tgt_s_rows'].value)" it still gives me 0. What is it that I am missing, please...

Elastic Search. Net With Nest: Term Filter For Elastic Property

filter,elasticsearch,nest,elastic
I am attempting to perform a query with filters. I can get it to filter on some properties but not the one I need. Here is my model: public class IndexItem { public DateTime CreatedDate { get; set; } [ElasticProperty(Index = FieldIndexOption.Analyzed)] public String Name { get; set; } [ElasticProperty(Index...

Get elasticsearch result based on two keys

elasticsearch,elastic
I want to get all docs who's "PayerAccountId" should equal to "123" and "UsageStartDate" should be in range [2015-05-01 TO 2015-05-10] I am expecting something to run like this, curl -X GET http://192.168.1.3:9200/_all/_search -d '{"query" : {"match" : { "PayerAccountId:\"156023466485\" AND UsageStartDate:[2015-01-01 TO 2015-01-10]" }}}' Obviously it's not working any...

Elasticsearch Facet List doesn't Match Results

ruby-on-rails,ruby,search,elasticsearch,elastic
Problem When I filter by a particular facet, that specific field's facets are correctly filtered in the result but the other facet fields remain the same. Best way to explain this is with the query and the response. Query { query: { match_all: {} }, filter: { and: [{ term:...

Getting the result of a SearchResponse in ElasticSearch

java,mongodb,elasticsearch,elastic
I am trying to use ES as the index for my MongoDB. I've managed to integrate them successfully, but I find the search API rather complex and confusing. The Java API is not too helpful either. I am able to find exact matches, but how can I get this result?...

My ES custom analyser is not used?

elasticsearch,mapping,elastic
I'm using Elasticsearch and create an index with the following information for mapping and settings. The problem I have is that my field geography.locality which should use the 'name_analyser' doesn't seem to use it. { "index": "programs", "body": { "settings": { "number_of_shards": 5, "analysis": { "filter": { "elision": { "type":...

Bulk insert documents to ElasticSearch without updating when document exists

elasticsearch,nest,elastic
Is it possible to bulk insert data to ES without updating document content if exists by Id. Only not existing documents should be inserted, without any updates.

Elasticsearch doesn't return results for a specific term search

elasticsearch,elastic
I am attempting to do a query where I filter on term for a specific term. This is the query I am attempting to run: { "query": { "filtered": { "filter": { "term": { "tags": "sports" } } } }, "sort": { "timestamp": "desc" } } When I run the...

Aggregations Size makes diffrent results

elasticsearch,aggregation,elastic
Hi I have simples aggregation like "aggs": { "firm_aggregation": { "terms": { "field": "experience.company_name.slug", "size": 10 } } } and this gives me result like "aggregations": { "firm_aggregation": { "buckets": [ ... (some others) { "key": "freelancer", "doc_count": 33 }, but when I increase aggregation size to 2000 i get...

ElasticSearch Date Field Mapping Malformation

elasticsearch,elastic
In my ElasticHQ mapping: @timestamp date yyyy-MM-dd HH:mm:ssZZZ ... date date yyyy-MM-dd HH:mm:ssZZZ In the above I have two types of date field each with a mapping to the same format. In the data: "@timestamp": "2014-05-21 23:22:47UTC" .... "date": "2014-05-22 05:08:09-0400", As above, the date format does not map to...

Elasticsearch postfiler cancel filter

java,filter,elasticsearch,lucene,elastic
In the following query I want to filter the query results to size medium and color blue but I want aggregations to ignore that the color blue is applied. { "query":{ "bool" { "must": { "query_string": { "query": "foo" } }, "should": { // deferred } } }, "filter": {...

And + Or query using elastic search REST API

elasticsearch,elastic
How do we combine AND/OR Query in elastic search. For Eg, Below are the documents we have {"title": "MI2", "brand": "Xiomi","operatingSystem": "Android"}, {"title": "Nexus","brand": "Google","operatingSystem": "Android"}, {"title": "Samsung Note","brand": "Samsung","operatingSystem": "Android"}, {"title": "Samsung Galaxy","brand": "Samsung","operatingSystem": "Android"} Now How do we query Android phones from either Samsung or Xiomi. ( operatingSystem...

Elastic Search JDBC River Plugin SQL Server Integrated Security

jdbc,elasticsearch,elasticsearch-plugin,elasticsearch-jdbc-river,elastic
So I've been working on implementing elastic search using the JDBC River plugin to get data from our SQL Server DB into elastic search. I've got it working fine using the SQL Server credentials, but trying to use integrated security doesn't work. It will create the index, but it doesn't...

TransportError(503, u'') when tying to use a recently created elasticsearch index

python,elasticsearch,elastic
I'm creating an Elasticseach index using Python API like this: from elasticsearch import Elasticsearch es = Elasticsearch() index_body = {"mappings": {".percolator": {"properties": {"message": {"type": "string", "analyzer": "english"}}}}} # Creates the index if it doesn't exist if not es.indices.exists('test'): es.indices.create(index='test', body=index_body) print es.exists(index='test', id='1') The index is created successfully, but when...

Using an aggregation on data with forward slash in elasticsearch

elasticsearch,kibana-4,elastic
I have data, that has an attribute like this apiUrl:/REST/endpoint/123 Now I would like to show all the urls and I am trying to use an aggregate function (apiUrl.raw is not_analyzed part of the multifield): POST /index/type/_search { "aggregations": { "application": { "terms": { "field": "apiUrl.raw" } } } }...