FAQ Database Discussion Community


Elasticsearch boost per field with function score

elasticsearch,lucene,solr-boost
I have a query with different query data for different fields and ORed results. I also want to favor hits with certain fields. Ideally this would only increase ranking but would not cause results that did not contain some of the terms in the other fields. This would skew results...

How to check the tokens generated for different tokenizers in Elasticsearch

elasticsearch
I have been using different type of tokenizers for test and demonstration purposes. I need to check how a particular text field is tokenized using different tokenizers and also see the tokens generated. How can I achieve that?...

Elasticsearch aggregations over regex matching in a list

regex,elasticsearch
My documents in elasticsearch are of the form { ... dimensions : list[string] ... } I'd like to find all dimensions over all the documents that match a regex. I feel like an aggregation would probably do the trick, but I'm having trouble formulating it. For example, suppose I have...

How to compute the scores based on field data in elasticsearch

elasticsearch
I have the following fields in documents { name: "Pearl", age : 43, weight: 54, bodyWeight : 103, height : 1.8 } Now i want to get scores for the documents based on the bodyWeight to height ratio of the documents. How to do that?...

Elasticsearch - Query document missing an array value

elasticsearch
I would like to query my elasticsearch index in order to retrieve the documents that don't contain a specific value in an array. For instance, if my query is : { "query": { "bool": { "must": [ { "match_all": {} } ], "must_not": [], "should": [] } }, "from": 0,...

ElasticSearch - doc_value in murmur3 hash

elasticsearch
Continuing a previous question. What am I trying to do? Thanks to @AndreiStefan I'm trying to put a murmur3 hash off the heap, using doc_values: "dynamic_templates": [ { "murmur3_hashed": { "mapping": { "index": "not_analyzed", "norms": { "enabled": false }, "fielddata": { "format": "doc_values" }, "doc_values": true, "type": "string", "fields": {...

Elastic Search analyzing fields and Kibana Graphs Legends

elasticsearch,kibana-4
I'm using elasticsearch 1.5.2 and Kibana 4.0.3. I'm having a problem with the analyzers, using the default one the legends in the kibana graphs split the field by &, / and -. Which is the default behavior. I've tried setting the fields to not_analyzed and then the kibana legends display...

Date type property in fields of query_string giving error always

date,elasticsearch
I have a type in a index which have a property of date type. I need to search by this date in my query_string. So here is the mapping of that type. ....... ....... "created": { "type": "date", "format": "yyyy-MM-dd'"'T'"'HH:mm:ss.SSSZ" }, ....... ....... And the query is "query_string": { "fields":...

ElasticSearch REST - insert JSON string without using class

elasticsearch,elastic,elasticsearch-net
I am looking for an example where we can push below sample JSON string to ElasticSearch without using classes in REST api. { "UserID":1, "Username": "Test", "EmailID": "[email protected]" } We get the input as xml and we convert it to JSON string using NewtonSoft.JSON dll. I know REST api is...

Bad scoring due to different maxDocs of IDF

elasticsearch
I have two documents with a field title of: News New Website If I search for the term new website the score for the News document is much higher than the other one which is obviously not what I want. I wrapped an explain around it and got: 'hits': [{'_explanation':...

Re-index object with new fields

elasticsearch,nest
It seems like as long as the id field is maintained, its super easy to re-index a document by simply calling Index(), but is there a way to given an object was updated and new fields were added, to have it include these new fields in the index? I'm not...

ElasticSearch asynchronous post

database,post,asynchronous,elasticsearch,get
I'm posting data on my ElasticSearch database. I've noticed that data is not immediately available, it requires some milliseconds to show up in a GET request. I can live with that (after all, the calls are asynchronous so this behavior is expected) but in my test code I need to...

Strategy to persist the node's data for dynamic Elasticsearch clusters

elasticsearch,docker,mesos,marathon
I'm sorry that this is probably a kind of broad question, but I didn't find a solution form this problem yet. I try to run an Elasticsearch cluster on Mesos through Marathon with Docker containers. Therefore, I built a Docker image that can start on Marathon and dynamically scale via...

logstash tab separator not escaping

elasticsearch,logstash
I have tab separated data which I want to input into logstash. Here is my configuration file: input { file { path => "/*.csv" type => "testSet" start_position => "beginning" } } filter { csv { separator => "\t" } } output { stdout { codec => rubydebug } }...

Elasticsearch. Nested field type and raw string subfield

elasticsearch
I has mapping: { "properties":{ /*some fields ommited*/, "properties":{ "type":"nested", "properties":{ "FieldA":{ "type":"string", "fields":{ "raw":{ "index":"not_analyzed", "type":"string" } } } } } } } and some documents: { /*some fields ommited*/, "properties":{ "FieldA":"one" } } { /*some fields ommited*/, "properties":{ "FieldA":"two" } } and so on. I try to make...

Elasticsearch nested filter query

elasticsearch
Here's an example of my json document I'm trying to create a query against. The "params" field is mapped as "nested". { "images": [ { "name": "1907183375555f7c44126f23.67610475.png" }, { "name": "5693836375575c567764bc2.87695507.png" } ], "sku": "MYSKU", "class": "some_class", "params": [ { "name": "item1", "value": "item1value" }, { "name": "item2", "value": "item2value"...

Elasticsearch aggregation on part of string, not full string

node.js,mongodb,elasticsearch,aggregation
Basically, what I'm trying to do here is get the second-level-down categories from a hierarchically stored string. The problem is that the level of hierarchy vary and one product category could have six levels and another only four, otherwise I would have just implemented predefined levels. I have some products...

Performing searches on JSON data in Elasticsearch

json,elasticsearch,logstash
I have mapped JSON data into Elasticsearch via Logstash which has worked, it has imported the data in and I can see it in Elasticsearch-Head. My problem is querying the data. I can run a search for a field but it returns the entire type within the index as a...

get buckets count in elasticsearch aggregations

elasticsearch,elastic
I am using elasticsearch to search a database with a lot of duplicates. I am using field colapse and it works, however it returns the amount of hits (including duplicates) and not the amount of buckets. "aggs": { "uniques": { "terms": { "field": "guid" }, "aggs": { "jobs": { "top_hits":...

Bonsai automatic indexing

mongodb,heroku,express,elasticsearch,bonsai-elasticsearch
I use MongoDb for storage with an express, Heroku-hosted app. I would like to provide search capabilities to some MongoDb collections, using ElasticSearch via the Bonsai Heroku add-on. However, according to the Bonsai faq, Elasticsearch River plugins are not supported. How can I automatically re-index my MongoDb collections (when a...

How to use special document fields in scripts in elastic?

elasticsearch,elastic
I'm trying to write query with custom script in elasticsearch: https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-script-filter.html#query-dsl-script-filter https://www.elastic.co/guide/en/elasticsearch/reference/master/modules-scripting.html. This is useful when you need to compare two document fields. Everything worked fine, until I decide to use special document field (ex: _id, _uid, etc). The query always returns empty results and there is no errors if...

Elasticsearch Function Scoring based on max score within array / nested

elasticsearch,scoring
I have a field within my document, which stores an array of Integers. Java Class: public class Clazz { public List<Foo> foo; public static Foo { public Integer bar; public Integer baz; } } Mapping: "properties" : { "foo" : { "properties" : { "bar" : { "type" : "integer"...

Logstash not writing to Elasticsearch with Shield

elasticsearch,logstash,elasticsearch-plugin,logstash-configuration
I have been trying to make logstash write to elasticseach with shield without success. My setup was working nromally before installing the shield plugin to elasticsearch. I've followed this guide from elastic.co and created a new user for the logstash user role using: esusers useradd logstashadmin -r logstash I've also...

Filtered Query in Elastic Search

elasticsearch,nosql
Filtered Query query not working in elastic search. It gives a error Query Parsing exception with filter malformed, no field after start_object { "query": { "filtered": { "query": { "match_all": {} }, "filter": { "bool": { "must": [], "should": [ { "_expires": null }, { "_expires": { "gte": 1433947304884 }...

Elasticsearch : _score always 0 in Groovy script

groovy,elasticsearch
I have this kind of Groovy script: def multiplier = doc['data'].value if (multiplier <= 0) { multiplier = 1 } multiplier * _score I use it as a script_score, and my score is always 0. It seems like _score is always 0. With a mvel script, it works. mvel script...

Elastic Search multi-value field aggregation

elasticsearch,aggregation,multivalue
My indexed documents have a schema: { ... 'authors': [{'first name': 'John', 'last name': 'Smith'}, {'first name': 'Mark', 'last name': 'Spencer'}] ... } I would like to search them and aggregate by the individual authors, so get a list with top authors which occurred in my hits. Terms aggregation seems...

Get document on some condition in elastic search java API

java,elasticsearch,elasticsearch-plugin
As I know we can parse document in elastic search, And when we search for a keyword, It will return the document using this code of java API:- org.elasticsearch.action.search.SearchResponse searchHits = node.client() .prepareSearch() .setIndices("indices") .setQuery(qb) .setFrom(0).setSize(1000) .addHighlightedField("file.filename") .addHighlightedField("content") .addHighlightedField("meta.title") .setHighlighterPreTags("<span class='badge badge-info'>") .setHighlighterPostTags("</span>") .addFields("*", "_source")...

ElasticSearch (Nest) Terms sub aggregation of Terms - Not working as intended

elasticsearch,nest
Taking the following mapping in account : { "person": { "properties": { "id": { "type": "string" }, "name": { "type": "string" }, ... "trainings": { "properties": { "attendanceDate": { "type": "date", "format": "dateOptionalTime" }, "providerId": { "type": "string", "index": "not_analyzed" }, "trainingId": { "type": "string", "index": "not_analyzed" } ... }...

How to define a bucket aggregation where buckets are defined by arbitrary filters on a field (GROUP BY CASE equivalent)

elasticsearch
ElasticSearch enables us to filter a set of documents by regex on any given field, and also to group the resulting documents by the terms in a given (same or different field, using "bucket aggregations". For example, on an index that contains a "Url" field and a "UserAgent" field (some...

Specify fields with Elasticsearch Transport client

java,elasticsearch
Is there a way to specify the fields ES returns via Transport Client, specifically using BoolQueryBuilder? With REST API, it seems easy, e.g. Can I specify the result fields in elasticsearch query? But not sure how to do it with Transport Client. Looking at BoolQueryBuilder class, I don't see any...

And + Or query using elastic search REST API

elasticsearch,elastic
How do we combine AND/OR Query in elastic search. For Eg, Below are the documents we have {"title": "MI2", "brand": "Xiomi","operatingSystem": "Android"}, {"title": "Nexus","brand": "Google","operatingSystem": "Android"}, {"title": "Samsung Note","brand": "Samsung","operatingSystem": "Android"}, {"title": "Samsung Galaxy","brand": "Samsung","operatingSystem": "Android"} Now How do we query Android phones from either Samsung or Xiomi. ( operatingSystem...

Elasticsearch NumberFormatException when running two consecutive java tests

java,date,elasticsearch,numberformatexception,spring-data-elasticsearch
I have two test in a class, each of them containing the following query: SearchQuery searchQuery = new NativeSearchQueryBuilder().withQuery(matchAllQuery()).withFilter(rangeFilter("publishDate").lt(date)).build(); In one of the tests, the number of the results elasticsearchTemplate.count(searchQuery, Article.class), in the other one the returned values are verified elasticsearchTemplate.queryForPage(searchQuery,Article.class) If I run any of these two tests separately,...

How to read data in logs using logstash?

elasticsearch,logstash
I have just started log stash, i have log files in that log file whole object is printed in the logs, Since my object is huge i cant write the grok patterns to the whole object and also i expecting only two values out of those object. Can you please...

Javascript: Altering an object where dot notation is used [duplicate]

javascript,jquery,elasticsearch
This question already has an answer here: How to access object properties containing special characters? 1 answer I'm building an Elasticsearch search interface. My method is to build the initial query object, and then alter it depending on the user input. In the filter part of my object, I...

Finding ElasticSearch records matching empty and null values

python,python-2.7,elasticsearch
I have some elasticsearch records that are being stored as either an empty string, or a null value. I am trying to develop a query that will allow me to return these from the index. I came up with: { 'query': { 'filtered': { 'filter': { 'bool': { 'should': [...

Elasticsearch geospatial search, problems with index setup

elasticsearch,geospatial
I'm trying to search for documents previously added to an index, which has been configured to allow geospatial queries (or so I think). My elasticsearch instance is hosted on qbox.io. This is the code I wrote to create an index from the command line curl -XPOST username:[email protected]/events -d '{ "settings"...

create elasticsearch query with optional params

ruby,ruby-on-rails-4,elasticsearch
I'm trying to create an elasticsearch query that have the following logic: search by title, and if the type of record is given, also add it to the query. I did this def to_es_query { query: { match: { title_en: { query: @title } }.merge!(optional_params) } } end def optional_params...

Elasticsearch 1.6 reboot on ubuntu pid-file

linux,ubuntu,elasticsearch,pid
Has anyone here installed the latest version of Elasticsearch on a Ubuntu? If so you might have encountered the elasticsearch PID_DIR issue when rebooting the Ubuntu server. The file PID file is written to this directory: /var/run/elasticsearch but it's not actually there on Linux reboot because it gets deleted by...

NEST ElasticSearch.NET Escape Special Characters

c#,elasticsearch,nest
I have been experimenting with the use of the NEST client for Elastic Search, but seem to have hit a barrier when filtering on a term which contains special/reserved characters such as '/' Below is a JSON representation of my model.. "categories": { "count": 1, "default": "root/Hello/World/Category", } When submitting...

Cannot run elastic search in circleci to make my rspec for elasticsearch to pass?

ruby-on-rails-3,github,rspec,elasticsearch,circleci
i have configured my project in github with circleci. I am using searchkick for elastic search. my code to reindex looks like this: IN spec_helper.rb RSpec.configure do |config| .... config.before :each do Location.reindex end ... end My app/models/location.rb looks like this class Location < ActiveRecord::Base ... searchkick word_start: [:location_name], autocomplete:...

How to get a list of all document types

python,elasticsearch
I use elesticserach_dsl in Python to do searching, and I really like it. But the thing I do not know how to impement, is how to get a list of all different document types. The catch is type field plays for me almost the same role as table name in...

Get elasticsearch result based on two keys

elasticsearch,elastic
I want to get all docs who's "PayerAccountId" should equal to "123" and "UsageStartDate" should be in range [2015-05-01 TO 2015-05-10] I am expecting something to run like this, curl -X GET http://192.168.1.3:9200/_all/_search -d '{"query" : {"match" : { "PayerAccountId:\"156023466485\" AND UsageStartDate:[2015-01-01 TO 2015-01-10]" }}}' Obviously it's not working any...

What are the possible reasons behind the java lang outofmemoryerror java heap space in elasticsearch

java,elasticsearch,out-of-memory
I have seen lots of java lang outofmemoryerror java heap space in elasticsearch but I could find any help page that describes the possible reasons behind these errors in elastic search. I am getting errors for example: (2015-04-09 13:56:47,527 DEBUGaction.index Emil Blonsky observer: timeout notification from cluster service. timeout setting...

ElasticSearch Multiple Scrolls Java API

java,scroll,elasticsearch,parallel-processing
I want to get all data from an index. Since the number of items is too large for memory I use the Scroll (nice function): client.prepareSearch(index) .setTypes(myType).setSearchType(SearchType.SCAN) .setScroll(new TimeValue(60000)) .setSize(amountPerCall) .setQuery(MatchAll()) .execute().actionGet(); Which works nice when calling: client.prepareSearchScroll(scrollId) .setScroll(new TimeValue(600000)) .execute().actionGet() But, when I call the former method multiple times,...

Sorting after aggregation in Elasticsearch

sorting,elasticsearch,aggregation
I have docs with this structure: { FIELD1:string, FIELD2: [ {SUBFIELD:number}, {SUBFIELD:number}...] } I want to sort on the result of the sum of numbers in FIELD2.SUBFIELDs: GET myindex/_search { "size":0, "aggs": { "a1": { "terms": { "field": "FIELD1", "size":0 }, "aggs":{ "a2":{ "sum":{ "field":"FIELD2.SUBFIELD" } } } } }...

Heap memory Solr and Elasticsearch

solr,elasticsearch
I'm just reading the book Mastering Apache Solr and the writer recommends to set the minimum heap size (-Xms) to 2GB and the maximum heap size (-Xmx) to 12GB. Is 2GB necessary? I just use a 512MB server (which is low, I know) for Solr and I found it already...

How to filter results based on frequency of repeating terms in an array in elasticsearch

elasticsearch
I have an array field with a lot of keywords and i need to sort the documents on the basis on how many times a particular keyword repetation in those arrays. For eg,if my field name is "nationality" and for document 1, it consists of the following doc1 nationality :...

search with filter by token count

elasticsearch,token
Fields in documents are analyzed, to create token. {"message":"hello world"} -> token: ["hello", "world"] {"message":"hello"} -> token: ["hello"] {"message":"world"} -> token: ["world"] {"message":"hello java"} -> token: ["hello", "java"] {"message":"java"} -> token: ["java"] Is there a possibility to search all documents in which a specific field contains a given token and...

ElasticSearch - Configuration to Analyse a document on Indexing

elasticsearch
In a single request, I want to retrieve documents from a SOR, store them in ElasticSearch, and then search those documents using the ES search API. There seems to be some lag from the time the document is indexed and the time it is analyzed and ready to be searched....

Getting Parse error for elasticsearch-py

python,elasticsearch,elastic,elasticsearch-py
I am trying to search my entire elasticsearch data for a certain word "tsbu" within a time range. When I try running this, I get a SearchParseException and Parse Failure. es = Elasticsearch() doc = { "query": { "match" : { "message" : "tsbu" } }, "range" : { "@timestamp"...

elastic search sort in aggs by column

sorting,elasticsearch,group-by,order
I am trying to sort in elastic search in aggs, equivalent in mysql "ORDER BY Title ASC/DESC". Here is the index structure: 'body' => array( 'mappings' => array( 'test_type' => array( '_source' => array( 'enabled' => true ), 'properties' => array( 'ProductId' => array( 'type' => 'integer', 'index' => 'not_analyzed'...

Elasticsearch standard analyser stopwords

elasticsearch
I am trying to guess what is the default stopwords list in standard analyzer in elasticsearch. I run version 1.3.1, and it seems to me that the English list is used, because running a wildcard query like this { "wildcard" : { "name" : { "wildcard" : "*in*" } }...

NEST - Using GET instead of POST/PUT for searching

c#,elasticsearch,nest
Is there a way to tell NEST to use GET instead of POST when performing searches? Similar to how the ElasticSearch documentation shows CURL using GET I'd like to use GET when using NEST instead of using POST as it currently does.

How to store money in elasticsearch

python,elasticsearch
I've created an elasticsearch index and my current mapping stores the dollar amount of an item as a string. This is proving to be difficult as I can't search/filter on this value correctly. GET catalog/product/_search { "filter": { "range": { "price": { "from": 230.0, "to": 300.0 } } } }...

Insert data to ElasticSearch using NEST API

elasticsearch,nest
I am trying to insert\bulk insert data to Elastic using NEST API. Can someone provide me the example using NEST? Thanks, Sameer...

Elasticsearch error : Root type mapping not empty after parsing! Remaining fields

elasticsearch
I am trying to insert a new type 'place' in elasticsearch using its curl API. But it keeps failing. The command I'm using is: curl -XPUT host/my_index/_mapping/place --data '{ "place": { "_id": { "path": "place_id" }, "properties": { "place_id":{ "type": "string", "index": "not_analyzed", "store": true }, "location": { "type": "geo_point",...

Elasticsearch average over date histogram buckets

elasticsearch,facets,date-histogram
I've got a bunch of documents indexed in ElasticSearch, and I need to get the following data: For each month, get the average number of documents per working day of the month (or if impossible, use 20 days as the default). I already aggregated my data into months buckets using...

Elasticsearch - Order search results ASC

c#,sorting,elasticsearch,nest
having a problem with my elasticsearch. Setup: Having a Company-Class with the data field "companyName". My search shall search and response all companys with the searched term. If I try to sort via .Sort(x=> x.OnField(x => x.CompanyName).Descending()) The data aren't sorted rightly - reference stackOverflow I tried the given solution,...

How to do a wildcard or regex match on _id in elasticsearch?

elasticsearch
From below sample elasticsearch data I want to apply wildcard say *.000ANT.* on _id so as to fetch all docs whose _id contains 000ANT. Please help. "hits": [ { "_index": "data_collector", "_type": "agents", "_id": "Org000LAN_example1.com", "_score": 1, "fields": { "host": [ "example1.com" ] } }, { "_index": "data_collector", "_type": "agents",...

How can i disable the automatic index creation in elasticsearch?

elasticsearch
I need to disable automatic index creation for an index but need to permit for another one. How can I disable the automatic index creation by elasticsearch for a particular index only? I tried action.auto_create_index: false in elasticsearch.yml file,but it seems to disable all the automatic indexing for all. Can...

FIELDDATA Data is too large

elasticsearch,kibana-4
I open kibana and do a search and i get the error where shards failed. I looked in the elasticsearch.log file and I saw this error: org.elasticsearch.common.breaker.CircuitBreakingException: [FIELDDATA] Data too large, data for [@timestamp] would be larger than limit of [622775500/593.9mb] Is there any way to increase that limit of...

Elasticsearch Filter by day of week

elasticsearch
I have the following document structure: { "@version" : "1", "@timestamp" : "2015-05-25T13:31:43.848Z", "type" : "myevent", "value": 1 } Is it possible to query all documents that occured on a Sunday?...

indexing names in json using elasticsearch in couchdb

json,elasticsearch,couchdb,elasticsearch-river
I am trying to implement full-text query for my json documents. I want to search by title. My json is as follows: { "release":{ "genres":{ "genre":"Electronic" }, "identifiers":{ "identifier":[ { "description":"A-Side", "value":"MPO SK 032 A1 G PHRUPMASTERGENERAL T27 LONDON", "type":"Matrix / Runout" }, { "description":"B-Side", "value":"MPO SK 032 B1", "type":"Matrix...

Docker container http requests limit

http,elasticsearch,docker
I'm new to Docker so, most likely, I'm missing something. I'm running a container with Elasticsearch, using this image. I'm able to setup everyhing correctly. After that I was a using a script developed by a collegue in order to insert some data, basically querying a MySQL database and making...

Elasticsearch python client: Getting the ES version through API call

python,elasticsearch
I want to get the current Elasticsearch version through the python API. I could easily get it through a http call like import requests requests.get(http://endpoint:9200) But I am wondering is there any way to get the version through the API call instead of http request to the endpoint. Like from...

Use a PHP file to Symfony

php,symfony2,elasticsearch
I have a PHP file which is connected to elasticsearch, where I am indexing my documents. My elasticIndex.php file: **class elasticIndex{ function elasticFun(){ require 'vendor/autoload.php'; $client = new Elasticsearch\Client(); $feed = 'http://blaasd.zasdp.tv/Xml'; $xml = simplexml_load_file($feed); foreach ($xml-> ....... ......}** Now the problem is I am working on Symfony framework, where...

ElasticSearch + Kibana - Unique count using pre-computed hashes

elasticsearch,kibana-4
update: Added I want to perform unique count on my ElasticSearch cluster. The cluster contains about 50 millions of records. I've tried the following methods: First method Mentioned in this section: Pre-computing hashes is usually only useful on very large and/or high-cardinality fields as it saves CPU and memory. Second...

How to write search queries in kibana using Query DSL for Elasticsearch aggregation

elasticsearch,querydsl,kibana-4
I am working on ELK stack to process Apache access logs. Spent a lot of time understanding Query DSL format so that more complex queries can be written. Currently am facing issues with running the queries in kibana interface but the same queries work just fine when posted using curl...

django-haystack with elastic search not building index for newly created objects

python,django,elasticsearch,django-haystack
using Django==1.8 django-haystack==2.3.1 setting.py: HAYSTACK_CONNECTIONS = { 'default': { 'ENGINE':'haystack.backends.elasticsearch_backend.ElasticsearchSearchEngine', 'URL': 'http://127.0.0.1:9200/', 'INDEX_NAME': 'haystack', 'TIMEOUT': 60, }, } Newly created(updated or deleted )objects are not updating in my elastic search index. entering .manage.py rebuild_index command each time when creating or updating objects which working perfectly and updating each change. Is...

How to correctly setup 2-node ElasticSearch cluster

elasticsearch
I have small 2-node cluster node1 is "always on" and placed on production server node2 is "sometimes on" and placed on notebook for developing proposes they both have simple unicast config discovery.zen.ping.multicast.enabled: false discovery.zen.ping.unicast.hosts: [ "other node IP" ] so node2 is very often offline from node1 node1 is USUALLY...

Elastic4s - finding multiple exact values for one term

scala,elasticsearch,elastic4s
I'm trying to filter a term to be matching one of the values in an array. relaying on the ES https://www.elastic.co/guide/en/elasticsearch/guide/current/_finding_multiple_exact_values.html GET /my_store/products/_search { "query" : { "filtered" : { "filter" : { "terms" : { "price" : [20, 30] } } } } } I tried this: val res...

How to avoid to extend the mapping on commiting an object

elasticsearch,nest
Let's assume folowing class which I want to index: private class User { public User() { Id = Guid.NewGuid(); Added = DateTime.Now; } public Guid Id { get; protected set; } public string LastName { get; set; } public DateTime Added { get; protected set; } // Unimportant for search...

ElasticSearch: How to search on different fields that are not related that are arrays of objects

elasticsearch
I want to search on different fields that are not related that are arrays of objects. I cannot find out how. Given the following mapping and data entry: I want to give the user the ability to search all possible fields in any combination. The user would use a form...

How do I implement a PatternAnalyzer in elastic4s and elasticsearch to exclude result with a certain field

elasticsearch,playframework,elastic4s
I'm trying to perform a query on my index and get all reviews that do NOT have a reviewer with a gravatar image. To do this I have implemented a PatternAnalyzerDefinition with a host pattern: "^https?\\:\\/\\/([^\\/?#]+)(?:[\\/?#]|$)" that should match and extract host of urls like: https://www.gravatar.com/avatar/blablalbla?s=200&r=pg&d=mm becomes: www.gravatar.com The mapping:...

How to display “ALL” the nested documents in an object in separate rows from elasticsearch?

elasticsearch,nested,elasticsearch-plugin
I have a nested object in the following form: { "name": "Multi G. Enre", "books": [ { "name": "Guns and lasers", "genre": "scifi", "publisher": "orbit" }, { "name": "Dead in the night", "genre": "thriller", "publisher": "penguin" } ] } I tried the following JSON query for the above document: {...

How to use arrays in lambda expressions?

c#,elasticsearch,nest
I am writing a program with NEST library of ElasticSearch. I want to write a lambda expression for a function with this argument: HighlighDescriptor<parentdocument> HighlighDescriptor.onFields (param Action<HighlightFieldDescriptor<ParentDocument>>[] fieldHighlighters) I don't know what is the array in the function argument?...

Creating Index in Elasticsearch using Java API giving NoClassFoundException

java,indexing,elasticsearch
I'm trying to create a node based client using Java API and index a JSON document. Here's the code : import java.util.Date; import java.util.HashMap; import java.util.Map; import org.elasticsearch.action.deletebyquery.DeleteByQueryResponse; import org.elasticsearch.client.Client; import org.elasticsearch.node.Node; import static org.elasticsearch.node.NodeBuilder.*; public class Els { public static void main (String args[]){ Els p = new Els();...

Elasticsearch and C# - query to find exact matches over strings

c#,.net,database,elasticsearch,nest
I need a way to search documents using a plain exact match over two or multiple fields which are of type "string" and "integer". I'd like to avoid standard query as I don't care about scoring or best match, just a yes/no outcome if both the fields match or not....

How to append two SearchDescriptors in NEST

c#,elasticsearch,nest
I am taking input from a client to build up an elasticsearch query using NEST. I start out with the basics, like so: var search = esClient.Search<MyData>(s => s .From(pageNum * pageSize) .Take(pageSize) .QueryRaw(@"{""match_all"": {} }") I then parse out the request and see if an optional sorting parameter was...

How to get duplicate field values in elastic search by field name without knowing its value

elasticsearch
I have a field "EmployeeName" in an elastic search index - and I would like to execute a query that will return me all the cases where there are duplicate values of "EmployeeName". Can this be done? I found more_like_this but this requires field value for "like_text". But my requirement...

Java method name analyzer

java,elasticsearch
I am indexing the request logs from a java app and the logs contain key method names and time taken. The method names are full method names like com.example.domain.File[] com.example.service.FolderService.getFiles() throws com.example.Exception In order to search for a method name or type (like getfiles or File) I believe I need...

ElasticSearch- “No query registered for…”

search,indexing,elasticsearch
ElasticSearch returns me "No query registered for [likes_count]" error when trying to look up entries using the following query. The field likes_count is a new field of documents and does not exist in every document. The same query works without the sort part. Why does this error appear? Thanks {...

elasticsearch aggregation group by null key

elasticsearch
here is the data in my elasticsearch server: {"system": "aaa"}, {"system": "bbb"}, {"system": null} I want to get the statistics for system. then I did the query: { "aggs" : { "myAggrs" : { "terms" : { "field" : "system" } } } it gives me the result: { "key":...

Parsing Google Custom Search API for Elasticsearch Documents

json,python-2.7,elasticsearch,google-search-api
After retrieving results from the Google Custom Search API and writing it to JSON, I want to parse that JSON to make valid Elasticsearch documents. You can configure a parent - child relationship for nested results. However, this relationship seems to not be inferred by the data structure itself. I've...

ElasticSearch - how to get the auto generated id from an insert query

c#,mysql,database,elasticsearch,nest
On my ElasticSearch database I need to get the autogenerated id from my insert query (I'm using .NET C#). How to do it? I tried debugging the readRecords response but I didn't find such id. Basically I need the equivalent of the MySQL LAST_INSERT_ID() command. var readRecords = elasticClient.Search<HistoryRecord>(s =>...

Not able to access Kibana running in a Docker container on port 5601

elasticsearch,docker,dockerfile,kibana-4
I have built a docker image with the following Docker file. # gunicorn-flask FROM devdb/kibana MAINTAINER John Doe <[email protected]> ENV DEBIAN_FRONTEND noninteractive RUN apt-get update RUN apt-get install -y python python-pip python-virtualenv gunicorn # Setup flask application RUN mkdir -p /deploy/app COPY gunicorn_config.py /deploy/gunicorn_config.py COPY app /deploy/app RUN pip install...

Delete records of a certain type from logstash/elasticsearch

elasticsearch,logstash
I'm about to embark upon importing a large number of records into elasticsearch (via logstash). I'm sure I will make a few mistakes. As such, I would like to be able to easily delete the imported records from elasticsearch. For now, I can just delete the indicies containing the imports....

ElasticSearch Nest Query with Dynamic Match

elasticsearch,nest
I want to build an ElasticSearch Nest SearchDescriptor dynamically using conditional Match. The UI has two check boxes "Exclude Email" and "Exclude Loose Files". If both are checked, the search result will match 0 hits. These 4 queries work, but I can't figure out how to make the Match parts...

Elasticsearch: How to query using partial phrases in quotation marks

elasticsearch
I am trying to implement a search behavior that supports partial phrases. A possible search input could look like this: example "hello world" elasticsearch Now I want to get all documents, that contain the words example and elasticsearch as well as the phrase hello world. As this is a very...

Operator '??' cannot be applied to operands of type IQueryContainer and lambda expression

c#,elasticsearch,nest
I am trying to create a method to process a certain query. I follow an example posted on the Nest repository (line 60), but still the MatchAll is not recognized by the compiler and if I try to build the solution, the error that shows is: Operator '??' cannot be...

Passing Elasticsearch and Kibana config file to docker containers

elasticsearch,docker,kibana,kibana-4
I have found a docker image devdb/kibana which runs Elasticsearch 1.5.2 and Kibana 4.0.2. However I would like to pass into this docker container the configuration files for both Elasticsearch (i.e elasticsearch.yml) and Kibana (i.e config.js) Can I do that with this image itself? Or for that would I have...

elasticsearch -check if array contains a value

elasticsearch
I want to check on an field of an array long type that includes some values. the only way I found is using script: ElasticSearch Scripting: check if array contains a value but it still not working fore me: Query: { "query": { "filtered": { "query": { "match_all": {} },...

Query returns both documents instead of just one

c#,.net,elasticsearch,nest
var res = esclient.Search<MyClass>(q => q .Query(fq => fq .Filtered(fqq => fqq .Query(qq => qq.MatchAll()) .Filter(ff => ff .Bool(b => b .Must(m1 => m1.Term("macaddress", "mac")) .Must(m2 => m2.Term("another_field", 123)) ) ) ) ) ); As far as I can understand the bool and must together are the equivalent of the...

ElasticSearch - Setting doc_values on existing index

elasticsearch
According to this article: It is possible to change the field data format (and the field data settings in general) on a live index by using the update mapping API. But when I try: curl -XPUT 'http://es1:123/my_index/my_type/_mapping' -d ' { "my_type": { "properties": { "my_prop": { "index": "not_analyzed", "fielddata": {...

MultiMatch query with Nest and Field Suffix

c#,elasticsearch,nest
Using Elasticsearch I have a field with a suffix - string field with a .english suffix with an english analyser on it as shown in the following mapping ... "valueString": { "type": "string", "fields": { "english": { "type": "string", "analyzer": "english" } } } ... The following query snippet won't...

Elasticsearch data model

data,elasticsearch,model,nested,parent-child
I'm currently parsing text from internal résumés in my company. The goal is to index everything in elasticsearch to perform search on them. for the moment I have the following JSON document with no mapping defined : Each coworker has a list of project with the client name { name:...

Unexpected case sensitivty

elasticsearch
I am a noob running elastic search 1.5.9. I want to pull out all of the documents that have the field "PERSON" set to "Johnson." (Note the mixed casing). If I manually look at elastic search head, I can see a document with exactly those attributes. The docs explain that...

Strange behaviour of limit in Elasticsearch

python,elasticsearch
I tried two queries. First one looks like this (it simply lists all data): # listing 1 from elasticsearch import Elasticsearch from elasticsearch_dsl import Search, Q, F .... .... connection etc s = Search(using=db,index="reestr") rows = s.execute() for r in rows: print(r) listing 1 prints out all documents from the...

How to have multiple regex based on or condition in elasticsearch?

elasticsearch
I want to get all 000ANT and 0BBNTA from id, is there something similar to terms which works with regexp or is there any other way? Otherwise I will have to query elasticsearch for each item say 000ANT and 0BBNTA. Please help. Below is something that I am trying out...

Elasticsearch daily rolling index contains duplicate _id

elasticsearch,logstash
this maybe a silly question but I am using the daily rolling index to save my events with logstash, the config is simple as: input: {..source..} filter: {..filter..} output: { elasticsearch: { document_id: %{my_own_guarantee_unique_id} index: myindex-%{+YYYY.MM.DD} } } what I found was if there are events with same my_own_guarantee_unique_id appears...