elasticsearch,nest , ElasticSearch (Nest) Terms sub aggregation of Terms - Not working as intended


ElasticSearch (Nest) Terms sub aggregation of Terms - Not working as intended

Question:

Tag: elasticsearch,nest

Taking the following mapping in account :

{
    "person": {
        "properties": {
            "id": {
                "type": "string"
            },
            "name": {
                "type": "string"
            },
            ...
            "trainings": {
                "properties": {
                    "attendanceDate": {
                        "type": "date",
                        "format": "dateOptionalTime"
                    },
                    "providerId": {
                        "type": "string",
                        "index": "not_analyzed"
                    },
                    "trainingId": {
                        "type": "string",
                        "index": "not_analyzed"
                    }
                    ...
                }
            }
        }
    }
}

This is currently my elastic search query

{
    "aggs": {
        "trainning_agg": {
            "terms": {
                "field": "trainings.trainingId"
            },
            "aggs": {
                "provider_agg": {
                    "terms": {
                        "field": "trainings.providerId"
                    }
                }
            }
        }
    }
}

Unfortunately, elastic search will return to me a list of buckets by trainings containing a list of buckets of all aggregated providers (ignoring the training id).

Instead of a list of bucket by trainings containing a list of buckets of all the providers filtered by the parent training.

Any help would be grateful, and let me know what details I could provide to help you understand my problem!

Thanks!

Edits:

Let me precise that currently every training has currently one, and only provider, so every "person" will currently have one "trainings" which will contains a single element. (This is not always the case, but lets assume so for the sake of the simplicity of the example)

There is currently 12 distinct trainings, each trainings as one provider.

Please note that currently the provider id is the same as the training id (This is not always the case, but lets assume so for the sake of the simplicity of the example) This is currently my response :

{
        ...
        "aggregations": {
            "training_agg": {
                ...
                "buckets": [
                    {
                        "key": "17effec3-1284-4609-af41-2252b4dde8b5",
                        "doc_count": 8808,
                        "provider_agg": {
                            ...
                            "buckets": [
                                {
                                    "key": "17effec3-1284-4609-af41-2252b4dde8b5",
                                    "doc_count": 8808
                                },
                                {
                                    "key": "560be929-134b-4b67-befd-f38fdbb2f7ad",
                                    "doc_count": 8808
                                },
                                {
                                    "key": "96b16133-51ef-413c-afa5-24f74263126b",
                                    "doc_count": 8808
                                },
                                {
                                    "key": "96c36f93-90ce-4378-a9f2-8a8aaa780fd2",
                                    "doc_count": 8808
                                },
                                {
                                    "key": "e9467886-42e0-467c-a338-a7fcbc66125f",
                                    "doc_count": 8808
                                },
                                {
                                    "key": "f05554c2-778b-44b6-972e-c2fea01d924f",
                                    "doc_count": 8808
                                },
                                {
                                    "key": "ab40dcda-e253-4ff3-95e9-314e4ce5ba39",
                                    "doc_count": 8807
                                },
                                {
                                    "key": "5aaa2f31-6aa6-4d5a-86ea-8a25cb11247e",
                                    "doc_count": 8081
                                },
                                {
                                    "key": "d1c45ad7-046b-43de-b65e-012185c17f6a",
                                    "doc_count": 4686
                                },
                                {
                                    "key": "c06e0cee-1050-41b6-ac83-ea3a00f0abb7",
                                    "doc_count": 3135
                                }
                            ]
                        }
                    }
                ]
            }
        }
    }

And this is what i'm trying to get :

{
    ...
    "aggregations": {
        "training_agg": {
            ...
            "buckets": [
                {
                    "key": "17effec3-1284-4609-af41-2252b4dde8b5",
                    "doc_count": 8808,
                    "provider_agg": {
                        ...
                        "buckets": [
                            {
                                "key": "17effec3-1284-4609-af41-2252b4dde8b5",
                                "doc_count": 8808
                            }
                        ]
                    }
                }
            ]
        }
    }
}

Documents examples (Had to remove some info, confidentiality agreement :O) N.B.: The fact that the providerId and the trainingId is the same is pure laziness from me when I inserted the data, it could be two UUID completely different.

    {
    "hits": [
        {
            ...
            "_source": {
                "id": "d9753b0fd674f9d55b8cd125258b2713b191499828dc5301162dffc17f3ddf4b",
                "name": "Do Van Tien",
                ...
                "trainings": [
                    {
                        "trainingName": "Certification - 4C",
                        "providerName": "Hidden Provider",
                        "trainingId": "c06e0cee-1050-41b6-ac83-ea3a00f0abb7",
                        "providerId": "c06e0cee-1050-41b6-ac83-ea3a00f0abb7",
                        "attendanceDate": "2012-07-07T00:00:00"
                    },
                    {
                        "trainingName": "GAP - Safety & Hygiene",
                        "providerName": "Hidden Provider",
                        "trainingId": "96c36f93-90ce-4378-a9f2-8a8aaa780fd2",
                        "providerId": "96c36f93-90ce-4378-a9f2-8a8aaa780fd2",
                        "attendanceDate": "2014-04-16T00:00:00"
                    },
                    {
                        "trainingName": "GAP - Crop protection",
                        "providerName": "Hidden Provider",
                        "trainingId": "ab40dcda-e253-4ff3-95e9-314e4ce5ba39",
                        "providerId": "ab40dcda-e253-4ff3-95e9-314e4ce5ba39",
                        "attendanceDate": "2014-04-16T00:00:00"
                    },
                    {
                        "trainingName": "GAP - Post Harvest",
                        "providerName": "Hidden Provider",
                        "trainingId": "d1c45ad7-046b-43de-b65e-012185c17f6a",
                        "providerId": "d1c45ad7-046b-43de-b65e-012185c17f6a",
                        "attendanceDate": "2012-07-07T00:00:00"
                    },
                    {
                        "trainingName": "GAP - Soil Management & Fertilizer Application",
                        "providerName": "Hidden Provider",
                        "trainingId": "17effec3-1284-4609-af41-2252b4dde8b5",
                        "providerId": "17effec3-1284-4609-af41-2252b4dde8b5",
                        "attendanceDate": "2012-08-16T00:00:00"
                    },
                    {
                        "trainingName": "GAP - Irrigation & Water Management",
                        "providerName": "Hidden Provider",
                        "trainingId": "560be929-134b-4b67-befd-f38fdbb2f7ad",
                        "providerId": "560be929-134b-4b67-befd-f38fdbb2f7ad",
                        "attendanceDate": "2012-07-07T00:00:00"
                    },
                    {
                        "trainingName": "GAP - Composting",
                        "providerName": "Hidden Provider",
                        "trainingId": "96b16133-51ef-413c-afa5-24f74263126b",
                        "providerId": "96b16133-51ef-413c-afa5-24f74263126b",
                        "attendanceDate": "2013-08-22T00:00:00"
                    },
                    {
                        "trainingName": "Certification - UTZ - Code of Conduct",
                        "providerName": "Hidden Provider",
                        "trainingId": "5aaa2f31-6aa6-4d5a-86ea-8a25cb11247e",
                        "providerId": "5aaa2f31-6aa6-4d5a-86ea-8a25cb11247e",
                        "attendanceDate": "2013-08-22T00:00:00"
                    },
                    {
                        "trainingName": "Economic - Business skills",
                        "providerName": "Hidden Provider",
                        "trainingId": "e9467886-42e0-467c-a338-a7fcbc66125f",
                        "providerId": "e9467886-42e0-467c-a338-a7fcbc66125f",
                        "attendanceDate": "2013-08-22T00:00:00"
                    },
                    {
                        "trainingName": "GAP - Pruning",
                        "providerName": "Hidden Provider",
                        "trainingId": "f05554c2-778b-44b6-972e-c2fea01d924f",
                        "providerId": "f05554c2-778b-44b6-972e-c2fea01d924f",
                        "attendanceDate": "2013-05-09T00:00:00"
                    }
                ]
            }
        },
        {
            ...
            "_source": {
                "id": "dca2d4b86bf1bea73492bde614e0bb3d7cadf2f4f44e9fe3c5b4cf930d2cca00",
                "name": "Le Thi Than",
                ...
                "trainings": [
                    {
                        "trainingName": "Certification - UTZ - Code of Conduct",
                        "providerName": "Hidden Provider",
                        "trainingId": "5aaa2f31-6aa6-4d5a-86ea-8a25cb11247e",
                        "providerId": "5aaa2f31-6aa6-4d5a-86ea-8a25cb11247e",
                        "attendanceDate": "2012-06-25T00:00:00"
                    },
                    {
                        "trainingName": "GAP - Crop protection",
                        "providerName": "Hidden Provider",
                        "trainingId": "ab40dcda-e253-4ff3-95e9-314e4ce5ba39",
                        "providerId": "ab40dcda-e253-4ff3-95e9-314e4ce5ba39",
                        "attendanceDate": "2014-04-23T00:00:00"
                    },
                    {
                        "trainingName": "Economic - Business skills",
                        "providerName": "Hidden Provider",
                        "trainingId": "e9467886-42e0-467c-a338-a7fcbc66125f",
                        "providerId": "e9467886-42e0-467c-a338-a7fcbc66125f",
                        "attendanceDate": "2012-06-25T00:00:00"
                    },
                    {
                        "trainingName": "GAP - Soil Management & Fertilizer Application",
                        "providerName": "Hidden Provider",
                        "trainingId": "17effec3-1284-4609-af41-2252b4dde8b5",
                        "providerId": "17effec3-1284-4609-af41-2252b4dde8b5",
                        "attendanceDate": "2012-07-25T00:00:00"
                    },
                    {
                        "trainingName": "GAP - Composting",
                        "providerName": "Hidden Provider",
                        "trainingId": "96b16133-51ef-413c-afa5-24f74263126b",
                        "providerId": "96b16133-51ef-413c-afa5-24f74263126b",
                        "attendanceDate": "2012-06-25T00:00:00"
                    },
                    {
                        "trainingName": "GAP - Pruning",
                        "providerName": "Hidden Provider",
                        "trainingId": "f05554c2-778b-44b6-972e-c2fea01d924f",
                        "providerId": "f05554c2-778b-44b6-972e-c2fea01d924f",
                        "attendanceDate": "2013-05-21T00:00:00"
                    },
                    {
                        "trainingName": "GAP - Irrigation & Water Management",
                        "providerName": "Hidden Provider",
                        "trainingId": "560be929-134b-4b67-befd-f38fdbb2f7ad",
                        "providerId": "560be929-134b-4b67-befd-f38fdbb2f7ad",
                        "attendanceDate": "2012-06-25T00:00:00"
                    },
                    {
                        "trainingName": "GAP - Safety & Hygiene",
                        "providerName": "Hidden Provider",
                        "trainingId": "96c36f93-90ce-4378-a9f2-8a8aaa780fd2",
                        "providerId": "96c36f93-90ce-4378-a9f2-8a8aaa780fd2",
                        "attendanceDate": "2013-05-21T00:00:00"
                    },
                    {
                        "trainingName": "GAP - Post Harvest",
                        "providerName": "Hidden Provider",
                        "trainingId": "d1c45ad7-046b-43de-b65e-012185c17f6a",
                        "providerId": "d1c45ad7-046b-43de-b65e-012185c17f6a",
                        "attendanceDate": "2012-06-25T00:00:00"
                    }
                ]
            }
        }
    ]
}

Answer:

Ok, then what you need is simply to declare your trainings property with type: nested in your mapping, like this:

{
  "mappings": {
    "person": {
      "properties": {
        "id": {
          "type": "string"
        },
        "name": {
          "type": "string"
        },
        "trainings": {
          "type": "nested",       <----- add "nested" here
          "properties": {
            "attendanceDate": {
              "type": "date",
              "format": "dateOptionalTime"
            },
            "providerId": {
              "type": "string",
              "index": "not_analyzed"
            },
            "trainingId": {
              "type": "string",
              "index": "not_analyzed"
            }
          }
        }
      }
    }
  }
}

And then you need to change your aggregation a bit to work with your new nested type, like this:

{
  "size": 0,
  "aggs": {
    "trainning_agg": {
      "nested": {            <----- use nested aggregation
        "path": "trainings"
      },
      "aggs": {
        "trainings": {
          "terms": {
            "field": "trainings.trainingId"
          },
          "aggs": {
            "provider_agg": {
              "terms": {
                "field": "trainings.providerId"
              }
            }
          }
        }
      }
    }
  }
}

And the response will be as you expect:

{
  ...
  "aggregations" : {
    "trainning_agg" : {
      "doc_count" : 19,
      "trainings" : {
        "doc_count_error_upper_bound" : 0,
        "sum_other_doc_count" : 0,
        "buckets" : [ {
          "key" : "17effec3-1284-4609-af41-2252b4dde8b5",
          "doc_count" : 2,
          "provider_agg" : {
            "doc_count_error_upper_bound" : 0,
            "sum_other_doc_count" : 0,
            "buckets" : [ {
              "key" : "17effec3-1284-4609-af41-2252b4dde8b5",
              "doc_count" : 2
            } ]
          }
        }, {
          "key" : "560be929-134b-4b67-befd-f38fdbb2f7ad",
          "doc_count" : 2,
          "provider_agg" : {
            "doc_count_error_upper_bound" : 0,
            "sum_other_doc_count" : 0,
            "buckets" : [ {
              "key" : "560be929-134b-4b67-befd-f38fdbb2f7ad",
              "doc_count" : 2
            } ]
          }
        }, {
          "key" : "5aaa2f31-6aa6-4d5a-86ea-8a25cb11247e",
          "doc_count" : 2,
          "provider_agg" : {
            "doc_count_error_upper_bound" : 0,
            "sum_other_doc_count" : 0,
            "buckets" : [ {
              "key" : "5aaa2f31-6aa6-4d5a-86ea-8a25cb11247e",
              "doc_count" : 2
            } ]
          }
        }, {
          "key" : "96b16133-51ef-413c-afa5-24f74263126b",
          "doc_count" : 2,
          "provider_agg" : {
            "doc_count_error_upper_bound" : 0,
            "sum_other_doc_count" : 0,
            "buckets" : [ {
              "key" : "96b16133-51ef-413c-afa5-24f74263126b",
              "doc_count" : 2
            } ]
          }
        }, {
          "key" : "96c36f93-90ce-4378-a9f2-8a8aaa780fd2",
          "doc_count" : 2,
          "provider_agg" : {
            "doc_count_error_upper_bound" : 0,
            "sum_other_doc_count" : 0,
            "buckets" : [ {
              "key" : "96c36f93-90ce-4378-a9f2-8a8aaa780fd2",
              "doc_count" : 2
            } ]
          }
        }, {
          "key" : "ab40dcda-e253-4ff3-95e9-314e4ce5ba39",
          "doc_count" : 2,
          "provider_agg" : {
            "doc_count_error_upper_bound" : 0,
            "sum_other_doc_count" : 0,
            "buckets" : [ {
              "key" : "ab40dcda-e253-4ff3-95e9-314e4ce5ba39",
              "doc_count" : 2
            } ]
          }
        }, {
          "key" : "d1c45ad7-046b-43de-b65e-012185c17f6a",
          "doc_count" : 2,
          "provider_agg" : {
            "doc_count_error_upper_bound" : 0,
            "sum_other_doc_count" : 0,
            "buckets" : [ {
              "key" : "d1c45ad7-046b-43de-b65e-012185c17f6a",
              "doc_count" : 2
            } ]
          }
        }, {
          "key" : "e9467886-42e0-467c-a338-a7fcbc66125f",
          "doc_count" : 2,
          "provider_agg" : {
            "doc_count_error_upper_bound" : 0,
            "sum_other_doc_count" : 0,
            "buckets" : [ {
              "key" : "e9467886-42e0-467c-a338-a7fcbc66125f",
              "doc_count" : 2
            } ]
          }
        }, {
          "key" : "f05554c2-778b-44b6-972e-c2fea01d924f",
          "doc_count" : 2,
          "provider_agg" : {
            "doc_count_error_upper_bound" : 0,
            "sum_other_doc_count" : 0,
            "buckets" : [ {
              "key" : "f05554c2-778b-44b6-972e-c2fea01d924f",
              "doc_count" : 2
            } ]
          }
        }, {
          "key" : "c06e0cee-1050-41b6-ac83-ea3a00f0abb7",
          "doc_count" : 1,
          "provider_agg" : {
            "doc_count_error_upper_bound" : 0,
            "sum_other_doc_count" : 0,
            "buckets" : [ {
              "key" : "c06e0cee-1050-41b6-ac83-ea3a00f0abb7",
              "doc_count" : 1
            } ]
          }
        } ]
      }
    }
  }
}

Related:


Elasticsearch: How to query using partial phrases in quotation marks


elasticsearch
I am trying to implement a search behavior that supports partial phrases. A possible search input could look like this: example "hello world" elasticsearch Now I want to get all documents, that contain the words example and elasticsearch as well as the phrase hello world. As this is a very...

Elasticsearch - Order search results ASC


c#,sorting,elasticsearch,nest
having a problem with my elasticsearch. Setup: Having a Company-Class with the data field "companyName". My search shall search and response all companys with the searched term. If I try to sort via .Sort(x=> x.OnField(x => x.CompanyName).Descending()) The data aren't sorted rightly - reference stackOverflow I tried the given solution,...

Not able to access Kibana running in a Docker container on port 5601


elasticsearch,docker,dockerfile,kibana-4
I have built a docker image with the following Docker file. # gunicorn-flask FROM devdb/kibana MAINTAINER John Doe <[email protected]> ENV DEBIAN_FRONTEND noninteractive RUN apt-get update RUN apt-get install -y python python-pip python-virtualenv gunicorn # Setup flask application RUN mkdir -p /deploy/app COPY gunicorn_config.py /deploy/gunicorn_config.py COPY app /deploy/app RUN pip install...

Elasticsearch - Query document missing an array value


elasticsearch
I would like to query my elasticsearch index in order to retrieve the documents that don't contain a specific value in an array. For instance, if my query is : { "query": { "bool": { "must": [ { "match_all": {} } ], "must_not": [], "should": [] } }, "from": 0,...

How to have multiple regex based on or condition in elasticsearch?


elasticsearch
I want to get all 000ANT and 0BBNTA from id, is there something similar to terms which works with regexp or is there any other way? Otherwise I will have to query elasticsearch for each item say 000ANT and 0BBNTA. Please help. Below is something that I am trying out...

Get document on some condition in elastic search java API


java,elasticsearch,elasticsearch-plugin
As I know we can parse document in elastic search, And when we search for a keyword, It will return the document using this code of java API:- org.elasticsearch.action.search.SearchResponse searchHits = node.client() .prepareSearch() .setIndices("indices") .setQuery(qb) .setFrom(0).setSize(1000) .addHighlightedField("file.filename") .addHighlightedField("content") .addHighlightedField("meta.title") .setHighlighterPreTags("<span class='badge badge-info'>") .setHighlighterPostTags("</span>") .addFields("*", "_source")...

Operator '??' cannot be applied to operands of type IQueryContainer and lambda expression


c#,elasticsearch,nest
I am trying to create a method to process a certain query. I follow an example posted on the Nest repository (line 60), but still the MatchAll is not recognized by the compiler and if I try to build the solution, the error that shows is: Operator '??' cannot be...

MultiMatch query with Nest and Field Suffix


c#,elasticsearch,nest
Using Elasticsearch I have a field with a suffix - string field with a .english suffix with an english analyser on it as shown in the following mapping ... "valueString": { "type": "string", "fields": { "english": { "type": "string", "analyzer": "english" } } } ... The following query snippet won't...

ElasticSearch - Configuration to Analyse a document on Indexing


elasticsearch
In a single request, I want to retrieve documents from a SOR, store them in ElasticSearch, and then search those documents using the ES search API. There seems to be some lag from the time the document is indexed and the time it is analyzed and ready to be searched....

ElasticSearch: How to search on different fields that are not related that are arrays of objects


elasticsearch
I want to search on different fields that are not related that are arrays of objects. I cannot find out how. Given the following mapping and data entry: I want to give the user the ability to search all possible fields in any combination. The user would use a form...

Elasticsearch standard analyser stopwords


elasticsearch
I am trying to guess what is the default stopwords list in standard analyzer in elasticsearch. I run version 1.3.1, and it seems to me that the English list is used, because running a wildcard query like this { "wildcard" : { "name" : { "wildcard" : "*in*" } }...

Re-index object with new fields


elasticsearch,nest
It seems like as long as the id field is maintained, its super easy to re-index a document by simply calling Index(), but is there a way to given an object was updated and new fields were added, to have it include these new fields in the index? I'm not...

How to write search queries in kibana using Query DSL for Elasticsearch aggregation


elasticsearch,querydsl,kibana-4
I am working on ELK stack to process Apache access logs. Spent a lot of time understanding Query DSL format so that more complex queries can be written. Currently am facing issues with running the queries in kibana interface but the same queries work just fine when posted using curl...

How to read data in logs using logstash?


elasticsearch,logstash
I have just started log stash, i have log files in that log file whole object is printed in the logs, Since my object is huge i cant write the grok patterns to the whole object and also i expecting only two values out of those object. Can you please...

indexing names in json using elasticsearch in couchdb


json,elasticsearch,couchdb,elasticsearch-river
I am trying to implement full-text query for my json documents. I want to search by title. My json is as follows: { "release":{ "genres":{ "genre":"Electronic" }, "identifiers":{ "identifier":[ { "description":"A-Side", "value":"MPO SK 032 A1 G PHRUPMASTERGENERAL T27 LONDON", "type":"Matrix / Runout" }, { "description":"B-Side", "value":"MPO SK 032 B1", "type":"Matrix...

Javascript: Altering an object where dot notation is used [duplicate]


javascript,jquery,elasticsearch
This question already has an answer here: How to access object properties containing special characters? 1 answer I'm building an Elasticsearch search interface. My method is to build the initial query object, and then alter it depending on the user input. In the filter part of my object, I...

Elasticsearch NumberFormatException when running two consecutive java tests


java,date,elasticsearch,numberformatexception,spring-data-elasticsearch
I have two test in a class, each of them containing the following query: SearchQuery searchQuery = new NativeSearchQueryBuilder().withQuery(matchAllQuery()).withFilter(rangeFilter("publishDate").lt(date)).build(); In one of the tests, the number of the results elasticsearchTemplate.count(searchQuery, Article.class), in the other one the returned values are verified elasticsearchTemplate.queryForPage(searchQuery,Article.class) If I run any of these two tests separately,...

Elasticsearch geospatial search, problems with index setup


elasticsearch,geospatial
I'm trying to search for documents previously added to an index, which has been configured to allow geospatial queries (or so I think). My elasticsearch instance is hosted on qbox.io. This is the code I wrote to create an index from the command line curl -XPOST username:[email protected]/events -d '{ "settings"...

NEST ElasticSearch.NET Escape Special Characters


c#,elasticsearch,nest
I have been experimenting with the use of the NEST client for Elastic Search, but seem to have hit a barrier when filtering on a term which contains special/reserved characters such as '/' Below is a JSON representation of my model.. "categories": { "count": 1, "default": "root/Hello/World/Category", } When submitting...

Query returns both documents instead of just one


c#,.net,elasticsearch,nest
var res = esclient.Search<MyClass>(q => q .Query(fq => fq .Filtered(fqq => fqq .Query(qq => qq.MatchAll()) .Filter(ff => ff .Bool(b => b .Must(m1 => m1.Term("macaddress", "mac")) .Must(m2 => m2.Term("another_field", 123)) ) ) ) ) ); As far as I can understand the bool and must together are the equivalent of the...

ElasticSearch (Nest) Terms sub aggregation of Terms - Not working as intended


elasticsearch,nest
Taking the following mapping in account : { "person": { "properties": { "id": { "type": "string" }, "name": { "type": "string" }, ... "trainings": { "properties": { "attendanceDate": { "type": "date", "format": "dateOptionalTime" }, "providerId": { "type": "string", "index": "not_analyzed" }, "trainingId": { "type": "string", "index": "not_analyzed" } ... }...

Parsing Google Custom Search API for Elasticsearch Documents


json,python-2.7,elasticsearch,google-search-api
After retrieving results from the Google Custom Search API and writing it to JSON, I want to parse that JSON to make valid Elasticsearch documents. You can configure a parent - child relationship for nested results. However, this relationship seems to not be inferred by the data structure itself. I've...

Bad scoring due to different maxDocs of IDF


elasticsearch
I have two documents with a field title of: News New Website If I search for the term new website the score for the News document is much higher than the other one which is obviously not what I want. I wrapped an explain around it and got: 'hits': [{'_explanation':...

How to compute the scores based on field data in elasticsearch


elasticsearch
I have the following fields in documents { name: "Pearl", age : 43, weight: 54, bodyWeight : 103, height : 1.8 } Now i want to get scores for the documents based on the bodyWeight to height ratio of the documents. How to do that?...

How to define a bucket aggregation where buckets are defined by arbitrary filters on a field (GROUP BY CASE equivalent)


elasticsearch
ElasticSearch enables us to filter a set of documents by regex on any given field, and also to group the resulting documents by the terms in a given (same or different field, using "bucket aggregations". For example, on an index that contains a "Url" field and a "UserAgent" field (some...

elastic search sort in aggs by column


sorting,elasticsearch,group-by,order
I am trying to sort in elastic search in aggs, equivalent in mysql "ORDER BY Title ASC/DESC". Here is the index structure: 'body' => array( 'mappings' => array( 'test_type' => array( '_source' => array( 'enabled' => true ), 'properties' => array( 'ProductId' => array( 'type' => 'integer', 'index' => 'not_analyzed'...

Docker container http requests limit


http,elasticsearch,docker
I'm new to Docker so, most likely, I'm missing something. I'm running a container with Elasticsearch, using this image. I'm able to setup everyhing correctly. After that I was a using a script developed by a collegue in order to insert some data, basically querying a MySQL database and making...

Strange behaviour of limit in Elasticsearch


python,elasticsearch
I tried two queries. First one looks like this (it simply lists all data): # listing 1 from elasticsearch import Elasticsearch from elasticsearch_dsl import Search, Q, F .... .... connection etc s = Search(using=db,index="reestr") rows = s.execute() for r in rows: print(r) listing 1 prints out all documents from the...

How to get duplicate field values in elastic search by field name without knowing its value


elasticsearch
I have a field "EmployeeName" in an elastic search index - and I would like to execute a query that will return me all the cases where there are duplicate values of "EmployeeName". Can this be done? I found more_like_this but this requires field value for "like_text". But my requirement...

Get elasticsearch result based on two keys


elasticsearch,elastic
I want to get all docs who's "PayerAccountId" should equal to "123" and "UsageStartDate" should be in range [2015-05-01 TO 2015-05-10] I am expecting something to run like this, curl -X GET http://192.168.1.3:9200/_all/_search -d '{"query" : {"match" : { "PayerAccountId:\"156023466485\" AND UsageStartDate:[2015-01-01 TO 2015-01-10]" }}}' Obviously it's not working any...

Elasticsearch boost per field with function score


elasticsearch,lucene,solr-boost
I have a query with different query data for different fields and ORed results. I also want to favor hits with certain fields. Ideally this would only increase ranking but would not cause results that did not contain some of the terms in the other fields. This would skew results...

How to use arrays in lambda expressions?


c#,elasticsearch,nest
I am writing a program with NEST library of ElasticSearch. I want to write a lambda expression for a function with this argument: HighlighDescriptor<parentdocument> HighlighDescriptor.onFields (param Action<HighlightFieldDescriptor<ParentDocument>>[] fieldHighlighters) I don't know what is the array in the function argument?...

NEST - Using GET instead of POST/PUT for searching


c#,elasticsearch,nest
Is there a way to tell NEST to use GET instead of POST when performing searches? Similar to how the ElasticSearch documentation shows CURL using GET I'd like to use GET when using NEST instead of using POST as it currently does.

Elasticsearch and C# - query to find exact matches over strings


c#,.net,database,elasticsearch,nest
I need a way to search documents using a plain exact match over two or multiple fields which are of type "string" and "integer". I'd like to avoid standard query as I don't care about scoring or best match, just a yes/no outcome if both the fields match or not....

Elasticsearch aggregations over regex matching in a list


regex,elasticsearch
My documents in elasticsearch are of the form { ... dimensions : list[string] ... } I'd like to find all dimensions over all the documents that match a regex. I feel like an aggregation would probably do the trick, but I'm having trouble formulating it. For example, suppose I have...

ElasticSearch REST - insert JSON string without using class


elasticsearch,elastic,elasticsearch-net
I am looking for an example where we can push below sample JSON string to ElasticSearch without using classes in REST api. { "UserID":1, "Username": "Test", "EmailID": "[email protected]" } We get the input as xml and we convert it to JSON string using NewtonSoft.JSON dll. I know REST api is...

Creating Index in Elasticsearch using Java API giving NoClassFoundException


java,indexing,elasticsearch
I'm trying to create a node based client using Java API and index a JSON document. Here's the code : import java.util.Date; import java.util.HashMap; import java.util.Map; import org.elasticsearch.action.deletebyquery.DeleteByQueryResponse; import org.elasticsearch.client.Client; import org.elasticsearch.node.Node; import static org.elasticsearch.node.NodeBuilder.*; public class Els { public static void main (String args[]){ Els p = new Els();...

elasticsearch aggregation group by null key


elasticsearch
here is the data in my elasticsearch server: {"system": "aaa"}, {"system": "bbb"}, {"system": null} I want to get the statistics for system. then I did the query: { "aggs" : { "myAggrs" : { "terms" : { "field" : "system" } } } it gives me the result: { "key":...

ElasticSearch asynchronous post


database,post,asynchronous,elasticsearch,get
I'm posting data on my ElasticSearch database. I've noticed that data is not immediately available, it requires some milliseconds to show up in a GET request. I can live with that (after all, the calls are asynchronous so this behavior is expected) but in my test code I need to...

ElasticSearch- “No query registered for…”


search,indexing,elasticsearch
ElasticSearch returns me "No query registered for [likes_count]" error when trying to look up entries using the following query. The field likes_count is a new field of documents and does not exist in every document. The same query works without the sort part. Why does this error appear? Thanks {...