elasticsearch,nested , 'Associated' data in ElasticSearch

'Associated' data in ElasticSearch


Tag: elasticsearch,nested

For an ecommerce platform, we're looking to index products. Default fields are simple as: name_en, name_de, name_fr, description. But, price and stock are dependant on another value:

I was thinking of nested objects, but is that even an option?

- name_en: Product A
- description_en: Product A description
- webshops: [{
    - key: webshop_id
      value: 1
    - key: price
      value: 1.99
    - key: stock
      value: 10
    - key: categories
      value: [1, 10, 50]
    - key: webshop_id
      value: 2
    - key: price
      value: 5.99
    - key: stock
      value: 5
    - key: categories
      value: [9, 90, 500]

Is it easy querying like this? Can we easily get the entire document, with the values where webshop.key.webshop_id.value = 1, or webshop.key.categories.value = 500?

Is my thinking wrong, any pointers in the right direction?


You can nest as you did, but it will get difficult to update the price or stock of a product in a single webshop, because you'll have to reindex the whole webshops array. There are ways to around it, but that's convoluted.

Instead of having a nested structure, you can also denormalize the webshop part and simply include the price, stock and categories fields in the documents like this.

Document 1:
- name_en: Product A
- description_en: Product A description
- webshop_id: 1
- price: 1.99
- stock: 10
- categories: [1, 10, 50]

Document 2:
- name_en: Product A
- description_en: Product A description
- webshop_id: 2
- price: 5.99
- stock: 5
- categories: [9, 90, 500]

Then in your queries you can simply add a constraint for webshop = 1 or webshop = 2 (or both) depending on which webshop you're querying against. It's also much easier to update the price, stock and categories of a product in a specific shop, all you have to do is update the corresponding document.

This means that your product data (name, description, etc) will be copied once per webshop but it's not a big deal usually (pretty common in the NoSQL world), you just have to update 2 documents instead of a single, but _bulk will help there. At least, when you add new webshops, you don't need to reindex all your data (!!!) and you change the prices, stocks in one webshop without interfering with the others.


How to iterate an object and build a new one

I am using JavaScript to iterate customers in an object which is constructed as follows: [ { "customer": "Customer 1", "project": "1" }, { "customer": "Customer 2", "project": "2" }, { "customer": "Customer 2", "project": "3" } ] I wish to build a new object in iteration which would be...

Nested promises in ui-router resolve

I have been struggling for a while with the following problem: I would like to get some data before the view shows, (resolve). But some of the data is dependend on the result of another promise. It goes like this: I get the job id (index) from the $stateParams and...

How to compute the scores based on field data in elasticsearch

I have the following fields in documents { name: "Pearl", age : 43, weight: 54, bodyWeight : 103, height : 1.8 } Now i want to get scores for the documents based on the bodyWeight to height ratio of the documents. How to do that?...

Operator '??' cannot be applied to operands of type IQueryContainer and lambda expression

I am trying to create a method to process a certain query. I follow an example posted on the Nest repository (line 60), but still the MatchAll is not recognized by the compiler and if I try to build the solution, the error that shows is: Operator '??' cannot be...

ElasticSearch - Configuration to Analyse a document on Indexing

In a single request, I want to retrieve documents from a SOR, store them in ElasticSearch, and then search those documents using the ES search API. There seems to be some lag from the time the document is indexed and the time it is analyzed and ready to be searched....

How to write search queries in kibana using Query DSL for Elasticsearch aggregation

I am working on ELK stack to process Apache access logs. Spent a lot of time understanding Query DSL format so that more complex queries can be written. Currently am facing issues with running the queries in kibana interface but the same queries work just fine when posted using curl...

Angular ui.router. Deep nested routes

Here is an example to check http://embed.plnkr.co/uVMlkk/preview When we navigate to 'page2' route there is a 'hey, I'm a subroute' note. But once we navigate anywhere else that note will disappear forever. The goal is to make some nested states to be shown right away (as a default ones). I...

elasticsearch aggregation group by null key

here is the data in my elasticsearch server: {"system": "aaa"}, {"system": "bbb"}, {"system": null} I want to get the statistics for system. then I did the query: { "aggs" : { "myAggrs" : { "terms" : { "field" : "system" } } } it gives me the result: { "key":...

Elasticsearch and C# - query to find exact matches over strings

I need a way to search documents using a plain exact match over two or multiple fields which are of type "string" and "integer". I'd like to avoid standard query as I don't care about scoring or best match, just a yes/no outcome if both the fields match or not....

Parsing Google Custom Search API for Elasticsearch Documents

After retrieving results from the Google Custom Search API and writing it to JSON, I want to parse that JSON to make valid Elasticsearch documents. You can configure a parent - child relationship for nested results. However, this relationship seems to not be inferred by the data structure itself. I've...

How to have multiple regex based on or condition in elasticsearch?

I want to get all 000ANT and 0BBNTA from id, is there something similar to terms which works with regexp or is there any other way? Otherwise I will have to query elasticsearch for each item say 000ANT and 0BBNTA. Please help. Below is something that I am trying out...

Re-index object with new fields

It seems like as long as the id field is maintained, its super easy to re-index a document by simply calling Index(), but is there a way to given an object was updated and new fields were added, to have it include these new fields in the index? I'm not...

decltype does not resolve nested vectors. How can I use templates for nested vectors?

I am trying to overload the + operator to deal with nested vectors. I thought the function would call itself until the nested vectors resolve to basic types, but instead I get a huge list of errors when I compile it. My vector operations that I have defined work for...

nested ajax parent completed before child had finished

$.ajax( { type: 'POST', url: '@Url.Action("EkranListeleSektor")', dataType: 'json', async: false, //added async later, no changes data: { sektorID: $("#ekranSektorID").val() }, success: function (items) { $.each(items, function (i, ekranlar) { $.ajax({ type: 'POST', url: '@Url.Action("IlAdGetir")', dataType: 'json', data: { id: ekranlar.il }, success: function (ilAd) { strIlAd = ilAd; }, error:...

Defining a “vector_cast” template function that works with a variable amount of nested vectors

I want to be able to do something like this for example. std::vector<std::vector<int>> e = {{ 1 },{ 2 }}; std::vector<std::vector<double>> q = {{1.5},{2.5}}; q = vector_cast<double>(e); But also have vector_cast be able to handle any number of nested vectors. Here is my attempt at this. The code works if...

Not able to access Kibana running in a Docker container on port 5601

I have built a docker image with the following Docker file. # gunicorn-flask FROM devdb/kibana MAINTAINER John Doe <[email protected]> ENV DEBIAN_FRONTEND noninteractive RUN apt-get update RUN apt-get install -y python python-pip python-virtualenv gunicorn # Setup flask application RUN mkdir -p /deploy/app COPY gunicorn_config.py /deploy/gunicorn_config.py COPY app /deploy/app RUN pip install...

How to read data in logs using logstash?

I have just started log stash, i have log files in that log file whole object is printed in the logs, Since my object is huge i cant write the grok patterns to the whole object and also i expecting only two values out of those object. Can you please...

show values of parent row in ng-repeat

I'm a bit stuck here i have an array of 'parcel' objects, but i want to display a value of the parent object of this parcel how can i achieve this? i have created this fiddle: i would like to show also the buyer var myApp = angular.module('myApp', []); myApp.filter('createarray',...

ElasticSearch (Nest) Terms sub aggregation of Terms - Not working as intended

Taking the following mapping in account : { "person": { "properties": { "id": { "type": "string" }, "name": { "type": "string" }, ... "trainings": { "properties": { "attendanceDate": { "type": "date", "format": "dateOptionalTime" }, "providerId": { "type": "string", "index": "not_analyzed" }, "trainingId": { "type": "string", "index": "not_analyzed" } ... }...

indexing names in json using elasticsearch in couchdb

I am trying to implement full-text query for my json documents. I want to search by title. My json is as follows: { "release":{ "genres":{ "genre":"Electronic" }, "identifiers":{ "identifier":[ { "description":"A-Side", "value":"MPO SK 032 A1 G PHRUPMASTERGENERAL T27 LONDON", "type":"Matrix / Runout" }, { "description":"B-Side", "value":"MPO SK 032 B1", "type":"Matrix...

NEST ElasticSearch.NET Escape Special Characters

I have been experimenting with the use of the NEST client for Elastic Search, but seem to have hit a barrier when filtering on a term which contains special/reserved characters such as '/' Below is a JSON representation of my model.. "categories": { "count": 1, "default": "root/Hello/World/Category", } When submitting...

Is there a syntax for incomplete nested type without a forward declaration?

The following program produces a diagnostic error. #include <memory> class Containing { // class Nested; // [1]: This line seems required. typedef std::shared_ptr<class Nested> Ptr; class Nested { Ptr & ptr (); void foo (const Ptr &p) { p->ptr() = ptr()->ptr(); // [2]: Error here without [1] } }; };...

Strange behaviour of limit in Elasticsearch

I tried two queries. First one looks like this (it simply lists all data): # listing 1 from elasticsearch import Elasticsearch from elasticsearch_dsl import Search, Q, F .... .... connection etc s = Search(using=db,index="reestr") rows = s.execute() for r in rows: print(r) listing 1 prints out all documents from the...

Get document on some condition in elastic search java API

As I know we can parse document in elastic search, And when we search for a keyword, It will return the document using this code of java API:- org.elasticsearch.action.search.SearchResponse searchHits = node.client() .prepareSearch() .setIndices("indices") .setQuery(qb) .setFrom(0).setSize(1000) .addHighlightedField("file.filename") .addHighlightedField("content") .addHighlightedField("meta.title") .setHighlighterPreTags("<span class='badge badge-info'>") .setHighlighterPostTags("</span>") .addFields("*", "_source")...

Elasticsearch aggregations over regex matching in a list

My documents in elasticsearch are of the form { ... dimensions : list[string] ... } I'd like to find all dimensions over all the documents that match a regex. I feel like an aggregation would probably do the trick, but I'm having trouble formulating it. For example, suppose I have...

Elasticsearch NumberFormatException when running two consecutive java tests

I have two test in a class, each of them containing the following query: SearchQuery searchQuery = new NativeSearchQueryBuilder().withQuery(matchAllQuery()).withFilter(rangeFilter("publishDate").lt(date)).build(); In one of the tests, the number of the results elasticsearchTemplate.count(searchQuery, Article.class), in the other one the returned values are verified elasticsearchTemplate.queryForPage(searchQuery,Article.class) If I run any of these two tests separately,...

NEST - Using GET instead of POST/PUT for searching

Is there a way to tell NEST to use GET instead of POST when performing searches? Similar to how the ElasticSearch documentation shows CURL using GET I'd like to use GET when using NEST instead of using POST as it currently does.

Elasticsearch - Query document missing an array value

I would like to query my elasticsearch index in order to retrieve the documents that don't contain a specific value in an array. For instance, if my query is : { "query": { "bool": { "must": [ { "match_all": {} } ], "must_not": [], "should": [] } }, "from": 0,...

ElasticSearch- “No query registered for…”

ElasticSearch returns me "No query registered for [likes_count]" error when trying to look up entries using the following query. The field likes_count is a new field of documents and does not exist in every document. The same query works without the sort part. Why does this error appear? Thanks {...

Javascript: Altering an object where dot notation is used [duplicate]

This question already has an answer here: How to access object properties containing special characters? 1 answer I'm building an Elasticsearch search interface. My method is to build the initial query object, and then alter it depending on the user input. In the filter part of my object, I...

Query returns both documents instead of just one

var res = esclient.Search<MyClass>(q => q .Query(fq => fq .Filtered(fqq => fqq .Query(qq => qq.MatchAll()) .Filter(ff => ff .Bool(b => b .Must(m1 => m1.Term("macaddress", "mac")) .Must(m2 => m2.Term("another_field", 123)) ) ) ) ) ); As far as I can understand the bool and must together are the equivalent of the...

Elasticsearch standard analyser stopwords

I am trying to guess what is the default stopwords list in standard analyzer in elasticsearch. I run version 1.3.1, and it seems to me that the English list is used, because running a wildcard query like this { "wildcard" : { "name" : { "wildcard" : "*in*" } }...

ElasticSearch: How to search on different fields that are not related that are arrays of objects

I want to search on different fields that are not related that are arrays of objects. I cannot find out how. Given the following mapping and data entry: I want to give the user the ability to search all possible fields in any combination. The user would use a form...

Get elasticsearch result based on two keys

I want to get all docs who's "PayerAccountId" should equal to "123" and "UsageStartDate" should be in range [2015-05-01 TO 2015-05-10] I am expecting something to run like this, curl -X GET -d '{"query" : {"match" : { "PayerAccountId:\"156023466485\" AND UsageStartDate:[2015-01-01 TO 2015-01-10]" }}}' Obviously it's not working any...

How to use parameters from data frame in R and loop through time holding them constant

I have a function (weisurv) that has 2 parameters - sc and shp. It is a function through time (t). Time is a sequence, i.e. t<-seq(1:100). weisurv<-function(t,sc,shp){ surv<-exp(-(t/sc)^shp) return(surv) } I have a data frame (df) that contains a list of sc and shp values (like 300+ of them). For...

MultiMatch query with Nest and Field Suffix

Using Elasticsearch I have a field with a suffix - string field with a .english suffix with an english analyser on it as shown in the following mapping ... "valueString": { "type": "string", "fields": { "english": { "type": "string", "analyzer": "english" } } } ... The following query snippet won't...

How to use arrays in lambda expressions?

I am writing a program with NEST library of ElasticSearch. I want to write a lambda expression for a function with this argument: HighlighDescriptor<parentdocument> HighlighDescriptor.onFields (param Action<HighlightFieldDescriptor<ParentDocument>>[] fieldHighlighters) I don't know what is the array in the function argument?...

Elasticsearch boost per field with function score

I have a query with different query data for different fields and ORed results. I also want to favor hits with certain fields. Ideally this would only increase ranking but would not cause results that did not contain some of the terms in the other fields. This would skew results...

ElasticSearch asynchronous post

I'm posting data on my ElasticSearch database. I've noticed that data is not immediately available, it requires some milliseconds to show up in a GET request. I can live with that (after all, the calls are asynchronous so this behavior is expected) but in my test code I need to...

How to get duplicate field values in elastic search by field name without knowing its value

I have a field "EmployeeName" in an elastic search index - and I would like to execute a query that will return me all the cases where there are duplicate values of "EmployeeName". Can this be done? I found more_like_this but this requires field value for "like_text". But my requirement...

ElasticSearch REST - insert JSON string without using class

I am looking for an example where we can push below sample JSON string to ElasticSearch without using classes in REST api. { "UserID":1, "Username": "Test", "EmailID": "[email protected]" } We get the input as xml and we convert it to JSON string using NewtonSoft.JSON dll. I know REST api is...

Inject a template in parent's view with ui-router

I have an abstract state declared as $stateProvider.state('myapp', { abstract: true, templateUrl : 'index.html' }); And all the <ui-view> in the 'index.html' template will be populated by the children states: $stateProvider.state('myapp.home', { url : "/", views : { "main" : { controller : 'HomeCtrl', templateUrl : 'includes/home.html' } } });...

Elasticsearch: How to query using partial phrases in quotation marks

I am trying to implement a search behavior that supports partial phrases. A possible search input could look like this: example "hello world" elasticsearch Now I want to get all documents, that contain the words example and elasticsearch as well as the phrase hello world. As this is a very...

Nested animation in iOS: combine animated and non-animated code in animateWithDuration?

Nested animation In short the nested animation allows to run 2 independent animations simultaneously. But what to do if I have a complex code which consists of both animated and non-animated code? I mean the following situation: [UIView animateWithDuration:... animations:^{ ...//code1 - with animation ...//code2 - also contains code which...

elastic search sort in aggs by column

I am trying to sort in elastic search in aggs, equivalent in mysql "ORDER BY Title ASC/DESC". Here is the index structure: 'body' => array( 'mappings' => array( 'test_type' => array( '_source' => array( 'enabled' => true ), 'properties' => array( 'ProductId' => array( 'type' => 'integer', 'index' => 'not_analyzed'...

Elasticsearch geospatial search, problems with index setup

I'm trying to search for documents previously added to an index, which has been configured to allow geospatial queries (or so I think). My elasticsearch instance is hosted on qbox.io. This is the code I wrote to create an index from the command line curl -XPOST username:[email protected]/events -d '{ "settings"...

Bad scoring due to different maxDocs of IDF

I have two documents with a field title of: News New Website If I search for the term new website the score for the News document is much higher than the other one which is obviously not what I want. I wrapped an explain around it and got: 'hits': [{'_explanation':...

Docker container http requests limit

I'm new to Docker so, most likely, I'm missing something. I'm running a container with Elasticsearch, using this image. I'm able to setup everyhing correctly. After that I was a using a script developed by a collegue in order to insert some data, basically querying a MySQL database and making...

Partial specialization of nested template types produces “internal error” in VC++ 2012 compiler

In the following example, I have an object class which defines a reference class. Both accept mutability as a template argument. In the case that 'obj' is 'Const', I want to disallow a reference of type 'Non_Const'. The example produces the ambiguous message, "An internal error has occurred in the...

Creating a number list with nested For loops in Python

I've been working on this now for well over four hours and i've tried to check several resources. I'm trying to get something like this: 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4...