json,elasticsearch,logstash , Performing searches on JSON data in Elasticsearch


Performing searches on JSON data in Elasticsearch

Question:

Tag: json,elasticsearch,logstash

I have mapped JSON data into Elasticsearch via Logstash which has worked, it has imported the data in and I can see it in Elasticsearch-Head.

My problem is querying the data. I can run a search for a field but it returns the entire type within the index as a single search result. I have tried a few variations but have not had any luck.

Here is the logstash shipper file:

input {
   exec {
     type => "recom_db"
     command => "curl -s -X GET http://www.test.com/api/edselastic/recom_db.json"
     interval => 86400
     codec => "json"
   }
   exec {
     type => "recom_ki"
     command => "curl -s -X GET http://www.test.com/api/edselastic/recom_ki.json"
     interval => 86400
     codec => "json"
   }
   exec {
     type => "recom_un"
     command => "curl -s -X GET http://www.test.com/api/edselastic/recom_un.json"
     interval => 86400
     codec => "json"
   }
}
output {
        elasticsearch {
                host => localhost
                index => "lib-recommender-%{+yyyy.MM.dd}"
                template_name => "recommender-template"
        }
}

and the Elasticsearch index is in the following form:

{
    "_index": "lib-recommender-2015.06.11",
    "_type": "recom_un",
    "_id": "qoZE4aF-SkS--tq_8MhH4A",
    "_version": 1,
    "_score": 1,
    "_source": {
        "item": [{
            "name": "AAM219 -- reading lists",
            "link": "http://www.test.com/modules/aam219.html",
            "description": "AAM219 -- reading lists",
            "terms": {
                "term": ["AAM219"]
            }
        },
        {
            "name": "AAR410 -- reading lists",
            "link": "http://www.test.com/modules/aar410.html",
            "description": "AAR410 -- reading lists",
            "terms": {
                "term": ["AAR410"]
            }
        }
        ...

Anyway so I have tried querying the data the various ways I have seen in the Elasticsearch documentation but am unable to a desired result. Here is one of the many queries I have tried:

curl -XPOST "http://localhost:9200/lib-recommender/recom_un/_search" -d'
{
    "fields": ["item.name", "item.link"],
    "query":{
        "term": {
                "item.terms.term": "AAM219"
                        }
                }
        }
}'

but it returns the entire type within the index (with the correct fields selected but disjointed and all of them):

{
    "took": 13,
    "timed_out": false,
    "_shards": {
        "total": 5,
        "successful": 5,
        "failed": 0
    },
    "hits": {
        "total": 1,
        "max_score": 0.006780553,
        "hits": [{
            "_index": "lib-recommender-2015.06.11",
            "_type": "recom_un",
            "_id": "qoZE4aF-SkS--tq_8MhH4A",
            "_score": 0.006780553,
            "fields": {
                "item.link": ["http://www.test.com/modules/aam219.html",
                "http://www.test.com/modules/aar410.html",
                "http://www.test.com/modules/ac1201.html",
                "http://www.test.com/modules/aca401.html",

I am after the following result:

{
    "took": 5,
    "timed_out": false,
    "_shards": {
        "total": 5,
        "successful": 5,
        "failed": 0
    },
    "hits": {
        "total": 1,
        "max_score": 0.006780553,
        "hits": [{
            "_index": "lib-recommender-2015.06.11",
            "_type": "recom_un",
            "_id": "qoZE4aF-SkS--tq_8MhH4A",
            "_score": 0.006780553,
            "_source": {
                "item": [{
                    "name": "AAM219 -- reading lists",
                    "link": "http://www.test.com/modules/aam219.html",
                    "description": "AAM219 -- reading lists",
                    "terms": {
                        "term": ["AAM219"]
                    }
                }
            }
        }
    }
}

What am I missing? Is the mapping of the index wrong for this kind of search (so should I manually make a mapping file for elasticsearch before importing data). Is there a parameter I am missing in the query? I have been looking for an answer but feel like I am running around in circles now, I am guessing it is something simple that I am overlooking but not sure.


Answer:

Props to Val's answer above. It was mostly what but with another level of nesting. Here is the mapping:

{
  "recom_un": {
    "properties": {
      "item": {
        "type": "nested",
        "properties": {
          "name": {
            "type": "string"
          },
          "link": {
            "type": "string"
          },
          "description": {
            "type": "string"
          },
          "terms": {
            "type": "nested",
            "properties": {
              "term": {
                "type": "string"
              }
            }
          }
        }
      }
    }
  }
}

The search query I used to get what I wanted:

curl -XPOST "http://localhost:9200/lib-recommender/recom_un/_search" -d'
{
  "_source": false,
  "query": {
    "filtered": {
      "filter": {
        "nested": {
          "path": "item",
          "query": {
            "nested": {
              "path": "item.terms",
              "query": {
                "match": {
                  "term": "AAM219"
                }
              }
            }
          },
          "inner_hits": { }
        }
      }
    }
  }
}'

Related:


Response 200 OK but Jquery shows error?


javascript,jquery,json
I am stuck in kind of a weird problem where I am trying to make a JSONP request to the Open Weather history API to get the weather information for the previous years on the a particular date. fetchHistoryUrl = 'http://78.46.48.103/data/3.0/days?day=0622&mode=list&lat=39.77476949&lon=-100.1953125'; $.ajax({ method: 'get', url: fetchHistoryUrl, success: function(response){ console.log(response); },...

KendoUI Grid - Complex JSON with inconsistent keys


javascript,json,kendo-ui,kendo-grid
I know this is probably been asked before, but having tried to find an answer - I am guessing either I am not comprehending some of the answers right, or I am looking at the problem all wrong. I am using a complex SLC loopback query - and the api...

Sencha/Extjs rest call with all parameters


json,rest,extjs,sencha-touch
I'm using ExtJs 5.1.1 and I've written a simple view with a grid, and selecting one row the corresponding model property are editable in some text fields. When editing is completed the button 'save' call Model.save() method, which use the rest proxy configured to write the changes on the server....

Not able to access Kibana running in a Docker container on port 5601


elasticsearch,docker,dockerfile,kibana-4
I have built a docker image with the following Docker file. # gunicorn-flask FROM devdb/kibana MAINTAINER John Doe <[email protected]> ENV DEBIAN_FRONTEND noninteractive RUN apt-get update RUN apt-get install -y python python-pip python-virtualenv gunicorn # Setup flask application RUN mkdir -p /deploy/app COPY gunicorn_config.py /deploy/gunicorn_config.py COPY app /deploy/app RUN pip install...

Nested JSON structure to build side-by-side d3.js charts


javascript,json,d3.js
I'm currently stuck on how to traverse a JSON structure (I created) in order to create side-by-side donut charts. I think I've created a bad structure, but would appreciate any advice. I'm working from Mike Bostock's example here: http://bl.ocks.org/mbostock/1305337 which uses a .csv file for source data. In that example...

do calculation inside JSONArray in Java


java,arrays,json
I have a simple issue but cannot solve it as I am not very good at algorithm! I have an JSONArray in this form: [{"values":[{"time":1434976493,"value":"50"}, {"time":1434976494,"value":"100"}],"counter":"counter1"}, {"values":[{"time":1434976493,"value":"200"}, {"time":1434976494,"value":"300"}],"counter":"counter2"}, {"values":[{"time":1434976493,"value":"400"}, {"time":1434976494,"value":"600"}],"counter":"total"}] What I want to do is to get the integer value for counter 1 and counter 2 and then divide...

How to rearrange CSV / JSON keys columns? (Javascript)


javascript,json,csv,papaparse
I am converting a JSON object array to CSV using Papa Parse JavaScript Library. Is there a way to have the CSV columns arranged in a certain way. For e.g; I get the column as: OrderStatus, canOp, OpDesc, ID, OrderNumber, FinishTime, UOM, StartTime but would like to be arranged as:...

How to use arrays in lambda expressions?


c#,elasticsearch,nest
I am writing a program with NEST library of ElasticSearch. I want to write a lambda expression for a function with this argument: HighlighDescriptor<parentdocument> HighlighDescriptor.onFields (param Action<HighlightFieldDescriptor<ParentDocument>>[] fieldHighlighters) I don't know what is the array in the function argument?...

How use jquery getJSON with a local variable


javascript,jquery,ajax,json,ace-editor
i want to extend the completions of Ace Editor, and i need a json of word, this is my code: function load_completions() { var object = $('#preview').attr('class'); $.ajax({ type: "POST", url: "test_editor_ajax.php", data: {action:'load_completion', object:object}, cache: false, success: function(json){ var adminCompleter = { getCompletions: function(editor, session, pos, prefix, callback) {...

Serializing a java bean into a cookie: Is it bad?


java,json,cookies
In the organization that i work for, there was a serious debate about the following. Scenario: There is a POJO with 6 different properties all are of type Strings. These values need to be persisted as cookies so that it can be picked back when someone does a booking on...

Uncaught error: Invalid type for google table column


javascript,json,google-maps,google-visualization
I recently started using google visualization due to its simplicity. But while implementing in my recent project i seem to have run into a bit of trouble. The program shown is used to pull data from a JSON object and utilize it to display data in the google tables. Now...

How to work with django-rest-framework in the templates


json,django,django-templates,django-rest-framework
Please bear with me. I am just learning django-rest-framework. And I really can't seem to grab it. model: class Day(models.Model): date = models.DateField(default=date.today) class ToDo(models.Model): date = models.ForeignKey(Day) name = models.CharField(max_length=100) very_important = models.BooleanField(default=False) finished = models.BooleanField(default=False) normal view: def home(request): days = Day.objects.all() return render(request, 'test.html', { 'days':days })...

convert from json to array


javascript,php,json
I am trying to get data to display in a table. I don't know what I am doing wrong, but when I get the data from my page it is an array of single characters. I could parse this myself but would prefer to know what I am doing wrong....

Deserializing Json data to c# for use in GridView - Error Data source is an invalid type


c#,json,gridview,serialization,.net-4.5
I am relatively new to working with C# (v5.0) and Json data (Newtonsoft.Json v6.0) and am seeking assistance in resolving an error when attempting to populate a .Net 4.5 GridView control. My sample Json data as returned by a web service is: { PersonDetails: [ { Book: "10 ", FirstName:...

Bad scoring due to different maxDocs of IDF


elasticsearch
I have two documents with a field title of: News New Website If I search for the term new website the score for the News document is much higher than the other one which is obviously not what I want. I wrapped an explain around it and got: 'hits': [{'_explanation':...

Replacing elements in an HTML file with JSON objects


javascript,json,replace
I am writing some Javascript to: Read in and Parse a JSON file Read in an HTML file (created as a template file) Replace elements in the HTML file with elements from the JSON file. It is only replacing the first element obj.verb. Can someone please help me figure out...

String comparison in AngularJS


javascript,json,angularjs,ionic-framework,string-comparison
I'm trying to compare two strings in AngularJS, and I've seen examples online. As I understand it, you can use angular.equals(str1, str2), you can use ===, you can use == if you're sure that both are strings... I've tried all three, but I don't get the result. Something must be...

Get document on some condition in elastic search java API


java,elasticsearch,elasticsearch-plugin
As I know we can parse document in elastic search, And when we search for a keyword, It will return the document using this code of java API:- org.elasticsearch.action.search.SearchResponse searchHits = node.client() .prepareSearch() .setIndices("indices") .setQuery(qb) .setFrom(0).setSize(1000) .addHighlightedField("file.filename") .addHighlightedField("content") .addHighlightedField("meta.title") .setHighlighterPreTags("<span class='badge badge-info'>") .setHighlighterPostTags("</span>") .addFields("*", "_source")...

Why i get can not resolve method error in class android?


android,json
I use this tutorial: JSON and part of code to this: protected String doInBackground(String... urls) { TextView lbl=(TextView) findViewById(R.id.provCODE); person = new Person(); person.setName(lbl.getText().toString()); //person.setCountry("1853"); //person.setTwitter("1892"); return POST(urls[0],person); } but in the this line: person.setName(lbl.getText().toString()); i get this error: Can not resolve method. How can i solve that?what happen? my...

Link to another resource in a REST API: by its ID, or by its URL?


json,api,rest,api-design,hateoas
I am creating some APIs using apiary, so the language used is JSON. Let's assume I need to represent this resource: { "id" : 9, "name" : "test", "customer_id" : 12, "user_id" : 1, "store_id" : 3, "notes" : "Lorem ipsum example long text" } Is it correct to refer...

Can't save json data to variable (or cache) with angularjs $http.get


json,angularjs,web-services,rest
I have weird angularjs problem. I'm trying to fetch data from Rest Webservice. It works fine, but I can't save json data to object. My code looks like: services.service('customerService', [ '$http', '$cacheFactory', function($http, $cacheFactory) { var cache = $cacheFactory('dataCache'); var result = cache.get('user'); this.getById = function(id){ $http.get(urlList.getCustomer + id).success(function(data, status,...

Sorting in Ruby on rails


ruby-on-rails,json,sorting
I'm very new to Ruby on rails. I try to edit the following api. I want to sorting "can_go" which is true are shown at the top of list. I added this row before sending data, but the result is still order by "user_id". user_infos.sort { |a, b| - (a['can_go']<=>b['can_go'])...

Codeigniter Select JSON, Insert JSON


json,codeigniter,select,insert,routing
I have very simple users database: user_id, user_name, user_email My model this: class Users extends CI_Model { private $table; private $table_fields; private $table_fields_join; function __construct() { parent::__construct(); $this->table = 'users'; $this->table_fields = array( $this->table.'.user_id', $this->table.'.user_name', $this->table.'.user_email' ); $this->table_fields_join = array(); } function select(){ $this->db->select(implode(', ', array_merge($this->table_fields, $this->table_fields_join)));...

Call function on Server from iOS app - Objective C


ios,objective-c,json,server,backend
If you are familiar with Parse.com's Javascript SDK, this is what I am trying to do for my own server for my iOS app (Objective-c). I want to be able to send some a string to the function that is on my server, have the server run its function and...

NEST ElasticSearch.NET Escape Special Characters


c#,elasticsearch,nest
I have been experimenting with the use of the NEST client for Elastic Search, but seem to have hit a barrier when filtering on a term which contains special/reserved characters such as '/' Below is a JSON representation of my model.. "categories": { "count": 1, "default": "root/Hello/World/Category", } When submitting...

access the json encoded object returned by php in jquery


php,jquery,ajax,json
I want to post some data to php function by ajax, then get the encoded json object that the php function will return, then I want to get the information (keys and values) from this object, but I don't know how, here is my code: $.ajax({ url: "functions.php", dataType: "JSON",...

Getting front value from array/json


php,arrays,json
I want to get my image value wich could be images/thispicture.png for example. I have an Array or Json with some values, but the problem is i only want the image from array. This is what i get now from the array: {"image_intro":"images\/16748007940_1cc8f06622_small.jpg","float_intro":"","image_intro_alt":"","image_intro_caption":"","image_fulltext":"","float_fulltext":"","image_fulltext_alt":"","image_fulltext_caption":""}"><img...

JQuery mutiple post in the same time


javascript,php,jquery,json
I have three functions ,and every function post to special php page to get data.. and every function need some time because every php script need some time .. function nb1() { $.post("p1.php", { action: 1 }, function(data) { console.log(data); }, "json") .fail(function(data) { console.log("error"); }); } function nb2() {...

How to parse JSON array of string arrays


c#,json
Lets say I get the following json data from a web service which I can't change. [ [ "Header1", "Header2", "Header3", "Header4" ], [ "FirstValue1", "FirstValue2", "FirstValue3", "FirstValue4" ], [ "SecondValue1", "SecondValue2", "SecondValue3", "SecondValue4" ] ] jsonlint.com tells me that it is valid json and from what I know I...

php array returns undefined but print_r shows otherwise [duplicate]


php,arrays,json,undefined
This question already has an answer here: PHP: “Notice: Undefined variable” and “Notice: Undefined index” 11 answers foreach($jsonmediationinfo as $value1) { echo $value1['mstatus']; print_r($jsonmediationinfo); } Output: 1Array ( [0] => Array ( [mstatus] => 1 [mhearingnum] => first [mminutes] => adakjaflafjlarjkelfkalfkd;la ) [1] => Array ( [mhearingnum] => second...

elastic search sort in aggs by column


sorting,elasticsearch,group-by,order
I am trying to sort in elastic search in aggs, equivalent in mysql "ORDER BY Title ASC/DESC". Here is the index structure: 'body' => array( 'mappings' => array( 'test_type' => array( '_source' => array( 'enabled' => true ), 'properties' => array( 'ProductId' => array( 'type' => 'integer', 'index' => 'not_analyzed'...

Use JSON file to insert data in database


javascript,json,mongodb,meteor,data
I'm using my JSON file like this to insert data in my collection : var content = JSON.parse(Assets.getText('test.json')); console.log('inserting...'); Profiles.insert({ user: id, data:content }; But I would like to have a "data's tree" like that : [ user: "rtegert23423131", firstname:"test", surname:"test2", // ... ] Not like that : [ user:...

PHP how to extract info from json string wtihout knowing the keys


php,json
I am querying a service if a person have phone number(s) (also maybe not). I have a json string (as return value) like the following: $json = '{"data":[{"tel1":"1102"},{"tel2":"3220"}],"found":true}'; I convert this string to json_decode() function. $jd = json_decode($json); Then I want to get the phone numbers only into an array...

'utf8' codec can't decode byte 0xf3


python,json,character-encoding
I am using python 2.7 to read the json file. my code is import json from json import JSONDecoder import os path=os.path.dirname(os.path.abspath(__file__))+'/json' print path for root, dirs, files in os.walk(os.path.dirname(path+'/json')): for f in files: if f.lower().endswith((".json")): fp=open(root + '/'+f) data = fp.read() print data.decode('utf-8') But I got the following error....

LineChart with C3 using JSON


json,c3
I can't understand because my chart is not working properly. Anything is displayed. Below my simple code: <div id="chartc3"></div> <script> var scene; $.getJSON('assets/json/chartc3.json', function(data) { scene=data; var chart = c3.generate({ bindto: '#chartc3', data: { json: scene, keys: { value: ['round','val'], } } }); }); </script> and the following json file:...

How can I get json objects without the object number?


javascript,jquery,json,rest
I have a simple json object that spits out 4 items that have completely different properties inside each one. I have got the json being displayed with the 4 objects that are called meta.work_content like so: [Object, Object, Object, Object] I can open these in console and see the objects...

Check for duplicates in JSON


javascript,jquery,json,duplicates
I have a json object: var object1 = [{ "value1": "1", "value2": "2", "value3": "3", }, { "value1": "1", "value2": "5", "value3": "7", }, { "value1": "6", "value2": "9", "value3": "5", }, { "value1": "6", "value2": "9", "value3": "5", }] Now I want to take each record out of that...

Create n:m objects using json and sequelize?


javascript,json,node.js,sequelize.js
I am trying to learn sequelize, but am having trouble getting a n:m object created. So far I have my 2 models that create 3 tables (Store, Product, StoreProducts) and the models below: models/Store.js module.exports = (sequelize, DataTypes) => { return Store = sequelize.define('Store', { name: { type: DataTypes.STRING, },...

Parsing Google Custom Search API for Elasticsearch Documents


json,python-2.7,elasticsearch,google-search-api
After retrieving results from the Google Custom Search API and writing it to JSON, I want to parse that JSON to make valid Elasticsearch documents. You can configure a parent - child relationship for nested results. However, this relationship seems to not be inferred by the data structure itself. I've...

How to use ajax response JSON data?


javascript,jquery,ajax,json
I have made my ajax request, which works great and returns JSON. But how do i use it? My response is something like this [{ "id": "5", "reviewID": "2389", "serviceID": "50707", "title": "well done" }] Now in my success function I am trying to use the data like such: success:...

Insert data in collection at Meteor's startup


javascript,json,meteor,data,startup
I would like to insert data at Meteor's startup. (And after from a JSON file) At startup, I create a new account and I would like to insert data and link it to this account once this one created. This is the code that creates the new account at startup:...

How to transform $.post to $.ajax?


javascript,jquery,ajax,json,servlets
I have this $.post peace of code: $.post("../admin-login", { dataName:JSON.stringify({ username:uname, password:pass, }) }, function(data,status){ console.log("Data:"+data); answer = data; } ); and I wont to transform it to $.ajax. On the servlet side I am demanding request.getParamter("dataName") but I do not know how to write data: section in $.ajax so...

why i don't get return value javascript


javascript,jquery,html,json,html5
When i debug my code i can see i have value but i don't get value to createCheckBoxPlatform FN function createCheckBoxPlatform(data) { var platform = ""; $.each(data, function (i, item) { platform += '<label><input type="checkbox" name="' + item.PlatformName + ' value="' + item.PlatformSK + '">' + item.PlatformName + '</label>' +...

SwiftyJSON Reading JSON Array issue


json,swift,swifty-json
I'm trying to read a json array data as below with my code but I'm getting empty result all the time while I can print the whole data. Please where would be my issue? I've read this topic but I couldn't get the result. Topic Link {"Cars":[{"Brand":"Alfa Romeo"},{"Brand":"Audi"},{"Brand":"BMW"},{"Brand":"Citroen"},{"Brand":"Dacia"},{"Brand":"Fiat"},.......... My code...

Complicated Json processing - PHP


php,json
I am using this JSON: http://steamcommunity.com/id/mitch8910/inventory/json/730/2/ I am sorting through the "rgDescriptions" data with this: $data = file_get_contents('http://steamcommunity.com/id/mitch8910/inventory/json/730/2/'); $json = json_decode($data); foreach ($json->rgDescriptions as $mydata) { //checking and adding data to a database } Each section in "rgDescriptions" contains a lot of information on a certain item. For each item...

Fatal error catched by register_shutdown_function and update json_encode


php,json,fatal-error
I can catch a fatal error with register_shutdown_function('shutdown') function shutdown(){ $error = error_get_last(); $result['message'] = $error['message']; } and I have an echo json_encode($result); Unfortunately this echo json_encode shows nothing because json is not updated with the message of the fatal error. What can I do to fix this?...

NEST - Using GET instead of POST/PUT for searching


c#,elasticsearch,nest
Is there a way to tell NEST to use GET instead of POST when performing searches? Similar to how the ElasticSearch documentation shows CURL using GET I'd like to use GET when using NEST instead of using POST as it currently does.

Parse JSON output from AlamoFire


json,swift,nsurlrequest
I am using the latest Alamofire to manage a GET http request to my server. I am using the following to GET and parse the JSON: Alamofire.request(.GET, "*******") .responseJSON {(request, response, JSON, error) in if let statusesArray = JSON as? NSArray{ if let aStatus = statusesArray[0] as? NSDictionary{ //OUTPUT SHOWN...

Deserializing JSON into c# class


c#,json,couchdb
I have a JSON in the following format (from couchDB view) {"rows":[ {"key":["2015-04-01","524","http://www.sampleurl.com/"],"value":1}, {"key":["2015-04-01","524","http://www.sampleurl2.com/"],"value":2}, {"key":["2015-04-01","524","http://www.sampleurl3.com"],"value":1} ]} I need to create a "service" to get this data from couchDB and insert it on SQL Server (for generating reports..) in a efficient way. My first bet was to bulk insert this json...