json,parsing,elasticsearch,logstash,dsl , Need to extract the timestamp from a logstash elasticsearch cluster


Need to extract the timestamp from a logstash elasticsearch cluster

Question:

Tag: json,parsing,elasticsearch,logstash,dsl

I'm trying to determine the freshness of the most recent record in my logstash cluster, but I'm having a bit of trouble digesting the Elasticsearch DSL.

Right now I am doing something like this to extract the timestamp: curl -sX GET 'http://localhost:9200/logstash-2015.06.02/' -d'{"query": {"match_all": {} } }' | json_pp | grep timestamp

which gets me; "@timestamp" : "2015-06-02T00:00:28.371+00:00",

I'd like to use an elasticsearch query directly with no grep hackiness.

The raw JSON (snipped for length) looks like this:

{
   "took" : 115,
   "timed_out" : false,
   "hits" : {
      "hits" : [
         {
            "_index" : "logstash-2015.06.02",
            "_source" : {
               "type" : "syslog",
               "@timestamp" : "2015-06-02T00:00:28.371+00:00",
               "tags" : [
                  "sys",
                  "inf"
               ],
               "message" : "    2015/06/02 00:00:28 [INFO] serf: EventMemberJoin: generichost.example.com 10.1.1.10",
               "file" : "/var/log/consul.log",
               "@version" : 1,
               "host" : "generichost.example.com"
            },
            "_id" : "AU4xcf51cXOri9NL1hro",
            "_score" : 1,
            "_type" : "syslog"
         },
      ],
      "total" : 8605141,
      "max_score" : 1
   },
   "_shards" : {
      "total" : 50,
      "successful" : 50,
      "failed" : 0
   }
}

Any help would be appreciated. I know the query is simple, I just don't know what it is.


Answer:

You don't need to use the DSL for this. You can simply cram everything into the URL query string, like this:

curl -s XGET 'localhost:9200/logstash-2015.06.02/[email protected]&size=1&[email protected]:desc&format=yaml'

So:

The output would look like this:

- _index: "logstash-2015.06.02"
  _type: "syslog"
  _id: "AU4xcf51cXOri9NL1hro"
  _score: 1.0
  _source:
    @timestamp: "2015-06-02T00:00:28.371+00:00"

You don't need json_pp anymore, you can still simply grep @timestamp to get the data you need.

Note that in 1.6.0, there will be a way to filter out all the metadata (i.e. _index, _type, _id, _score) and only get the _source for a search result using the filter_path parameter in the URL.


Related:


'utf8' codec can't decode byte 0xf3


python,json,character-encoding
I am using python 2.7 to read the json file. my code is import json from json import JSONDecoder import os path=os.path.dirname(os.path.abspath(__file__))+'/json' print path for root, dirs, files in os.walk(os.path.dirname(path+'/json')): for f in files: if f.lower().endswith((".json")): fp=open(root + '/'+f) data = fp.read() print data.decode('utf-8') But I got the following error....

php array returns undefined but print_r shows otherwise [duplicate]


php,arrays,json,undefined
This question already has an answer here: PHP: “Notice: Undefined variable” and “Notice: Undefined index” 11 answers foreach($jsonmediationinfo as $value1) { echo $value1['mstatus']; print_r($jsonmediationinfo); } Output: 1Array ( [0] => Array ( [mstatus] => 1 [mhearingnum] => first [mminutes] => adakjaflafjlarjkelfkalfkd;la ) [1] => Array ( [mhearingnum] => second...

Display only values containing specific word in array


php,arrays,json,string,if-statement
Right now I am working on a search function for my json decoded results. The code that I got looks like this: <?php foreach($soeg_decoded as $key => $val){ $value = $val["Value"]; $seotitle = $val["SEOTitle"]; $text = $val["Text"]; echo '<tr><td>' . $value . '</td><td>' . $seotitle . '</td><td>' . $text ....

Codeigniter Select JSON, Insert JSON


json,codeigniter,select,insert,routing
I have very simple users database: user_id, user_name, user_email My model this: class Users extends CI_Model { private $table; private $table_fields; private $table_fields_join; function __construct() { parent::__construct(); $this->table = 'users'; $this->table_fields = array( $this->table.'.user_id', $this->table.'.user_name', $this->table.'.user_email' ); $this->table_fields_join = array(); } function select(){ $this->db->select(implode(', ', array_merge($this->table_fields, $this->table_fields_join)));...

do calculation inside JSONArray in Java


java,arrays,json
I have a simple issue but cannot solve it as I am not very good at algorithm! I have an JSONArray in this form: [{"values":[{"time":1434976493,"value":"50"}, {"time":1434976494,"value":"100"}],"counter":"counter1"}, {"values":[{"time":1434976493,"value":"200"}, {"time":1434976494,"value":"300"}],"counter":"counter2"}, {"values":[{"time":1434976493,"value":"400"}, {"time":1434976494,"value":"600"}],"counter":"total"}] What I want to do is to get the integer value for counter 1 and counter 2 and then divide...

Serializing a java bean into a cookie: Is it bad?


java,json,cookies
In the organization that i work for, there was a serious debate about the following. Scenario: There is a POJO with 6 different properties all are of type Strings. These values need to be persisted as cookies so that it can be picked back when someone does a booking on...

Link to another resource in a REST API: by its ID, or by its URL?


json,api,rest,api-design,hateoas
I am creating some APIs using apiary, so the language used is JSON. Let's assume I need to represent this resource: { "id" : 9, "name" : "test", "customer_id" : 12, "user_id" : 1, "store_id" : 3, "notes" : "Lorem ipsum example long text" } Is it correct to refer...

Insert data in collection at Meteor's startup


javascript,json,meteor,data,startup
I would like to insert data at Meteor's startup. (And after from a JSON file) At startup, I create a new account and I would like to insert data and link it to this account once this one created. This is the code that creates the new account at startup:...

PHP how to extract info from json string wtihout knowing the keys


php,json
I am querying a service if a person have phone number(s) (also maybe not). I have a json string (as return value) like the following: $json = '{"data":[{"tel1":"1102"},{"tel2":"3220"}],"found":true}'; I convert this string to json_decode() function. $jd = json_decode($json); Then I want to get the phone numbers only into an array...

LineChart with C3 using JSON


json,c3
I can't understand because my chart is not working properly. Anything is displayed. Below my simple code: <div id="chartc3"></div> <script> var scene; $.getJSON('assets/json/chartc3.json', function(data) { scene=data; var chart = c3.generate({ bindto: '#chartc3', data: { json: scene, keys: { value: ['round','val'], } } }); }); </script> and the following json file:...

SwiftyJSON Reading JSON Array issue


json,swift,swifty-json
I'm trying to read a json array data as below with my code but I'm getting empty result all the time while I can print the whole data. Please where would be my issue? I've read this topic but I couldn't get the result. Topic Link {"Cars":[{"Brand":"Alfa Romeo"},{"Brand":"Audi"},{"Brand":"BMW"},{"Brand":"Citroen"},{"Brand":"Dacia"},{"Brand":"Fiat"},.......... My code...

Getting front value from array/json


php,arrays,json
I want to get my image value wich could be images/thispicture.png for example. I have an Array or Json with some values, but the problem is i only want the image from array. This is what i get now from the array: {"image_intro":"images\/16748007940_1cc8f06622_small.jpg","float_intro":"","image_intro_alt":"","image_intro_caption":"","image_fulltext":"","float_fulltext":"","image_fulltext_alt":"","image_fulltext_caption":""}"><img...

Uncaught error: Invalid type for google table column


javascript,json,google-maps,google-visualization
I recently started using google visualization due to its simplicity. But while implementing in my recent project i seem to have run into a bit of trouble. The program shown is used to pull data from a JSON object and utilize it to display data in the google tables. Now...

Check for duplicates in JSON


javascript,jquery,json,duplicates
I have a json object: var object1 = [{ "value1": "1", "value2": "2", "value3": "3", }, { "value1": "1", "value2": "5", "value3": "7", }, { "value1": "6", "value2": "9", "value3": "5", }, { "value1": "6", "value2": "9", "value3": "5", }] Now I want to take each record out of that...

How to instantiate lexical.Scanner in a JavaTokenParsers class?


scala,parsing,lexical-scanner
I am writing a parser which inherits from JavaTokenParsers in that I have a function as follow: import scala.util.parsing.combinator.lexical._ import scala.util.parsing._ import scala.util.parsing.combinator.RegexParsers; import scala.util.parsing.combinator.syntactical.StdTokenParsers import scala.util.parsing.combinator.token.StdTokens import scala.util.parsing.combinator.lexical.StdLexical import scala.util.parsing.combinator.lexical.Scanners import scala.util.parsing.combinator.lexical.Lexical import...

How to parse JSON array of string arrays


c#,json
Lets say I get the following json data from a web service which I can't change. [ [ "Header1", "Header2", "Header3", "Header4" ], [ "FirstValue1", "FirstValue2", "FirstValue3", "FirstValue4" ], [ "SecondValue1", "SecondValue2", "SecondValue3", "SecondValue4" ] ] jsonlint.com tells me that it is valid json and from what I know I...

How to transform $.post to $.ajax?


javascript,jquery,ajax,json,servlets
I have this $.post peace of code: $.post("../admin-login", { dataName:JSON.stringify({ username:uname, password:pass, }) }, function(data,status){ console.log("Data:"+data); answer = data; } ); and I wont to transform it to $.ajax. On the servlet side I am demanding request.getParamter("dataName") but I do not know how to write data: section in $.ajax so...

How can I get json objects without the object number?


javascript,jquery,json,rest
I have a simple json object that spits out 4 items that have completely different properties inside each one. I have got the json being displayed with the 4 objects that are called meta.work_content like so: [Object, Object, Object, Object] I can open these in console and see the objects...

JQuery mutiple post in the same time


javascript,php,jquery,json
I have three functions ,and every function post to special php page to get data.. and every function need some time because every php script need some time .. function nb1() { $.post("p1.php", { action: 1 }, function(data) { console.log(data); }, "json") .fail(function(data) { console.log("error"); }); } function nb2() {...

How use jquery getJSON with a local variable


javascript,jquery,ajax,json,ace-editor
i want to extend the completions of Ace Editor, and i need a json of word, this is my code: function load_completions() { var object = $('#preview').attr('class'); $.ajax({ type: "POST", url: "test_editor_ajax.php", data: {action:'load_completion', object:object}, cache: false, success: function(json){ var adminCompleter = { getCompletions: function(editor, session, pos, prefix, callback) {...

String parsing with batch scripting


windows,string,parsing,batch-file,xml-parsing
I have a file called pictures.xml and it contains some pictures information like: <ResourcePicture Name="a.jpg"> <GeneratedPicture Name="b.jpg"/> <GeneratedPicture Name="c.jpg"/> </ResourcePicture> <ResourcePicture Name="z1.jpg"> <GeneratedPicture Name="z2.jpg"/> <GeneratedPicture Name="z3.jpg"/> <GeneratedPicture Name="z4.jpg"/> </ResourcePicture> What I want do do is to get each line in for loop and print the names of the pictures. Sample...

Sencha/Extjs rest call with all parameters


json,rest,extjs,sencha-touch
I'm using ExtJs 5.1.1 and I've written a simple view with a grid, and selecting one row the corresponding model property are editable in some text fields. When editing is completed the button 'save' call Model.save() method, which use the rest proxy configured to write the changes on the server....

Fatal error catched by register_shutdown_function and update json_encode


php,json,fatal-error
I can catch a fatal error with register_shutdown_function('shutdown') function shutdown(){ $error = error_get_last(); $result['message'] = $error['message']; } and I have an echo json_encode($result); Unfortunately this echo json_encode shows nothing because json is not updated with the message of the fatal error. What can I do to fix this?...

convert from json to array


javascript,php,json
I am trying to get data to display in a table. I don't know what I am doing wrong, but when I get the data from my page it is an array of single characters. I could parse this myself but would prefer to know what I am doing wrong....

Response 200 OK but Jquery shows error?


javascript,jquery,json
I am stuck in kind of a weird problem where I am trying to make a JSONP request to the Open Weather history API to get the weather information for the previous years on the a particular date. fetchHistoryUrl = 'http://78.46.48.103/data/3.0/days?day=0622&mode=list&lat=39.77476949&lon=-100.1953125'; $.ajax({ method: 'get', url: fetchHistoryUrl, success: function(response){ console.log(response); },...

Extracting strings from HTML with Python wont work with regex or BeautifulSoup


python,regex,parsing,beautifulsoup,python-requests
Im using Python 2.7, BeautifulSoup4, regex, and requests on windows 7. I've scraped some code from a website and I am having problems parsing and extracting the bits I want and storing them in a dictionary. What I'm after is text that is presented as follows in the code: @CAD_DTA\">I...

How to work with django-rest-framework in the templates


json,django,django-templates,django-rest-framework
Please bear with me. I am just learning django-rest-framework. And I really can't seem to grab it. model: class Day(models.Model): date = models.DateField(default=date.today) class ToDo(models.Model): date = models.ForeignKey(Day) name = models.CharField(max_length=100) very_important = models.BooleanField(default=False) finished = models.BooleanField(default=False) normal view: def home(request): days = Day.objects.all() return render(request, 'test.html', { 'days':days })...

How to use ajax response JSON data?


javascript,jquery,ajax,json
I have made my ajax request, which works great and returns JSON. But how do i use it? My response is something like this [{ "id": "5", "reviewID": "2389", "serviceID": "50707", "title": "well done" }] Now in my success function I am trying to use the data like such: success:...

Use JSON file to insert data in database


javascript,json,mongodb,meteor,data
I'm using my JSON file like this to insert data in my collection : var content = JSON.parse(Assets.getText('test.json')); console.log('inserting...'); Profiles.insert({ user: id, data:content }; But I would like to have a "data's tree" like that : [ user: "rtegert23423131", firstname:"test", surname:"test2", // ... ] Not like that : [ user:...

Parse JSON output from AlamoFire


json,swift,nsurlrequest
I am using the latest Alamofire to manage a GET http request to my server. I am using the following to GET and parse the JSON: Alamofire.request(.GET, "*******") .responseJSON {(request, response, JSON, error) in if let statusesArray = JSON as? NSArray{ if let aStatus = statusesArray[0] as? NSDictionary{ //OUTPUT SHOWN...

Deserializing Json data to c# for use in GridView - Error Data source is an invalid type


c#,json,gridview,serialization,.net-4.5
I am relatively new to working with C# (v5.0) and Json data (Newtonsoft.Json v6.0) and am seeking assistance in resolving an error when attempting to populate a .Net 4.5 GridView control. My sample Json data as returned by a web service is: { PersonDetails: [ { Book: "10 ", FirstName:...

String comparison in AngularJS


javascript,json,angularjs,ionic-framework,string-comparison
I'm trying to compare two strings in AngularJS, and I've seen examples online. As I understand it, you can use angular.equals(str1, str2), you can use ===, you can use == if you're sure that both are strings... I've tried all three, but I don't get the result. Something must be...

How to rearrange CSV / JSON keys columns? (Javascript)


javascript,json,csv,papaparse
I am converting a JSON object array to CSV using Papa Parse JavaScript Library. Is there a way to have the CSV columns arranged in a certain way. For e.g; I get the column as: OrderStatus, canOp, OpDesc, ID, OrderNumber, FinishTime, UOM, StartTime but would like to be arranged as:...

.NET wep api won't accept %2E or . in api request uri


c#,jquery,asp.net,ajax,json
We're trying to create our first web api using the .net framework. To try this we've used this demo project: http://www.codeproject.com/Articles/549152/Introduction-to-ASP-NET-Web-API In this project we've changed the find() function of the AJAX script so it only sends one var to our new democontroller.: <script> var uri = 'api/Demo'; function find()...

Create n:m objects using json and sequelize?


javascript,json,node.js,sequelize.js
I am trying to learn sequelize, but am having trouble getting a n:m object created. So far I have my 2 models that create 3 tables (Store, Product, StoreProducts) and the models below: models/Store.js module.exports = (sequelize, DataTypes) => { return Store = sequelize.define('Store', { name: { type: DataTypes.STRING, },...

Nested JSON structure to build side-by-side d3.js charts


javascript,json,d3.js
I'm currently stuck on how to traverse a JSON structure (I created) in order to create side-by-side donut charts. I think I've created a bad structure, but would appreciate any advice. I'm working from Mike Bostock's example here: http://bl.ocks.org/mbostock/1305337 which uses a .csv file for source data. In that example...

Deserializing JSON into c# class


c#,json,couchdb
I have a JSON in the following format (from couchDB view) {"rows":[ {"key":["2015-04-01","524","http://www.sampleurl.com/"],"value":1}, {"key":["2015-04-01","524","http://www.sampleurl2.com/"],"value":2}, {"key":["2015-04-01","524","http://www.sampleurl3.com"],"value":1} ]} I need to create a "service" to get this data from couchDB and insert it on SQL Server (for generating reports..) in a efficient way. My first bet was to bulk insert this json...

Find element by class name


python,parsing,selenium,selenium-webdriver,css-selectors
I'm trying to find one tag using we.find_element_by_css_selector('p.p1.transfer strong.ng-binding').text The problem is that there is sometimes a 'strong' tag before the tag I'm searching for which is very similar but it's class is: class="ng-binding ng-hide" instead of class="ng-binding". But when I try to find it it finds the first tag....

why i don't get return value javascript


javascript,jquery,html,json,html5
When i debug my code i can see i have value but i don't get value to createCheckBoxPlatform FN function createCheckBoxPlatform(data) { var platform = ""; $.each(data, function (i, item) { platform += '<label><input type="checkbox" name="' + item.PlatformName + ' value="' + item.PlatformSK + '">' + item.PlatformName + '</label>' +...

Sorting in Ruby on rails


ruby-on-rails,json,sorting
I'm very new to Ruby on rails. I try to edit the following api. I want to sorting "can_go" which is true are shown at the top of list. I added this row before sending data, but the result is still order by "user_id". user_infos.sort { |a, b| - (a['can_go']<=>b['can_go'])...

access the json encoded object returned by php in jquery


php,jquery,ajax,json
I want to post some data to php function by ajax, then get the encoded json object that the php function will return, then I want to get the information (keys and values) from this object, but I don't know how, here is my code: $.ajax({ url: "functions.php", dataType: "JSON",...

Call function on Server from iOS app - Objective C


ios,objective-c,json,server,backend
If you are familiar with Parse.com's Javascript SDK, this is what I am trying to do for my own server for my iOS app (Objective-c). I want to be able to send some a string to the function that is on my server, have the server run its function and...

Can't save json data to variable (or cache) with angularjs $http.get


json,angularjs,web-services,rest
I have weird angularjs problem. I'm trying to fetch data from Rest Webservice. It works fine, but I can't save json data to object. My code looks like: services.service('customerService', [ '$http', '$cacheFactory', function($http, $cacheFactory) { var cache = $cacheFactory('dataCache'); var result = cache.get('user'); this.getById = function(id){ $http.get(urlList.getCustomer + id).success(function(data, status,...

Complicated Json processing - PHP


php,json
I am using this JSON: http://steamcommunity.com/id/mitch8910/inventory/json/730/2/ I am sorting through the "rgDescriptions" data with this: $data = file_get_contents('http://steamcommunity.com/id/mitch8910/inventory/json/730/2/'); $json = json_decode($data); foreach ($json->rgDescriptions as $mydata) { //checking and adding data to a database } Each section in "rgDescriptions" contains a lot of information on a certain item. For each item...

Replacing elements in an HTML file with JSON objects


javascript,json,replace
I am writing some Javascript to: Read in and Parse a JSON file Read in an HTML file (created as a template file) Replace elements in the HTML file with elements from the JSON file. It is only replacing the first element obj.verb. Can someone please help me figure out...

Read JSON double dimensional array in Java


java,android,arrays,json
I'm trying to get a string from a JSONObject. My JSON looks like this : { "agent": [ {"name":"stringName1", "value":"stringValue1"}, {"name":"stringName2", "value":"stringValue2"} ] } I want to get the values from stringName1 and stringName2. So first, I tried to get the "agent" using this : JSONObject agent = (JSONObject) jsonObject.get("agent");...

Why i get can not resolve method error in class android?


android,json
I use this tutorial: JSON and part of code to this: protected String doInBackground(String... urls) { TextView lbl=(TextView) findViewById(R.id.provCODE); person = new Person(); person.setName(lbl.getText().toString()); //person.setCountry("1853"); //person.setTwitter("1892"); return POST(urls[0],person); } but in the this line: person.setName(lbl.getText().toString()); i get this error: Can not resolve method. How can i solve that?what happen? my...

Parsing Google Custom Search API for Elasticsearch Documents


json,python-2.7,elasticsearch,google-search-api
After retrieving results from the Google Custom Search API and writing it to JSON, I want to parse that JSON to make valid Elasticsearch documents. You can configure a parent - child relationship for nested results. However, this relationship seems to not be inferred by the data structure itself. I've...

Parse text from a .txt file using csv module


python,python-2.7,parsing,csv
I have an email that comes in everyday and the format of the email is always the same except some of the data is different. I wrote a VBA Macro that exports the email to a text file. Now that it is a text file I want to parse the...

KendoUI Grid - Complex JSON with inconsistent keys


javascript,json,kendo-ui,kendo-grid
I know this is probably been asked before, but having tried to find an answer - I am guessing either I am not comprehending some of the answers right, or I am looking at the problem all wrong. I am using a complex SLC loopback query - and the api...