FAQ Database Discussion Community


Kibana: filter events for today

elasticsearch,kibana
I'm using Kibana on top of logstash and I want to filter items in the index to today, or the last 24 hours is fine too. So apparently this requires me to run a range query against the underlying ElasticSearch engine that would look like: "range" : { "timestamp" :...

Why do I need a broker for my production ELK stack + machine specs?

elasticsearch,redis,logstash,kibana
I've recently stood up a test ELK stack Ubuntu box to test the functionality and have been very happy with it. My use case for production would involve ingesting at least 100GB of logs per day. I want to be as scalable as possible, as this 100GB/day can quickly rise...

Unable to show location in tile map of kibana

elasticsearch,logstash,kibana,kibana-4
I am using Elasticsearch-1.5.1, Kibana-4.0.2-linux-x86, Logstash-1.4.2. My logstash conf is like this input{ redis{ data_type=>'list' key=>'pace' password=>'bhushan' type=>pace } }filter { geoip { source => "mdc.ip" target => "geoip" database => "/opt/logstash-1.4.2/vendor/geoip/GeoLiteCity.dat" add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ] add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ] } } output{ if[type]=="pace"{ elasticsearch{ template_overwrite...

Kibana to display @timestamp field into a readable format

elasticsearch,logstash,kibana
I setup an ELK stack (Elasticsearch, Logstash, and Kibana) I wonder how to display the @timestamp field into a readable format which contains this format YYYY-MM-DDTHH:mm:ss.SSSZ I want the Kibana to display something like HH:mm:ss DD MM YYYY Note: I'd like to configure the @timestamp field in Kibana because I...

Multiple indices under a single name in kibana is possible?

indexing,customization,kibana,indices
I want multiple indices to be under one name.. How can i do that in kibana... as shown in this picture..so that the logs ( app,server,db) of the server1 and the logs of server 2 under it. How can we customize to make multiple indices under one name. { http://i.stack.imgur.com/t3eV8.png...

Kibana - SearchParseException on histogram using custom timestamp

exception,timestamp,histogram,logstash,kibana
A SearchParseException is thrown on my Kibana histogram, although the data seems to be captured correctly. I am using a custom timestamp field that I called logtimestamp. A log file line example would be: 2015-01-28 17:09:51,059 DEBUG [main] - processStatus = [MATCHED] And the related logtimestamp field being added to...

Does Kibana take data from ElasticSearch or LogStash directly?

elasticsearch,logstash,kibana
This might be quite a dumb question but I couldn't find the explanation in any architectural overview of the ELK stack. When data are displayed in real time, does Kibana use LogStash directly or Kibana get the information from ElasticSearch after it's been put there by LogStash? I'm asking this...

ElasticSearch and index strategy

iis,logging,elasticsearch,kibana
We are currently setting up an ELK stack to consume our various logs. Currently we are just using an index per day and only have logs from various applications we develop (approx 100m docs). Next step is to look at other types of logs from e.g. IIS, event logs from...

Passing Elasticsearch and Kibana config file to docker containers

elasticsearch,docker,kibana,kibana-4
I have found a docker image devdb/kibana which runs Elasticsearch 1.5.2 and Kibana 4.0.2. However I would like to pass into this docker container the configuration files for both Elasticsearch (i.e elasticsearch.yml) and Kibana (i.e config.js) Can I do that with this image itself? Or for that would I have...

Logstash/Elasticsearch/Kibana resource planning

elasticsearch,logstash,kibana,high-load
How to plan resources (I suspect, elasticsearch instances) according to load: With load I mean ≈500K events/min, each containing 8-10 fields. What are the configuration knobs I should turn? I'm new to this stack....

How to set authentication in kibana

kibana,kibana-4
Is it possible to enable authentication in Kibana in order to restrict access to a dashboard to only be accessible to particular users?

filter json in logstash

json,logstash,kibana,elasticsearch-plugin
I have a json file with records like this one {"id":1,"first_name":"Frank","last_name":"Mills","date":"5/31/2014","email":"[email protected]","country":"France","city":"La Rochelle","latitude":"46.1667","longitude":"-1.15" and I'm trying to filter the fields in logstash, unsuccessfully so far. I tried the grok debugger and the grokconstructor but cannot make it work. My last attempt is input { file{ path => ["C:/logstash-1.4.2/mock_data.json"] type => "json"...

Configuring Logstash to Decode Its Own Event Format JSON

logging,logstash,kibana,logstash-forwarder,logstash-logback-encoder
I have a java log file for a webbapp that was created using SLF4J, Logback and the logstash-logback-encoder for use in logstash 1.4.2. While various configurations have succeeded from retrieving data from the logs, none has actually resulted in proper json being returned. Based on every guide I have read,...

entries not entering logstash filter

ruby-on-rails,logstash,kibana,beaver
I've been trying to parse rails log entries sent from beaver to my logstash indexer. But certain entries are not entering the filter section at all, but are appearing on my kibana dashboard in their original state (i.e without the fields being extracted). beaver conf path:[path to the log] exclude:[.gz]...

Sum total of two queries - Kibana Lucene syntax

lucene,kibana
I have the following queries I am running to get counts from multiple log files with custom tags. The issue is I am not able to get the sum total of 2 queries combined. Query 1: tags:ppr AND tags:api AND loglevel:ERROR Result: 203445 Query 2: tags:ppr AND tags:api NOT response:200&302...

Replace values with predefined mapping with ELK

elasticsearch,logstash,kibana
I have a file that I read with Logstash containing a certain parameter called type. The possible values for this parameter are 0,1,2,3,4,5 (it is actually represented as an Enum in my Java code). Let's say type 0 means book, type 1 means newspaper, type 2 means magazine, etc. Currently...

What is the purpose of “analysed” button in Kibana 4?

elasticsearch,kibana,elasticsearch-plugin,kibana-4
Description: ​I have set up Kibana 4 and also configured an index for analytics. So in the discover page,of Kibana,I can see my index name and the corresponding fields in the index there. To the right side of the fields panel,there is a settings icon. When I click that,four options...

Logstash: 1-hour difference in custom timestamp

timestamp,logstash,kibana
I am using a custom timestamp field in Logstash (one present in my log file instead of Logstash's @timestamp field), and although this timestamp is created and usable in Kibana, there seems to always be a 1-hour difference with the actual timestamp I am fetching. Here is, for example, actual...

Elasticsearch Date Range Aggregation with Sum

elasticsearch,kibana,elastic
I've followed Elastic's docs and successfully queried my index to return data for two date ranges. What I can't get right, is adding a sum of units to each range. I've managed to add a unit sum (see query below) but this only returns a sum for the entire range....

SearchPhaseExecutionException[Failed to execute phase [query], all shards failed]

elasticsearch,logstash,windows-server-2012,kibana
Recently our server was rebootet without correctly shutting down the Elastic Search / Kibana. After that reboot both applications were running but no indices where created anymore. I checked logstash setup in debugg mode and he is sending data to Elastic Search. now all my created windows report this error:...

can't force GROK parser to enforce integer/float types on haproxy logs

types,mapping,logstash,kibana,grok
Doesn't matter if integer/long or float, fields like time_duration (all time_* really ) map as strings in kibana logstash index. I tried using mutate (https://www.elastic.co/blog/little-logstash-lessons-part-using-grok-mutate-type-data) did not work either. How can i correctly enforce numeric type instead of strings on these fields? My /etc/logstash/conf.d/haproxy.conf: input { syslog { type =>...

Is it possible to extract a certain part of a string field and cast it to some other type?

elasticsearch,logstash,kibana
I have a file input in logstash, which reads from /var/log/syslog. The log message goes into message field. I didn't think about extracting some parts of the message beforehand, but now I would like to find all entries with the message field that have a word WORD in them and...

Can Kibana 3 and Kibana 4 Be Used Side-By-Side?

kibana,kibana-4
I'm currently using Kibana 3, and would like to upgrade to Kibana 4. However, I would like to keep Kibana 3 around until I have my dashboards in Kibana 4 configured. Can the two be run side-by-side without interfering with one another? I know that Kibana stores its settings in...

Scripted Fields for if/else condition in Kibana 4

elasticsearch,lucene,kibana,kibana-4
I have some numeric fields in elasticsearch, I have to implement some logic for which I need to create some scripted fields. I am new to kibana 4's scripted fields feature, so I need some help regarding a basic format that could be used for writing a basic if else...

how to use logstash-generated fields in kibana?

logstash,kibana
I have some Logstash-generated fields which I can see in Kibana (in discovery mode). They are not present in the sidebar, though: In the example above who is well present in the data but not on the list of fields on the left. Is there a particular action to take...

How to remove date from LogStash event

log4j,logstash,kibana,kibana-4,logstash-grok
I have the following message in my log file... 2015-05-08 12:00:00,648064070: INFO : [pool-4-thread-1] com.jobs.AutomatedJob: Found 0 suggested order events This is what I see in Logstash/Kibana (with the Date and Message selected)... May 8th 2015, 12:16:19.691 2015-05-08 12:00:00,648064070: INFO : [pool-4-thread-1] com.pcmsgroup.v21.star2.application.maintenance.jobs.AutomatedSuggestedOrderingScheduledJob: Found 0 suggested order events The date...

elasticsearch/kiabana - analyze and visualize total time for transactions?

elasticsearch,logstash,kibana
Parsing log files using logstash, here is the json sent to elasticsearch looks like: For log lines contaning transaction start time, i add db_transaction_commit_begin_time field with the time it is logged. { "message" => "2015-05-27 10:26:47,048 INFO [T:3 ID:26] (ClassName.java:396) - End committing transaction", "@version" => "1", "@timestamp" => "2015-05-27T15:24:11.594Z",...

Kibana 4 configuration for production

node.js,azure-web-sites,kibana,kibana-4
I'm trying to deploy Kibana 4 to Azure Websites. I can't use bin/kibanta.bat file since Azure Websites uses start script in package.json to bootstrap application. I tried to update package.json start script to run bin\kibana.js file and environment variables in it. After that Azure starts running Kibana server but I'm...

multiple group by in elasticsearch including missing values

php,elasticsearch,group,kibana
I'm trying to do a group by in elasticsearch, by multiple fields. I know that nested aggregation exists, but what I want is including in a certain bucket the record for which the field I'm grouping by is empty. Say that we have this kind of data structure: SONG_ID |...

ElasticSearch/Lucene query string — select “field X exists”

elasticsearch,lucene,kibana
How do I query ElasticSearch through Kibana to select items that have field X? For example, I have a mapping with fields {"a": {"type": "string"}, "b": {"type": "string"}}, and two documents {"a": "lalala"} {"a": "enoheo", "b": "nthtnhnt"} I want to find the second document without knowing what its b actually...

Bringing in single and multi-line App log records to ELK (some contain JSON objects)

logging,logstash,kibana
I'm trying to take log records from a custom (node.js) application that will be putting data into elastic search and then processed by Kibana. My environment is Ubuntu with ELK (Elasticsearch, Logstash and Kibana) and the log generating Application is in Node.JS I'm already processing the standard system log files,...

Kibana 4 Metric visualization show latest value

elasticsearch,kibana,kibana-4
I'm new to Kibana and Elastic Search and i have run into this problem: My ES contains (among other stuff) also data containing the current value of one custom performance counter and i would like my dashboard to show this value, e.g., as a big number - therefore i tried...

How to smoothly load 200MB data to browser for visualization?

javascript,d3.js,leaflet,data-visualization,kibana
The 200MB data is collected among 300 days, about 600kb daily. Currently I use d3.tsv to load one file containing all data, and then use setTimeout to loop through each day. But the question is to load 200MB data to client's browser, it can take a few minutes... How to...

Visualizing a percentage of current view

elasticsearch,kibana,kibana-4
I am trying to display a users chat messages as a percentage of the current view/query. I can get a visualization of the counts per user easily but is there a way to turn this into a percent of the all the selected users?...

Kibana visualizations splitting fields with dashes in them

elasticsearch,kibana
I am building visualizations in Kibana for AWS CloudWatch metrics, and have run into a bit of an issue creating Metric Tables.. Kibana is splitting my fields that contain dashes (instance ID, region, etc..) Rather than having an individual row with an instance ID, for example, i-7bb06dzz, it is creating...

Elasctic Search is working at port 9200 but Kibana is not working

elasticsearch,kibana,kibana-4
Hello I am starting work with kibana and elasticsearch. I am being able to run elasticsearch at port 9200 but kibana is not running at port 5601. The following two images are given for clarification Kibana is not running and showing the page is not available...

How to overwrite field value in Kibana?

elasticsearch,logstash,kibana
I am using Logstash to feed data into Elasticsearch and then analyzing that data with Kibana. I have a field that contains numeric identifiers. These are not easy to read. How can I have Kibana overwrite or show a more human-readable value? More specifically, I have a 'ip.proto' field. When...

Log storage location ELK stack

elasticsearch,logstash,kibana,logstash-forwarder,elk-stack
I am doing centralized logging using logstash. I am using logstash-forwarder on the shipper node and ELK stack on the collector node.I wanted to know the location where the logs are stored in elasticsearch i didn't see any data files created where the logs are stored.Do anyone has idea about...

Search for parse errors in logstash/grok

logstash,kibana,grok,kibana-4
I´m using the elk stack to analyze log data and have to handle large volumes of log data. It looks like all the logs can be parsed with logstash/grok. Is there a way to search with kibana for loglines that couldn´t be parsed?...

worker_connections are not enough

nginx,kibana
I am trying access kibana application deployed in nginx,but getting below URL :- http://127.0.0.1/kibana-3.1.2 2015/02/01 23:05:05 [alert] 3919#0: *766 768 worker_connections are not enough while connecting to upstream, client: 127.0.0.1, server: , request: "GET /kibana-3.1.2 HTTP/1.0", upstream: "http://127.0.0.1:80/kibana-3.1.2", host: "127.0.0.1" Kibana is deployed at /var/www/kibana-3.1.2 I have tried to increase...

Setting default index in Kibana 4

elasticsearch,kibana,kibana-4
I can't define default index in Kibana 4.0.2, as you can see on the image below. It doesn't saves if I enter it manually in "Advanced" tab, it only glitches when I click on "Set as default index" button but it doesn't make needed changes. I'm using Couchbase 3.0.3, transport-couchbase...

Nested Object in Kibana visualize

elasticsearch,kibana,kibana-4
I have uploaded the JSON file in elasticsearch and my mapping contains of some nested objects. The problem is that, in Kibana, in visualize I can not see them This is my mapping: "comments":{"type": "nested", "properties":{ "count":{ "type": "integer" }, "data": { "type": "nested", "properties": { "created_time": { "type": "integer"...

Kibana3: long to IP in terms panel

long-integer,logstash,kibana,ipv4
For an ELK(Kibana is v3) setup I feed logs from some firewalls and src_ip/dst_ip fields are defined as type "ip". eg. "dst_ip" : {"type" : "ip"} Mappings are also correct: curl -XGET http://localhost:9200/logstash-2015.03.10/_mapping/field/src_ip?pretty { "logstash-2015.03.10" : { "mappings" : { "screenos" : { "src_ip" : { "full_name" : "src_ip", "mapping":{"src_ip":{"type":"ip"}}...

How to trigger a ticket on kibanahud from python code

python,curl,elasticsearch,kibana
I am stuck at a problem What I want to do is once a certain threshold is reached I want to trigger a ticket on KibanaHud from my python code. I am creating a json file with all the data that I need for the ticket -> ticket.json I am...

Courier Fetch: shards failed

elasticsearch,kibana,kibana-4
Why do I get these warnings after adding more data to my elasticsearch? And the warnings are different every time I browse the dashboard. "Courier Fetch: 30 of 60 shards failed." More details: It's a sole node on a CentOS 7.1 /etc/elasticsearch/elasticsearch.yml index.number_of_shards: 3 index.number_of_replicas: 1 bootstrap.mlockall: true threadpool.bulk.queue_size: 1000...

Anyone know what's the data source of http://logstash.openstack.org?

logstash,openstack,kibana
I'm new to OpenStack and I'd like to do some mining on OpenStack logs. So I found this webpage: http://logstash.openstack.org It gives a lot of logs which seems interesting. Anyone know how these data are generated and where they are from? Thanks a lot for your help! Best Regards...

Kibana and fixed time spans

elasticsearch,kibana,kibana-4
Is it possible to set a fixed timespan for a saved visualization or a saved search in Kibana 4? Scenario: I want to create one dashboard with 2 visualizations with different time spans. A metric counting unique users within 10 min (last 10 minutes) A metric counting todays unique users...

How to do a time range search in Kibana

elasticsearch,logstash,kibana,kibana-4
We are using the ELK for log aggregation. Is it possible to search for events that occured during a particular time range. Lets say I want to see all exceptions that occurred between 10am and 11am in last month. Is it possible to extract the time part from @timestamp and...

Need a logstash-conf file to extract the count of different strings in a log file

logstash,kibana
How to write a logstash configuration file to separate two different (S:Info & S:Warn) strings from a log file and display the respective count in Kibana? Tried using the 'grep' filter in logstash but not sure of getting the count of two different strings (Info and Warn) in Kibana. Below...

Can I use mutate filter in Logstash to convert some fields to integers of a genjdbc input?

jdbc,filter,elasticsearch,logstash,kibana
I am using genjdbc input plugin for Logstash to get data from a DB2 database. It works perfectly, I get in Kibana all the database columns as fields. The problem I have is that in Kibana all fields are string type, and I want the numeric fields to be integers....

What would be a good approach for sending logs from multiple servers a centralized logging server?

elasticsearch,logstash,syslog,kibana,rsyslog
I am trying to send logs, lots of logs, from a php application hosted on multiple ec2 instances. Instead of going with the standard approach of having logstash installed on each server and using logstash-forwarder to send the logs to a logging server with logstash parsing the logs and feeding...

Is it possible to access elasticsearch's internal stats via Kibana

kibana,kibana-4
I can see from querying our elasticsearch nodes that they contains internal statistics that for example show disk, memory and CPU usage (for example via GET _nodes/stats API). Is there anyway to access these in Kibana-4?...

Packetbeat dashboard for Application logs

dashboard,kibana,packetbeat
Can packetbeat is used to monitor the tomcat server logs and windows logs?? or it will only monitor the database i.e., network monitoring?

Reducing elasticsearch's index size

elasticsearch,logstash,kibana
I currently have a large amount of log files being analyzed by Logstash, and therefore a consequent amount of space being used in Elasticsearch. However, a lot of this data is useless to me, as everything is not being displayed in Kibana. So I'm wondering: is there a way to...

Kibana4 search bar

iframe,elasticsearch,kibana
I just started using Kibana4 and they have this new feature of embedding the dashboard into your app using an iframe generated in Kibana. Pretty cool and easy to use. However, the search bar is NOT embedded as well with the dashboard and it's very very necessary that I have...

Kibana multi graph

elasticsearch,kibana
I am currently using kibana3 and elasticsearch to monitor my system and I'm having trouble to create 3 line charts on the same graph. Example : I have the following line : { "@version":"1","@timestamp":"2015-02-03T10:06:35.539Z","host":"localhost","plugin":"load","collectd_type":"load","shortterm":0.4,"midterm":0.36,"longterm":0.33 } On kibana, I can just graph one of these : shortterm, midterm, longterm I would...

Make Kibana 4 remain running after disconnecting SSH session

linux,ubuntu,elasticsearch,kibana
I've installed ElasticSearch and Kibana edge versions on a Ubuntu Linux 14 box. So that's Kibana 4 on ElasticSearch 1.4.4. Runs and works like a charm through: ./bin/kibana However, as soon as I disconnect my Putty session, Kibana stops working. ElasticSearch keeps listening on port 9200, but Kibana cannot be...

ELK Type Conversion - Not a number but a string

elasticsearch,logstash,kibana
I'm trying to set up an elk dashboard to see some numbers like total bytes, avg load time, etc. I'm forcing some conversions in logstash to make sure these fields aren't strings convert => [ "bytes", "integer" ] convert => [ "seconds", "float" ] convert => [ "milliseconds", "integer" ]...

Kibana displays “Searching…”, displays no results. Confirmed data exists in ElasticSearch

elasticsearch,kibana
I am trying to build a new index, but I have run into some issues in Kibana. The Discover page has displayed "Searching..." for several hours now, on just a handful of data points. I think there may be an issue with the formatting? The indices page shows that Average,...

Kibana 4 Color Scheme

kibana,kibana-4
I tend towards darker color schemes when building GUIs, and some of the nicer screenshots that I've seen of Kibana support this. Now that we are kicking the tires on Kibana 4, the vizualizations and dashboards seem to be light background only. Is there a button staring me in the...

Kibana3 - Graph to plot amount of bytes per minute

kibana
I have apache logs with size of each request and the time , I need to plot graph on amount of data transfered per unit time. A sample document looks like below { "@timestamp" : "2015-01-01T00:00:00", "bytes" : 20 } For each minute , I want to take sum of...

Elastic Search Index in python

python,elasticsearch,kibana,kibana-4,pyelasticsearch
I want to push my data from a dictionary to Elasticsearch. How can I create an index for the same? I tried using the Curl commands as well on Linux server curl -XPUT 'http://localhost:9200/osint/' -d ' index: number_of_shards: 5 number_of_replicas: 2 ' but this also didnt help out I wrote...

Kibana 3 - How to bind different query for different panels

elasticsearch,kibana
In Kibana , I have made many panels. Currently a single query is working for the entire set of panels. Instead , I would like to keep a different query for each panel and no queries at all for a single panel. How can i do this using Kibana 3...

Kibana 4 'Discover' search error

elasticsearch,kibana
I indexed a dataset of geo-data records in ElasticSearch for analysis in Kibana. My issue is that the 'Discover' tab doesn't pick up the data but instead displays the error message Discover: An error occurred with your request. Reset your inputs and try again. In 'Settings', I could configure my...

Histogram not displayed properly in Kibana-3

kibana
The histogram i created with my data is shown as a thick block. There are valid hits numbers and dates on the y and x axis respectively, which indicates the data is proper. What can be done to scale it down to an understandable one?

How to set time in log as main @timestamp in elasticsearch

elasticsearch,logstash,kibana,logstash-grok
Im using logstash to index some old log files in my elastic DB. i need kibana/elastic to set the timestamp from within the logfile as the main @timestamp. Im using grok filter in the following way: %{TIMESTAMP_ISO8601:@timestamp} yet elasticsearch sets the time of indexing as the main @timestamp and not...

separate indexes on logstash

elasticsearch,logstash,kibana
Currently I have logstash configuration that pushing data to redis, and elastic server that pulling the data using the default index 'logstash'. I've added another shipper and I've successfully managed to move the data using the default index as well. My goal is to move and restore that data on...

Display 2 different query results in table with Kibana

logging,elasticsearch,kibana,rsyslog
I'm trying to show logs from 2 different servers in Kibana. Here's picture: http://i.imgur.com/U0JkmK1.png In first table I'd like to show logs about Server 1 and in table 2( on the right) logs from another Server. Is this even possible with Kibana? If I write query, both tables data change....