FAQ Database Discussion Community


Configuring Logstash to Decode Its Own Event Format JSON

logging,logstash,kibana,logstash-forwarder,logstash-logback-encoder
I have a java log file for a webbapp that was created using SLF4J, Logback and the logstash-logback-encoder for use in logstash 1.4.2. While various configurations have succeeded from retrieving data from the logs, none has actually resulted in proper json being returned. Based on every guide I have read,...

Update @timetamp field in logstash with custom timestamp value

elasticsearch,logstash,grok,logstash-grok,logstash-forwarder
I have following logstash config file for parsing following exception stack trace. stacktrace 2015-03-02 09:01:51,040 [com.test.MyClass] ERROR - execution resulted in Exception com.test.core.MyException <exception line1> <exception line2> 2015-03-02 09:01:51,040 [com.test.MyClass] ERROR - Encountered Exception, terminating execution Config File: input { stdin {} } filter { multiline { pattern => "(^%{TIMESTAMP_ISO8601})...

Logstash - How to filter by [tags]

logstash,logstash-forwarder
Logstash filter by tags for different websites Issue: I have multiple websites inside a single IIS Server.. I want to add a "Tag" for each of the log files i am sending towards logstash This is my logstash forwarder config Each log file represents a different website.. so i want...

Log storage location ELK stack

elasticsearch,logstash,kibana,logstash-forwarder,elk-stack
I am doing centralized logging using logstash. I am using logstash-forwarder on the shipper node and ELK stack on the collector node.I wanted to know the location where the logs are stored in elasticsearch i didn't see any data files created where the logs are stored.Do anyone has idea about...

Logstash architecture decisions

amazon-ec2,elasticsearch,logstash,logstash-forwarder
So we have a bunch of servers running on EC2 Amazon Web Services, and are looking to set up logstash/elasticsearch for distributed logging. From what I read there are several options generally chosen: logstash on each server node, using the File input filter and going directly to ElasticSearch cluster as...

Is there any indication that logstash forwarder finished processing a file?

logstash,logstash-forwarder
I would like to delete files after logstash forwarder sent them (otherwise I get too many files open error). Is there any indication that logstash forwarder is done with the file?

How to process multilines in logstash with multiple worker threads?

regex,multithreading,logstash,multiline,logstash-forwarder
I would like to process mulitline logs with logstash using multiple worker threads for performance but multiline filter doesn't work: - https://github.com/elastic/logstash/pull/1591 - https://github.com/elastic/logstash/issues/1590 solutions for now: using multiple logstash-forwarder and send them to different lumberjack port (scales very poorly: new logstash-forwarder for each logfile that has multilines) using an...

Sending logs every 2 hours using logstash-forwarder without using cronjob

logstash,logstash-forwarder,logstash-configuration
Is there a way I can send data using the logstash-forwarder every 2 hours or more without using a cronjob script to start and stop the forwarder every time I want to send the data?