Loading presentation...

Present Remotely

Send the link below via email or IM

Copy

Present to your audience

Start remote presentation

  • Invited audience members will follow you as you navigate and present
  • People invited to a presentation do not need a Prezi account
  • This link expires 10 minutes after you close the presentation
  • A maximum of 30 users can follow your presentation
  • Learn more about this feature in our knowledge base article

Do you really want to delete this prezi?

Neither you, nor the coeditors you shared it with will be able to recover it again.

DeleteCancel

Make your likes visible on Facebook?

Connect your Facebook account to Prezi and let your likes appear on your timeline.
You can change this under Settings & Account at any time.

No, thanks

Centralized Logging

No description
by

Jim Turpin

on 18 June 2013

Comments (0)

Please log in to add your comment.

Report abuse

Transcript of Centralized Logging

Centralized Logging
Making the logs work for you
Managing the logs
Why centralized logging
Logs are a critical part of any system, they give you insight into what the system is doing.
Getting the logs
Virtually every running process generates a log in some form.
LOG WRANGLIN'
MAKING LOGS WORK FOR YOU
Reduce time spent searching through logs to find something specific.
Get automatic alerts for log anomalies.
Allow access to certain logs to certain people.
Be a DevOps ALL STAR!

Do you need to normalize the logs?
Do you need to remove sensitive data?
Just need to pull stats from the logs?

What challenges do you face in shipping your logs?

Do your logs contain sensitive data?
Are there special networking requirements?
What are your OS requirements?
What do your logs look like?
What regulatory requirements do you face?

GUIs
File Replication
A simple approach is to setup replication of your logs via rsync + cron. Most systems have some sort of rsync / scp / sftp client.
Syslog
Syslog is likely already installed on the majority of systems (windows excluded). Most people are familiar with rsyslog or syslog-ng.
Distributed Log Collectors
High volume / high throughput log and event collection. Most of these are event streaming and processing systems.


Logstash Example
input {
irc {
channels => ["#reprap"]
host => "irc.freenode.net"
port => 6667
type => "reprap"
debug => "true"
}
}

output {
#stdout {}
gelf {
facility => "logstash-gelf"
host => '192.168.230.5'
}
}
Log Retrieval
- File Replication
- Syslog
- Distributed Log Collectors

Pros:
Cons:
Easy to setup
Exact copies of your logs
Logs sent through secure channels
Timely access to data
No aggregation
Logstash - log shipping / parsing and indexing
Graylog - A UI for searching and analyzing logs. Also provides GELF logging format to overcome some syslog limitations.
fluentd - Similar to logstash, small footprint
Flume - Apache project for collecting, aggregating and moving log data.
Scribe - Used and created by Facebook.
Kafka - Developed by LinkedIn for their activity stream processing. Not ideal for logs but can be used.
Pros:
Cons:
Highly configurable
Available on most distros
Windows versions available
Not all OS's supported natively
Default settings can lead to dropped messages
Limited number of categories
Not all apps can use syslog
Examples:
Troubleshooting!
Logstash
"logstash is a tool for managing events and logs. You can use it to collect logs, parse them, and store them for later use (like, for searching)."
Input
Filter
Output
amqp
drupal_dblog
elasticsearch
eventlog
exec
file
ganglia
gelf
gemfire
generator
graphite
heroku
imap
irc

log4j
lumberjack
lumberjack2
pipe
rabbitmq
redis
relp
snmptrap
sqs
stdin
stomp
syslog
tcp

xtwitter
udp
varnishlog
websocket
mpp
zenoss
zeromq
alter
anonymize
checksum
clone
csv
date
dns
environment
gelfify
geoip
grep

grok
grokdiscovery
json
kv
metrics
multiline
mutate
noop
prune
ruby
sleep

split
syslog_pri
translate
urldecode
useragent
xml
zeromq

amqp
boundary
circonus
cloudwatch
datadog
elasticsearch
elasticsearch_http
elasticsearch_river
email
exec
file
ganglia
gelf
gemfire

graphite
graphtastic
hipchat
http
internal
irc
juggernaut
librato
loggly
lumberjack
metriccatcher
mongodb
nagios
nagios_nsca
null
opentsdb
pagerduty
pipe

rabbitmq
redis
riak
riemann
sns
sqs
statsd
stdout
stomp
syslog
tcp
websocket
xmpp
zabbix
zeromq
Graylog
Kibana
Splunk
Graphite
Large user base
Great graphing ability
Runs on Windows for the all Windows shops
Paid Support

Open Source
Multiple access levels
Fast
Open Source
Great for graphing data
Fast
Pros:
Cons:
Can't configure individual access for users.
Pros:
Cons:
Graphing leads room for improvement
Pros:
Cons:
Not Open Source
Painfully expensive
Quick searching through logs (No more grep)
Visual representations of what's going on
Simplify searching for non-technical users
Not all vendors follow RFCs.
Timestamps seem especially troublesome for certain vendors (Cisco that means you)
Removing sensitive data
Discarding worthless information
Why would you ever possibly need to modify logs!?!
Pros:
Cons:
Great Visualization
Can also graph server statistics
Can't configure individual access for users.
Can be confusing to navigate to the desired data
Logstash Example #2
input {
file {
path => "/var/log/fans"
type => "fan_stats"
format => "plain"
}
}

filter {
grok {
type => fan_stats
pattern => %{MONTH}-%{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND} %{HOSTNAME} %{APPNAME} %{GREEDYDATA:message}
}
}

output {
#stdout {}
gelf {
facility => "logstash-gelf"
host => '192.168.230.5'
}
}
File contents:
06-17 14:18:03 atticpi Attic_Temp[1]: The current temp is: 106.59
Hosted Solutions
loggly
papertrail
logentries

Windows and Syslog
Snare vs. nxlog
Both have open source/free versions.
nxlog is open source while Snare has a limited feature set free version
nxlog is extremely configurable
nxlog is cross platform

Now: Back to Graylog2
Message Flow
Forensically sound logs.
Full transcript