Skip to content

Latest commit

 

History

History
160 lines (95 loc) · 5.73 KB

10.md

File metadata and controls

160 lines (95 loc) · 5.73 KB

+++ date = "2015-09-30" draft = false weight = 10 title = "Lab 10 - Logs" +++

Lab Duration: 10 minutes

Lab Objective

The objective of this lab is to show how to locate the various logs for services supporting OpenStack, thereby allowing admins to monitor their OpenStack deployment. The second part of this lab is designed as a reference for how to build an rsyslog server for your OpenStack infastruture. This lab has no dependency on other labs.

1. Where are the logs?

Most OpenStack services use the convention of writing their log files to sub directories of /var/log/

  1. Display the nova-* service logs

    [root@controller ~]# ls /var/log/nova

    nova-api.log   nova-conductor.log    nova-manage.log      nova-scheduler.log
nova-cert.log  nova-consoleauth.log  nova-novncproxy.log

  2. Display the glance-* service logs

    [root@controller ~]# ls /var/log/glance

    api.log  api.log-20151004.gz  api.log-20160113  registry.log  registry.log-20160111

  3. Display the cinder-* service logs

    [root@controller ~]# ls /var/log/cinder

    api.log  api.log-20151004.gz  api.log-20160113  registry.log  registry.log-20160111

  4. Display the keystone-* service logs

    [root@controller ~]# ls /var/log/keystone

    keystone.log  keystone.log-20151004.gz  keystone.log-20160111

  5. Display the horizon service logs

    [root@controller ~]# ls /var/log/horizon

    horizon.log

  6. SSH to compute node #1!

    [root@controller ~]# ssh root@compute1

  7. CD to the directory that contains the instances

    [root@compute1 ~]# cd /var/lib/nova/instances

  8. List the directory and note the instances

    [root@compute1 instances]# ll

    drwxr-xr-x. 2 nova nova 69 Oct  4 20:20 07334a4c-dac3-474b-83f0-1d8f4db1f093 <-- Instance#1
drwxr-xr-x. 2 nova nova 69 Nov  4 20:26 6a56b82f-547d-440b-9ebc-04813a343a2a <-- Instance#2
drwxr-xr-x. 2 nova nova 53 Nov  4 20:25 _base
-rw-r--r--. 1 nova nova 73 Nov  4 21:15 compute_nodes
drwxr-xr-x. 2 nova nova 91 Nov  4 20:25 locks

  9. Change directory into Instance#1

    [root@compute1 instances]# cd iiii where iiii = instance#1 shown above

  10. List the files in this directory

    [root@compute1 07334a4c-dac3-474b-83f0-1d8f4db1f093]# ll

    -rw-rw----. 1 nova qemu   28356 Oct  3 00:03 console.log  <--cat console.log | tail -20
-rw-r--r--. 1 qemu qemu 1703936 Oct  3 00:04 disk         <-- BORING to look at, ephemeral storage
-rw-r--r--. 1 nova nova      79 Oct  2 23:24 disk.info    <-- cat disk.info
-rw-r--r--. 1 nova nova    2715 Oct  2 23:27 libvirt.xml  <--less libvirt.xml

  11. Inspect the log files shown above. No special analysis required on your part, just check out the logs

    Use the cat or less command to look at thes files. Hints of what to use are shown above

2. Introduction to rsyslog (or remote logging) (Reference ONLY)

All systems generate and update log files recording their actions and any problems they encounter. In a distributed or cloud computing environment that contains many systems, and it may be unreasonable to expect admins to collect log files from these many locations. Remote logging is the solution to this problem.

The rsyslog service provides a centralized logging server, and configuration for individual systems to send their log files to the centralized logging server.

  1. WARNING WARNING WARNING You won't actually do these steps, but it might be useful to see how they would be performed in a production system.

  2. Firstly, install the rsyslog server on the system you want to use as the centralized logging server. All other nodes will be configured to send logs to this location, and they too will need rsyslog installed. We'll assume an installation onto a CentOS system (this is what you're using in class)

    • # yum install rsyslog <--Read Step 1 above, this is FYI

    • To be clear, you'd install rsyslog on the centralize logging server, as well as every node you wish to have report logging.

  3. Allow SELinux to allow rsyslog traffic (centralized server only)

    • # semanage -a -t syslogd_port_t -p udp 514

  4. Configure /etc/sysconfig/iptables to allow rsyslog traffic, then restart the service so the new rules take effect (centralized server only)

    • Note -- IP TABLES ACCEPT rules need to appear before REJECT rules

    • # vim /etc/sysconfig/iptables

    • -A INPUT -m state --state NEW -m udp -p udp --dport 514 -j ACCEPT <- place this line inside of /etc/sysconfig/iptables

    • # service iptables restart

  5. Configure /etc/rsyslog.conf to define where the logs will be saved to. Place the following snippet in rsyslog.conf (centralized server only)

    • # vim /etc/sysconfig/iptables
    $template TmplAuth, "/var/log/%HOSTNAME%/%PROGRAMNAME%.log" 

authpriv.*   ?TmplAuth
*.info,mail.none,authpriv.none,cron.none   ?TmplMsg

  6. Before you exit and save /etc/rsyslog.conf, uncomment the following lines

    $ModLoad imudp
$UDPServerRun 514

  7. The central server is now configured-- time to turn our attention to the individual nodes.

  8. On each node (not the centralized server), edit the /etc/rsyslog.conf, and specify the address of your centeralized logging server with the following:

    • *.* @REPLACE_WITH_IP_ADDRESS_OF_SERVER:PORT_SERVER_LISTENING_ON

    • The port used in the above statement would be 514 if you are following this procedure as it is written

  9. Start rsyslog, and then confirm that rsyslog will always run on startup (do this on the centralized server AND all nodes)

    • # service rsyslog start

    • # chkconfig rsyslog on