The first challenge you may encounter when trying to monitor files on a Linux server is when you’re trying to access files your Splunk user is not allowed to read.

Some people would dismiss this problem by adding their Splunk user to the root/admin group. However, this is not the best solution as it opens the way to a lot of trouble. Your Splunk instance or user could be compromised and be used for a privilege escalation on the server.

The issue:

Let’s take /var/log/messages as an example. Use this command to find the warnings:

index="_internal"  file='/var/log/messages' log_level="WARN" component=TailReader

You can add host="<name_of_your_host>" if you’re not monitoring locally.

You should find this event:

09-24-2019 19:39:45.847 +0100 WARN  TailReader - Insufficient permissions to read file='/var/log/messages' (hint: Permission denied , UID: 1001, GID: 1001). 

At the Linux level, you can check the permissions with the command:
ls -lart

-rw-------.  1 root   root        0 Sep 22 02:10 messages 

As a reminder you can use UGO to figure out how the permissions work:
U = User/Owner of the file (Root here)
G = Its Group (Root here)
O = Others (Spunk for example)

And we can definitely see that only Root has got read and write on that file. Splunk user will not be able to read it.

The Solution: Access Control Lists (ACLs)

While there are different ways to go about solving these permission issues, ACLs have been created specifically for this scenario.

The downside is that you need to have root access or a sudo role to be able to modify the file’s permissions. This can also be solved by asking your Linux admin, if you’re not the owner of the server.

In our case, let’s pretend we have root access and we can directly change the permissions. Here is what should be done:

Go to the /var/log folder and use getfacl command on the messages file:

[root@localhost log]# getfacl messages
# file: messages
# owner: root
# group: root

[root@localhost log]# setfacl -m g:splunk:r /var/log/messages
[root@localhost log]# getfacl messages
# file: messages
# owner: root
# group: root

Now we can see that Splunk group has been added to the list. However, it is not yet sufficient as next time the file will rotate, this will disappear. We need to make it persistent.

Linux rotates its logs following a configuration that is defined in /etc/logrotate.conf and it also reads and applies everything defined in /etc/logrotate.d/ . We will try not to modify /etc/logrotate.conf directly and add a custom file instead. You can deploy this file automatically when provisioning new servers in the future.

Create a file in that folder and name it splunk_logaccess or anything that helps you identify it.

Add these lines and save the file:

        /usr/bin/setfacl -m g:splunk:r /var/log/messages

That’s it, you only need to restart Splunk and the logs will be picked up!

Posted by:Laurent Lacoste

Laurent Lacoste has been part of iDelta for 3 years and a half and has been focused on providing his knowledge and expertise to our clients since then. He is Splunk Certified Consultant and recently re-certified for Architect 7.3. He holds an IT Engineer Diploma and comes from a security and linux administration background.