The first challenge you may encounter when trying to monitor files on a Linux server is when you’re trying to access files your Splunk user is not allowed to read, such as root-owned files.

Some people would dismiss this problem by adding their Splunk user to the root/admin group. However, this is not the best solution as it opens the way to a lot of trouble. Your Splunk instance or user could be compromised and be used for a privilege escalation on the server. A much better solution is to use Access Control Lists, as described below.

The issue: Insufficient permissions to read root-owned files

Let’s take /var/log/messages as an example. Use this command to find the warnings:

index="_internal"  file='/var/log/messages' log_level="WARN" component=TailReader

You can add host="<name_of_your_host>" if you’re not monitoring locally.

You should find this event:

09-24-2019 19:39:45.847 +0100 WARN  TailReader - Insufficient permissions to read file='/var/log/messages' (hint: Permission denied , UID: 1001, GID: 1001). 

At the Linux level, you can check the permissions with the command:
ls -lart

-rw-------.  1 root   root        0 Sep 22 02:10 messages 

As a reminder you can use UGO to figure out how the permissions work:
U = User/Owner of the file (Root here)
G = Its Group (Root here)
O = Others (Spunk for example)

Here we can definitely see that only Root has got read and write permissions on that file. Splunk user will not be able to read this file.

The Solution: Access Control Lists (ACLs)

While there are different ways to go about solving these permission issues, ACLs have been created specifically for this scenario.

The downside is that you need to have root access or a sudo role to be able to modify the file’s permissions. This can also be solved by asking your Linux admin, if you’re not the owner of the server.

In our case, let’s pretend we have root access and we can directly change the permissions. Here is what should be done:

Go to the /var/log folder and use getfacl command on the messages file:

[root@localhost log]# getfacl messages
# file: messages
# owner: root
# group: root

[root@localhost log]# setfacl -m g:splunk:r /var/log/messages
[root@localhost log]# getfacl messages
# file: messages
# owner: root
# group: root

This shows us that the Splunk group has been added to the list, and we can monitor this root-owned file in Splunk. However, it is not yet sufficient as next time the file will rotate, this will disappear. So, we need to make it persistent.

Linux rotates its logs following a configuration that is defined in /etc/logrotate.conf and it also reads and applies everything defined in /etc/logrotate.d/ . We will try not to modify /etc/logrotate.conf directly and add a custom file instead. You can deploy this file automatically when provisioning new servers in the future.

Create a file in that folder and name it splunk_logaccess or anything that helps you identify it.

Add these lines and save the file:

        /usr/bin/setfacl -m g:splunk:r /var/log/messages

That’s it, you only need to restart Splunk and the logs will be picked up!

Alternative method

Alternatively, if you want to use setfacl to grant the Splunk user access to a directory and all the sub-directories and files therein, then you should use the command below. It has the added advantage that all new files and directories created in that location will also inherit the setfacl settings – so no need for the postrotate script shown above.

-R = recursive
-d = make this the default for new files and directories
-m = modify

For example:

[root@localhost log]# setfacl -Rdm g:splunk:r /var/log/
[root@localhost log]# setfacl -Rdm g:splunk:r /applogs/

For 2021 we’ve committed to posting a new Splunk tip every week!

If you want to keep up to date on tips like the one above then sign up below:

Subscribe to our newsletter to receive regular updates from iDelta, including news and updates, information on upcoming events, and Splunk tips and tricks from our team of experts. You can also find us on Twitter and LinkedIn.


* indicates required
Posted by:Laurent Lacoste

Laurent Lacoste has been part of iDelta for 3 years and a half and has been focused on providing his knowledge and expertise to our clients since then. He is Splunk Certified Consultant and recently re-certified for Architect 7.3. He holds an IT Engineer Diploma and comes from a security and linux administration background.