PCI DSS 10.6 requires that logs are reviewed on a daily basis. If the proper alerting mechanisms are in place you can bypass the need to look through thousands of lines of logs, but you still need a trail of what actions are taken after an alert is received.
PCI DSS 10.6
10.6
Are logs for all system components reviewed at least daily? Log reviews must include those servers that perform security functions like intrusion detection system (IDS) and authentication, authorization, and accounting protocol (AAA) servers (for example, RADIUS). Note: Log harvesting, parsing, and alerting tools may be used to achieve compliance with Requirement 10.6.
I am using XMLRPC and a perl script to automatically create issues in Jira when a Splunk alert is triggered. Although you can use Splunk to perform log review by tagging events I thought it would be more beneficial to use our currently installed work flow system so that we don’t have to allow everyone to log into Splunk and we could use some of the other features inside of Jira as well. So what do you need to do to make this work?
1. Install necessary CPAN modules (assuming you already have perl) on your splunk server.
#cpan -e
cpan> install XMLRPC::Lite
...
cpan> install Data::Dumper
...
1.b Enable XMLRPC in Jira under administration -> Plugins -> RPC Plugin.
2. Copy the following script in $SPLUNK_HOME/bin/scripts/openJiraTicket.pl
#!/usr/bin/perl
#
# * Your search can trigger a shell script.
# * Specify the name of the shell script to run.
# * Place the script in $SPLUNK_HOME/bin/scripts.
# * Command line arguments passed to the script are:
# * $0 = script name.
# * $1 = number of events returned.
# * $2 = search terms.
# * $3 = fully qualified query string.
# * $4 = name of saved splunk.
# * $5 = trigger reason (i.e. "The number of events was greater than 1").
# * $6 = link to saved search.
# * $7 = a list of tags belonging to this saved search.
# * $8 = file where the results for this search are stored (contains raw results).
use strict;
use warnings;
#
use XMLRPC::Lite;
use Data::Dumper;
#
my $summary = $ARGV[3];
my $description = "An alert was triggered in splunk with the following information:" . "\n\nSearch String: " . $ARGV[1] . "\nNumber of Results:" . $ARGV[0] . "\nAlert Name:" . $ARGV[3] . "\nReason Alarm Triggered:" . $ARGV[4] . "\n\nLink to Search:" . $ARGV[5];
#
my $jira = XMLRPC::Lite->proxy('http://jira.domain.com:8080/rpc/xmlrpc');
my $auth = $jira->call("jira1.login", "username", "password")->result();
my $call = $jira->call("jira1.createIssue", $auth, {
'project' => 'AI',
'type' => 30,
'summary' => $summary,
'description' => $description,
'assignee' => 'usertoassignto',
});
my $fault = $call->fault();
if (defined $fault) {
die $call->faultstring();
} else {
print "issue created:\n";
print Dumper($call->result());
}
$jira->call("jira1.logout", $auth);
3. Add a new input to your savedsearches.conf file in $SPLUNK_HOME/etc/system/local/savedsearches.conf. Note the action_script is set to openJiraTicket.pl which means whenever the relation field is met the script openJiraTicket.pl will be executed.
[PCIDSS-10_3_4-LinuxFailedLogins]
action_email = linux.admins@domain.com
action_rss = 0
action_script = openJiraTicket.pl
counttype = number of events
enableSched = 1
quantity = 0
relation = greater than
role = productionLinux
schedule = 0 * * * *
search = index=production pam_unix failure startminutesago=60
sendresults = 1
userid = myuser
4. Now create a few failed login attempts via ssh to a server that is logging to Splunk and you should see an issue get opened in Jira by Splunk.
Some other enhancements that can be made to this setup are:
1. Creating an event inside of Splunk itself when the issue is opened in Jira. Then log when the Jira ticket is closed to Splunk as well and measure the variance in time of the opening and closing of issues as a KPI of the operations staff.
2. Extend the functionality of the perl script with an array (or maybe use sqlite) to match the alert name in Splunk to a component in Jira. This would allow you to compartmentalize the alerts into groups in Jira (i.e windows in one component, linux in another, network in another, etc).
3. Use encrypted password in the perl script and use XMLRPC over ssl. I will be doing this shortly. A quick howto
Encrypt your password file
openssl rand 32 -out key.temp
openssl des3 -e -pass file:key.temp -in passwd -out passwd.enc
openssl des3 -d pass file:key.dec -in passwd.enc -out passwd.dec
openssl des3 -d -pass file:key.dec -in passwd.enc -out passwd.dec
openssl des3 -d -pass file:key.temp -in passwd.enc -out passwd.dec
Unencrypt your password from within the script using the key. I learned chomp is VERY important 
if ($enpasswd) {
$passwd = `openssl des3 -d -pass file:/etc/pki/tls/private/dbi.key -in $enpasswd |awk NF`;
chomp($passwd);
}
