Hi there 👋

This article compares the performance of AMD-based hardware against our existing Intel based hardware of the same generation. The discussion includes analysis of impact on Splunk performance, disk performance, and CPU performance with NUMA enabled and disabled. The full post is available on medium as Splunk Indexing Tier: Intel and AMD Hardware Performance Comparison. A Splunk lantern post is pending.

This article compares the data balance between indexers when using an intermediate forwarding tier, in comparison with forwarders sending direct to indexers without the intermediate tier. Data balance is compared between the two environments using a few indexes and criteria is provided for when an intermediate tier may be of benefit. The full post is available on medium as Splunk load balancing — does adding an intermediate tier actually help?. A Splunk lantern post is pending.

A deep dive into a failure to cleanly restart Splunk while running under systemd on Linux. This post discusses tracing options, problems found with Oracle Linux version 8 and Splunk restarts with particular TA’s. The full post is available on medium, Splunk, systemd, and a clean shutdown, and on Splunk Lantern as Troubleshooting slow Splunk platform restarts when running under systemd on Linux

This article summarises the process for using the Splunk TA OTel collector with a custom built OpenTelemetry (OTel) collector. Examples are provided to reduce the size of the overall application with the intention to run OS host monitoring using a smaller custom built OTel collector. The full post is available on medium, Building a custom OpenTelemetry collector (OTel), and on Splunk Lantern as Building a custom OpenTelemetry collector.

The pstack and eu-stack utilities do not exist inside the Splunk container, and therefore if your using the Splunk operator for Kubernetes it can be difficult to gather stacks for support purposes. This article demonstrates how to build a container that includes eu-stack’s for debugging purposes and a script you can use to obtain stacks when working with Splunk support. Stacks are useful for any situation where performance of the instance is an issue. ...

A year has passed since the article, Splunk Operator for Kubernetes (SOK) — Lessons from our implementation, this article discusses new learnings using the SOK. The full post is available on medium, Splunk Operator for Kubernetes (SOK) — Lessons from our implementation part 2 . A Splunk lantern version is available as Splunk Operator for Kubernetes: Advanced operational learnings.

This series of articles provides performance tuning tips for the Splunk platform at search head, indexing and forwarder tiers. Each article contains links to practical examples of how the tips can be implemented in another environment. The original posts are availabe on medium as Splunk — performance tuning tips — forwarding tier, Splunk — performance tuning tips — search head tier, and Splunk — performance tuning tips — indexing tier The lantern versions of the article are available as Performance tuning the forwarding tier, Performance tuning the search head tier and Performance tuning the indexing tier.

A discussion around Splunk metrics indexes and how using different ingestion methods changes the bucket. In our production environment, we encountered challenges with high cardinality metrics indexes, specifically those receiving metric data with a large number of unique dimension values. The full post is available on medium, Splunk indexers — metrics data — HEC fields vs INDEXED_EXTRACTIONS changes the bucket, and on Splunk lantern, Preventing premature bucket rolling in metrics indexes.

In addition to the Splunkbase apps, I have written various automation components related to Splunk. Each repository has a unique purpose as described in the following sections. Splunk This repository contains various scripts that I have built over the years, the most popular would be the transfersplunkknowledgeobjects.py script. This first script is documented in the README.md of the repository and exists to assist in search head to search head (or search head cluster to search head cluster) migrations using the REST API. Note that POST’ing too many searches to the savedsearches REST API can be very disruptive to the Splunk scheduler and this works best for small migrations. ...

Over the years I have created a number of Splunk-related applications. This post exists to summarise the applications I have created, along with why they might be useful within your environment. Alerts For Splunk Admins GitHub link First published in 2017, this application was created after previous Splunk conf presentations would demonstrate an interesting platform-focussed dashboard, and then, they would not provide any source code or example searches to build the dashboard. I decided to create a presentation catered to Splunk administrators with all searches and dashboards available after the presentation. ...