In this age of Big Data, an organization’s ability to nimbly respond or adapt to variables affecting their operations — such as sales or supply chain issues, or even the weather — increasingly depends on how well it copes with the vast amounts of data and metadata flowing into its networks.
Splunk is a helpful tool for visualizing, monitoring, and analyzing all of that data and metadata.
Splunk is software for searching, monitoring, and analyzing machine-generated data from various sources through a Web-style interface. Machine data refers to data generated by websites, applications, sensors, RFID tags — any mobile or computing device you can imagine. This data can be collected and indexed in real time.
Digging Deep into Unstructured Data
It’s worth mentioning that Splunk is a corruption of the word “spelunk,” which means to explore caves. “Our customers told us that finding their IT problems was like ‘digging through caves with headlamps and helmets, crawling through the muck,’ ” notes the company on its website. Splunk adds:
“According to IDC, unstructured data, much of it generated by machines, accounts for more than 90% of the data in today’s organizations. This type of machine-generated data is massive in scale and contains a definitive record of transaction activity, system behavior, application performance, user actions, security threats and fraudulent activity. Traditional technologies built on relational or multi-dimensional databases cannot handle the complexity or scale of today’s massive volumes of machine data. Nor do they allow the flexibility to ask any question or get questions answered in real time — which is now an expectation of users.”
These are the challenges Splunk concentrates on solving for its users. Splunk is great for unstructured data and for monitoring all the inconsistent metadata we have. We have a lot of both. You do, too.
Getting Started with Splunk
Starting with Splunk is as easy as downloading the application. There are three different versions: Free, Splunk Light, Enterprise, and Cloud. After downloading the application, you’ll need some type of logs to analyze, which could beApache logs, or an Oracle alert log from your application server. The Splunk interface can run in several different environments. You will also need sufficient storage space. If you collect roughly 5 GB of data per day and plan on analyzing a month’s worth of data, about 200 GB of storage space would be a great place to start.
There are a lot of different metrics you can capture with Splunk — you are not restricted to gathering SQL execution data. You can look at Oracle alert logs, database exception logs, concurrent requests, code changes, or the processing times of running jobs. Splunk is flexible.
Splunk Customization and Collaboration
There are a lot of useful features in Splunk. In fact, there are too many to mention! With Splunk, you can segregate data, as well as manage access. Here are some of the other useful features:
- Permissions enable you to manage individual user access, which may include specifying users able to modify the dashboard, or those permitted to search the indexes.
- Forwarders are another helpful and very powerful feature that provide a fast way to get data from a variety of sources right into your Splunk instance.
- Splunk works with existing technologies such as MySQL, SQL Server and Oracle.
- Although Splunk isn’t designed to be a database monitoring tool, you could easily use it to monitor databases. This is one of our favorite uses for it here at Datavail. I even led a webinar on using Splunk to build a database monitoring tool, a recording of which can be found here.
- Splunk integrates with Hadoop, which allows even more flexibility and vast data collection capabilities.
- You can transport Splunk data to other custom applications, such as one created for Salesforce, using Splunk APIs.
- A mobile-related feature is available via BugSense Mobile that extends the capabilities into the mobile realm.
This should give you a basic idea about the breadth of tasks you can accomplish with Splunk. There is also a vibrant, connected developer community willing to help those wanting to implement and configure it.
Datavail Can Help
We can assist your staff with implementing and configuring Splunk to meet your database needs, whether with crafting appropriate tools, integration, or optimization.
If you need immediate startup help or ongoing assistance with Splunk or other database applications and tools, please contact one of Datavail’s customer service representatives to discuss a custom solution for your enterprise.
Image by omnimages/123RF.
EPM applications help measure the business performance. This post will help you choose the best EPM solutions for your organization’s needs and objectives.
It’s 2015 and you can now establish totally respectable MS SQL DBA credibility just by mentioning you have been in the game since SQL Server version 9. You may even get the same gasps of shock from some colleagues that used to be reserved for the version 6 veterans.