Log management isn’t a new concept. As long as there have been computers they have generated log data. Logging is the backbone of the IT monitoring and management process. And, until recently, the ease of generating log data was inherently greater than the ease of making this data actually useful, let alone actionable.
For every company, log data is generated by a variety of servers and applications from tracking the health of the system, to detailed specifics on events and processes. It is not enough to know just what needs to be monitored and regulated; it is imperative to also ensure the right tool is in place to efficiently drill down in the mountains of data to identify, extract, analyze and create timely response to the most valuable insights.
Log management requires bringing all the log data from any given environment and streamlining the volume and variety into one place. Log data generated from various devices, operating systems and platforms – that are also geographically scattered – pose a challenge when attempting to cohesively collect and input this disparate variety of log data in its entirety into one central solution. Additionally, the cost of storing all this data is an issue that has been addressed before by YourDailyTech and is an issue that is clearly a component of evaluating any log management solution.
Finding the right tool to overcome all these issues can be difficult when you are facing all the options available on the market. The first thing to know is that these tools come in two different types of products: on-premise solutions and cloud-based Software as a Service (SaaS) solutions.
Logentries’ Visual Search feature generates graphs and timelines based upon top trends found within your data without ever typing a query. Visual Search allows you to simply click to drill in and out of datasets to identify trends and anomalies within your own data, while automatically building the search query for you. Users quickly focus in on key information to better analyze, understand, and share insightful log data information to a company’s entire team.
Logentries by Rapid7 is a leading player in the cloud based log management and analytics service market. Logentries is a software as a service (SaaS) provider for log management and intelligence. Logentries has a key competitive distinction we want to point out up front: it is not built upon any other tools – like Elasticsearch – and thus does not depend upon nor is subject to limitation of that infrastructure.
Logentries has limitless scalability regardless of the daily data generation, the log format of the data, or the environment where log data is extracted from. Logs may be streamed to Logentries via agent, token, library, syslog, or directly from firewalls, load balancers, servers, routers, and more. Considering all these factors, with Logentries, logs can be easily organized to account for auto-scaling the environments.
Logentries does not require any reformatting of data for analyzing and understanding the information contained within the logs. With universal log format support and a SQL-like query language (LEQL), it is easy and possible to analyze your logs to identify key trends and correlations to reveal previously hidden or undiscovered insights.
Though with logs in the JSON format, there are several additional features not widely available to all formats including visual search. Most recently, Logentries has released visual search from beta to open availability.
This new visual search feature generates graphs and timelines based upon top trends found within your data. It allows for easy understanding and sharing of log data information to a company’s entire team. Visual search allows you to focus in on specific keys, queries, tags, and more to better understand, analyze, and isolate key information that is pivotal to your operations. Visual Search allows you to simply click to drill in and out of datasets to identify trends and anomalies within your own data and Visual Search automatically builds a search query for you.
Using Visual Search is easy even for those that aren’t as technologically gifted. First, you just choose the logs you would like to visualize. Then, without even having to get into the query builder, all you have to do is select Visual from the mode drop down in the Logentries interface. Then, without ever having to type up a query, you can begin to dig into the data by clicking on any of the charts and getting a better sense of patterns and anomalies within the log data.
Visual Search allows you to avoid learning another query language. However, if you preferred to use the Logentries query language, LEQL is quite intuitive and easy to use. LEQL empowers you to perform advanced calculations such as average, sum, min, max, and percentile, sort your results, specify by time slice and more.
There is also full Regual Expression (RegEx) support which accommodates Logentries users to search for patterns rather than specific keywords. Logentries offers full support of regular expressions, accessible directly in the search bar. Unstructured data can be analyzed with the RegEx field extraction by directly implementing it within the queries. Regular expressions enable the user to perform calculations and group together values extracted from strings of text.
With Logentries, real-time search means searching log events as they occur for immediate and reliable results. It is possible to tail all aggregated log files from one location in real-time. Live Tail mode enables customers to search a live stream of their log events as they occur for up-to-the-second monitoring and troubleshooting. This is achieved via the Logentries unique pre-processing layer which analyses your data in real-time as it streams into the Logentries service. To activate Live Tail simply click on the Live Tail button in Log View and any new log events will appear right away.
Not only can you troubleshoot and filter data in real-time by using Logentries’ search functionality, you can also append Tags to the log events to easily identify any important events such as ‘errors’ or ‘exceptions’.
Log data includes information that can be used for debugging, server monitoring, security compliance and more. Visualizing that log data enables trend and correlation analysis while allowing easy sharing with other team members.
The graphical dashboards found in the Logentries tool create powerful data visualization for log analysis. Produce multi-line graphs, bar charts, pie charts, histograms and more. If you prefer another graphing tool, it is also possible to export the dashboards to Hosted Graphite or Geckoboard or other preferred services with one click.
Once a user has queried the logs to filter for significant events, the Context button can be used to unfilter surrounding events to understand what occurred before and after to help identify a root cause. Additionally, with the feature of being able to create and assign custom tags, it is simple to identify the events that are the most meaningful to a company. Custom tags act as visual markers within the log stream and timeline.
When critical issues occur, the time it takes to find a solution is of utmost importance. Logentries not only streams a customer’s log data in real-time, users can also set up alerts to occur within seconds, not minutes. Alerts can be synced to a specific pattern found within the log data such as inactivity, when expected or scheduled events do not occur, or anomaly detection within the system. Logentries supports integrations with team communication tools including Slack (we love Slack!), HipChat, PagerDuty, and more. Notifications may also be sent to webhooks for any further integration.
Logentries has a robust REST API to automate reports, alerts, user management, integrations, tag creation, and more including REST Query API, REST User Management API, and REST Alerting API. It also affords new opportunities for users to use Logentries to interact programmatically, automate, and integrate with other systems easily. For instance, many customers integrate Logentries with Tableau (read our Tableau server review here) for further data science insight.
Logentries supports a wide variety of operating systems and distributions, including Linux, Windows, Mac, CoreOS, and SmartOS. Additionally, it supports agentless approaches like Syslog. It also integrates with platforms like AWS, Azure, Docker, Fastly, and more. They make it easy to log from any platform with just a few clicks. Logging directly from a customer’s application is achievable with one of Logentries’ many libraries. All popular languages, like Java, Ruby, and .NET are supported.
For a complete list, please visit the Logentries website: https://logentries.com/product/platform-support/.
Please see the graphic below for a complete listing of Logentries’ pricing. Those prices, including the free trial, include unlimited hosts & sources, basic analytic functions (LEQL), custom tagging, packs, and 30 day peak overage protection. Beyond the free trial, live tail, unlimited users, unlimited alerting, webhook integrations, advanced LEQL functions, RegEx, and S3 archiving are all available on the other plans.
For complete details, visit the Logentries support and pricing page: https://logentries.com/pricing/.
Primary buyers of Logentries include IT Operations Professionals, internal IT teams, and IT teams that practice DevOps. For teams that do not have time to custom setup Logentries or the precious time to learn another query language, Logentries offers free Packs which can help guide and ease the process of learning Logentries for customers. Two very popular packs include the Amazon Web Services Cloud Watch pack, which takes logs from all the different amazon products that a company has and streamlines the log into one place. This pack was built in conjunction with Amazon engineers so it follows best practices and is a top DevOps tool for many customers. Secondly, a popular choice for internal IT teams, the Microsoft SQL Pack provides out of the box functionality for Microsoft customers using the popular database solution. For a complete list of the Packs, view their Packs homepage: https://logentries.com/resources/packs/. A complete list of plug-ins is also available under the resource tab.
Logentries was originally a start-up that was incorporated in 2010. The story begun in 2008 when Viliam Holub joined the Performance engineering group in University College Dublin to work on a project with IBM. The project was aimed at developing tools for systems testers and was led by a colleague Trevor Parsons (co-founder) and prof. John Murphy.
The project was called the Run-Time Correlation Engine and collected, normalized, correlated, and presented log events in a convenient way for system testers. As it appeared to be fairly useful, they decided to develop a cloud based log management and analysis engine in 2010 as a general service available to the masses and (not only to) cloud systems. In October 2015, Rapid7, Inc. acquired Logentries for “an aggregate purchase price of approximately $68 million.”
As previously mentioned, the log management market is quite competitive. One of the leading organizations for on-premise solutions log management is Splunk Enterprise, which “monitors and analyzes machine data from any source to deliver Operational Intelligence to optimize a company’s IT, security and business performance. With intuitive analysis features, machine learning, packaged applications and open APIs, Splunk Enterprise is a flexible platform that scales from focused use cases to an enterprise-wide analytics backbone.” They also offer the Splunk Cloud, a solution that delivers the benefits of Splunk Enterprise “as a cloud-based service and enables companies to build custom apps. Splunk Cloud allows centralized visibility across cloud, hybrid and on-premises environments.”
Another alternative option is Loggly which includes JIRA software integration, New Relic integration, the ability to analyze any kind of logs. However, Logentries offers greater customization of the alert queuing and alert delivery customization. Another alternative option is Loggly which includes JIRA software integration, New Relic integration, the ability to analyze any kind of logs. However, Logentries offers greater customization of the alert queuing and alert delivery customization. Additionally, because Logentries is an offering from a larger organization, the support services and resources are more robust. Lastly, the Loggly price structure is higher than that of Logentries, which leads us to state Logentries is our preferred choice.
Logentries offers the most original and compelling solution, complete with the one-of-a-kind easy to use visual search and a fair pricing structure complete with all the tools needed to help your company use log data for bettering your systems and DevOps practices.
Register for the Logentries Feature Walkthrough Webinar
Learn how to collect, manage, monitor & analyze your logs
Are you ready to take control and get the most value from your log data? Register to watch Logentries Product Manager, Justin Buchanan, as he demonstrates exactly how to use Logentries for successful log management and analytics.
This webinar covers:
- Querying your data using LEQL
- Watching real-time data streaming in the Logentries UI with Live Tail
- Adding context to your logs with tags
- Visualizing your logs with dashboards
- Monitoring logs with alerts & popular integrations
- And more!