This blog will provide an overview of centralized log management and why it is so important. We will share our experience on what is most important when starting a logging project and how to avoid most common pitfalls. Also lemons.
Why centralized logging?
This is the question we want to answer today. Why do we need centralized logging and why should you implement it? What are the benefits and what can you expect? Interesting topics to be sure, but at the same time, why alone is quite a boring question. The reasons for having centralized logging are rather obvious but we don’t want to focus on that alone. The saying when life gives you lemons, make lemonade, fits quite well here. The logs are the lemon and you just need to know how to turn it into tasty lemonade with the aid of centralized log management.
The reason why we compared logs to lemons is the nature of logs that exist without any form of management. It can be very chaotic and leave you feeling sour after your storage has become full one too many times. Centralized log archiving can move the logs to another place, but that alone does not solve the issue of storage constraints. Those constraints are an example of why Centralized Log Management is important. It takes a step beyond just central archival of the logs.
Some can also see logs as a necessary evil, something that you are forced to store. There are quite many laws which require that companies storing personal data must also monitor and log access to it. Even logs themselves might contain personal data, like location or payment information. In EU, GDPR allows legitimate use of consumers personal information, as long as we take the necessary precautions to protect it and we are transparent on how we use it. This practically means monitoring and controlling who can access and view said data.
More local legal regulations can define for example how long a company must store it’s logs. Such legal requirements can quite easily push a company to adopt centralized log archiving in a quick manner. This is where things can get a bit messy and while logs are collected and stored, there is no real good way to leverage them. Using our previous example of comparing logs to lemons, we can make a comparison that the logs are lemons that still hang in a tree. When using central log archiving, we only collect those lemons into a basket. They’re all in the same place but we’re not using them. With centralized log management, we can process them into lemonade.
Tied to the legal requirements is security. Centralized log archiving can provide you with a single location to store and access all your audit, access and any other log. But security is not just storing logs, those logs need to be taken advantage of and analyzed. Systems like Security Information and Event Management (SIEM) can help with threat analyzing and detection. Alerts for certain actions or logins, dashboards for traffic based on access logs and traceability between applications with trace tokens to see what a user has done, are some of the possibilities that centralized log management helps enable. These features can help not only to improve a company’s security but also bring them value and important information from their systems.
Troubleshooting an issue from logs is something every developer is familiar with and having logs centralized somewhere with easy access will certainly improve developers’ life. Aforementioned trace and event logs from applications can be used for monitoring, troubleshooting and security purposes. Even the possibility of using your events and tracing for machine learning can help to understand your users and further improve your system.
To be able to take advantage of centralized logging you need to understand your needs for both, the present and the future. While sizing is more flexible today, especially when working in the cloud, the underlying architecture and logic needs to be planned beforehand. It is important to know the entire lifecycle of your logs, from the applications where they’re generated, all the way to the long-term storage where they’re retained.
Each log that is logged should have a clear purpose. This can’t be stressed enough. Every log has to mean something important. It’s useless to log anything that doesn’t provide any value.
There are exceptions, like when debug logs are needed for troubleshooting, but generally in its default state only the important, descriptive logs should be collected and stored. Configuring event or trace logs from applications can be very beneficial, but these need some work from the development teams. Centralized logging management projects are very much a group effort. Both developers, infrastructure, security and other parties need to work together and understand what they want and what they actually need from centralized logging management.
This all might seem like quite a lot of work and that’s because it is. But this is where good planning, proper project management and agile development comes into play. There should be a clear start and an end to the project. Not everything needs to be done immediately. Nothing is built over night, so take your time to improve and add features as time goes by and your system is ready for them. Just keep in mind what is the goal you want to achieve and what kind of value it is that you want to generate in the end. We will talk more about this in our next centralized log management blog post that dives deeper into how to start this type of a project.
There will be value
The value of centralized log management reaches further than just meeting requirements and helping accelerate development. The ability to transform your log data to alerts, dashboards, threat detection or even use them for machine learning can help to appraise the logs that are filling your disks. Centralized logging doesn’t need to be just a legal obligation or a development tool, it can be both. It can provide multiple avenues of value while meeting those pesky requirements. But as it was already said, achieving this will take time and the approach should be well planned and methodical.
With new technologies constantly emerging and becoming popular, it also challenges us to change with it. Containerized workloads running on orchestrators like Kubernetes challenges the way we think about our softwares lifecycle. And all this statelessness needs a centralized way of managing things as old ways are no longer applicable.
At least for your logging needs the transformation is easy, you can just contact us and we will help you to design, develop, deploy and take care of your centralized logging management. Or just keep reading out blogs as they come out.
Solita is a technology, data and design company with years of experience working in cloud and on-premises. We have helped companies in their transformation into technology driven organizations and brought centralized log management into their lives. Our experience rests both in large and small setups with data from few hundred gigabytes per day into terabytes of data. Technologies behind us are commonly know and popular, Elasticsearch, Opensearch, Graylog and cloud specific services like CloudWatch, to mention a few. These will be the vocal point in our examples as these are what we work with day-to-day basis. Migration from on-premise to cloud and changing technologies is also something we are very familiar with as cloud is constantly gaining more popularity.
In our next Centralized Logging Management blog we will talk about how this kind of project should be started and how it’s actually done properly from start to finish. At later date we will return for more in-depth technical view on different features and how to use them.