Information Security Assurance Frameworks

There are many ways to demonstrate the maturity level of information security management. National and international standards and frameworks can be used as criteria for measuring the level of information security. Here is a brief overview of common frameworks in use.

ISO/IEC 27001

International Organization for Standardisation (ISO)  and International Electrotechnical Commission (IEC) maintain and publish the ISO/IEC 27001 standard on information security management system (ISMS) and its requirements. It is part of the 27000 family of standards which address information security. ISO/IEC 27001 is probably the most famous one, because it is the one that can be certified. It emphasises risk-based approach, continuous improvement and commitment from the top-management.  The standard itself has seven mandatory clauses and Annex A, which defines controls in 14 groups to manage information security risks. ISO/IEC 27001 certification requires a third-party audit by an independent certification body, so certified organisations can be trusted to have an implemented and maintained information security management system. 

It should be noted that the audit does not necessarily provide assurance on how well the controls have worked, merely that they exist. It is also a good idea to examine the scope of the management system, as it might cover only some of the operations of the organisation. Statement of Applicability is another document that should be examined; it defines which controls have actually been implemented and which have been left out and why. 

Note: The standard is being reviewed and based on changes in ISO/IEC 27002  (implementation guide of the controls in Annex A) there will be some changes.

Mandatory clauses Annex A
Context of the organisation Information security policies
Leadership Organisation of information security
Planning Human resource security
Support Asset management
Operation Access control
Performance evaluation Cryptography
Improvement Physical and environmental security
Operations security
Communications security
System acquisition, development and maintenance
Supplier relationships
Information security incident management
Information security aspects of business continuity management
Compliance

 

Assurance Reports

ISAE 3000, ISAE 3402 and SOC 2® are standards and frameworks for assurance reports. Assurance reports provide independent attestation that a service provider has adequate controls of the subject matter, in this case information security. They are more common in the United States and Canada, but also used in Europe. Cloud providers or other service providers utilising cloud services often have some assurance report.

ISAE 3000 

International Standard on Assurance Engagement 3000 is a standard which defines how assurance engagements other than financial audits should be conducted. It does not define any controls in itself, but rather how the auditing should be done. The reader of an ISAE 3000 assurance report automatically knows that the assurance is conducted objectively and independently in a professional manner. It is up to the subject matter and the criteria whether it provides assurance and what sort of assurance on information security.

ISAE 3402

ISAE 3402 is also an international standard on assurance engagements, it focuses on internal controls of a service provider. Like ISAE 3000, it does not define any exact criteria to be used, but they have to be applicable to the subject matter. 

SOC 2®

SOC 2® or System and Organisational Controls 2 is AICPA’s (American Institute of Certified Public Accountants) framework for information security assurance for service providers. The abbreviation should not be confused with Security Operations Center! It uses Trust Service Criteria (TSC) as the criteria used to assess the level of information security. The TSC includes requirements on security, availability, processing integrity, confidentiality and privacy. For a SOC report, the security requirements are mandatory.  An official SOC 2® report can only be given by an AICPA’s certified public accountant, which is bypassed by the rest of the world with ISAE 3000 reports that are compliant with the SOC 2® framework. 

ISAE 3000, ISAE 3402 and SOC 2® can be done either as Type I or Type II reports. Type I provides assurance that the controls are described and suitable and is similar to an ISO/IEC 27001 certification. Type II provides assurance that, in addition to being described and suitable, the controls have also operated effectively during the audit period (typically 12 months). For example, for Type I report the auditor might inspect that a policy and procedure for incident management exists. For Type II report the auditor would also inspect a sample of incidents that occurred during the audit period to ensure the procedure was followed. 

It is worth noting that the actual reports are not publicly available, although there might be a summary of such assessment having been done. However, the reports can be requested from business partners or when negotiating possible partnerships. It also requires some level of expertise in security and auditing to assess controls descriptions and testing procedures in the reports. 

KATAKRI and PITUKRI

KATAKRI or Kansallinen turvallisuusauditointikriteeristö, literally national security audit criteria, is a comprehensive criteria published by the Finnish national security agency. It consists of three domains: security leadership, physical security and technical security. Katakri is used to assess the suitability of an organization to handle officials’ classified information, but as a public criteria can also be used by anyone as a benchmark criteria. 

PITUKRI or Pilvipalveluiden turvallisuuden auditointikriteeristö, literally cloud service security audit criteria is meant for assessing cloud service security in the context of Finnish requirements. 

PCI-DSS

PCI-DSS is an abbreviation for Payment Card Industry Data Security Standard. It is an international standard used to assess the security related to payment card transactions. It was created by major credit card companies and maintained by Payment Card Industry Security Standards Council. Compliance with the PCI-DSS is required in practice from businesses that process credit or debit card transactions. The standard has 12 requirements divided into six groups: secure networks and systems, cardholder data protection, vulnerability management, access control, network monitoring and testing and information security policy. 

The process of PCI-DSS compliance is a three step process of assessing, remediating and reporting. Assessing means identifying cardholder data and relevant assets and processes which are analysed to recognize vulnerabilities. Remediating means fixing the vulnerabilities which is followed by reporting to banks and card brands. Compliance with the standard requires an annual scoping of anything that has anything to do with the cardholder data. The assessment requires a qualified security assessor. 

What to think of certificates and information security assurance?

The importance of information security assurance depends on the business done with the party having (or not having) them. If your business partner, especially on the vendor side, has anything to do with processing or storing your information and data, you should be interested in their information security controls. And if you are providing such services, clients will come easier if they are assured their data is safe and secure. Certifications and assurance reports can also reduce the number of audits: every business partner does not have to do vendor audits if there is independent assurance provided.

As for vendor relationships, information security frameworks might have requirements for vendors and suppliers. Although the responsibility for these controls will be on the certificate holder, they might have effects on business partners too.

If you want to do business with public sector, there will probably be national regulation and requirements. For example, with Finnish public sector attention should be paid to possible Katakri requirements such as related to physical security and doing confidential work in Katakri approved areas. 

Trustworthy assurance requires independent and accredited parties to provide them, such as accredited certification body for ISO/IEC 27001 or CPA for ISAE 3000. The party providing assurance or certification should not provide consultation on the subject, at least not for the customer that is being certified by them. If implementation help is needed, another party should be used. For example, if you want to get ISO/IEC 27001 certified, Solita can help you in the implementation and then a certification body can conduct the audit and grant the certificate.

Most importantly everyone should be aware that certifications and assurance reports do not guarantee impenetrable security against breaches and incidents. Suppliers, customers and partners with a certificate or an assurance report are, however, more likely to be better prepared to recognise, mitigate and handle breaches and incidents when they occur. To get the most out of information security assurance, all interested parties should also know how they are achieved and what subjects they address. 

Cloud-technology is a tool

Us engineers are eager to get our hands dirty and dive into difficult architectural- and implementation tasks. We love to speak about technology and how to implement wonderful constructs. Good architecture and implementation is crucial, but we should not forget to put the customer to the spotlight. Do we have a solid business case and most important where is the value?

To project hardened seniors reading this, what would resonate in the manner to get you chills.

You know the feeling, when things really click. Your peer colleague succeeds in difficult project or the architecture comes together.

To us, who have experienced dozens of projects and clients, we tend to know the direction for the chills for success.

Meaning, we have created models from the past experiences. These models are formal structures explaining the world around.

The models

If we master and understand the models, it improves our communication, enables us to explain difficult scenarios, reason and cooperate with our teammates, predict the outcome from a solution and furthermore explore different options.

When our thinking is logical, consistent and based on past experiences, we are more likely to make wise choices. No matter if we speak about development team trying to implement difficult business case or just us cooking for our family.

The same is true, when a salesperson tries to find out what a customer truly needs or when we are implementing the next cloud-enabled solution for the customer.

Building value

It is all about building value. Is the food you prepare the value or is it merely a tool for you to have energy to play with your children?

Nevertheless I’m sure each and everyone of us have made up models, how to build value and get the chills in different situations.

Good architecture and implementation is crucial, but we should not forget to put the customer to the spotlight.

Do we have a solid business case and most important where is the value? We should not ask what  the customer wants, but what the customer needs.

Try 3 whys the next time you’re thinking of making a big decision:

“Why we are building the infrastructure to the cloud?”

“We need to grow our capacity and we need to be highly available and resilient”

“Why it is important to improve scalability and resilience?”

“The old infrastructure can not handle the increased traffic”

“Why the old infrastructure is not enough anymore?”

“We have analysed the platform usage. If we get more capacity on the rush hours, we can serve more customers.”

Productise

All right! We are convinced the customer needs the cloud migration and start working with the implementation.

Solita has productised the whole cloud journey, so we can quickly get into speed.

Ready made instructions, battle proven implementations and peer-support from earlier projects will help you up to speed.

Technical solutions follow a model, meaning there are few ways to provision a database, few ways to do networking in the cloud, few ways to optimize expenditure.

When we do not plan and develop everything from scratch, we build value.

Re-use

According to study published in the book Accelerate: The Science of Lean Software and DevOps, to be productive, everything should be in version control.

For sure the application code is already in the version control, but also system configuration, application configuration and scripts for automated build, test and delivery should be in version control.

The study revealed, keeping configurations and scripts in the version control correlated more to the software delivery performance than keeping the application code in the version control.

When building for cloud, the infrastructure must be defined in code. And the code should reside.. in version control!

This enables us to move faster without trade offs. We can recognise modules, implement them once, create automated tests and then re-use the codebase in customer repositories.

Every time we are able to do this, we build value.

Automate

Humans are erroneous. This is an experience based model. When computers handle the repetitive work, it enables people to solve problems.

We know from experience, automation requires investments and should be implemented from day one. Investments done here will result smaller development lead time, easier deployments and more quality code.

In cloud, we describe our infrastructure as code. This goes hand in hand with automation. Based on this model, we choose to automate recurring tasks such as building code, testing and making the deployments.

As a result we speed up the feedback loops, have repeatable results each time and enable developers make quality code and automated tests. We test our infrastructure in the pipeline and once again build value.

Deliver

Deliver continuously in small chunks is an experience based model.

You most surely want to have your piece of code tested and delivered to production before you forget, what you were doing.

Short lived branches or Trunk-Based Development predict performance. Failures in small changes are far easier to fix.

Also test automation is key part of continuous delivery. Reliable automated tests predict performance and improves quality. Diagram below is from the book Accelerate. 

High Performers who had adopted the Continuous Delivery model spent far more time on new work than in unplanned or rework.

Although unplanned, emergency and refactoring work is a necessity, value is built when implementing new features.

Software Delivery Performance

Measuring software delivery performance is difficult. An incorrect productivity metric can easily lead to poor decisions.

If you want to deliver the feature as quickly as possible without trading for quality, one key metric is development Lead Time. This is because code not in production is a waste.

For example Software Delivery Performance can be split to 4 topics:

  • Lead Time (from starting implementation to delivery to production)
  • Deployment Frequency (more often results smaller changes)
  • Mean Time to Restore (how to recover from failure)
  • Change Failure Rate (how often the changes cause a failure)

According to the study made by the authors of the book Accelerate, these are the measures from different types of Organisations:

Conclusion

Past experiences makes us what we are. Stop and think the models you have crafted. Challenge yourself, interact with your peers, find out new models for building value.

Together with the team and with the customer, you’ll will find the best solution for the opportunity at hand. Remember the actual implementation is only a fraction of the whole journey.

On pure technical aspect, we should re-use as much as possible and we should automate as much as possible. When talking about cloud migration, infrastructure must be described as code (IaC).

Does your organisation understand and use the models?

Does your organisation productise and re-use the codebase?

You are welcome to join our meetup to hear, how Solita does it:

Mitä ihmettä CloudBlox® tarkoittaa pilviasiantuntijan arjessa?

What does CloudBlox® mean in cloud consultants’ everyday life?

Let’s build value!

 

Avoid the pitfalls: what to keep in mind for a smooth start with cloud services

Many companies are looking for ways to migrate their data centre to the cloud platform. How to avoid potential pitfalls in migrating data centres to the public cloud? How to plan your migration so that you are satisfied with the end result and achieve the set goals?

Why the public cloud?

The public cloud provides the ability to scale as needed, to use a variety of convenient SAAS (Software as a Service), PAAS (Platform as a Service) and IAAS (Infrastructure as a Service) solutions, and to pay for the service exactly as much as you use it.

 The public cloud gives a company the opportunity for a great leap in development, the opportunity to use various services of a service provider during development, as those accelerate the development and help create new functionality.

 All of this can be conveniently used without having to house a personal data centre.

Goal setting

The first and most important step is to set a goal for the enterprise. The goal cannot be general; it must be specific and, if possible, measurable, so that it would be possible to assess at the end of the migration whether the set goal has been achieved or not.

Goal setting must take the form of internal collaboration between the business side and the technical side of the company. If excluding even one party, it is very difficult to reach a satisfactory outcome.

The goals can be, for example, the following:

  • Cost savings. Do you find that running your own data centre is too expensive and operating costs are very high? Calculate the cost, how much resource the company will spend on it, and set a goal of what percentage in savings you want to achieve. However, cost savings are not recommended as the main goal. Cloud providers also aim to make a profit. Rather, look for goals in the following areas to help you work more efficiently.
  • Agility, i.e. faster development of new functionalities and the opportunity to enter new markets.
  • Introduction of new technologies (ML or Machine Learning, IOT or Internet of Things, AI or Artificial Intelligence). The cloud offers a number of already developed services that have been made very easy to integrate.
  • End of life for hardware or software. Many companies are considering migrating to the cloud at the moment when their hardware or software is about to reach its end of life.
  • Security. Data security is a very important issue and it is becoming increasingly important. Cloud providers invest heavily in security. Security is a top priority for cloud providers because an insecure service compromises customer data and thus they are reluctant to buy the service.

The main reason for migration failure is the lack of a clear goal (the goal is not measurable or not completely thought out)

Mapping the architecture

The second step should be to map the services and application architecture in use. This mapping is essential to choose the right migration strategy.

In broad strokes, applications fall into two categories: applications that are easy to migrate and applications that require a more sophisticated solution. Let’s take, for example, a situation where a large monolithic application is used, the high availability of which is ensured by a Unix cluster. An application with this type of architecture is difficult to migrate to the cloud and it may not provide the desired solution.

The situation is similar with security. Although security is very important in general, it is especially important in situations where sensitive personal data of users, credit card data, etc. must be stored and processed. Cloud platforms offer great security solutions and tips on how to run your application securely in the cloud.

Security is critical to AWS, Azure, and GCP, and their security is invested into much more than individual customers could ever do.

Secure data handling requires prior experience. Therefore, I recommend migrating applications with sensitive personal data at a later stage of the migration, where experience has been gained. It is also recommended to use the help of different partners. Solita has previous experience in managing sensitive data in the cloud and is able to ensure the future security of data as well. Partners are able to give advice and draw attention to small details that may not be evident due to lack of previous experience.

This is why it is necessary to map the architecture and to understand what types of applications are used in the companies. An accurate understanding of the application architecture will help you choose the right migration method.

Migration strategies

‘Lift and Shift’ is the easiest way, transferring an application from one environment to another without major changes to code and architecture.

Advantages of the ‘Lift and Shift’ way:

  • In terms of labour, this type of migration is the cheapest and fastest.
  • It is possible to quickly release the resource used.
  • You can quickly fulfil your business goal – to migrate to the cloud.

 Disadvantages of the ‘Lift and Shift’ way:

  • There is no opportunity to use the capabilities of the cloud, such as scalability.
  • It is difficult to achieve financial gain on infrastructure.
  • Adding new functionalities is a bit tricky.
  • Almost 75% of migrations take place again within two years. Either they go back to their data centre or they use another migration method. At first glance, it seems like a simple and fast migration strategy, but in the long run, it will not open up the cloud’s opportunities and no efficiency gains will be achieved.

‘Re-Platform’ is a way to migrate where a number of changes are made to the application that enable the use of services provided by the cloud service provider, such as using the AWS Aurora database.

Benefits:

  • It is possible to achieve long-term financial gain.
  • It can be scaled as needed.
  • You can use a service, the reliability of which is the service provider’s responsibility.

 Possible shortcomings:

  • Migration takes longer than, for example, with the ‘Lift and Shift’ method.
  • The volume of migration can increase rapidly due to the relatively large changes made to the code.

‘Re-Architect’ is the most labour- and cost-intensive way to migrate, but the most cost-effective in the long run. During the re-architecture, the application code is changed sufficiently that it can be handled smoothly in the cloud. This means that the application architecture will take advantage of the opportunities and benefits offered by the cloud

Advantages:

  • Long-term cost savings.
  • It is possible to create a highly manageable and scalable application.
  • An application based on the cloud and micro services architecture enables to add new functionality and to modify the current one.

Disadvantages:

  • It takes more time and therefore more money for the development and migration.

Start with the goal!

Successful migration starts with setting and defining a clear goal to be achieved. Once the goals have been defined and the architecture has been thoroughly mapped, it is easy to offer a suitable option from those listed above: either ‘Lift and Shift’, ‘Re-Platform’ or ‘Re-Architect’.

Each strategy has its advantages and disadvantages. To establish a clear and objective plan, it is recommended to use the help of a reliable partner with previous experience and knowledge of migrating applications to the cloud.

Turbulent times in security

We are currently living very turbulent times: COVID-19 is still among us, and at the same time we are facing a geopolitical crisis in Europe not seen since the second world war. You can and you should prepare for the changed circumstances by getting the basics in order.

On top of the normal actors looking for financial gains, state-backed actors are now likely to activate their campaigns against services critical for society. This expands beyond the crisis zone, we have for example already seen Denial of Service attacks against banks. It is likely that different types of ransomware campaigns and data wipers will also be seen in western countries targeting utilities providers, telecommunications, media, transportation and financial institutions and their supply chains.

So what should be done differently during these times in terms of securing our business and environments? Often an old trick is better than a bagful of new ones, meaning that getting the basics right should always be the starting point. There’s no shortcuts in securing the systems, there’s no single magic box that can be deployed to fix everything.

Business continuity and recovery plans

Make sure you have the business continuity plan and recovery plan available and revised. Also require the recovery plans from your service providers. Make sure that roles and responsibilities are clearly defined and everyone knows the decision making tree. Check that the contact information is up to date, and your service providers and partners have your contact information correct. It is also a good idea to practice cyberattack scenarios with your internal and external stakeholders to see potential pitfalls of your plan in advance.

Know what you have out there!

How certain are you that the CMDB you have is 100% up-to-date? When’s the last time you have checked how your DNS records have been configured? Do you really know what services are visible to the internet? Are you aware what software and versions you are using in your public services? These questions are the same what malicious actors are going through when they are gathering information on where to attack and how. This information is available on the internet for everyone to find out, and this is something that all organizations should also use for their own protection. There are tools and services (such as Solita WhiteHat) available to perform reconnaissance checks against your environment. Use them or get a partner to help you in this.

Keep your software and systems updated

This is something that everyone of us hears over and over again, but still: It is utmost important to keep the software up-to-date! Every single software contains vulnerabilities and bugs which can be exploited. Vendors are nowadays patching vulnerabilities coming to their attention rather quickly, so use that as your own benefit and apply the patches.

Require MultiFactor Authentication and support strong passwords

This one is also on every recommendation list and it’s not there for nothing. Almost all services nowadays provide the possibility to enable MFA, so why not to require it. It is easy to set up and provides an additional layer of security for users, preventing brute forcing and password spraying. It doesn’t replace a good and strong password, so a rather small thing to help users in creating strong passwords and prevent using same passwords in multiple services is to provide them a password manager software, such as LastPass or 1Password. If you have SSO service in place, make sure you take the most out of it.

Take backups and exercise recovery

Make sure you are backing up your data and services. Also make sure that backups are stored somewhere else than in the production environment, to prevent for example ransom trojans making them useless. Of course, just taking backups is not enough, but the recovery should be tested periodically (at least yearly) to make sure that when recovery is needed it will actually work.

What if you get hit

One famous CEO once said that there are two types of companies: ones that have been hacked and ones who don’t know they have been hacked. So what should you do if you even suspect that you have been attacked:

Notify authorities

National authorities run CERT (Computer Emergency Response Team) teams, who maintain the situational awareness and coordinate the response actions on national level. For example in Finland its kyberturvallisuuskeskus.fi and in Sweden cert.se.  So if you suspect a possible data leak or attack, notify the local CERT and at the same time, file a police report. It is also advisable to contact a service provider who can help you to investigate and mitigate the situation. One good source to find a service provider providing Digital Forensics and Incident Response services is from dfir.fi.

Isolate breached targets and change/lock credentials

When you suspect a breach, isolate the suspected targets from the environment. If possible cut off network access and let the resources still run, this way you are not destroying possible evidence by turning off the services (shutting down servers, deleting cloud resources).  At the same time, lock the credentials suspected to be used in the breach and change all the passwords.

Verify logs

Check that you have logs available from the potentially breached systems. Best case would be that the logs are available outside of the system in question. If not, back them up to external storage, to make sure that it doesn’t get altered or removed by the attacker.

Remember to communicate

Communicate with stakeholders, remember your users, customers and public. Although it may feel challenging to tell these kinds of news, it’s much better to be open in the early stages than to get caught your pants down later on.

To summarise

The threat level is definitely higher due to above mentioned circumstances, but getting the basics in order helps you to react if something happens. Keep also in mind that you don’t have to cope in this situation alone. Security service providers have the means and capacity to support you in efficient way. Our teams are always willing to help to keep your business and operations secure.

 

Very Large Array, Socorro, United States

Alerting Estonian Citizens with Azure

Why not take advantage of the public cloud? Read how the system for transmitting alarm messages to consumers was born in Estonia. With this piece of writing, we go through one specific and technical real-life journey from the birth of an idea to its implementation in the Microsoft Azure cloud environment.

 

The author of the article works for the company Solita as a Data Engineer since 2019 and specialises in cloud-based data platforms. By now, he has accumulated more than 20 years of experience in various fields of the IT sphere – development, architecture, training. Many interesting projects have been made both in Estonia and abroad.

The beginning

In the digital state’s Hackathon event, Solita addressed the transmission of government’s cultural messages to televisions via an Android app.

In parallel, the Ministry of the Interior’s IT Centre (SMIT) pointed out a similar need. Sending alarm messages from the SITREP (Situation Reporting) system to the mobile application ‘Ole valmis!’ (‘Be ready!’) was considered. The purpose of that application became to warn the user of a nearby accident or deteriorating weather (such as a snowstorm, or a coastal flood).

In conclusion, since the pattern was the same, it seemed reasonable to create a single, unified solution.

Problems and objectives

SITREP did not have a public interface for requesting messages, but it did have the functionality to send messages. So it was possible to interface the system directly with the back-end of the ‘Ole valmis!’ (‘Be ready!’) application. The following goals emerged in the process of the Hackathon, which led to the development of a separate cloud-based system.

  • Transmission of messages (push and pull) must be possible to several channels in parallel, whoever the consumer is.
  • The messaging functionality must be separate from SITREP.
  • The interface must be secure so that a malicious actor cannot send false alarms.
  • It must be easy to administer.
  • It must be possible to subscribe / categorise messages by subject and location.
  • Setting up the system must be quick and easy.
  • The system must be flexible and inexpensive to maintain.
  • The available development time is a couple of man-months.

Why Microsoft Azure?

Solita participated in the Hackathon in partnership with Microsoft, which is why Azure was chosen as the cloud environment – although similar solutions could be created with the help of AWS or Google, for example. Azure also provides some direct benefits.

  • Most components can be easily integrated with Active Directory (although Active Directory was not an end in itself in the first iteration, this was one argument to consider in the future).
  • The range of services (in other words – the arsenal of ‘building blocks’ of the completed system) is really impressive and also includes exclusive components – in the following we will take a closer look at some of them.

For example, API Management is, to put it simply, a scalable API gateway service, and as a big bonus, it includes a public Web portal (Developer Portal) that is convenient for both the user and the administrator. In addition, the interface can be redesigned to suit your needs. The main value comes precisely from the ease of use – you don’t have to have much Azure-specific knowledge to sign up, send / receive messages, set the final destination, describe conversion rules.

The Developer Portal provides developers with pre-written sample code for consuming messages (presented in cURL, C#, and Python, for example). In addition, of course, a built-in firewall, resilience, and resistance to DDoS-type attacks are provided. All of the above saves a lot of time (and money!) from the user’s, administrator’s and developer’s point of view.

Infrastructure creation process

From the architect’s point of view, the aim was to create a system based on the most standard possible components (provided by Azure itself), and the installation of which would be simple enough for anyone with a basic knowledge of the working principle of the cloud. Account had also to be taken of the fact that the requirements were also still evolving.

From the beginning, we relied on the principle of IaC (Infrastructure as Code) – the entire infrastructure of the solution in the cloud is unambiguously described as human and machine readable code. In addition, the installation process would be incremental (if a new version is created, the existing infrastructure could be updated instead of recreating), configurable and automated; the code would be as clear and editable as possible. Figuratively speaking, you press ‘deploy’ and you don’t need much else.

All of the above is made possible by a tool called Terraform, which is quite common, especially among those who administer infrastructures – so-to-speak the de-facto standard for precisely cloud infrastructures. It is a free tool produced by HashiCorp that is perfect for situations like this – a person describes in a code what resources he needs, and Terraform interprets the code into instructions that can be understood by the (cloud) environment to create, modify or delete them.

Terraform has the following strengths that were the decisive factor:

  • its spread and wide platform support,
  • the ability to ‘remember’ the state of the installed infrastructure,
  • the simple but powerful HCL language that can be used to describe even complex logic.

The method officially supported by Microsoft for achieving the same are ARM templates (ARM templates are essentially structured static JSON or YAML). The entire Azure infrastructure can be described based on purely ARM templates, but then more code is created and the possibilities of directing the installation logic are greatly reduced.

Changing requirements and options

The first thing that the work continued on was creating a message store (for pull scenario and debugging).

The initial understanding of the message format was quite simple:

  • single-level JSON,
  • a few required attributes (timestamp, author, etc.),
  • rest of the schema was completely open.

Based on the above and on the principled decision to use only Microsoft Azure components + to install the entire solution with a single command, two options remained on the table for storing and reading JSON data without a defined schema:

  • Table Storage (default; although by operating principle it is a key / attribute type service),
  • Cosmos DB.

The ability to query data via HTTP(S) (less development work) and a lower cost (especially important in the prototype phase) spoke in favour of Table Storage; Cosmos DB had the advantage of storage flexibility, as it stores data in several regions. However, the situation was changed by the fact that SITREP’s messages came as a multi-level JSON and some of the critical attributes were at a ‘deeper’ level. Therefore, Table Storage no longer met the new requirement and the Cosmos DB component had to be introduced instead.

In addition, there was a real possibility that the system would be used in a context other than alarm messages – it had to be taken into account that the system could be used for transmitting virtually any message from many senders to different channels in different combinations. In essence, the goal became to create a messaging platform (Message Service Provider) that would functionally resemble the products of Twilio or MessageBird, for example.

Not a single line of ‘real’ code

So, by now, the following was architecturally in place:

  • all incoming messages and queries went through API Management,
  • all messages were stored in the Cosmos DB database.

 

At the same time, pushing messages to destinations through API Management remained an open issue. And exactly which component handles the database of messages and destination addresses?

Microsoft Azure offers options for almost any scenario, from an application hosted on a virtual machine to a serverless component called Azure Function. You can also use a number of intermediate variants (Docker, Kubernetes, Web App), where the user may or may not have direct access to the server hosting the application.

In the context of the present solution, all the above solutions would have meant the following:

  • a person with a developer background would have been needed to create the system,
  • the system as a whole could no longer be installed very easily – the application code would have been separate from the infrastructure code.

Fortunately, Azure has provided the Logic App technology that addresses the above issues. It’s a way to describe business logic as a flowchart – for example, you can visually ‘draw’ a ready-made Logic App ‘application’ in the Azure Portal, using the online interface.

It is true that in more complex cases, such as conversion operations, you will probably need to write a few lines of code, but this is far from traditional programming. Writing Logic App code is more akin to developing Excel macros than Python or Java.

The Logic App flow code can be saved as an ARM template, and the ARM template can be used as part of a Terraform installation script – making the Logic App a great fit for this context. Starting a single workflow in this solution costs in the order of 0.0005 euros per occasion (according to the consumption-based plan) – yes, there are even cheaper solutions like Azure Function, but in this case the infrastructure needs to be installed and developed separately.

Support components

Azure has well-thought-out tools for monitoring the operation of the system; in this case we focus on two of them: Azure Monitor and Log Analytics. The first, as the name suggests, is a set of dashboards provided by the Azure platform that help monitor the status of applications and components (including in real-time), such as load, memory usage, and user-defined metrics.

Since the Monitor is ‘included’ with every solution by default, it may not be correct to consider it as a separate component – it is simply a question of displayed indicators. Log Analytics, on the other hand, is a place to direct the logs of all components so that they can be conveniently analysed and queried later. This helps detect system errors and quickly track down errors. You can even query Log Analytics for metrics to display later in the Monitor, such as the number of errors per day.

Results and observations

In summary, the architecture of the solution came out as follows.

Messaging system architecture in Azure
Messaging system architecture in Azure

 

Broadly, the objectives set out at the start were achieved and the principles were followed (IaC, Azure components only, etc.). Clearly, Microsoft Azure offers a truly comprehensive suite of services with typically 99.95-99.99% SLAs; however, ‘the seven nines’ (99.99999%) or even higher are not uncommon. Such a high percentage is achieved through redundancy of components and data, optimised hardware usage, and exceptionally strict security measures in the region’s data centres.

Installing a system from scratch on an Azure account takes 45-60 minutes, and the lion’s share of this is provisioning API Management – a kind of heavyweight in Microsoft Azure, with a number of internal components hidden from the user (firewall, web server, load balancer, etc.).

There were no obstacles, but the development revealed that Terraform is a few steps behind Microsoft Azure as a third-party tool – in other words, when Microsoft launches a new Azure service, it will take time for HashiCorp developers to add functionality to their module. In this case, for example, the ARM template for the new component can be partially grafted into Terraform scripts, so that the creation of the infrastructure can be automated in any case.

In conclusion

Public cloud providers, such as Microsoft Azure, have hundreds of different services that can be considered Lego blocks – combining services to create the solution that best meets your needs.

The article describes how an MSP-like product was created from scratch that has reached the pre-live status by now. The same product can be assembled from different components – it all depends on the exact needs and on the possibilities to include other competencies, such as C# or Java developer’s. The public cloud is diverse, secure, affordable and ever evolving – there are very few reasons not to take advantage of it.

Thank you: Janek Rozov (SMIT), Timmo Tammemäe (SMIT), Märt Reose (SMIT), Kristjan Kolbre (‘Ole valmis!’), Madis Alesmaa (‘Ole valmis!’), Elisa Jakson (Women’s Voluntary Defence Organisation / ‘Ole valmis!’), Henrik Veenpere (Rescue Board).

Cloud services

Hybrid Cloud Trends and Use Cases

Let's look at different types of cloud services and learn more about the hybrid cloud and how this cloud service model can exist at an organisation. We’ll also try to predict the future a bit and talk about what hybrid cloud trends we are expecting.

As an IT person, I dare say that today we use the cloud without even thinking about it. All kinds of data repositories, social networks, streaming services, media portals – they work thanks to cloud solutions. The cloud now plays an important role in how people interact with technology.

Cloud service providers are inventing more and more features and functionalities, bringing them to the IT market. Such innovative technologies offer even more opportunities for organisations to run a business. For example, AWS, one of the largest providers of public cloud services, announces over 100 product or service updates each year.

Cloud services

Cloud technologies are of interest to customers due to their economy, flexibility, performance and reliability.

For IT people, one of the most exciting aspects of using cloud services is the speed at which the cloud provides access to a resource or service. A few clicks at a cloud provider’s portal – and you have a server with a multi-core processor and large storage capacity at your disposal. Or a few commands on the service provider’s command line tool – and you have a powerful database ready to use.

Cloud deployment models

In terms of the cloud deployment model, we can identify three main models:

• A public cloud – The service provider has publicly available cloud applications, machines, databases, storage, and other resources. All this wealth runs on the IT infrastructure of the public cloud service provider, who manages it. The best-known players in the public cloud business are AWS, Microsoft Azure and Google Cloud.

In my opinion, one of the most pleasant features of a public cloud is its flexibility. We often refer to it as elasticity. An organisation can embark on its public cloud journey with low resources and low start costs, according to current requirements. 

Major public cloud players offer services globally. We can easily launch cloud resources in a geographical manner which best fits our customer market reach. 

For example, in a globally deployed public cloud environment, an organization can serve its South American customers from a South American data centre. A data centre located in one of the European countries would serve European customers. This greatly improves the latency and customer satisfaction.

There is no need to invest heavily in hardware, licensing, etc. – organisation spends money over time and only on the resources actually used.

• A private cloud – This is an infrastructure for a single organisation, managed by the organisation itself or by a service provider. The infrastructure can be located in the company’s data centre or elsewhere.

The definition of a private cloud usually includes the IT infrastructure of the organisation’s own data centre. Most of these state-of-the-art on-premise solutions are built using virtualisation software. They offer the flexibility and management capabilities of a cloud.

Here, however, we should keep in mind that the capacity of a data centre is not unlimited. At the same time, the private cloud allows an organisation to implement its own standards for data security. It also allows to follow regulations where applicable. Also, to store data in a suitable geographical area in its data centre, to achieve an ultra low latency, for example. 

As usual, everything good comes with trade-offs. Think how complex activity it might be to expand the private cloud into a new region, or even a new continent. Hardware, connectivity, staffing, etc – organisation needs to take care of all this in a new operating area. 

• A hybrid cloud – an organisation uses both its data centre IT infrastructure (or its own private cloud) and a public cloud service. Private cloud and public cloud infrastructures are separate but interconnected.

Using this combination, an organisation can store sensitive customer data in an on-premise application according to regulation in a private cloud. At the same time, it can integrate this data with corporate business analytics software that runs in a public cloud. The hybrid cloud allows us to use the strengths of both cloud deployment models.

Hybrid cloud model

When is a hybrid cloud useful?

Before we dive into the talk about hybrid cloud, I’d like to stress that we at Solita are devoted advocates of cloud-first strategy, referring to public cloud. At the same time, cloud-first does not mean cloud-only, and we recognize that there might be use-cases when running a hybrid model is justified, be it regulation reasons or very low latency requirements.

Let’s look at some examples of when and how a hybrid cloud model can benefit an organisation. 

Extra power from the cloud

Suppose that the company has not yet made its migration to public cloud. Reasons can be lack of resources or cloud competence. It is running its private cloud in a colocation data centre. The private cloud is operating at a satisfactional level while the load and resource demand remains stable. 

However, the company’s private cloud lacks extra computing resources to handle future events of demand growth. But an increased load on the IT systems is expected due to an upcoming temporary marketing campaign. As a result of the campaign, the number of visitors to the organisation’s public systems will increase significantly. How to address this concern?

The traditional on-premise way used to be getting extra resources in the form of additional hardware. It means new servers, larger storage arrays, more powerful network devices, and so on. This causes additional capital investment, but it is also important that this addition of resources may not be fast.

The organisation must deliver, install, configure the equipment – and these jobs cannot always be automated to save time. After the load on the IT systems has decreased with the end of the marketing campaign, the situation may arise that the acquired additional computing power is not used any more.

But given the capabilities of the cloud, a better solution is to get additional resources from the public cloud. Public cloud allows to do this flexibly and on-demand, as much as the situation requires. The company spends and pays for resources only as it needs them, without large monetary commitments. Let the cloud adoption start 😊

The organisation can access additional resources from the public cloud in hours or even minutes. We can order these programmatically, and in automated fashion in advance, according to the time of the marketing campaign.

When the time comes and there are many more visitors, the company will still keep the availability of its IT systems. They will continue to operate at the required level with the help of additional resources. This method of use is known as cloud bursting, i.e. resources “flow over” to another cloud environment.

This is the moment when a cloud journey begins for the organization. It is an extremely important point of time when the organization must carefully evaluate its cloud competence. It needs to consider possible pitfalls on the road to cloud adoption. 

For an organisation, it is often effective to find a good partner to assist with cloud migration. The partner with verified cloud competence will help to get onto cloud adoption rails and go further with cloud migration. My colleagues at Solita have written a great blog post about cloud migration and how to do it right.

High availability and recovery

Implementing high availability in your data centre and/or private cloud can be expensive. As a rule, high availability means that everything must be duplicated – machines, disk arrays, network equipment, power supply, etc. This can also mean double costs.

An additional requirement can be to ensure geo-redundancy of the data and have a copy in another data centre. In such case, the cost of using another data centre will be added.

A good data recovery plan still requires a geographically duplicated recovery site to minimise risk. From the recovery site, a company can quickly get its IT systems back up and running in the event of a major disaster in a major data centre. Is there a good solution to this challenge? Yes, there is.

A hybrid cloud simplifies the implementation of a high availability and recovery plan at a lower cost. As in the previous scenario described above, this is often a good starting point for an organisation’s cloud adoption. Good rule of thumb is to start small, and expand your public cloud presence in controlled steps.

A warm disaster recovery site in the public cloud allows us to use cloud resources sparingly and without capital investment. Data is replicated from the main data centre and stored in the public cloud, but bulky computing resources (servers, databases, etc.) are turned off and do not incur costs.

In an emergency case, when the main data centre is down, the resources on the warm disaster recovery site turn on quickly – either automatically or at the administrator’s command. Because data already exists on the replacement site, such switching is relatively quick and the IT systems have minimal downtime.

Once there is enough cloud competence on board, the organisation will move towards cloud-first strategy. Eventually it would switch its public cloud recovery site to be a primary site, whereas recovery site would move to an on-premise environment.

Hybrid cloud past and present

For several years, the public cloud was advertised as a one-way ticket. Many assumed that organisations would either move all their IT systems to the cloud or continue in their own data centres as they were. It was like there was no other choice, as we could read a few years ago.

As we have seen since then, this paradigm has now changed. It’s remarkable that even the big cloud players AWS and Microsoft Azure don’t rule out the need for a customer to design their IT infrastructure as a hybrid cloud.

Hybrid cloud adoption

Organisations have good reasons why they cannot always move everything to a public cloud. Reasons might include an investment in the existing IT infrastructure, some legal reasons, technical challenges, or something else.

Service providers are now rather favouring the use of a hybrid cloud deployment model. They are trying to make it as convenient as possible for the customer to adopt it. According to the “RightScale 2020 State of the Cloud” report published in 2020, hybrid cloud is actually the dominant cloud strategy for large enterprises:

Hybrid cloud is the dominant strategy

Back in 2019, only 58% of respondents preferred the hybrid cloud as their main strategy. There is a clear signal that the hybrid cloud offers the strengths of several deployment models to organisations. And companies are well aware of the benefits.

Cloud vendors vs Hybrid

How do major service providers operate on the hybrid cloud front? Microsoft Azure came out with Azure Stack – a service that is figuratively speaking a public cloud experience in the organisation’s own data centre.

Developers can write the same cloud native code. It runs in the same way both in a public Azure cloud and in a “small copy” of Azure in the enterprise’s data centre. It gives the real cloud feeling, like a modern extension to a good old house that got small for the family.

Speaking of multi-cloud strategy as mentioned in the above image, Azure Arc product by Microsoft is worth mentioning, as it is designed especially for managing multi-cloud environments and gaining consistency across multiple cloud services.

AWS advertises its hybrid cloud offering portfolio with the message that they understand that not all applications can run in the cloud – some must reside on customers premises, in their machines, or in a specific physical location.

A recent example of hybrid cloud thinking is AWS’s announcement of launching its new service ECS Anywhere. It’s a service that allows customers to run and manage their containers right on their own hardware, in any environment, while taking advantage of all the ECS capabilities that AWS offers in the “real” cloud to operate and monitor the containers. Among other things, it supports “bare” physical hardware and Raspberry Pi. 😊

As we’ve also seen just recently, the next step for Amazon to win hybrid cloud users was the launch of EKS Anywhere – this allows customers using Kubernetes to enjoy the convenience of a managed AWS EKS service while keeping their containers and data in their own environment, on their own data centre’s machines.

As we see, public cloud vendors are trying hard with their hybrid service offerings. It’s possible that after a critical threshold of hybrid services users is reached, it will create the next big wave of cloud adoption in the next few years. 

Hybrid cloud trends

The use of hybrid cloud related services mentioned above assumes that there is cloud competence in the organisation. These services integrate tightly with the public cloud. It is important to have skills to manage these correctly in a cloud-native way.

I think we will see a general trend in the near future that the hybrid cloud will remain. Multi-cloud strategy as a whole will grow even bigger. Service providers will assist customers in deploying a hybrid cloud while maintaining a “cloud native” ecosystem. So that the customer has the same approach to developing and operating their IT systems. It will not matter whether the IT system runs in a “real” cloud or on a hybrid model. 

The convergence of public, private and hybrid models will evolve, whereas public cloud will continue to lead in the cloud-first festival. Cloud competence and skills around it will become more and more important. The modern infrastructure will not be achievable anymore without leveraging the public cloud.

Growing from a small cloud team to a community of over 70 people

The cloud community at Solita has grown substantially in the past years. While the industry has been developing rapidly, we've done our best to stay ahead of the game. Our recipe for growth has two main ingredients: a sensitive ear to our clients’ evolving needs and a clear vision to build a caring work culture where people can excel in their jobs.

We are long-term solitans, and we’ve had the opportunity to witness the growth of our cloud community from the start. The team was created when we saw the first signs from our clients that there is a need for cloud specialization. In the beginning, in 2014, we only had a couple of people in the team. But we had our eyes in the market, and we started developing our cloud services from a holistic point of view. We wanted to be a full-service cloud partner and kept this in mind when planning and developing our services.

In 2016 when we saw an increased number of opportunities in the market, we officially launched our Cloud & Connectivity unit. Since then, we have grown from a small team to a community of over 70 cloud professionals. These people are specialized in demanding cloud services, like infrastructure, cyber security, and public cloud platforms.

It all started from our clients’ needs 

The key driver in our growth has been the willingness to serve our clients better and pay close attention to their needs. When we saw a growing demand, we responded to it. In the past years, the world has turned more complex, and technologies have developed very fast. These days a big part of our job is to help our clients to choose the right solutions that serve their needs in the best possible way.

It’s also been an interesting journey to see how the attitudes towards cloud services have evolved. There was a lot of resistance and insecurity in the beginning. These days it’s quite the opposite. Our client portfolio has become diverse, including organizations from many different industries. We also work with several organizations in the public sector, and we’ve built cloud solutions for companies in the heavily regulated financial sector. Cloud has become mainstream, and we’ve been glad to be part of this development.

As an example, we’ve helped Savings Bank build a solid cloud foundation and supported them with their “cloud first” strategy, which states that as much as possible should be built on the public cloud. Solita CloudBlox has allowed them to get up and running quickly, and results so far have been great. As for billing and reporting, the contrast between the transparency of Solita CloudBlox and the traditional data center world could not be greater. For the first time ever, Savings Bank gets thoroughly granular reports on every hour worked, every euro spent, and every cloud component implemented – literally, an eye-opener for the bank.

Our people have enabled our growth

Solita is a value-driven and people-focused company, and our people play a key role in our growth equation. While managing growth is not always easy, during every step, we’ve tried to do our best to make sure that people are happy to work here. We have fostered our culture and kept the organization lean and agile.

In practice, it means that our people have a voice, and their opinions are appreciated. Our professionals get plenty of freedom to execute their work the best way they think they should, and everyone gets to have a say in which type of projects they want to work. Caring is one of our core values, and it’s visible in many ways. Our people here really care for one another. There is always help and support available; you are never left alone.

Cloud&Connectivity Day at the Biitsi 2021 / Nea Tamminen

But being a frontrunner company in many arenas also means that there

is a requirement to learn new things, explore and test new products, and investigate new services. For some, it could become overwhelming or uncomfortable. Also, the fact that we deliver a wide range of projects means that people must be adaptable, and managing oneself becomes essential. We live in a constant change in this industry, so we need to stay alert and aim to be a bit ahead of the game all the time. And it’s our job as leaders to provide the best possible set-up for our people to do so.

The growth in the cloud community has also provided a personal growth journey for both of us. We’ve understood that like this community, also we personally are constantly evolving and never “ready”. We’ve gone through tough moments as well, but when we’ve had the courage and humility to ask for help, there’s been a line of colleagues providing their support. That’s the beauty of our culture and something we also want to hold on to in the future.

Writers of this blog, Jari Jolula and Karri Lehtinen, are long-term Solitans and have been leading our Cloud offering since the beginning.

Would you have something to add in our culture? Please check our openings and contact us!

Want to know how we do managed cloud? Please visit our Solita CloudBlox site.

 

AWS re:Invent 2021: Sustainability in cloud, Local zone to Finland, developer experience and more

re:Invent was held in Las Vegas last week. During and around the event AWS has again announced a bunch of new services and features. Five of them got my attention: Sustainability for Well-Architected review, Local Zone in Finland, CPU chips, better developer experience and free-tier upgrade.

AWS re:Invent was held in Las Vegas, Nevada between Nov. 29th and Dec 3rd 2021. As a customer or developer, after the main event of cloud service provider (CSP) you can have mixed feelings. On the other hand, WOW feeling from the new service that solves my existing problem easily. On the other hand, I might have developed a feature for a month and now it’s available by checking a box. You will get used to these mixed feelings! The best way to avoid this is by having a strong relationship with CSP’s both in business and technical perspective. AWS’s CTO Dr. Werner Volgels said: “It’s our fault that AWS has hundreds of services with thousands of features!“. More than 90% of new services and features are based on direct customer feedback.

It’s All About Teamwork

For human beings, it is impossible to really know each service in a detailed manner. I work with several teams to find out the most suitable solutions for our projects during a typical work week for me. During our work we are taking into account constraints such as overall architecture, schedule, budget and current knowledge of the team. Developing something new is never a unicorn task but it is teamwork.

The other team members have done the groundwork for it already even if there is a technical problem that finally one person solves. It’s the same as when a group of people is trying to light a campfire and the last one succeeds. Without preliminary work from previous people, he probably wouldn’t have succeeded either.

Here are my top five AWS’s announcements from the last couple of weeks.

Sustainability Pillar for AWS Well-Architected Framework

Year after year, sustainability thinking has become the most important theme both in the world and in information technology. For example, we have noticed that sustainability has reflected our system tenders, where customers also score bids based on their sustainability values. This is the only right way!

AWS Well-Architected review includes sustainability

The usage of cloud providers has always been justified by economies of scale which also covers sustainability values like energy efficiency, recycling processes and so on. The Well-Architected Framework is a structured way to analyse your workload and now also it’s sustainability values.

The AWS’s Alex Casalboni sums up the new pillar nicely into six things:

Cloud's sustainability

  1. Understand your impact
  2. Establish sustainability goals
  3. Maximize utilization
  4. Anticipate and adopt new, more efficient hardware and software offerings
  5. Use managed services
  6. Reduce the downstream impact of your cloud workloads

Read more from Alex’s blog post.

Local Zone to Finland in 2022

Finland with Norway and Denmark will get its own AWS Local Zone in 2022. A Local Zone provides selected AWS services closer to end-users with single-digit latency. It is a pretty new concept starting from Las Vegas in December 2019. The concept is expanding with 30 new locations in 2022. It provides typical data center services like private networking (VPC), EC2 virtual machines, EKS and ECS for containers and EBS for storage. Only selected EC2 instance types are available in the Local Zone environment.

AWS Local Zone locations includes Finland

You can manage Local Zone resources seamlessly like resources in regions, for example via AWS Console. Each Local Zone has their “mother region” for configuration and larger integrations:

Local Zone settings under EC2 in mother region

The Local Zone can be very useful for many things like regulated systems and data (location inside Finland) and virtual desktop applications (low latency).

More information about Local Zones.

CPU Chip Competition Continues

For a long time, Intel had gained a superior market share in the CPU market. Then suddenly big ICT companies, like Apple and AWS, published their intention to start using self-made ARM M1 and Graviton processors. In re:Invent the 3rd generation of AWS Graviton chip was announced. With Graviton chips customers can benefit from lower overall cost due to better performance and cheaper instance pricing. The Graviton3 supports bfloat16 format which has been typically only supported by special made AI processors. EC2 C7g is the first type to use Graviton3.

EC2 C7g uses Graviton3 AWS ARM CPU

Services like Lambda (FaaS) and Fargate (serverless containers) are also now supporting Graviton2. For infrastructure as code, it is just a matter of changing a parameter and you are ready. But it is just not so simple for compiled software like Docker containers (OS libraries) or 3rd party libraries. The reason is the change of  processor architecture from x86 to ARM.

OpenJDK as example of multiple CPU architectureIn a migration you should start first producing one version for x86 and one for ARM. You need to execute adequate testing on a case-by-case basis. For example Docker image repositories support multiple processor architectures for a single image. In some cases with compiled 3rd party software you are just stuck and not able to use ARM based chips. My dear colleague Timo pointed out that there is no efficient way to run Oracle XE database in his M1 MacBook laptop.

Graviton3 and EC2 C7g (C stands for computing intensive workload) instance type.

List of Graviton enabled software.

Fargate support for Graviton2.

AWS Cloud Development Kit version 2 for better developer experience

AWS Cloud Development Kit (CDK) is roughly two years old. In the opinion of me and many others it is the most efficient way to handle AWS infrastructure because of two things: it is programmatic (Typescript, Python, Java, etc.) and the CDK library provides sensible defaults and best-practice security policies.

AWS Cloud Development Kit for better developer experience

The main focus of version 2 of the AWS CDK library is to make development experience better. The library structure changed totally. In v1 you handle tens of small CDK library packages (one for each AWS service). Now with v2, all stable libraries were consolidated to a single package. With experimental libraries the v1 model stays.

I strongly recommend most projects to start migrating to CDK v2 immediately, and only projects that are in solid maintenance mode now or soon could stay in v1. The v1 one is supported fully for the next 6 months and for bug fixes up to June 2023. Luckily, in most cases the migration is actually pretty easy and the code changes can be done in hours than in days. After code changes, you should test deployment thoroughly.  The migration guide for CDK v1 to v2.

AWS CDK has improved the developer experience also by speeding up code change deployments with hot swap and Watch mode. With hot swapping functionality you can deploy eg. lambda code changes in a few seconds. When no infrastructure changes exist, CDK deployment uses direct API calls and skips Cloudformation layer in hot swap mode. Remember to use this feature only in development and testing environments because Cloudformation stack is not up to date after the bypassing operation.

CDK watch mode makes development faster by monitoring your code and assets for changes and automatically performing the optimal form of deployment every time a file change is detected. This means that you do not need to run the cdk deploy command manually anymore.

cdk watch mode

Free Tier Data expansions

Cloudfront free tier

At least now, everybody should use Amazon CloudFront in front of any AWS based Web or media service. You can now use it for free up to 1000 GB per month (data out). And to remind, data transfer from AWS services to CloudFront edge locations (origin fetches) has been free of charge for some years now.

Also the free tier of regional Internet outbound traffic increased from 1GB to 100GB per month.

More details about the announcement.

 

What are the most important new services or features that you use? Zoph.io’s GitHub page (thanks Arto, check his Youtube channel) compiled a comprehensive listing of new services and features.

What would You log?

This is first part of our new blog series about centralized log management in Solita Cloud. We provide numerous log management platforms to our customers and now we would like to share our experience on why centralized log management is so important and how to avoid most common pitfalls.

This blog will provide an overview of centralized log management and why it is so important. We will share our experience on what is most important when starting a logging project and how to avoid most common pitfalls. Also lemons.

Why centralized logging?

This is the question we want to answer today. Why do we need centralized logging and why should you implement it? What are the benefits and what can you expect? Interesting topics to be sure, but at the same time, why alone is quite a boring question. The reasons for having centralized logging are rather obvious but we don’t want to focus on that alone. The saying when life gives you lemons, make lemonade, fits quite well here. The logs are the lemon and you just need to know how to turn it into tasty lemonade with the aid of centralized log management.

The reason why we compared logs to lemons is the nature of logs that exist without any form of management. It can be very chaotic and leave you feeling sour after your storage has become full one too many times. Centralized log archiving can move the logs to another place, but that alone does not solve the issue of storage constraints. Those constraints are an example of why Centralized Log Management is important. It takes a step beyond just central archival of the logs.

Some can also see logs as a necessary evil, something that you are forced to store. There are quite many laws which require that companies storing personal data must also monitor and log access to it. Even logs themselves might contain personal data, like location or payment information. In EU, GDPR allows legitimate use of consumers personal information, as long as we take the necessary precautions to protect it and we are transparent on how we use it. This practically means monitoring and controlling who can access and view said data.

More local legal regulations can define for example how long a company must store it’s logs. Such legal requirements can quite easily push a company to adopt centralized log archiving in a quick manner. This is where things can get a bit messy and while logs are collected and stored, there is no real good way to leverage them. Using our previous example of comparing logs to lemons, we can make a comparison that the logs are lemons that still hang in a tree. When using central log archiving, we only collect those lemons into a basket. They’re all in the same place but we’re not using them. With centralized log management, we can process them into lemonade.

Tied to the legal requirements is security. Centralized log archiving can provide you with a single location to store and access all your audit, access and any other log. But security is not just storing logs, those logs need to be taken advantage of and analyzed. Systems like Security Information and Event Management (SIEM) can help with threat analyzing and detection. Alerts for certain actions or logins, dashboards for traffic based on access logs and traceability between applications with trace tokens to see what a user has done, are some of the possibilities that centralized log management helps enable. These features can help not only to improve a company’s security but also bring them value and important information from their systems.

Troubleshooting an issue from logs is something every developer is familiar with and having logs centralized somewhere with easy access will certainly improve developers’ life. Aforementioned trace and event logs from applications can be used for monitoring, troubleshooting and security purposes. Even the possibility of using your events and tracing for machine learning can help to understand your users and further improve your system.

Moving forward

To be able to take advantage of centralized logging you need to understand your needs for both, the present and the future. While sizing is more flexible today, especially when working in the cloud, the underlying architecture and logic needs to be planned beforehand. It is important to know the entire lifecycle of your logs, from the applications where they’re generated, all the way to the long-term storage where they’re retained.

Each log that is logged should have a clear purpose. This can’t be stressed enough. Every log has to mean something important. It’s useless to log anything that doesn’t provide any value.

There are exceptions, like when debug logs are needed for troubleshooting, but generally in its default state only the important, descriptive logs should be collected and stored. Configuring event or trace logs from applications can be very beneficial, but these need some work from the development teams. Centralized logging management projects are very much a group effort. Both developers, infrastructure, security and other parties need to work together and understand what they want and what they actually need from centralized logging management.

This all might seem like quite a lot of work and that’s because it is. But this is where good planning, proper project management and agile development comes into play. There should be a clear start and an end to the project. Not everything needs to be done immediately. Nothing is built over night, so take your time to improve and add features as time goes by and your system is ready for them. Just keep in mind what is the goal you want to achieve and what kind of value it is that you want to generate in the end. We will talk more about this in our next centralized log management blog post that dives deeper into how to start this type of a project.

There will be value

The value of centralized log management reaches further than just meeting requirements and helping accelerate development. The ability to transform your log data to alerts, dashboards, threat detection or even use them for machine learning can help to appraise the logs that are filling your disks. Centralized logging doesn’t need to be just a legal obligation or a development tool, it can be both. It can provide multiple avenues of value while meeting those pesky requirements.  But as it was already said, achieving this will take time and the approach should be well planned and methodical.

With new technologies constantly emerging and becoming popular, it also challenges us to change with it. Containerized workloads running on orchestrators like Kubernetes challenges the way we think about our softwares lifecycle. And all this statelessness needs a centralized way of managing things as old ways are no longer applicable.

At least for your logging needs the transformation is easy, you can just contact us and we will help you to design, develop, deploy and take care of your centralized logging management. Or just keep reading out blogs as they come out.

Solita is a technology, data and design company with years of experience working in cloud and on-premises. We have helped companies in their transformation into technology driven organizations and brought centralized log management into their lives. Our experience rests both in large and small setups with data from few hundred gigabytes per day into terabytes of data. Technologies behind us are commonly know and popular, Elasticsearch, Opensearch, Graylog and cloud specific services like CloudWatch, to mention a few. These will be the vocal point in our examples as these are what we work with day-to-day basis. Migration from on-premise to cloud and changing technologies is also something we are very familiar with as cloud is constantly gaining more popularity.

In our next Centralized Logging Management blog we will talk about how this kind of project should be started and how it’s actually done properly from start to finish. At later date we will return for more in-depth technical view on different features and how to use them.

The learning curve of a Cloud Service Specialist at Solita

Tommi Ritvanen is part of Cloud Continuous Services Team in Solita. The team consists of dozens of specialist who ensure that customers' cloud solutions are running smoothly. Now Tommi shares his experiences of learning path, culture and team collaboration.

I’ve been working at Solita for six months as a Cloud Service Specialist. I’m part of the cloud continuous services team, where we take care of our ongoing customers and ensure that their cloud solutions are running smoothly. After our colleagues have delivered a project, we basically take over and continue supporting the customer with the next steps.

What I like about my job is that every day is different. I get to work and learn from different technologies; we work with all the major cloud platforms such as AWS, Microsoft Azure and Google Cloud Platform. What also brings variety in our days is that we have different types of customers that we serve and support. The requests we get are multiple, so there is no boring day in this line of work.

What inspires me the most in my role is that I’m able to work with new topics and develop my skills in areas I haven’t worked on before. I wanted to work with public cloud, and now I’m doing it. I like the way we exchange ideas and share knowledge in the team. This way, we can find ways to improve and work smarter.

We have this mentality of challenging the status quo positively. Also, the fact that the industry is changing quickly brings a nice challenge; to be good at your job, you need to be aware of what is going on. Solita also has an attractive client portfolio and a track record of building very impactful solutions, so it’s exciting to be part of all that too.

I got responsibility from day one

Our team has grown a lot which means that we have people with different perspectives and visions. It’s a nice mix of seniors and juniors, which creates a good environment for learning. I think the collaboration in the team works well, even though we are located around Finland in different offices. While we take care of our tasks independently, there is always support available from other members of the cloud team. Sometimes we go through things together to share knowledge and spread the expertise within the team.

The overall culture at Solita supports learning and growth, there is a really low barrier to ask questions, and you can ask for help from anyone, even people outside of your team. I joined Solita with very little cloud experience, but I’ve learned so much during the past six months. I’ve got responsibility from the beginning and learned while doing, which is the best way of learning for me.

From day one, I got the freedom to decide which direction I wanted to take in my learning path, including the technologies. We have study groups and flexible opportunities to get certified in the technologies we find interesting.

As part of the onboarding process, I did this practical training project executed in a sandbox environment. We started from scratch, built the architecture, and drew the process like we would do in a real-life situation, navigating the environment and practising the technologies we needed. The process itself and the support we got from more senior colleagues was highly useful.

Being professional doesn’t mean being serious

The culture at Solita is very people-focused. I’ve felt welcome from the beginning, and regardless of the fact that I’m the only member of the cloud continuous services team here in Oulu, people have adopted me as part of the office crew. The atmosphere is casual, and people are allowed to have fun at work. Being professional doesn’t mean being serious.

People here want to improve and go the extra mile in delivering great results to our customers. This means that to be successful in this environment, you need to have the courage to ask questions and look for help if you don’t know something. The culture is inclusive, but you need to show up to be part of the community. There are many opportunities to get to know people, coffee breaks and social activities. We also share stories from our personal lives, which makes me feel that I can be my authentic self.

We are constantly looking for new colleagues in our Cloud and Connectivity Community! Check out our open positions here!