Automatic Log Analysis using Deep Learning and AI for Microservices

Reading Time: 11 Minutes

by  | July 21, 2017 

What is Deep Learning?

Deep Learning is a type of Neural Network Algorithm that takes metadata as an input and process the data through some layers of the nonlinear transformation of the input data to compute the output.

This algorithm has a unique feature, i.e., automatic feature extraction. It means that this algorithm automatically grasps the relevant features required for the solution of the problem.

It reduces the burden on the programmer to select the features explicitly. It can be used to solve supervised, unsupervised or semi-supervised type of challenges.

In Deep Learning Neural Network, each hidden layer is responsible for training the unique set of features based on the output of the previous layer. As the number of hidden layers increases, the complexity and abstraction of data also increase.

It forms a hierarchy of low-level features to high-level features. With this, it becomes possible that Deep Learning Algorithm can be used to solve higher complex problems consisting of a vast number of nonlinear transformational layers.

Hidden Layers in Deep Learning

Deep Learning works by the architecture of the network and the optimum procedure employed by the architecture. The type of network followed is known as a Directed graph. The graph is designed in such a way that each hidden layer is connected to every hidden node.

So, combination and recombination of outputs from all units of hidden layer are performed in the context of the mix of their activation functions. This procedure is known as Non-Linear Transformation. After that optimum process is applied to the network to produce optimum weights for each unit of a layer.

It is the whole routine for the flow of information inside the hidden layers to produce required target output.

Too many hidden layers present in the algorithm is not feasible. It is because the neural network is trained with the simple gradient descent procedure. If a huge number of hidden layers are involved in the algorithm, then this gradient descent will be reduced that further affects the output.

You May also Love to Read Overview of Artificial Neural Networks & Its Applications

What is Machine Learning?

Machine Learning is a set of the technique used for the processing of large data by developing algorithms and set of rules to deliver the required results to the user. It is the method used for developing automated machines by execution of algorithms and set of defined rules.

In Machine Learning, data is fed and set of rules are executed by the algorithm. Therefore, techniques of Machine Learning can be categorized as instructions that are executed and learned automatically to produce optimum results.

It is performed without any human interference. It automatically turns the data into patterns and goes deep inside the system for the detection of production problem automatically.

What is Deep about Deep Learning?

The traditional neural network consists of at most two layers, and this type of structure of the Neural Network is not suitable for the computation of larger networks. Therefore, a neural network having more than 10 or even 100 layers are introduced.

This type of structure is meant for Deep Learning. In this, a stack of the layer of neurons is developed. The lowest layer in the stack is responsible for the collection of raw data such as images, videos, text, etc.

Each neuron of the lowest layer will store the information and pass the information further to the next layer of neurons and so on. As the information flows within the neurons of layers hidden information of the data is extracted.

So, we can conclude that as the data moves from lowest layer to highest layer (running deep inside the neural network), more abstracted information is collected.

You May also Love to Read Overview of Artificial Intelligence and Role of Natural Language Processing in Big Data

Classes of Deep Learning Architecture

• Deep Learning for Unsupervised Learning

This type of deep learning is used when labels of the target variable are not provided, and higher correlation has to be computed from observed units for Pattern Analysis.

• Hybrid Deep Networks

In this approach, the goal can be accomplished either by using supervised learning for performing pattern analysis or by using Unsupervised Learning.

Difference Between Neural Networks and Deep Learning Neural Networks

Neural Network is a network that can use any network such as feedforward or recurrent network having 1 or 2 hidden layers. But, when the number of hidden layers increases, i.e., more than two then that is known as Deep Learning Neural Network.

Neural Network is less complicated and requires more information about features for performing feature selection and feature engineering method. On the other hand, Deep Learning Neural Network does not need any information about features rather they show optimum model tuning and model selection on their own.

Why is Deep Learning Important?

In today’s generation usage of smartphones and chips have increased drastically. Therefore, more and more images, text, videos, and audios are created day by day. But, as we know that a single layer neural network can compute complex function.

On the contrary, for the computation of complex features, Deep Learning is needed. It is because deep nets within the deep learning method can develop a complex hierarchy of concepts.

Another point is that when unsupervised data is collected, and machine learning is executed on it, manually labeling of data has to be performed by the human being. This process is time-consuming and expensive. Therefore, to overcome this problem deep learning is introduced as they can identify the particular data.

Introduction to Deep Learning Neural Network

Various methods are introduced for the analysis of log file such as pattern recognition methods like K-N Algorithm, Support Vector Machine, Naive Bayes Algorithm, etc. due to the presence of a large amount of log data, these traditional methods are not feasible to produce efficient results.

Deep Learning Neural Network shows excellent performance in analyzing the log data. It consists of good computational power and automatically extracts the features required for the solution of the problem. Deep learning is a subpart of Artificial Intelligence. It is a deeply layered learning process of the sensor areas in the brain.

Deep Learning Techniques

Different techniques of Deep Learning are described below -

• Convolutional Neural Networks

It is a type of network that constitutes of learning weight and biases. Every input layer is composed of a set of neurons where at every input a dot product is performed and move further with the concept of non-linearity. It is a kind of fully-connected type of network that uses SVM/Softmax function as a loss function.

• Restricted Boltzmann Machine

It is a kind of stochastic neural network that consists of one layer of visible units, one layer of hidden units and a bias unit. The architecture is developed in such a way that each visible unit is connected to all hidden units, and bias units are attached to all visible and hidden units. During the learning process, the restriction is developed so that no visible unit is connected with any visible units and no hidden unit is connected with any hidden unit.

• Recursive Neural Network

It is the type of deep learning neural network that uses same weights recursively for performing structure prediction about the problem. The stochastic gradient is used for training the network using backpropagation algorithm.

5 Amazing Applications of Deep Learning

• Biological Analogs

In the case of the Artificial Neural Network, the lowest layer can extract only essential features of the dataset. Therefore, the convolutional layer is used with the combination of pooling layers. It is performed to increase the robustness of features extraction. The highest convolutional layer is developed from the features of previous layers. These most top layers are responsible for the detection of highly sophisticated features.

• Image Classification

To recognize the human face, first the edges are detected by the Deep Learning Algorithm to form the first hidden layer. Then, by combining the sides, next shapes are generated as a second hidden layer. After that shapes are combined to create the required human face. In this way, other objects can also be recognized.

• Natural Language Processing

Reviews of movies or videos are gathered together to train them using Deep Learning Neural Network for the evaluation of reviews of films.

• Automatic Text Generation

In this case, a large recurrent Neural Network is used to train the text so that relationships between the sequence of strings could be determined. After learning the model, the text is generated word by speech/character by character.

• Drug Discovery

Deep Learning Neural Network is trained on gene expression levels, and scores of activations are used for the prediction of therapeutic use categories.

Data Used for Deep Learning

Deep Learning can be applied to any data such as sound, video, text, time series, and images. The features needed within the data are described below:

• The data should be relevant according to the problem statement.

• To perform the proper classification, the dataset should be labeled. In other words, labels have to be applied to the raw data set manually.

• Deep Learning accepts vectors as an input. Therefore, the input data set should be in the form of vectors and same length. This process is known as Data Processing.

• Data should be stored in one storage place such as file system, HDFS (Hadoop Distributed File System). If the data is stored in different locations which are not inter-related with each other then, Data Pipeline is needed. The development and processing of Data Pipeline is a time-consuming task.

You May also Love to Read Ingestion & Processing of Data For Big Data & IoT Solutions

Deep Learning Application Areas

Deep learning neural network plays a major role in knowledge discovery, knowledge application, and last but least knowledge-based prediction. Areas of usage of deep learning are listed below -

• Power image recognition and tagging

• Fraud Detection

• Customer recommendations

• Used for analyzing satellite images

• Financial marketing

• Stock market prediction and much more

Deep Learning Approach for Log Analytics

How deep learning can be utilized for the analysis of log data is explained with examples.

Suppose we have to analyze the server log to extract the information about the events performed by the internal employees and thus helps to interpret how much data is leaked from the organization's server.

There are many existing solutions for the security of data, but the results produced by them are not up to mark. Therefore, administrator of the security department examines the flow of data by analyzing the server logs.

But, there is a drawback that the response time taken by the administrator is huge and efforts made for the detection of leakage of data is vain. Deep Learning provides best results in analyzing the server log files.

A system is proposed that uses Deep Learning Algorithm to examine the activities of the internal employees. First of all, the security log information is collected. This information consists of user's information documents, their personal information along with the user access rights.

It also consists of information regarding the leakage of data from the database. The Data Leakage procedure is defined by considering both the security log list and the purpose of the data leakage. After completion, a graph is developed using data leakage method.

This graph describes at what time the data is leaked and distinguish the personal information of each internal employee using different color palettes. After the graphical representation of data leakage, the deep learning is trained to classify the graphs into the normal and the abnormal behavior of internal employees.

Deep Learning Algorithm is implemented by using these graphs as an input and compare the similarity with the graph showing the data leakage by the internal employee. After receiving the information, the administrator will examine the path of the data leakage.

Now let's discuss another example about the analysis of log messages using deep learning algorithm. Log messages consist of messages in the form of text. Traditional algorithms like support vector machine etc. do not produce optimum results while performing text classification.

This is because these methods are not able to determine the semantic relationship between the words. Therefore, deep learning algorithm known as a Recurrent Neural Network is used for the Log Analysis.

The concept behind the recurrent neural network is that it consists of a hidden layer which acts as a memory that stores the internal state of the log data. When the new data arrived, the memory is updated, and decisions are made according to the current and previous input.

The input layer will consist of log messages and training is performed with the algorithm, whenever the abnormal behavior is depicted by the hidden layer, an alert will be given. This is one of the best approaches for the analysis of log files.

Now let’s discuss how log analytics is performed in the Big Data Platform using Deep Learning. Firstly, all types of log data are taken as input such as proxy infrastructure log, DNS infrastructure log and much more. Data Integration is performed by collecting all log data at one location.

There are various data integrated tools in Big Data Platform such as Apache Flume, Apache Nifi, and Apache Kafka. After the data collection, next step is to store the log data into the storage system such as HDFS (Hadoop Distributed File System), No-SQL Database like HBase, etc.

After storage, processing of the data is performed by the corresponding tool engine like Apache Spark, MapReduce and much more. Then deep learning techniques are executed, and patterns are identified as output.

The obtained output of Deep Learning in CSV format is stored in the storage system. While running Deep Learning security use cases are also performed parallelly. After that, the output is visualized in the form of Graphical User Interface (GUI).

Machine Learning Approach For Log Analytics

The basic concept of Machine Learning usage for log analytics can be explained with an example. As shown in fig three types of inputs are obtained. First input sources are system counters, CPU, memory, disk, and network.

Now the second input source is a large amount of distributed logs from different applications around your system. Third input source is consist of error logs, crashing of executable programs, improper shutting down of applications, etc.

After the collection of all these input sources, a relevant type of information from these logs is extracted automatically with the use of Bayesian Algorithm. Relevant logs are obtained as an output. Machine learning is used to aggregate the logs automatically into correlated categories. Then, newly log data will automatically incorporate into the corresponding category.

Let's take an example how machine learning can be used to detect system failure automatically. First, select the representation of features of log data and use them to fit the appropriate model according to the given dataset.

Training data is used to recognize the failure within the system. Now, evaluate the performance of the model using test data set. This is the process of supervised learning, i.e., log data patterns can be defined in advance.

On the contrary, if log data patterns cannot be defined in advance, unsupervised learning is introduced. In this approach, most relevant patterns are taken without the need of training data set provided by the human being.

You May also Love to Read Understanding Log Analytics, Log Mining & Anomaly Detection

How Can Don Help You?

Power your operational intelligence using smart logs to impact the customer experience.

NexaTrace - Monitoring, Alerting & Analytics Platform

NexaTrace is a Monitoring and Alerting Platform which helps in Monitoring the Infrastructure, Data Pipeline and Applications/Jobs running on Kubernetes Cluster. It helps in defining the Data Collection Layer which collects metrics stats of Cluster, Pipelines, Applications running on it using REST API, Agent-Based Collection using SNMP Protocols etc.

Collect Machine Data from all sources in real-time. Collect both structured and un-unstructured data to gain real-time visibility into physical, virtual or Cloud Infrastructure. Run your queries on real-time using Machine Learning Algorithms.