APPLICATION OF ARTIFICIAL NEURAL NETWORKS METHOD IN BIOTECHNOLOGY

There has recently been a paradigm shift from the processing of experimental, clinical and laboratory data to their multifactor and multivariate analysis [1]. Thus, the new medical and biological knowledge should be recognized in the obtained empirical information, to be later possibly used in practice. Nowadays, a lot of important diagnostic and prognostic data in biology and medicine are achieved with mathematical data processing. Moreover, many new methods of mathematical analysis were invented for the needs of biology and medicine. According to this review, several methods of mathematical data processing occupy a priority place in biology. There is so much data analysis in the modern biology that the researchers form the data arrays, databases. Consequently, researchers need appropriate approaches to process such data [1, 2] and the modern powerful methods of artificial neural networks (MANNs) attract their special attention. ANNs are successfully implemented in modern biology to solve both practical and purely theoretical problems in forecasting, classification or management. The reason for this is that ANNs are multifunctional and relatively easy to use to obtain important results. This article describes the ANN methods, their theoretical apparatus, and application perspectives for solving biotechnology problems. Then the examples of complex modern works in biology, which require combined sets of mathematical methods, are suggested. Such works require not only ANN methods, but also such modern powerful methods as cluster analysis, image processing, regression analysis, probability theory, etc. The application perspectives of such sets of methods for data sets analyzing in biotechnology has been demonstrated. In biology, ANN methods are used since the mid-1980s, with the increased interest in mathematical methods [3–5]. Although later the interest in ANN temporarily waned, it reemerged following a period of reduced research funding [6–7]. The much wider availability and the increased power of computing systems, together with new areas of research, expanded the range of potential MANN application [6]. This happened after the MANN potential to accurately describe characteristic of the extremely complex systems was recognized. Present article examines the contribution of various network methodologies to bioprocess modeling, their control, and pattern recognition. Industrial processes can benefit from the application of feedforward neural networks with sigmoid activation functions, radial basis function networks and autoassociative networks. The implementation of ANN for modeling and control in biotechnology is presented in [7], UDC 004:591.5:612:616-006 https://doi.org/10.15407/biotech10.04.005

There has recently been a paradigm shift from the processing of experimental, clinical and laboratory data to their multifactor and multivariate analysis [1].Thus, the new medical and biological knowledge should be recognized in the obtained empirical information, to be later possibly used in practice.Nowadays, a lot of important diagnostic and prognostic data in biology and medicine are achieved with mathematical data processing.Moreover, many new methods of mathematical analysis were invented for the needs of biology and medicine.According to this review, several methods of mathematical data processing occupy a priority place in biology.
There is so much data analysis in the modern biology that the researchers form the data arrays, databases.Consequently, researchers need appropriate approaches to process such data [1,2] and the modern powerful methods of artificial neural networks (MANNs) attract their special attention.ANNs are successfully implemented in modern biology to solve both practical and purely theoretical problems in forecasting, classification or management.The reason for this is that ANNs are multifunctional and relatively easy to use to obtain important results.This article describes the ANN methods, their theoretical apparatus, and application perspectives for solving biotechnology problems.Then the examples of complex modern works in biology, which require combined sets of mathematical methods, are suggested.Such works require not only ANN methods, but also such modern powerful methods as cluster analysis, image processing, regression analysis, probability theory, etc.The application perspectives of such sets of methods for data sets analyzing in biotechnology has been demonstrated.
In biology, ANN methods are used since the mid-1980s, with the increased interest in mathematical methods [3][4][5].Although later the interest in ANN temporarily waned, it reemerged following a period of reduced research funding [6][7].The much wider availability and the increased power of computing systems, together with new areas of research, expanded the range of potential MANN application [6].This happened after the MANN potential to accurately describe characteristic of the extremely complex systems was recognized.Present article examines the contribution of various network methodologies to bioprocess modeling, their control, and pattern recognition.Industrial processes can benefit from the application of feedforward neural networks with sigmoid activation functions, radial basis function networks and autoassociative networks.
The implementation of ANN for modeling and control in biotechnology is presented in [7], along with several models of general classes of ANN structures.A baker's yeast production is modeled using the classical feedforward structure with multilayer of static neurons; the Polak-Ribiera algorithm with Powell modification is applied for the adaptation of parameters.Learning sets of input-output patterns are obtained in a computer model, using data from an industrial plant.A modular structure of "ANN MISO systems" for on-line estimation of pH, ethanol partial pressure, biomass concentration and the cell (metabolic) respiratory quotient (RQ) based on observed RQ in gas phase was proposed in [7], and the application of ANN modules for integration into a control structure was discussed.
Methods of artificial neural networks: general information.MANN is widespread not only in science, but in technology (including biotechnology) also.It is one of the widely used methods in theoretical biology and medicine.From the machine learning point of view, the neural network is a partial case of clustering methods [1,2].
ANN definition.Artificial neural networks are computing devices, assembled from simple processing elements (processors, sometimes numerous processors) in a parallel manner, which are interacting with each other.ANN is a machine that can easily be adapted to solve various tasks.Each of such simple processor deals only with the signals that it periodically receives on the input, and the signals that it periodically outputs to the network constructed from other processors.Although the computational capabilities of each processor (artificial neuron -AN) are limited, their large numbers combining into a network with controlled interaction essentially increase their capabilities.The structure of links between the network elements reflects how they are combined, and the problem they target.ANNs have a layered organization (Fig. 1).
Different layers can realize different transformations of received signals to the input of the next AN.The processed signals are shifted from the AN inputs of the first layer to the outputs of the last layer.ANN is a mathematical model with the software or hardware implementation; it is built on the principle of organization and operation of the neural network in a living organism [1,2,8,9].The underlying idea is the development of methods for problem solving similarly to the human brain.
ANN characteristics.The ANN may be characterized by the following notions (Fig. 2).
1. Neurons.The artificial neurons (ANs), which receive input signals from other ANs, may be described by following characteristics: 1) activation, which depends on the discrete time parameter; 2) threshold that remains fixed until the learning function is changed; 3) activation function that calculates the new activation at a certain moment, as well as the new input value; 4) output function.The output function very often can serve as identification function.The incoming AN does not have a predecessor, but that AN serves as input interface for the entire network.Accordingly, the output AN has no neuron-successor and it serves as interface for the entire network.
The computing capabilities of neurons (simple processors) are usually limited by a certain rule of input signals combination, and by the activation rule.This allows calculating the output signal based on input signals.The output signal is transmitted to another element with a certain weighting factor, depending on the weight; the signal can either be amplified or faded.
2. Connection and weight.The network consists of connections ( "synapses" of natural brain networks).Each connection (predecessor) transmits a signal from input neuron to output neuron (successor).The "weight" must be assigned to each connection.
3. Distribution function.This function calculates the input value of neuron depending on previous AN outputs.4. Training of artificial neural network.Training rule is a rule or algorithm that modifies the ANN parameters in order to create a favorable exit to this network entry.This learning process usually means changing of weights and thresholds of the network.ANNs are not programmed in the usual sense of the word; they can learn.Ability to be trained is one of the main advantages of ANN in comparison with traditional algorithms.Technically, training must result in the coefficients of communication between the ANs.During learning, ANN is able to detect complex interdependencies between the incoming data and output, and to make conclusions (generalizations).This means that in the case of successful training, the network will be able to return the correct result based on the data missing in the training sample (as well as incomplete and/or "noisy", partially distorted data).
From the standpoint of machine learning, ANN is a separate case of image recognition methods, discriminant analysis, clustering methods, and so on.From the mathematical point of view, ANN training is a multiparameter problem of nonlinear optimization.In cybernetics, ANNs are used for adaptive control and as algorithms for robotics.In the development of computer technology and programming, the neural network is a way of solving the problem of effective parallelism.From the point of view of artificial intelligence, ANN is the main direction of modeling natural intelligence with the help of computer algorithms.
Theoretical parallels between ANN and natural neural networks of the brain.Basic artificial "brain-like" model.The neural networks are inherently attractive because they are based on the well-known biological model of the nervous system.In time, such neurobiological models can lead to the emergence of elaborate "thinking computers".Meanwhile, "simple" neural networks already are powerful elements of applied statistics.
Other parallels between an artificial and a natural brain neuron: the single neuron level (Fig. 2).The theoretical similarities of ANN and a biological neural system can be given as: • ANN receives input signals (input data or output signals from previous "predecessor" neurons of the neural network) through several input channels.Each input passes through a connection with a certain intensity (or weight).This weight corresponds to the synaptic activity of biological neuron.A certain threshold value is associated with each neuron.The weighted sum of inputs is calculated, the threshold value is subtracted from it, and as a result, certain levels of neural activation are obtained (also known as the neuron's postsynaptic potential, PSP).• The activation signal is converted by activation function (or transfer function), resulting in the output signal formation.
One of activation functions is a threshold function, when the neural output is zero if the input is negative and one if the input is zero or positive.If this function is used, then such AN will work exactly as the natural neuron described above (to subtract the threshold from the weighted sum and then compare the result with zero is the same as comparing the weighted sum with a threshold value).In fact, as we will see soon, threshold functions are rarely used in artificial neural networks.Note that the weights may be negative, which means that the "synapse" input is inhibiting, not stimulating the neuron.
• Connections between neurons.If the network is intended for something, then it should have inputs (values of the significant variables from outside) and outputs (predictions or control signals).The inputs and outputs correspond to the sensory and motor nerves, for example those of eyes and hands.In addition, however, there may be many more intermediate (hidden) neurons in the network that perform internal functions.Input, hidden, and output neurons must be interconnected.
The key question here is feedback [1,2,9].The simplest network is built as direct signal transmission: the signals pass through input and hidden elements and eventually come to the output elements.This structure is stable.If the network is recurrent (containing links leading back from the more distant neurons), then it can be unstable and have very complex behavior.Though the recurrent neural networks are of great interest to researchers, the structures of direct transmission proved to be most useful in solving practical problems, at least until now.A typical direct signal transmission network is shown at Fig. 3.The neurons are organized in layers in a regular manner.The input layer serves simply to enter the values of the input variables.Each of the hidden and output neurons is connected to all elements from previous layer.It is possible to consider networks in which a neuron would be connected with only a few neurons of the previous layer.However, for most applications, the network with a full communication system is preferable.
As can be seen in Fig. 3, when this network is used, the input variables are applied to inputs elements.Then the intermediate, and later the output neuron layers are engaged.Each of them calculates its activation value, subtracting the threshold value from the weighted sum of the previous layer's outputs.Then the activation value is converted using the activation function, and as a result, the neural output is obtained.When the entire network finishes the operation, the output values from output layer elements are taken as output of the entire network.
Training of artificial neural network.A somewhat oversimplified description of the neural network conversion of the input to output signals is given above.Now the following important question arises: how to apply a neural network to specific tasks, for example, for biotechnology tasks?
ANN implementation in solving different tasks.The type of tasks that can be solved using a neural network is determined by how the network works and how it is trained.During its operation, the neural network assumes the values of input variables and outputs the values of resulted variables.Thus, it can be used if you know certain information and want to obtain some yet unknown information from ANN [1,2,9].
Typically, a neural network is used when the exact relationship between inputs and outputs is unknown.If it were known, then the link could be simulated directly.Another essential feature of neural networks is that the relationship between input and output is defined in its training process.For that training, two types of algorithms are used (different types of networks use different types of training): 1) controlled ("training with a teacher") and 2) non-managed ("without a teacher").The most commonly used is training Reviews 9 with a teacher, and this is the method we would like to examine now.
To manage the network, the user must prepare a set of training data.These data are examples of input data and the corresponding output data.The network learns how to establish connections between the former and the latter.Typically, the data for training are derived in experiment.The neural network is then trained using one of the managed learning algorithms (the most known of them is the backpropagation algorithm [2]) in which the available data are used to adjust the weights and thresholds of the network to minimize the forecast error on the training set.If the network is trained well, it will be able to model an unknown function that binds the input and output variables.Subsequently, this network can be used to predict in a situation where output values are unknown.
Data acquisition.If the problem must be solved with the help of ANN, then its training requires data collection.The training dataset is a set of data for which input and output variables are specified.The first question is choosing the variables and the amount and type of data.
The choice of variables (at least initial) is carried out intuitively.Identifying the important variables becomes easier with experience.When working with standard software STNN package, one can arbitrarily select variables and cancel the previous selection; this package can experimentally select the useful variables.For the beginning, it makes sense to include all the variables that can probably influence the result.Later, it is possible to reduce this set.
Peculiarities of the variables for ANN processing 1) Neural networks can work with numeric data within a certain limited range.This becomes problematic when data has a nonstandard scale, with missing values, and when data is non-numeric.Some standard computing packages (including STNN package) have the tools to overcome all these difficulties.Numeric data are scaled to the appropriate range for the network, and the missing values can be replaced with the average or other statistic values for all available training examples [2].
2) Working with non-numerical data is more difficult.Very often non-numeric data are presented in the form of nominal variables that could be represented in numerical form, and standard computing packages (including STNN package) have the means for working with it.However, neural networks do not produce satisfactory results for nominal variables that can obtain many different values.Nominal variables can be two-digit (for example, Gender = {Man, Woman}) or multivalued (i.e., accept more than two values or states).A two-digit nominal variable can easily be converted into numeric (for example, Male = 0, Woman = 1).It is more complicated with multivalued nominal variables.They can also be represented by a single numerical value (for example, Dog = 0, Sheep = 1, Cat = 2).However, with this the values of a nominal variable may be falsely ordered: in the example above, "Sheep" will be something between the "Dog" and "Cat".Hence, a more precise 1-of-N encoding method emerged, where one nominal variable is represented by several numerical variables.The number of numerical variables is equal to the number of possible values of the nominal variable; in each case exactly only one of the N variables assumes a nonzero value (for example, Dog = {1,0,0}, Sheep = {0,1,0}, Cat = {0,0,1}).Unfortunately, a nominal variable with a large number of possible states will require a very large number of numerical variables to be encoded by the 1-of-N method.This increases the network size and complicates its training.In such situations, it is possible (though not always is enough) to simulate a nominal variable using a single numerical index.Still, it would be better to find another way to represent data.
3) The non-numerical data of other types could either be converted to numerical form or declared nonessential.Values of dates and time, if necessary, can be converted into numerical ones, subtracting from them the initial date (time).
Estimating the amount of observation needed for networking is often difficult.There are a few heuristic rules, linking the number of necessary observations to the network size.The simplest of them says that the number of observations should be ten times the number of links in the network.In fact, this number also depends on the (previously unknown) complexity of the image that ANN tries to reproduce.As the number of variables increases, the number of required observations increases nonlinearly, so that a rather small (f.e., fifty) number of variables may require a lot of observations.For most practical tasks, hundreds or thousands of observations are enough.For even more complex tasks, one may need even more.A task requiring fewer than a hundred observations is very rare.If number of the data is less than what is said here, then in fact you do not have enough information to train the network, and the best thing you can do is try to fit some linear model into the data.
BIOTECHNOLOGIA ACTA, V. 10, No 4, 2017 Many real problems have to deal with not quite reliable data.Values of some of the variables may be distorted by noise or partially absent.The STNN package specializes on with missed values (they can be replaced by the average value of this variable or its other statistics), so if you do not have enough data, you can include cases with missing values (though, of course, it is better to avoid this).In addition, neural networks are generally resistant to noise.However, this stability has a limit.For example, outliers of some variables may distort the result of the training.In such cases, it is better to try detecting and removing these fluctuations (either removing relevant observations or converting "outliers" into missed values).
Pre/post processing.Each neural network accepts numerical values as input and outputs numerical values.The transfer function for each network element is selected usually so that its input argument can obtain any values, but the output values belong to a strictly limited range ("flattening").In this case, although the input takes any value, there is a saturation effect if the element is sensitive to only the input values lying in a certain limited range.As long as the output values belong to a limited range, and all information is presented in numerical form, MANN would require preprocessing and post-processing of data.There are appropriate means in the STNN package.
Prediction.The prediction goal can be divided into two main classes: classification and regression.The classification infers determining which of the several specified classes belong to this input set.Examples include loan provision (whether this person is a high or low risk group), diagnosis of cancer (tumor or healthy), and signature recognition (fake, genuine).In all these cases, obviously, only one nominal variable is required at the output.Often, the classification problems are two-valued (as in the examples), although there are problems with several possible states.
Using the method of regression the value of a variable that accepts (as a rule) continuous numerical values can be predicted.In this case, one numeric variable is required as output.The neural network can solve multiple regression and/or classification tasks simultaneously, but usually only one task is solved at a single moment.Thus, in most cases, the neural network will have only one output variable.If there are more than one output variables, several output elements may be required (postprocessing the data to convert the information of output elements to the output variable).
In some standard computing software packages (including STNN package), special prepost and post-processing tools are implemented.They allow to represent the raw data numerically for ANN processing and to convert the output of the neural network back to the input data format.The neural network serves as a "layer" between pre-and post-processing, and the result is given in necessary form.For example, the name of output class is given in the classification task.In addition, in such standard packages the user can directly access the internal activation network parameters.
Concept of perceptron: two stages of development.To solve problems in biotechnology, a basic ANN form is used, known as the perceptron.This is a mathematical or computer model of information perception by the brain.The perceptron concept was developed in the second half of the XX century; below we present this idea at two different stages of its development.
1) Perceptron.At the beginning the perceptron was invented as a cybernetic model of a brain, one of the first ANN models.It was implemented as an electronic machine Mark-1 (1960, the first neurocomputer).The perceptron is capable of training and solving quite complex tasks, such as linear separation of random nonlinear sets (socalled linear separability).The perceptron consists of three element types, namely the signals which are received from the sensors, transmitted to the associative elements, and then to the responders.Thus, a set of "associations" between the input stimuli and necessary reaction at output can be constructed using such perceptron.In biological terms, this corresponds to the transformation of, for example, visual information into the physiological response of motor neurons.The perceptrons can be classified as artificial neural networks that, in modern terms, have: 1) one hidden layer; 2) a threshold transfer function; 3) the direct propagation of the signal.
2) Multilayer Perceptron (MLP) today has another meaning.This network architecture is now probably the most used.It was proposed by Rumelhart, McClelland [2] and discussed in detail in almost all textbooks on neural networks.Briefly, this type of network has been described above.Each network element builds a weighted sum of its inputs with a correction term, and then skips this activation value through the transfer function, obtaining the output value for this element.The elements are layered, and there is a direct signal transmission.This network can easily be interpreted as an input-output model in which weights and thresholds (offsets) are free parameters of the model.It can simulate a function of almost any complexity, and the number of layers and the number of elements in each layer determines the complexity of the function.Determining the number of intermediate layers and the number of elements in them is an important issue in the MLP design [2].
Combined application of MANN and some other contemporary mathematic methods.Solving of many modern complicated problems in science requires the combination of several mathematic methods.Method of ANNs is often used with such methods as cluster methods, regression analysis, probability theory, etc. Below we review several examples of such contemporary problems.
Pattern recognition and classification.Images can be represented with different objects: text symbols, images, samples of sounds, etc.When training the network, various samples of images are offered, and their class is indicated.The sample, as a rule, is assembled as a vector of attribute values.In this case, the total set of signs must uniquely determine the class to which the model belongs.If the number of signs is insufficient, the network may relate one sample to several classes, which is incorrect.After the completion of network training, it can consider previously unknown images and refer them to a defined class.
Application of ANN methods for images recognition.Implementation of MANNs is one of the most progressive practical applications of modern science.One of such areas is computer analysis of images -chromatographic studies of chemical and biochemical compounds and samples, etc.The specific difference in biochemical composition of a sample would be reflected in chromatographic images.Another instance of applying the computer analysis method to chromatographic images is the identification of the sample in the presence of specific proteins, marker substances, etc., or for distinguishing between cancer cells and normal cells.An algorithm for image analysis of biological object [1,8,[10][11][12][13][14][15]] is considered.An example of recognition of different images is given at Fig. 4.
ANN method is used in the multilayer perceptron modification for the recognition of normal and pathological areas at the histological images in oncology [16][17][18][19][20][21].The algorithm of processing histological images for further separation of normal and pathological areas using MANN is suggested at Fig. 4.This algorithm has following stages: 1) making images with a web-camera; 2) converting the colored images into black-and-white; 3) choosing the necessary image areas (image segmentation and separation of boundaries); 4) digitalizing images, obtaining the contour image.Digitalization (binarization) of an image means that each pixel can be black or white only and thus may obtain notions "0" or "1".MANN permits to perfect diagnostics in medicine, object separations in biochemistry Our detected mitosis are circled green (true positives) and red (false positives); cyan denotes mitosis not detected by approach [22].Top right: details of three areas (full-size results on the whole dataset in supplementary material).Note the challenging appearance of mitotic nuclei and other very similar non-mitotic structures.Bottom: overview of detection approach [22] and is widely applied in modern practice for such purposes.
Thus, it is shown that: • The considered mathematical method, MANN is used in complex applications, because many problems of modern biology are so complex that they require complicated mathematical apparatus [1].We considered a series of works in which the combined application of methods of cluster analysis, image processing, and neural networks is demonstrated.It is shown that in the image analysis (histological sections, tomographic and endoscopic images), methods of image processing can be supplemented with great success by methods of neural networks [1,8,[10][11][12][13][14][15].
• It is important to choose the variables for ANN that would influence the result.It is possible to directly work with numerical and nominal variables in some standard computing packages (including STNN package).Variables of other types should be converted to specified types or declared nonessential.If necessary, observations containing missed values can be analyzed.Data fluctuations may present a difficulty and must be removed if possible.If there is enough data, it is necessary to remove the missing values from the study.
• This type of analysis requires the results of hundreds or thousands observations; the more variables are there in the problem, the more observations are needed.In some standard computing packages (including STNN package) it is possible to recognize the meaningful variables, thus all variables should be examined.
• Methods of neural networks have proven effective in solving the medical differential diagnosis problems.The theory of this group of methods is observed briefly.It has been demonstrated how the neural network method was used for prognostic studies [19].The method of neural networks is a modern approach to the personification of medical treatment and will determine the effectiveness of such radical treatment.
• Because of high responsibility of the work done by the physicians and biologists, neither the artificial neural network nor the computer diagnostics can play a decisive role in the choice of their methods.The purpose of such methods is to only facilitate the doctor's work and to make diagnostic processes and medical treatment correct.

Fig. 2 .
Fig. 2. Theoretical parallels between artificial neuron and natural brain neuron: description of separate neuron (details see in text)

Fig. 4 .
Fig. 4. Using the ANN method for differentiating cells of different types Top left: one image (4 MPixels) corresponding to one of the 50 high power fields represented in the dataset.Our detected mitosis are circled green (true positives) and red (false positives); cyan denotes mitosis not detected by approach[22].Top right: details of three areas (full-size results on the whole dataset in supplementary material).Note the challenging appearance of mitotic nuclei and other very similar non-mitotic structures.Bottom: overview of detection approach[22]