Eugeny Shtoltc - Machine learning in practice – from PyTorch model to Kubeflow in the cloud for BigData

Machine learning in practice – from PyTorch model to Kubeflow in the cloud for BigData
Название: Machine learning in practice – from PyTorch model to Kubeflow in the cloud for BigData
Автор:
Жанр: Программирование
Серии: Нет данных
ISBN: Нет данных
Год: 2020
О чем книга "Machine learning in practice – from PyTorch model to Kubeflow in the cloud for BigData"

In this book, the Chief Architect of the Department of Architecture and Management of Technical Architecture (Cloud Native Competence Center and the Corporate University of Architects) of Sberbank shares his knowledge and experience with readers in the field of ML. received in the work of the School of Architects and. Author: * guides the reader through the process of creating, learning and developing a neural network, showing in detail with examples * increases horizons, showing how it can take place in BigData from the point of view of the Architect * introduces real models of use in the product environment

Бесплатно читать онлайн Machine learning in practice – from PyTorch model to Kubeflow in the cloud for BigData


About the book

The book is structured like a textbook – from simple to complex. The reader will be able to:

* in the first three chapters, create the simplest neural network for image recognition and classification,

* in the following – to delve into the device and architecture for optimization,

* further expand the understanding of the company's ecosystem as a whole, in which neural networks operate, as its integral part,

and she interacts with and uses surrounding technologies,

* finish the study by deploying a full-scale production system in the full-cycle cloud.

Almost every chapter begins with the general information needed for the practical part that follows. In the practical part:

* demonstrates the process of preparing the environment, but more often free ready-made cloud services are used,

* demonstrates the writing process when with a parsing of the written and an overview of alternative solutions,

* analysis of the result and the formation of a vision of options for further development.

The book consists of sections:

* Introduction to Machine Learning. This is the only chapter without a practical part to get you started.

understanding the limits of their applicability, advantages over other methods and their general structure for beginners. Also produced

classification of neural networks according to the principles laid down in them, and the selection of a group, which will be discussed in the book.

* Basics for writing networks. It provides the basic knowledge necessary to write the first network in PyTorch, familiarity with the development environment

Jupyter in the Google Colab cloud service, which is a simplified version of the Google ML cloud platform, running the code in it and using the PyTorch framework for writing neural networks.

* We create the first network. The author demonstrates for the reader's practice how to create a simple neural network on PyTorch in

Colab with a detailed analysis of the written code, training it on the MNIST image dataset and launch it.

* Improving the recognition of the neural network on complex images. Here the author demonstrates to the reader not practice

training and prediction of neural networks for color pictures, methods to improve the quality of network predictions. Understands in detail

device, pitfalls in writing and training effective neural networks.

* Modern architectures of neural networks. The architectural principles used in modern neural networks for

improving the quality of predictions. An analysis of various neural network architectures that have made a breakthrough in the quality of training is given.

and brought approaches. Various architectural universal quality enhancement patterns such as ensemble are discussed.

* Using pre-trained networks. The use of already trained layers in their networks is demonstrated.

* ML scaling. Examples of preparing the environment for launching them in a cloud infrastructure are given.

* Receiving data from BigData. It tells how you can connect to various sources from Jupyter

data, including BigData, for training models.

* Big data preparation. This section describes BigData technologies such as Hadoop and Spark, which

are data sources for training models.

* ML in an industrial environment. This section covers systems such as Kubeflow and MLflow. The reader can try to expand

platform, set up the learning process and run the model in the cloud, as is done in companies.

Introduction Machine Learning

Artificial intelligence is a field at the intersection of many sciences. One of the ways to achieve it is machine learning, when we supply it with data and, on its basis, we learn how to find solutions and identify patterns and data that were not there before. For training, statistical algorithms can be used, for example, in the R language, depth-first search in Prolog or breadth-first search in Refal, as well as adaptive structures – neural networks. Neural networks, depending on the tasks, will hide according to different principles, have a structure and learn in different ways. Recently, the greatest breakthrough has been received by neural networks of data representation (Representation learning), which are engaged in identifying patterns in information, since they cannot remember the information itself due to its size. Deep neural networks with many levels of features give great effects, features at subsequent levels are built on the basis of features from previous levels, which will be discussed in this chapter.

Machine learning (ML) refers to the adaptation of the transformation of input data into outputs, depending on the history of decisions. This class of problems is solved either in an algorithmic way, or with the help of neural networks, and we will talk about where and in what situation it is better to apply what solution. For example, let's take uppercase numbers from zero to nine, which we will compare with printed ones. If the uppercase letters exactly fall into the contour, then everything is simple, we just need to go through the contours of the printed ones and get a suitable option. Such a task is not relevant to machine learning tasks. Now let's complicate the task – our numbers don't exactly match the pattern. If the uppercase numbers do not fit a little into the contour, we just find some kind of deviation. And this is where the difficulty arises when categorizing an uppercase number into zero and nine, when the size of the tail separates the careless spelling of zero from nine. Another point in the categorization of eight and nine. So, if the tip is bent, this is ten, and if it is bent and touches, then it is eight. To solve such a situation, you need to divide the figure into areas and depending on and assign them different coefficients. So, the connection of the tail of the lower part has a very high value than the shape of the circles themselves in the classification into eights and nines. Statistical data on a pre-given sample of the correspondence of figures to eights and nines will help to determine, where the researcher will be able to determine when it is already possible to calculate the lower ring closed and talk about the correspondence of the figure eight, and when not, talk about the correspondence of the nine. But we can programmatically divide the numbers into sectors and assign coefficients to them.

Another difficulty may be that the digit may not be in the observed area, but in an arbitrary one, for example, in a corner. To analyze the digit itself, we need to move the analyzing window to the place where the digit is. For simplicity, for now, we will assume that the dimensions of the analyzing window are equal to the dimensions of the figure under study. To solve this problem, an analyzing layer is put in front of the network, which forms a map of finding the numbers. The task of this layer is to determine the location of the number in the picture. Let's take a black image on a white sheet for simplicity. We need to go through the digit analyzer line by line throughout the sheet and determine the locations. Let us take the black area on the indicator as an indicator. After passing over a piece of paper and determining the area, we get a matrix with the numbers of the areas in black. Where there are more areas of black color, in that place the figure fits into the indicator as much as possible. Converting an image into a matrix of areas is called a convolution operation, and if it is performed by a neural layer, it is called a convolution layer. Neural networks that have a Conv Layers are called Convolutional Neural Networks (CNNs). Such networks are used in image recognition, now they have been adapted for speech recognition. The principle of operation was borrowed from the biological optic nerve.


С этой книгой читают
In this book, the Chief Architect of the Department of Architecture and Management of Technical Architecture of the Cloud Native Competence Center of Sberbank shares the knowledge and experience with readers, accumulated in the development of their own and assessment of other people's architectures, providing a basis for professional and career growth.
In this book, the chief Architect of the Cloud Native Competency Architecture Department at Sberbank shares the knowledge and experience with readers gained in developing their own and evaluating other people's architectures, providing a basis for professional and career growth.
In this book, the Chief Architect of the Cloud Native Competence Architecture Department at Sberbank shares his knowledge and experience with the reader on the creation and transition to the cloud ecosystem, as well as the creation and adaptation of applications for it. In the book, the author tries to lead the reader along the path, bypassing mistakes and difficulties. To do this, practical applications are demonstrated and explained so that the
Цель книги – познакомить читателей с существующими подходами и решениями в области моделирования бизнес-архитектуры предприятия. В книге освещаются различные аспекты данной проблематики, в том числе такие вопросы как базовые подходы к моделированию и возможности современных инструментальных средств.Особое внимание уделяется специфике организации проектов по разработке моделей бизнес-архитекуры. На основе практического опыта реализации проектов по
В этой книге автор, сам прошедший путь от разработчика до менеджера в сфере IT, рассказывает неочевидные моменты, которые являются критически важными для правильного управления. Почему разработчики увольняются после повышения зарплаты? Как делать FixedPrice проекты? Почему Scrum не упрощает менеджмент? Книга содержит ответ на эти и многие другие вопросы. В книге есть много баек, которые показывают тяжёлую, но интересную жизнь менеджера в разработ
Это не учебное пособие по информатике для старшеклассников, абитуриентов, студентов и аспирантов с двухзначным IQ. Это научно фантастическое эссе, главным героем которого является «межплатформенный искусственный разум». Занимаясь более 30 лет информационными технологиями, автор твердо убежден, что сегодня в мире нет более важной темы, чем эта. Ибо именно через искусственный интеллект на Земле наступить то самое Облачное Царствие, о котором так мн
Эта книга состоит из двух частей: художественной, где рассказывается удивительная история мальчика, захотевшего стать белым хакером (хотя в ней, конечно же, есть и девочка, а также его друзья), и научно-популярной, в которой дается обучение (она так и называется в книге, ты увидишь ее, когда начнешь читать) программированию и разным другим компьютерным вещам, очень нужным в современном мире. Но на самом деле это единая история, позволяющая задума
В настоящем издании представлены узловые положения фундаментальной социологической теории исторического развития отдельно взятого общества А. С. Шушарина «Полилогия современного мира (Критика запущенной социологии)», которая генерализирует известное учение К. Маркса «Капитал» и включает его в себя как составную часть. Дальнейшее развитие этой метатеории позволяет встать на путь научного познания, по-новому отнестись к современным тенденциям общес
Тайная организация «Монолит» обретает небывалое могущество. Отныне её интересует не только захват активов, деньги и незаконные операции, но и власть. Выходцы из спецслужб активно вмешиваются в политические процессы, одновременно расширяя зону своего контроля на финансовые операции и фондовый рынок. Сможет ли один человек противостоять огромной машине подавления? Пока одни воюют за нефть, доллары и дивиденды, другие – за правду, любовь и честь офи
Давным-давно мой мир погрузился во тьму. День как ночь, ночь как день. Каждый день я спрашиваю – почему? И каждую ночь я тебя ищу. Но кто ты? Кто? Во сне ты тихо шепчешь мне – я тебя люблю. Хочу сказать, что тоже тебя люблю, но ты убегаешь. Пытаюсь поймать тебя за руку, но ты исчезаешь. Где мне найти тебя?
В книге описан опыт автора путешествий по России в режиме SmartТУРИЗМ, позволяющем без проблем, не бросая работу, в выходные дни посетить любой город страны. Кратко рассказана история города, приведены его достопримечательности.