Understanding Deep Neural Networks Training Course

Part 1 – Deep Learning and DNN Concepts

Foundations of Artificial Intelligence, Machine Learning, & Deep Learning

• Historical context, core principles, and standard operational applications of artificial intelligence, distinguishing practical capabilities from speculative theories.
• Collective Intelligence: The aggregation of knowledge shared across distributed virtual agents.
• Genetic Algorithms: Evolutionary processes applied to populations of virtual agents through selection mechanisms.
• Standard Learning Machines: Formal definitions and foundational structures.
• Task Classification: Supervised learning, unsupervised learning, and reinforcement learning paradigms.
• Operational Objectives: Classification, regression, clustering, density estimation, and dimensionality reduction.
• Representative Machine Learning Algorithms: Linear regression, Naive Bayes, and Random Forests.
• Machine Learning vs. Deep Learning: Identifying domains where traditional Machine Learning methods (e.g., Random Forests, XGBoost) remain the optimal state of the art.

Core Neural Network Concepts (Application: Multi-layer Perceptron)

• Review of essential mathematical foundations.
• Definition of neural network structures: Classical architectures, activation functions,
• Weighted integration of previous activations and network depth considerations.
• Network Learning Mechanics: Cost functions, back-propagation, Stochastic Gradient Descent, and maximum likelihood estimation.
• Neural Network Modeling: Structuring input and output data based on problem type (e.g., regression, classification). Addressing the curse of dimensionality.
• Differentiation between multi-feature data and signal processing. Selection of cost functions based on data characteristics.
• Function Approximation via Neural Networks: Theoretical presentation and practical examples.
• Distribution Approximation via Neural Networks: Theoretical presentation and practical examples.
• Data Augmentation: Strategies for achieving dataset balance.
• Generalization capabilities of neural network outcomes.
• Network Initialization and Regularization: L1/L2 regularization techniques and Batch Normalization.
• Optimization and convergence algorithms.

Standard Machine Learning / Deep Learning Tools

A concise overview covering the advantages, limitations, ecosystem positioning, and operational uses of key tools.

• Data Management Frameworks: Apache Spark, Apache Hadoop tools.
• Machine Learning Libraries: NumPy, SciPy, Scikit-learn.
• High-Level Deep Learning Frameworks: PyTorch, Keras, Lasagne.
• Low-Level Deep Learning Frameworks: Theano, Torch, Caffe, TensorFlow.

Convolutional Neural Networks (CNN).

• Overview of CNNs: Fundamental principles and applicable domains.
• Basic CNN Operations: Convolutional layers, kernel utilization,
• Padding, stride, feature map generation, and pooling layers. Extensions for 1D, 2D, and 3D applications.
• Review of CNN architectures that established state-of-the-art classification performance.
• Image Processing Architectures: LeNet, VGG, Network in Network, Inception, ResNet. Analysis of architectural innovations (e.g., 1x1 convolution, residual connections) and broader applications.
• Integration of attention models.
• Application to standard classification tasks (text or image data).
• Generative Applications of CNNs: Super-resolution, pixel-to-pixel segmentation. Overview of
• Key strategies for enhancing feature maps in image generation tasks.

Recurrent Neural Networks (RNN).

• Overview of RNNs: Fundamental principles and applicable domains.
• Basic RNN Operations: Hidden activations, back-propagation through time, and unfolded network representations.
• Architectural Evolution: Gated Recurrent Units (GRU) and Long Short-Term Memory (LSTM).
• Analysis of internal states and advancements introduced by these architectures.
• Convergence challenges and the vanishing gradient problem.
• Standard Architectures: Temporal series prediction, classification, and related tasks.
• RNN Encoder-Decoder architectures. Implementation of attention models.
• Natural Language Processing (NLP) Applications: Word/character encoding and translation.
• Video Processing Applications: Prediction of subsequent frames in video sequences.

Generative Models: Variational AutoEncoder (VAE) and Generative Adversarial Networks (GAN).

• Overview of generative models and their relationship with CNNs.
• Auto-encoders: Dimensionality reduction and constrained generation.
• Variational Auto-encoders: Generative modeling and distribution approximation. Definition and application of latent space. Reparameterization trick. Analysis of applications and observed limitations.
• Generative Adversarial Networks: Fundamental principles.
• Dual Network Architecture (Generator and Discriminator) with alternating training processes and available cost functions.
• GAN convergence mechanisms and associated operational difficulties.
• Convergence Enhancements: Wasserstein GAN, Began, and Earth Mover’s Distance.
• Applications in image/photograph synthesis, text generation, and super-resolution.

Deep Reinforcement Learning.

• Overview of reinforcement learning: Agent control within defined environments.
• Formulation based on state definitions and potential action spaces.
• Utilization of neural networks to approximate state-value functions.
• Deep Q-Learning: Experience replay mechanisms and application to video game control.
• Policy Optimization: On-policy and off-policy methods. Actor-Critic architecture. A3C implementation.
• Applications: Control of single video games or complex digital systems.

Part 2 – Theano for Deep Learning

Theano Fundamentals

• Introduction to Theano.
• Installation and configuration procedures.

Theano Functions

• Inputs, outputs, update mechanisms, and givens.

Training and Optimizing Neural Networks with Theano

• Neural Network Modeling.
• Logistic Regression.
• Hidden Layer Configuration.
• Network Training Processes.
• Computation and Classification.
• Optimization Techniques.
• Log Loss Calculation.

Model Testing

Part 3 – DNN using TensorFlow

TensorFlow Fundamentals

• Creation, initialization, saving, and restoration of TensorFlow variables.
• Data feeding, reading, and preloading in TensorFlow.
• Leveraging TensorFlow infrastructure for large-scale model training.
• Model visualization and evaluation using TensorBoard.

TensorFlow Mechanics

• Data Preparation.
• Data Acquisition.
• Inputs and Placeholders.
• Graph Construction.
  • Inference Operations.
  • Loss Function Definition.
  • Training Operations.
• Model Training Execution.
  • Graph Structure.
  • Session Management.
  • Training Loop Implementation.
• Model Evaluation.
  • Evaluation Graph Construction.
  • Evaluation Output Generation.

The Perceptron

• Activation Functions.
• Perceptron Learning Algorithm.
• Binary Classification using Perceptrons.
• Document Classification using Perceptrons.
• Limitations of the Perceptron Model.

Transition from Perceptrons to Support Vector Machines

• Kernels and the Kernel Trick.
• Maximum Margin Classification and Support Vectors.

Artificial Neural Networks

• Nonlinear Decision Boundaries.
• Feedforward and Feedback Artificial Neural Networks.
• Multilayer Perceptrons.
• Cost Function Minimization.
• Forward Propagation.
• Back-Propagation.
• Strategies to Enhance Neural Network Learning.

Convolutional Neural Networks

• Objectives.
• Model Architecture.
• Core Principles.
• Code Structure and Organization.
• Model Launch and Training.
• Model Evaluation.

Foundational Overviews for the Following Modules (Brief introductions provided subject to time availability):

TensorFlow – Advanced Usage

• Threading and Queue Management.
• Distributed TensorFlow.
• Documentation Standards and Model Sharing.
• Customization of Data Readers.
• Manipulation of TensorFlow Model Files.

TensorFlow Serving

• Introduction.
• Basic Serving Tutorial.
• Advanced Serving Tutorial.
• Serving Inception Model Tutorial.