Artificial neural networks (ANNs) are computational structures of layered, interconnected units that learn mappings from inputs to outputs by optimizing connection weights, a paradigm widely used within Machine Learning and contemporary Deep Learning. According to Encyclopædia Britannica, neural networks are software systems inspired by biological neural processing and are trained to recognize patterns through weight adjustments in layered architectures. Britannica, Britannica.

History and foundational results

• –1943: Warren McCulloch and Walter Pitts introduced a formal model of neurons as threshold logic units and proved networks of such units can compute logical functions, establishing a bridge between neurophysiology and computation. Cambridge Core.
• –1958: Frank Rosenblatt’s Perceptron introduced a probabilistic learning machine that adjusts weights to classify inputs, marking an early learning algorithm in the ANN lineage. APA PsycNet summary via DOI.
• –1969: Marvin Minsky and Seymour Papert analyzed single‑layer perceptrons’ limitations (e.g., inability to learn XOR without additional features), steering research toward multilayer models and theory. MIT Press.
• –1986: Backpropagation enabled effective training of multilayer networks by propagating gradients through layers, revitalizing neural network research. Nature. This algorithm is now the canonical method for training networks; see Backpropagation. Nature.
• –1989–1991: Universal approximation theorems established that feedforward networks with at least one hidden layer and suitable nonlinearities can approximate continuous functions on compact sets to arbitrary accuracy, providing theoretical grounding for expressivity. SpringerLink/MCSS reference via DOI listing, DeepDyve metadata.
• –1997: Long Short‑Term Memory (LSTM) addressed vanishing gradients in recurrent networks, enabling long‑range sequence learning. MIT Press/Neural Computation (context), OA.mg metadata for LSTM.
• –1998: Convolutional neural networks (CNNs) achieved state‑of‑the‑art document and handwriting recognition with gradient‑based learning. Proceedings of the IEEE. See Convolutional Neural Network.
• –2006: Layer‑wise pretraining of deep belief nets offered practical routes to training deep architectures prior to the dominance of purely supervised training. Neural Computation.
• –2012: A deep CNN (“AlexNet”) won the ImageNet challenge by a large margin, catalyzing the modern deep learning era. NeurIPS proceedings.
• –2015–2016: Residual networks (ResNet) enabled very deep models; reinforcement learning with deep networks achieved human‑level control in Atari; AlphaGo combined deep policy/value networks with tree search to defeat top human players. arXiv, Nature (via DOI metadata: Opendata index), Google Research.
• –2017: Transformers dispensed with recurrence and convolution in favor of self‑attention, becoming the dominant architecture for sequence modeling and large language models. See Transformer. arXiv.

Architecture and components

• –Neuron model and layers: Each artificial neuron computes a weighted sum of inputs and applies a nonlinearity; neurons are arranged in input, hidden, and output layers connected by weighted edges. Britannica, Goodfellow–Bengio–Courville.
• –Activation functions: Logistic and tanh were early standards; rectified linear units (ReLU) and variants improved optimization by mitigating saturation and enabling sparse activations. PMLR/AISTATS, Goodfellow–Bengio–Courville.
• –Losses and outputs: Cross‑entropy with softmax for multiclass classification and various regression or ranking losses are typical, selected by task. Goodfellow–Bengio–Courville.

Learning and optimization

• –Training objective: Networks minimize an empirical risk (loss) over training data by gradient‑based optimization using backpropagation to compute parameter gradients. Nature, Goodfellow–Bengio–Courville.
• –Optimizers: Stochastic gradient descent (SGD) and momentum variants are common; Adam is a widely used adaptive first‑order method for large‑scale problems. arXiv, Goodfellow–Bengio–Courville.
• –Regularization and stabilization: Dropout reduces overfitting by randomly removing units during training; batch normalization accelerates and stabilizes training by normalizing layer inputs. JMLR, arXiv.
• –Learning paradigms: Supervised learning (labeled data), unsupervised/self‑supervised learning (representation learning via autoencoders or contrastive objectives), and reinforcement learning (learning policies from reward signals) are all used with ANNs. Goodfellow–Bengio–Courville, Nature review.

Principal network families

• –Feedforward multilayer perceptron (MLP): Directed acyclic graphs mapping fixed‑size inputs to outputs; universal approximators under mild conditions. DOI summary for Cybenko, DeepDyve metadata for Hornik et al..
• –Convolutional neural networks (CNNs): Share weights via convolution to exploit local stationarity in images and similar data, achieving state‑of‑the‑art recognition. Proceedings of the IEEE, NeurIPS 2012. See Convolutional Neural Network.
• –Recurrent neural networks (RNNs) and LSTM: Maintain state over sequences for language, speech, and time series; LSTM mitigates vanishing gradients for long‑term dependencies. OA.mg LSTM metadata, Goodfellow–Bengio–Courville.
• –Transformers: Use multi‑head self‑attention to model dependencies without recurrence, enabling high parallelism and scaling in language and multimodal models. See Transformer. arXiv.

Applications and impact

• –Computer vision: CNNs advanced image classification, detection, and recognition, notably on MNIST and ImageNet. Proceedings of the IEEE, NeurIPS 2012.
• –Natural language and speech: Sequence models and Transformers improved machine translation, language modeling, and speech recognition through attention and large‑scale pretraining. arXiv, Nature review.
• –Decision‑making: Deep reinforcement learning achieved human‑level control in Atari games and, combined with tree search, enabled AlphaGo to defeat top human players. Nature (DQN) metadata, Google Research/AlphaGo.

Key methods and practices

• –Weight initialization, learning‑rate schedules, normalization layers, and architectural motifs (e.g., residual connections) are standard engineering practices to train deep networks reliably at scale. arXiv, arXiv, Goodfellow–Bengio–Courville.
• –Interpretability, generalization, and data efficiency remain active research areas; survey overviews document capabilities and open challenges across domains. Nature review, Goodfellow–Bengio–Courville.

Terminology and internal links

• –The perceptron is a single‑layer linear classifier; see Perceptron. MIT Press.
• –Backpropagation is the gradient‑based learning procedure used for end‑to‑end training; see Backpropagation. Nature.
• –Convolutional Neural Network and Transformer denote two major architecture families; see Convolutional Neural Network and Transformer. Proceedings of the IEEE, arXiv.