⚛️

Physics-Informed Neural Networks

Learning the Physics of Engineering Systems by Embedding Differential Equations

Status

Research Complete

Equation

1D Heat Equation

Role

Scientific ML Engineer

Research Motivation

Traditional approaches to solving PDEs in engineering have fundamental limitations:

Numerical Solvers: Finite difference/element methods require dense grid discretization. Expensive computationally.
Data-Driven ML: Pure neural networks need massive labeled datasets. Don't generalize across parameter ranges.
Physics Agnostic: Standard models can predict solutions outside physical domain (negative temperature, impossible states).

PINN Paradigm Shift: Combine neural networks with physics constraints. The network learns to satisfy differential equations directly, reducing data requirements and ensuring physical validity.

Physics-Informed Training

The Heat Equation (Physics Loss)

∂u/∂t = α ∂²u/∂x²

u(x,t) = temperature at position x and time t
α = thermal diffusivity (material property)
∂u/∂t = time derivative (computed via autograd)
∂²u/∂x² = second spatial derivative (computed via autograd)

Composite Loss Function

L_total = L_data + L_physics

L_data: MSE on initial/boundary conditions
Trains network to satisfy: u(x,0) = u₀(x) [initial temp] and u(0,t) = T₁, u(L,t) = T₂ [boundary temps]

L_physics: PDE residual
L_physics = (1/N_f) Σ(∂u/∂t - α ∂²u/∂x²)²
Forces network output to satisfy differential equation at collocation points

Why This Works

• Network outputs continuous function u(x,t) ∈ ℝ
• PyTorch autograd computes exact derivatives: ∂u/∂x, ∂²u/∂x², ∂u/∂t
• Optimizer minimizes both data loss AND PDE residual
• Result: Network learns physics-consistent solution

Architecture: PINN Feed-Forward Network

Input: (x, t) ∈ ℝ² ↓ ┌─────────────────────┐ │ Fully Connected │ layers: 128 → 128 → 128 → 128 │ Layer 1-4 │ activation: Tanh (smooth, good for PDEs) ├─────────────────────┤ │ Output Layer │ 128 → 1 └─────────────────────┘ ↓ Output: u(x, t) ∈ ℝ ↓ Compute derivatives via torch.autograd.grad() ↓ Calculate PDE residual: ∂u/∂t - α·∂²u/∂x²

Key Design Choice: Tanh activation functions. Why?

• Smooth, infinitely differentiable → stable autograd
• ReLU causes kinks → discontinuous derivatives → poor PDE satisfaction
• Range [-1, 1] aids training stability

Key Innovation: Automatic Differentiation

PyTorch autograd enables exact derivative computation:

# Forward pass u = network(x, t) # Compute du/dt (time derivative) du_dt = torch.autograd.grad(u, t, create_graph=True)[0] # Compute d²u/dx² (second spatial derivative) du_dx = torch.autograd.grad(u, x, create_graph=True)[0] d2u_dx2 = torch.autograd.grad(du_dx, x, create_graph=True)[0] # PDE residual (Heat Equation) pde_residual = du_dt - alpha * d2u_dx2 physics_loss = (pde_residual**2).mean()

Advantage: No finite difference approximations needed. Network learns exact symbolic solution to PDE.

Training Strategy

Data Points

• Boundary conditions: u(x=0,t), u(x=L,t) for all t
• Initial conditions: u(x,t=0) = u₀(x) for all x
• Collocation points: Random (x,t) pairs to satisfy PDE

Optimization

• Optimizer: Adam (adaptive learning rates)
• Loss weighting: L_data + λ·L_physics
• λ typically ~1.0 (balance both losses)
• Convergence: ~1000-5000 epochs

Results & Validation

Validation Metrics

Compared PINN predictions against analytical solutions and finite difference reference:

• MSE on test domain: ~1e-4 (excellent match to analytical)
• PDE residual: <1e-3 across domain (physics satisfied)
• Boundary conditions: ±0.001 error (near perfect)

Generalization

PINN generalizes to α values not seen during training. Extrapolates beyond training domain, maintaining physical validity. Pure data-driven models would fail.

Interactive Streamlit Dashboard

Live parameter tuning and visualization:

• Adjust initial temperature profile, domain boundaries, thermal diffusivity α
• Train model in real-time with progress tracking
• 3D surface plot: u(x,t) from PINN prediction
• Error heatmap: Difference from analytical solution
• Download predictions and trained model weights

Mathematical Foundation

Fourier Transform Solution

For verification, we compute analytical solution using Fourier series:

u(x,t) = Σ [A_n · cos(nπx/L)] · exp(-α(nπ/L)²t)

PINN learns to approximate this series implicitly through network weights.

References

• Raissi, M., Perdikaris, P., & Karniadakis, G. E. (2019). "Physics-informed neural networks." SIAM Review.
• Karniadakis, G. E., et al. (2021). "Physics-informed machine learning approach for short-term electricity demand forecasting."

Tech Stack

Core ML

• PyTorch (autodiff)
• PyTorch Lightning (training loop)
• NumPy/SciPy (math)

Visualization & Deployment

• Streamlit (interactive app)
• Matplotlib (2D plots)
• Plotly (3D surface)

Future Directions

• Wave Equation: Extend to hyperbolic PDEs (acoustic propagation)

• Burgers' Equation: Nonlinear PDE with shock discontinuities

• Navier-Stokes: Fluid dynamics (3D CFD acceleration)

• Inverse Problems: Learn PDE parameters from observations

• Transfer Learning: Pre-train on simple PDEs, fine-tune on complex systems

Links & Resources

View on GitHub