The Well: A 15TB Physics Simulation Dataset That Solves Physical-AI’s Data Problem

In today’s AI and large-model race, what really separates teams is data quality, not model tricks — the model technology is largely comparable. The same is true for scientific machine learning: the painful part is usually swapping in a better dataset, not a better model. Fluid dynamics, plasma, acoustics, and biological systems each come with their own grids, variable names, and time steps, so training across datasets means a lot of cleaning and adaptation up front. Polymathic AI’s open-source project The Well tackles exactly this by organizing numerical simulations from 16 physical systems into a single trainable, comparable dataset.

Not a model, but a “data well”

The Well collects about 15TB of spatio-temporal physics simulation data covering fluid dynamics, acoustic scattering, magnetohydrodynamics, supernova evolution, active matter, and biological systems. What it preserves is how the physical fields evolve over time — not just a final result image. It can serve as a single-domain data source or as a unified benchmark for scientific machine learning. The point is not to claim a model has “solved physics,” but to let different models finally be compared under consistent data conditions.

Getting different physical systems to speak the same data language

Different equations describe very different phenomena, yet after numerical simulation they tend to collapse into the same structure: many trajectories, many time steps, one or more spatial dimensions, and several physical quantities. The Well fixes that common shape: data is stored as HDF5 with companion YAML metadata; scalar, vector, and tensor fields are kept separately; and coordinates, boundary conditions, and units share a unified description. For model builders this means no more guessing array dimensions and field meanings for every new dataset. The project uses a unified grid with equal-interval time sampling and pre-split train/validation/test sets — it does not erase the differences between problems, but it consolidates the common read, slice, and batch logic.

Beyond the data: a training entry point you can plug into directly

The Well does not just drop a pile of big files. It ships a Python package and a WellDataset that plug straight into PyTorch’s DataLoader; researchers simply pick a dataset and a split. Data can be downloaded locally or streamed via Hugging Face — the latter is great for exploration, while the docs explicitly recommend local data for real large-model training to avoid a constant remote-read bottleneck.

A complete benchmark

The project ships baselines — FNO, TFNO, U-Net, and ConvNeXt U-Net — with their training scripts, targeting the classic forward-prediction task: given a short history of states, predict the next time snapshot. The baselines are deliberately restrained: reproducible on a single NVIDIA H100 within a 12-hour budget, not tuned to maximize scores on every dataset. That makes them a better common starting line for new methods — confirm the data, task, and compute budget first, then argue about whether your model is actually better.

Start from a single dataset

It installs on Python 3.10+. You do not need to download everything at once — pick one domain and one split to begin. The full collection is about 15TB, while individual datasets range from roughly 6.9GB to 5.1TB. Validate your pipeline on a small slice first, then decide whether to scale up to local, large-scale training.

Who should care

If you work on PDE surrogate models, physical-field prediction, scientific foundation models, or want to test whether an architecture generalizes across different physical systems, The Well is worth adopting as your data and benchmark entry point. It may not help general-purpose AI directly, but for scientific machine learning it solves a plain yet critical problem: turning hard-to-reuse simulation data into material that can be trained on and discussed together.

GitHub: PolymathicAI/the_well