Results for "sequence modeling"

Attention where queries/keys/values come from the same sequence, enabling token-to-token interactions.

Injects sequence order into Transformers, since attention alone is permutation-invariant.

Predicts masked tokens in a sequence, enabling bidirectional context; often used for embeddings rather than generation.

Predicting protein 3D structure from sequence.

Learning structure from unlabeled data, such as discovering groups, compressing representations, or modeling data distributions.

Learning from data by constructing “pseudo-labels” (e.g., next-token prediction, masked modeling) without manual annotation.

Penalizes confident wrong predictions heavily; standard for classification and language modeling.

Converting text into discrete units (tokens) for modeling; subword tokenizers balance vocabulary size and coverage.

Training objective where the model predicts the next token given previous tokens (causal modeling).

Modeling interactions with environment.

Modeling environment evolution in latent space.

Modeling chemical systems computationally.