A generative pre-trained transformer (GPT) is a large language model architecture that generates text by predicting the next token, pre-trained on vast text and then adapted to specific tasks. In visual AI its relevance comes through vision-language models that extend the same recipe to images.
A GPT is a transformer-based model trained to predict the next token in a sequence, learned from enormous amounts of text and then instruction-tuned or aligned for real use. That autoregressive, predict-the-next-token approach is what powers modern chat and text-generation systems. GPT is not itself a computer vision model, but its architecture and its pre-training recipe are the foundation for vision-language models, which pair an image encoder with a GPT-style model so the same approach can describe, answer questions about, and reason over images.
Key takeaways
A GPT is a transformer that generates text by predicting the next token, pre-trained at scale.
It is a language model, not a vision model, but the same recipe extends to multimodal vision-language models.
In visual AI it shows up as vision-language models for captioning, visual question answering, and zero-shot labeling from text prompts.
How it relates to visual AI
Vision-language models: an image encoder plus a GPT-style decoder for captioning and visual question answering.
Open-vocabulary and zero-shot: text prompts drive image tasks with no task-specific training.
Data workflows: describing, tagging, or querying image datasets in natural language.
How it works, and how FiftyOne fits
FiftyOne integrates vision-language models you can prompt to caption, tag, or zero-shot label images, so the GPT lineage shows up in a visual AI pipeline as a labeling and search accelerator rather than as a text tool. See the FiftyOne model zoo and FiftyOne integrations in the docs.
GPT vs related terms
Definitions for GPT, LLM, and VLM
Definitions for GPT, LLM, and VLM
Term
What it is
GPT
A text-generating transformer, pre-trained then adapted
Large language model (LLM)
The broader class of large text models GPT belongs to
Vision language model (VLM)
A model that takes images and text, the GPT recipe extended to vision
Why it matters
For a visual AI team, the useful framing is that GPT's real contribution to vision is the recipe, not the model. Information-gain insight: large-scale self-supervised pre-training plus a promptable interface is exactly what lets vision-language models and SAM-style foundation models label, describe, and segment images from a text prompt with no task-specific training. The lesson transferred to vision even though the text model did not, which is why "prompt it in natural language" is now a labeling strategy and not just a chat feature.
Frequently asked questions
What is a generative pre-trained transformer?
A transformer that generates text by predicting the next token, pre-trained on large text corpora and then adapted to tasks.
Is GPT a computer vision model?
No. It is a language model, but vision-language models extend its architecture and recipe to images.
How does GPT relate to visual AI?
Through vision-language models used for captioning, visual question answering, and prompting image tasks in natural language.
Related terms
Foundation model labeling, Zero-shot labeling, Multimodal annotation, Segment Anything Model (SAM), Annotation