Generative pre-trained transformer
A generative pre-trained transformer (GPT) is a large language model architecture that generates text by predicting the next token, pre-trained on vast text and then adapted to specific tasks. In visual AI its relevance comes through vision-language models that extend the same recipe to images.
In this article
Building visual or physical AI?
Let's talk.
What is a generative pre-trained transformer?
A GPT is a transformer-based model trained to predict the next token in a sequence, learned from enormous amounts of text and then instruction-tuned or aligned for real use. That autoregressive, predict-the-next-token approach is what powers modern chat and text-generation systems. GPT is not itself a computer vision model, but its architecture and its pre-training recipe are the foundation for vision-language models, which pair an image encoder with a GPT-style model so the same approach can describe, answer questions about, and reason over images.
Key takeaways
- A GPT is a transformer that generates text by predicting the next token, pre-trained at scale.
- It is a language model, not a vision model, but the same recipe extends to multimodal vision-language models.
- In visual AI it shows up as vision-language models for captioning, visual question answering, and zero-shot labeling from text prompts.
How it relates to visual AI
- Vision-language models: an image encoder plus a GPT-style decoder for captioning and visual question answering.
- Open-vocabulary and zero-shot: text prompts drive image tasks with no task-specific training.
- Data workflows: describing, tagging, or querying image datasets in natural language.
How it works, and how FiftyOne fits
FiftyOne integrates vision-language models you can prompt to caption, tag, or zero-shot label images, so the GPT lineage shows up in a visual AI pipeline as a labeling and search accelerator rather than as a text tool. See the FiftyOne model zoo and FiftyOne integrations in the docs.
GPT vs related terms
| Term | What it is |
|---|---|
| GPT | A text-generating transformer, pre-trained then adapted |
| Large language model (LLM) | The broader class of large text models GPT belongs to |
| Vision language model (VLM) | A model that takes images and text, the GPT recipe extended to vision |
Why it matters
For a visual AI team, the useful framing is that GPT's real contribution to vision is the recipe, not the model. Information-gain insight: large-scale self-supervised pre-training plus a promptable interface is exactly what lets vision-language models and SAM-style foundation models label, describe, and segment images from a text prompt with no task-specific training. The lesson transferred to vision even though the text model did not, which is why "prompt it in natural language" is now a labeling strategy and not just a chat feature.
Frequently asked questions
What is a generative pre-trained transformer?
A transformer that generates text by predicting the next token, pre-trained on large text corpora and then adapted to tasks.
Is GPT a computer vision model?
No. It is a language model, but vision-language models extend its architecture and recipe to images.
How does GPT relate to visual AI?
Through vision-language models used for captioning, visual question answering, and prompting image tasks in natural language.
Related terms
Foundation model labeling, Zero-shot labeling, Multimodal annotation, Segment Anything Model (SAM), Annotation.
Try it in FiftyOne
Prompt a vision-language model over your images in the Gemini in FiftyOne tutorial.
Learn more
See how foundation models are reshaping labeling in What SAM 3 Means for Data Annotation.
Jesse Mostipak
SEO & Content
Jesse Mostipak is the SEO and Content Manager at Voxel51, where the work is helping humans find and trust what the brand knows, and teaching the Google knowledge graph and the LLMs answering on their behalf to do the same. That question, how knowledge gets built inside a system, is one Jesse has been chasing for years. Earlier versions of it ran through a New York City high school science classroom, data science and machine learning, and developer relations at Kaggle, Posit (formerly RStudio), and Baseten. The answer doesn't change much depending on whether the learner is a teenager, a software engineer, or a knowledge graph. Jesse holds a Master's in Education from CUNY Hunter College. LinkedIn
See all articles by Jesse Mostipak
Last updated July 1, 2026
Building visual or physical AI?
Let's talk.
© 2026 Voxel51 All Rights Reserved