Multimodal model
A multimodal model can handle more than one type of data, such as text, images, audio or video, as input, as output, or both.
Vision-language models read images together with text. Omni models may also take audio and video, and image and video generators produce pictures or clips from text. Every model page on this site lists its input and output types, for example text and image in, text out.
Multimodal ability often arrives as extra files: a vision encoder next to the language model, or a separate text encoder for an image generator. Check the download size on the model page, which covers every component needed to run the model.
Related terms
- Diffusion model A diffusion model generates images or video by starting from random noise and removing the noise step by step until a picture emerges.
- Embedding model An embedding model turns text, images or other data into a list of numbers that captures its meaning, so that similar items end up close together.