Music generation

Music generation models turn a text description, and often lyrics, into audio: full songs with vocals, instrumental tracks, live streams of music or sound effects.

  • Models and architecture
  • Updated

Open music models do four different jobs. Song models such as MiniMax Music 3, YuE2, ACE-Step 1.5 and HeartMuLa take lyrics and a style description and sing them. Instrumental models such as Stable Audio 3 make music without vocals, plus sound effects. Live models such as Magenta RealTime 2 stream audio continuously and react to text, audio or MIDI while it plays. Sound-effect models such as MOSS-SoundEffect make short clips from a sentence. Most work on compressed audio, either as tokens that a language model predicts or as latents that a diffusion model cleans up, and many combine both.

Quality is hard to measure, and the few benchmark tables come from the developers themselves, so listen to samples before you choose. Check the license of the weights, which can be non-commercial or tied to revenue, and note that developers describe their training data to very different degrees. Ask a lawyer about copyright before you sell generated music.

Related terms

  • Diffusion model A diffusion model generates images or video by starting from random noise and removing the noise step by step until a picture emerges.
  • Sampling steps Sampling steps are the number of passes a diffusion model makes to turn noise into an image or video. Fewer steps mean faster results, usually with some loss of detail or variety.
  • Non-commercial license A non-commercial license lets you use a model for research, study and personal projects, but not to make money from it.
  • Revenue cap (revenue threshold) A revenue cap is a license condition that makes a model free below a certain income and requires a separate agreement above it.
  • LoRA LoRA, short for low-rank adaptation, fine-tunes a model by training a small set of added weights instead of changing all of its parameters.

Keep reading

All terms