Riffusion is gaining traction as a creative AI tool that transforms text into immersive audio clips via spectrograms. But what if you want to upload your own audio to Riffusion for remixing, style transfer, or spectrogram-based editing? That capability isn’t immediately obvious if you’re only familiar with the default text-to-music interface.
In this guide, we’ll explore how to upload your own audio into Riffusion—from working through demo limitations to integrating via API. Expect step-by-step instructions, real user insights, and tips that will help you build on this open-source tool.
Can You Upload Audio in the Web Version?
The Riffusion web demo (e.g., riffusion.com) allows free use of text prompts, but it does not support uploading your own audio. Reddit users confirm this as a known limitation:
“I’ve tried uploading .wav, .ogg and .m4a… but the ‘Analyzing file…’ never ends”
The demo is designed strictly for text-to-spectrogram conversion, not audio import. If you need to process your own recordings, you’ll have to explore developer-level options.
Developer Route: Using the Riffusion Inference API
For advanced users, Riffusion offers an API endpoint for uploading audio through compatible platforms like UseAPI.net. Here’s how it works in practice:
Supported Formats and File Size
Accepts
.mp3,.m4a, and.wavformatsNo strict upper size limit, but files under 50?MB are recommended
Sample CURL Request
bash curl "https://api.useapi.net/v1/riffusion/music/upload-audio/?file_name=myclip" \ -H "Authorization: Bearer YOUR_TOKEN" \ -H "Content-Type: audio/mpeg" \ --data-binary @myclip.mp3
A successful response returns an audio_upload_id, which can then be used with other API endpoints to process or integrate Riffusion workflows.
Visual Interface Integration
Some developer-friendly UIs—such as custom front ends built with Streamlit or Flask—include “Browse” buttons that wrap this API. These apps streamline the upload, convert the uploaded audio file into a spectrogram, and feed it into the Riffusion pipeline.
Running Riffusion Locally with Audio Upload Feature
The most flexible method for uploading your audio is to run Riffusion locally, using its open-source code available on GitHub:
Clone the main repo (e.g.,
riffusion-hobbyorriffusion-app)Install Python, torchaudio, and ffmpeg
Use command-line or UI to invoke audio import
For example:
bash python -m riffusion.cli image-to-audio \ --image my_spectrogram.png \ --audio output_clip.wav
Or, in Python scripts, you can load a .wav with torchaudio, convert it to a spectrogram tensor, and feed it directly into the model for inference. This approach grants full control over your workflow.
Why Upload Audio to Riffusion?
Uploading your own audio extends Riffusion’s creative use cases:
Style transfer and remixes – input your vocal or beat track and convert it into a stylized spectrogram.
Audio augmentation – process loops, drums, or ambient recordings through the AI for texture remix.
Research and development – train or fine-tune using your own dataset in local setups.
Interactive installations – interfaces that accept live audio inputs (mic, stream) and generate real-time Riffusion soundscapes.
These use cases leverage the open-source spectrogram pipeline to go beyond text-based novelty.
Common Roadblocks and Solutions
Problem: Upload hangs or returns no response
Solution: Ensure your code triggers model inference; UI wrappers may miss back-end hooks.
Problem: File not supported
Solution: Convert to WAV or MP3 with ffmpeg before upload.
Problem: High GPU load
Solution: Reduce inference steps or resize spectrogram resolution; run locally on GPUs like RTX 3070 or A10G for real-time results.
FAQ: Uploading Audio to Riffusion
Can I just upload audio in the default web app?
No. Web demo does not support audio upload. It’s text-prompt only.
What formats are accepted by the API?
MP3, M4A, and WAV are supported, with file sizes ideally under 50?MB.
Do I need an API key?
Yes. Services like UseAPI.net require authentication to access the upload endpoint.
Can I remix uploaded audio with AI?
Absolutely. Once uploaded via API or locally, you can treat it as input for generation or interpolation.
Is conversion required before feeding to the model?
Yes. Internal pipelines expect spectrogram inputs, so conversion tools like torchaudio + Griffin-Lim are used.
Do I need coding skills?
Yes, for API or local use. However, some open-source GUIs make it easier without deep coding.
Conclusion: Upload Audio to Riffusion, Unlock Creative Freedom
So, how do you upload audio to Riffusion? The answer depends on your setup:
Default web version: Not possible—text only
API method: Use endpoints like
music/upload-audiowith proper authenticationLocal installation: Best option for full audio input control and creative customization
Uploading your own audio extends Riffusion from a text-generated novelty to a powerful audio manipulation engine. With the open-source code at your fingertips, you can experiment, remix, and evolve the tool to fit your unique creative workflow.
Learn more about AI MUSIC