Build a Co-Producer That Sounds Like You: Setting Up Co-Produce AI

📚 This is Part 1 of a 4-part series. Part 1 (this post) covers setup and architecture. Part 2 walks the full creative pipeline — organize → train → generate → package. Part 3 is the reference layer: costs, engines, serverless, the SaaS backend, and scaling it into a product.

Most AI beat tools hand you the same model everyone else is using. Co-Produce AI takes the opposite bet: your crate is the moat. This series is a developer-facing walkthrough of the toolkit, drawn straight from the repo, with every script and command you need to run it yourself.

1. What is Co-Produce AI?

What it is. A complete, self-hosted music-production AI suite built around one idea: your catalog is the moat. Instead of a generic model, you fine-tune on your own sounds and lyrics so the output sounds like you. It spans the whole journey — cleaning and labeling a messy sample library, analyzing and tagging every file, fine-tuning open audio models, generating one-shots/loops/beats/full songs, remixing across genres, writing lyrics in your voice, rendering through your real VST plugins, and shipping provenance-verified sample packs.

Who it's for. Producers and small AI-audio services who want owned, rights-clean, genre-deep output — not the homogenized sound of shared models. It covers hip-hop, rock/metal, dubstep, and DnB from raw crate to finished, rights-traced product.

The end-to-end demo from the repo says it best — a folder of beats becomes a trained model and a finished pack:

$ python scripts/prepare_dataset.py --input raw_beats --output dataset --name-contains _instrumental
Done. 1500 source files processed.  Log: dataset/prepare_log.txt

$ python scripts/validate_dataset.py --dataset dataset
Files: 3120   Total audio: 9.7 h   Dataset is ready for training.

$ # ... train on a pod ... then:
$ python scripts/build_pack.py --input processed --pack-name "Vol 1" --out packs
Pack built: packs/Vol1  (200 samples)

2. How it all fits together

Every box in the system is a script — and a tab in the dashboard. You can run the entire chain or any single step, because steps pass data to each other through small sidecar files (.caption.json, .tags.json, .genius.json, .caption.txt) that sit next to each audio file. That sidecar design is what makes the toolkit composable.

          YOUR RAW MATERIAL                  YOUR MODELS              OUTPUT
  library ─ organize ─┐                  ┌──────────────────┐   ┌──────────────────┐
  songs ─ remove_vocals ─ raw_beats ─┐   │ Stable Audio 3   │   │ one-shots / loops│
  lyrics ────────────────────────────┘   │ LoRA (cloud pod) │   │ beats / songs    │
  your lyrics ─ lyric_analyze ─ lyric_generate ─ lyric_to_beat ─┘   remix / a2a / beat_builder
  your VSTs ─ plugin_scan ─ vst_instrument / vst_chain ─────────┘   ACE Studio vocals

  deep_listen ─ auto_tag ─ genius_lookup ─ build_captions ─ prepare ─ validate
        ─▶ TRAIN ─▶ generate ─▶ postprocess ─▶ build_pack ─▶ provenance ─▶ .zip

Read the flow left to right: raw material on the left (your library, songs, lyrics, plugins), your trained models in the middle, finished output on the right. The analysis row — deep_listen → auto_tag → genius_lookup → build_captions — is what turns raw audio into rich training captions before prepare/validate hand off to training.

Each feature section in the toolkit follows the same shape: a plain-English What it is, a Demo, the Setup & run steps, and Optional / good-to-have extras. Anything that needs a GPU shows the cloud pod path first.

3. Quick start

Clone, install, and launch the web UI:

git clone https://github.com/pjcampbe11/Co-Produce-AI.git
cd Co-Produce-AI
pip install -r requirements.txt
python dashboard.py          # web UI for everything, or use the CLIs below

The fastest path to an actual result needs no training at all — point the beat builder at an organized sample folder and it sequences a finished beat from your own samples:

python scripts/beat_builder.py --library "F:/SoundBankAI" --style boom_bap --bpm 90 --bars 4 --count 4 --out beats

4. Install & setup

Python 3.11 is required. 3.9 is no longer supported — thinc/spaCy and parts of the audio stack dropped it. Always make a virtual environment first so the toolkit's packages stay isolated from system Python:

python -m venv .venv
.venv\Scripts\Activate.ps1        # macOS/Linux: source .venv/bin/activate
pip install -r requirements.txt

Optional features pull extra packages — each pipeline section lists its own, and the requirements appendix (covered in Part 3) breaks down every line of requirements.txt: what it is and which feature needs it.

ffmpeg is needed for MP3/M4A decoding (used by yt-dlp and the librosa fallback):

winget install ffmpeg

Route big model downloads to another drive so the model caches don't fill your C: drive. Run cloud/use_F_drive.ps1, or set these once in an Admin PowerShell:

[Environment]::SetEnvironmentVariable("HF_HOME","F:\ai_cache\huggingface","Machine")
[Environment]::SetEnvironmentVariable("TORCH_HOME","F:\ai_cache\torch","Machine")
[Environment]::SetEnvironmentVariable("PANNS_DATA_DIR","F:\ai_cache\panns_data","Machine")
[Environment]::SetEnvironmentVariable("AUDIO_SEPARATOR_MODELS","F:\ai_cache\audio_separator","Machine")

Optional / good-to-have: a HuggingFace account (gated models need you to accept their terms and run hf auth login).

5. Cloud GPU pod (the default for everything GPU)

What it is. Training and large-model generation run on a rented GPU pod, not your local card — it's faster, cheaper than it sounds, and avoids dependency pain. This is the default for every GPU step in the toolkit; a local GPU is only a fallback.

The loop is: spin up a pod, run a step, pull the results back to your PC.

# on the pod (RTX 4090, /workspace volume):
$ bash /workspace/toolkit/cloud/sa3_setup.sh
Ready. Train a LoRA with:  uv run python scripts/train_lora.py --model medium-base ...

$ runpodctl send /workspace/lora_beats/lora_step2500.safetensors
Code is: 8338-galileo-...   # receive on your PC with: runpodctl receive <code>

Pick a pod at or above these minimums:

CPU cores and RAM are your choice — more cores speed up data prep and audio I/O; 8+ vCPU / 32 GB is comfortable. You provide the account and payment — the toolkit can't rent it for you.

Setup (RunPod example): deploy a PyTorch 2.x / CUDA 12 template, RTX 4090+, with a persistent volume mounted at /workspace sized for your data (input + output; e.g. 80 GB). Then:

bash /workspace/toolkit/cloud/sa3_setup.sh        # Stable Audio 3 + LoRA
# or cloud/runpod_setup.sh for the SAO full-fine-tune path

Move data with runpodctl send/receive (peer-to-peer) or rclone. Terminate the pod when done — it bills per second.

Optional / good-to-have: size the volume generously (a 2-min, multi-stem render can need 25 GB of scratch); batch several GPU steps (train + tag + generate) in one session before terminating; route the HF cache to the persistent volume.

6. Legal & licensing

Read this before commercializing. This is not legal advice — consult an IP attorney.

• Training data: only train on audio you own or have explicit ML-training rights to. Owning a sample pack or a record does not grant the right to train a generative model on it and sell the output — that's a separate, unsettled rights question. Safest sources: your own productions, libraries explicitly cleared for AI/ML, public-domain/CC, or cleared-sample services.
• Base-model license: Stable Audio (Community License) is free for commercial use under US $1M annual revenue; enterprise license above that. HeartMuLa is Apache-2.0 (no revenue cap) — cleaner footing for vocal songs. ACE Studio vocals are yours per its license.
• Provenance is your friend: provenance.py records training sources, run id, generation seeds, and per-file hashes into a certificate — evidence you sourced responsibly. It's evidence, not a license; the underlying rights still have to exist.
• Lossless ≠ rights: a WAV ripped from YouTube is as infringing as the MP3. Format is sonic; acquisition/clearance is legal.