A basic but functional Unity Editor Component for image generation using Stable Diffusion Automatic 1111 webui from within the Unity Editor.
The full guide, FAQ, and troubleshooting notes are in AI-SUPPORT.md. That file is meant to be handed to an AI assistant so answers stay on what this project actually does.
- Download
AI-SUPPORT.md. Open the link and save the file (on GitHub, use Raw, then save the page). - Upload or attach that file in the AI assistant you already use.
- Ask the question, or describe what went wrong. Include your Unity version and the Console text when the problem depends on them.
Tell the assistant to follow that file and not to invent buttons, settings, or WebUI features the file does not describe. If you would rather read a shorter page yourself, start with Getting started, the user guide, troubleshooting, or the FAQ.
Unresolved bugs can be reported at https://github.com/dobrado76/Stable-Diffusion-Unity-Integration/issues . Include the Unity version, whether you cloned the repo or imported the unitypackage, the component name, and the Console error. Remove passwords, private prompts, and personal paths first. This project does not run a support desk and does not promise a reply.
If you like it, feel free to Buy me a Coffee (https://www.buymeacoffee.com/dobrado76)
Cheers!
Simple server configuration with StableDiffusionConfiguration component and SDSettings ScriptableObject assets.
Easy Text-to-Image generation for 3D models:
Straightforward Text-to-Image generation for UI elements:
- Text-to-Image generation with Prompt and Negative prompt for: a) texturing 3D models having a MeshRenderer component with tiling option and generation of Normal/Bump maps, b) UI component having a Image or RawImage component (Canvas, Panel, Image, Button, etc.)
- Image-to-Image generation with Prompt and Negative prompt for: a) UI component having a Image or RawImage component (Canvas, Panel, Image, Button, etc.) b) a MeshRenderer via StableDiffusionImage2Material (no Generate button in the Inspector; see the user guide)
- Editor only components for scene/level design (no runtime dependencies to Stable Diffusion)
- Image generation using any Stable Diffusion models available in the server model folder
- Standard parameters control over image generation (Prompt and Negative Prompt, Sampler, Nb. of Steps, CFG Scale, Image dimension and Seed)
- All images saved in the Assets folder for persistent storage and reference
- Can run from a local Stable Diffusion server, which requires no API key, or from an online surver such as a Runpod Virtual instance.
This Unity Editor tool requires access to a Stable Diffusion Automatic 1111 server. Installation or usage of such server is outside the scope of this documentation. Please refer to the SD A111 WebUI repo: https://github.com/AUTOMATIC1111/stable-diffusion-webui
-
Launch Stable DIffusion WebUI Automatic 1111 with the '--api' argument in the command line define in the "webui-user.bat" command file, in the root folder of your SD server. Without Authentication: set COMMANDLINE_ARGS= --api ; Or with Authentication: set COMMANDLINE_ARGS=--listen --api --api-auth YOUR_USERNAME:YOUR_PASSWORD --cors-allow-origins=*
-
Either, Download the stable-diffusion-unity-integration repository, for example by running
git clone https://github.com/dobrado76/Stable-Diffusion-Unity-Integration.git. Then, Open the stable-diffusion-unity-integration project in Unity. The demo scene isAssets/Stable-Diffusion-Unity-Integration/Scenes/DemoScene.unity.
Or;
-
Create and open a new Unity project for Built-in Pipeline, download and import the stable-diffusion-unity-integration.unitypackage file into the project (https://github.com/dobrado76/Stable-Diffusion-Unity-Integration/blob/main/Stable-Diffusion-Unity-Integration.unitypackage). By starting from a new project and importing the package, none of the demo pre-generated images will be available in the Asset folder but the project should work as-is, since the textures and images are embedded in the Unity project.
-
Open the demo scene
Assets/Stable-Diffusion-Unity-Integration/Scenes/DemoScene.unity. There is a version with Post-Processing (DemoScene.unity; install the Post-Processing package, then re-open the scene) and a version without it (DemoScene-NoPostProcessing.unity). -
Select the StableDiffusionConfiguration component and make sure the selected setting point to the correct URL and Port (default: http://127.0.0.1:7860/).
-
Click on the
List Modelsbutton to get the list of all available models from your local Stable Diffusion server.
- Select any existing GameObject with a MeshRenderer (or that contains at least one child with a MeshRenderer) or an Image (or RawImage) and click Generate from the StableDiffusionText2Material, StableDiffusionText2Image or StableDiffusionImage2Image component to generate a new image using the specified parameters.
- If any error occured, it should be catched by a try/catch and the exception displayed in the Unity Console.
- The StableDiffusionText2Material component requires to be added to a GameObject that has a MeshRenderer (if apply recursively is not used) or have at least one children GameObject (if apply recursively is used).
- The StableDiffusionText2Image component requires to be added to a UI GameObject (in a Canvas hierachy) that has an Image or RawImage component. The older names StableDiffusionMaterial and StableDiffusionImage were renamed in v1.2.4.
Be aware of a few limitations:
- This repo has been testing only in some versions of Unity (2019, 2020, 2021 and Unity 6). It is believed that it has only basic low level Unity dependencies and should work in any versions. The current scripts do not compile on Unity 2019.
FindAnyObjectByTypeis present in 2021.3.18f1 and later, and this project'sProjectVersion.txtis 2021.3.33f1. Seedocs/getting-started.md. - The components for generating materials are designed for the Built-in render pipeline and Universal Render Pipeline (URP) only. It may be easy to get it to work for HDRP but no effort has been done in that regard. Feel free to contribute your changes if you make it work.
To contribute, clone the repository, make your changes, commit and push to your clone, and submit a pull request (PR).
Before submitting a PR, make sure that:
- you do not add multiple unrelated things in same PR.
- your changes do not break anything before submitting.
- you do not submit PRs where you just take existing lines and reformat them without changing what they do.
If you think you found a bug, please first make sure that you are using the latest version of both this repo (as well as the Stable Diffusion repo) and that the bug has not already been fixed. If you are submitting a bug fix, there must be a way for me to reproduce the bug. Please, provide step-by-step instructions on how to reproduce the bug.
Thanks to UnityCoder for a straightforward and functional algorithm to automatically generate a normal map from a texture. - https://github.com/unitycoder/NormalMapFromTexture. Also, thanks to ALBRRT, FOXYTOCIN, PeixuanL for their contributions here. Additional thank you to TeoVibe for adding suport to online cloud Runpod server, as well as a more solid error catching/handling in various situations.
The LICENSE file at the repository root is the MIT License (copyright 2025 Dominic Branchaud). The paragraphs below are the older README description and still say LGPL; they do not match the current LICENSE file. This note is not legal advice.
This repository is under Lesser General Public License (LGPL) v2.1, that you can find in the LICENSE file (https://github.com/dobrado76/Stable-Diffusion-Unity-Integration/blob/72560920786b909e829b567567699f332256ea19/LICENSE).
Mainly, it allows you to:
- use or distribute any artwork generated using this tool without the need to mention this tool, repo or author.
- use, modify, and distribute this code base and other copyrighted work.
- create derivative works to be distributed under a different license than the original.
Requires that the source code for the derative work be made available to others. This does not introduce any limitations on your own Unity projects, since this tool is a Design time, Editor only tool (your game or application is obviously not required to be open source). Requires that any partial, full or modified distribution of this repo mentions the original source (this repo/author).
2025-03-30: v1.3.0 - Merged Pull Request from TeoVibe, thanks for your contribution adding support for online cloud Runpod server.
2023-03-15: v1.2.6 - Added StableDiffusionImage2Material, thanks to PeixuanL.
2023-03-10: v1.2.5 - Merged Pull Request from FOXYTOCIN to add support for Authentication API keys and URP materials.
2023-02-02: v1.2.4 - Merged Pull Request from ALBRRT to add support for Image-2-Image for UI Image and RawImage. To that effect, renamed StableDiffusionImage to StableDiffusionText2Image and StableDiffusionMaterial to StableDiffusionText2Material, and added a new StableDiffusionImage2Image component.
2023-01-31: v1.2.3 - Merged Pull Request from ALBRRT to fix an issue with progress bar when generating an image with StableDiffusionImage. Not perfect but somewhat/sometimes working ok.
2023-01-31: v1.2.2 - Changed the call to Stable Diffusion server to be asynchronous and non-blocking. Also added a Progress Bar for generation progress feedback.
2023-01-30: v1.2.1 - Added some comments in the code and updated the package with new codebase, to include last merge and post-processing settings for the demo scene with post-processing.
2023-01-25: v1.2.0 - Merged Pull Request from ALBRRT to enable Editor assets refresh only when not in Play mode, or returning from Play mode. Also added the display of the GUID for easier reference between the Component and the Image file.
2023-01-24: v1.1.0 - Added handling of the samplers as generation parameter. Manual list in StableDiffusionConfiguration that can be culled to remove samplers that you don't use.
2023-01-23: v1.0.0 - Initial public distribution version 1.0.0 with brief documentation.