It was only a matter of time before someone combined Stable Diffusion AI image generator and Minecraft command-line. This is probably one of the best uses of AI image generators I’ve seen so far, essentially using a combination of Minecraft screenshots and text prompts to create highly detailed images each time the user makes changes to their design within the Minecraft game world.
Sean Simon, more commonly known as ThoseSixFaces (or Lozmosis on Reddit (opens in new tab)), posted the above video yesterday showing off a sped-up version of their scripts running live as they messed around in-game.
It works by taking screenshots from a specific angle as the game is played, resizing or cropping them to 512×512 pixels and saving them to a folder. Each frame is generated on “a fixed interval, or whenever the player places/removes a block.” Using a version of InvokeAI (opens in new tab) that Sean modified, the model then goes about “targeting the latest image in the folder and outputting the Stable Diffusion image.” Another script then displays the output image using tkinter (opens in new tab).
Stable Diffusion (opens in new tab) is the main thing translating each frame into gorgeous AI generated images, however, which Sean explains “denoises on a random value between 0.47-0.5.” So although Sean is able to use prompts to push the AI in the preferred direction using image2image (opens in new tab), there’s still a little leeway in how random the generated results can be.
As it stands right now, the process involves running several scripts, though Sean is “keen to compress this all into a single script.”
It’s a fascinating project, and one my art teachers from college would be very excited to see working; they seemed to have an obsession with translating work into different mediums, then translating it back. On that note, one Reddit commenter jokingly suggests a reverse process to Sean, in which the AI would “turn arts into minecrafts.”
Sean responds with, “I’ve started cooking something up,” so it may not be too long before we see Minecraft worlds being generated using AI image prompts. Or indeed Minecraft worlds being translated into images, then back into Minecraft worlds.
Whew, what a world we live in.
Sadly, Sean’s scripts are not yet available to download from Github, but do keep an eye on their Twitter (opens in new tab) for updates.
Now, we’ve seen Stable Diffusion being used in VR (opens in new tab), and the CEO of Discord image bot Midjourney speculates that future consoles will have a “giant AI chip and all the games will be dreams (opens in new tab).” This Minecraft project is yet another addition to the exciting, AI-focussed narrative springing up in the gaming sphere right now.