From Words to Pictures: Text to Image Generation with Stable Diffusion XL
12 min readAug 11, 2023
I had previously used the stable-diffusion model and now with its new version out trying to see what has changed in Stable Diffusion XL. I deployed on AWS with SageMaker. My notebook is is available in github (see text_to_image_stability_xl).
Multi-prompting looks be a new change but it does not work as a I would expect and not much documentation to go around.
TextPrompt(text=”sunny day in a futuristic san francisco skyline in the background”, weight=5),
TextPrompt(text=”clouds”, weight=-1)]
TextPrompt(text=”the person is running around a track”, weight=2),
TextPrompt(text=”the person is holding a flower”, weight=1.5)]
However using my previous prompts the new images are clearly much superior and its just fun to see what comes out! Hands still have more than necessary number of fingers and biases are still similar though.
Here are some more. The prompts are labeled in the image itself.
Recursive
Emotions and Gender
Cultural references and biases
Similar to previous version, the doctor is always a man. Men, women and facial features are generally European. But it does know cultural differences when requested. Looks like there is some filtering to block any requests if the word “child” is used so still no ugly children!
Faces
Emotions and Facial Expressions
“Emotion: Sadness\nFacial Expression: Downturned corners of the mouth, drooping eyelids, and a furrowed brow.\nDescription: Sadness reflects a feeling of sorrow or unhappiness. The facial expression may appear dejected and lack the energy associated with other emotions.”,
“Emotion: Anger\nFacial Expression: Furrowed eyebrows, narrowed eyes, tensed jaw, and tightened lips.\nDescription: Anger is marked by a feeling of hostility or frustration. The facial expression is intense, with the focus often on the eyes and mouth.”,
“Emotion: Surprise\nFacial Expression: Widened eyes, raised eyebrows, and an open mouth.\nDescription: Surprise is characterized by a sudden and unexpected reaction to something. The facial expression conveys astonishment and can be accompanied by a quick inhalation.”,
“Emotion: Fear\nFacial Expression: Wide-open eyes, raised eyebrows, open mouth, and tensed facial muscles.\nDescription: Fear is a response to a perceived threat or danger. The facial expression is marked by heightened alertness and readiness for a fight-or-flight response.”,
“Emotion: Disgust\nFacial Expression: Wrinkled nose, raised upper lip, and a squinting or narrowing of the eyes.\nDescription: Disgust is a reaction to something unpleasant or offensive. The facial expression often reflects a sense of aversion.”,
“Emotion: Contempt\nFacial Expression: One corner of the mouth raised in a sneer or smirk, along with a slight raising of the chin.\nDescription: Contempt signifies a feeling of superiority or disdain for someone or something. The facial expression conveys a sense of arrogance.”,
“Emotion: Joy\nFacial Expression: Bright eyes, wide smile, raised cheeks, and an open and relaxed posture.\nDescription: Joy represents a deeper, more intense form of happiness. The facial expression is exuberant and may include laughter or other expressions of delight.”,
“Emotion: Confusion\nFacial Expression: Furrowed brow, tilted head, and a puzzled or uncertain look.\nDescription: Confusion arises when there is uncertainty or lack of understanding. The facial expression often conveys a sense of searching for answers.”,
“Emotion: Contemplation\nFacial Expression: Chin resting on the hand, raised eyebrows, and a thoughtful gaze.\nDescription: Contemplation reflects deep thinking or introspection. The facial expression indicates a person lost in thought.”
Tangible objects
Intangibles
Famous people and places
Spanish
I tried Hindi but that is not supported. Spanish gave pretty good results.
6 year old’s prompts
I let my 6 year old type in some prompts and you can see where they went!
Fantasy and Fiction
Generated Prompts
With stability XL the results are remarkable and these are simple single text prompts.
image_generation_prompts = [
"Generate a portrait of an elderly man with a kind smile and wise eyes.",
"Create a realistic image of a young woman with curly hair and piercing green eyes.",
"Generate a portrait of a young man with a rugged look and a slight beard.",
"Create an image of a middle-aged woman with short hair and a friendly expression.",
"Generate a portrait of a woman with rosy cheeks and bright eyes, full of wonder and curiosity.",
"Create an image of a person with a serious expression and strong facial features.",
"Generate a portrait of a woman with a unique hairstyle and bold makeup.",
"Create an image of a man with a beard and glasses, looking contemplative.",
"Generate a portrait of a person with piercing blue eyes and a mysterious expression.",
"Create an image of a person with a bright smile and sparkling eyes, radiating happiness and joy."
]
image_generation_prompts = [
"Imagine you're flying on the back of a giant eagle, soaring over mountains and valleys, feeling the wind rushing through your hair.\n",
"Picture yourself walking through a forest of glowing mushrooms, the light casting a surreal glow over everything around you.\n",
"You find yourself in a magical library, where every book holds a world of its own. You can choose any book and step into its pages, living out the story as if it were real.\n",
"You step through a shimmering portal into a world of floating islands and soaring airships, where the sky is an endless expanse of vibrant colors and you can explore to your heart's content.\n",
"You're a guest at a royal ball, where you dance the night away with elegant and exotic creatures from all corners of the realm, the music carrying you away into a world of enchantment.\n",
"You venture deep into a mysterious cavern, where you find a glowing crystal that shows you visions of distant lands and times, revealing secrets of the universe.\n",
"You awaken in a magical garden, filled with strange and wondrous creatures, each with a unique power or gift to share with you.\n",
"You find yourself on a vast plain, surrounded by a circle of standing stones, and you suddenly realize that you have the power to control the elements of nature - the wind, the rain, the sun, and the earth.\n",
"You're on a quest to find a legendary artifact, journeying through dark forests, treacherous mountains, and ancient ruins, facing challenges and meeting allies along the way.\n",
"You discover a hidden portal that takes you to a realm of dreams, where you can explore the subconscious mind and unlock the secrets of the human psyche.\n"
]
image_generation_prompts = [
"Imagine you're standing in the heart of New York City's Times Square, surrounded by the bright lights and buzzing energy of one of the world's most famous destinations. The towering billboards and electronic displays flash advertisements and messages, while street performers and vendors add to the lively atmosphere. You can feel the pulse of the city as people from all over the world rush past you, each on their own mission.",
"Picture yourself walking through the ornate halls of Buckingham Palace, home to the British royal family and a symbol of centuries of history and tradition. You can hear the echo of your footsteps on the marble floors as you take in the opulent furnishings and intricate artwork. You might even catch a glimpse of a member of the royal family, as they move through the palace's private chambers.",
"You find yourself on the beaches of Rio de Janeiro, where you can dance the samba, play soccer on the sand, and soak up the vibrant culture of one of Brazil's most iconic cities. The warm sand and cool ocean breeze invite you to relax and enjoy the sun, while the sounds of music and laughter fill the air. You can taste the delicious local cuisine and join in the lively conversation with the friendly locals.",
"You're sitting in the front row of a concert by your favorite musician, feeling the energy and excitement of the crowd and losing yourself in the music. The lights and sound of the show are intense, and you can feel the bass vibrating through your body. You might even get a chance to meet the artist backstage after the show.",
"You're exploring the winding streets and hidden alleyways of Venice, Italy, admiring the historic architecture and charming canals of this timeless and romantic city. You can smell the aroma of freshly baked bread and hear the sound of church bells ringing in the distance. The gondolas and water taxis glide past you on the shimmering canals, adding to the magical atmosphere.",
"You're at the top of the Eiffel Tower, looking out over the City of Lights and marveling at the breathtaking views of one of the world's most famous landmarks. The city below spreads out before you like a patchwork quilt, with the Seine River winding through the heart of it. You can see the iconic buildings and monuments of Paris from a unique perspective, and feel the breeze as it rushes by you at this lofty height.",
"You find yourself in the heart of Tokyo, surrounded by the neon lights and vibrant culture of one of the world's most bustling and dynamic cities. The streets are packed with people, all moving with purpose and urgency. The food stalls and restaurants offer a dizzying array of options, while the anime shops and karaoke bars are testament to the city's unique blend of old and new.",
"You're at a Hollywood movie premiere, walking the red carpet and rubbing shoulders with some of the biggest names in the entertainment industry. The cameras flash as you strike a pose for the paparazzi, and you can hear the murmur of excited fans gathered behind the velvet ropes. The air is electric with anticipation as you make your way into the theater to watch the latest blockbuster film.",
"You're standing in front of the Great Pyramid of Giza, one of the most famous and awe-inspiring structures in human history, marveling at the scale and craftsmanship of this ancient wonder. The sun beats down on the sandy landscape as you take in the enormity of the pyramid, which has stood for more than 4,500 years. You can feel the weight of history and culture as you stand in its shadow.",
"You're sitting in the stands at the Super Bowl, watching the biggestfootball game of the year with millions of viewers tuning in around the world. The atmosphere is electric as the two teams take to the field, and you can feel the excitement building with every play. The halftime show is a spectacle of music and dance, while the commercials are some of the most talked-about of the year. You can taste the salty snacks and cold drinks as you cheer on your favorite team to victory."
]
My text_to_image_stable_xl notebook is in this Github repo.
If these topics interest you then reach out to me and I will appreciate any feedback. If you would like to work on such problems you will generally find open roles as well! Please refer to LinkedIn.
