Design document: Finesse Generative Challenge

Goal

Generative learning model capable of generating clothing designs
Items must be realistic but also feasible to produce
Generation solution must be efficient and scalable to large number of requests

Model Choices

When it comes to text to image generation, I had two primary architectures to choose from Diffusion models and GANs

Metric	Diffusion Models	GANs
Image Quality	Very realistic and coherent images. Captures fine details well.	Lower image quality with more artifacts.
Training Stability	More stable training. No issues like mode collapse.	Can suffer from training instability and mode collapse.
Conditioning	Excellent interpretation of text to generate accurate images.	Good but may miss some text details.
Diversity	Tend to produce more uniform outputs.	Can generate more diverse outputs.
Resolution	Rapidly improving but lower than GANs so far.	Can generate higher resolution images.

Since text conditioning and high quality outputs were our main requirements with stable generation, I chose to go with Diffusion models.

However diffusion model endpoints are easily available via Huggingface making integration easier.

Model endpoints were extracted from huggingface (v-1-5) and stabilityai diffusion pipeline used to produce inference on test prompts

Data

I created a HuggingFace dataset which can be directly loaded to the finetuning script:
- HF Dataset: https://huggingface.co/datasets/sanctia/finesse_image_generation

Prompt Structure:

The current dataset has some useful textual attributes which can be used to create an effective prompt

Prompt currently consists of

A realistic photo of model wearing Title: <PRODUCT_TITLE> Category: <MAIN_CATEGORY> Subcategory: <SUBCATEGORY> Description: <DESCRPIPTION>

[Observations]:
- Although this prompt structure has most of the information, some information is not prominent due to the conversational nature of the description

[Future/Next steps]:

Extract tags from the product json to make clothing details more prominent. Some tags can be in terms of color, fit, style, occasion, material etc. Making sure product descriptions have these tags will ensure better generations.

Some methods to extract tags can be:

Use BERT based classifiers to extract tags from predefined class categories

Use language generation model such as GPT as instruct it to create prompts which capture these tags explicitly

Original json:

"title": "Dani Halter Mini-Dress"
"created_date": "2021-01-06 00:00:00"
"main_category": "Dresses"
"sub_category": "Dress"
"description": "The ultimate cool girl dress. For lighting up in the met gala bathrooms or rolling your eyes at strangers at the grocery store, the Dani Halter will be sure to cause a stir. That icy mint fit will steal all their breaths away "

ChatGPT generated (example)

Title: Dali Halter Dress
Main Category: Dresses
Subcategory: Dress
Color: Icy mint
Occasion: Casual
Fit: Halter
Description: The ultimate cool girl dress, suitable for various occasions, featuring an attention-grabbing icy mint color.

Training Approach (LoRA finetuning)

Using pre-trained model checkpoints for diffusion models especially fashion tuned models give decent results but does not serve our requirement.