Material Transfer

Material Transfer is based on zero-shot material transfer to an object in an input image given a material exemplar image. For example, given a subject image (e.g., a photo of an bird) and a single material example image (e.g., marble), Material transfer model can transfer the marble material from the material image onto the bird.

This process allows for the realistic transfer of material properties from one image to another, even when the two images have different structures and lighting conditions. It’s a powerful tool in image editing.

Key Components of Material Transfer

Under the hood of Material Transfer is a combination of IP Adapter + ControlNet Depth + Inpainting.

1. An image encoder (IP Adapter) understands unique characteristics of a certain material and captured from a reference image. This image is known as the material image.

2. ControlNet Depth: In this step, the input image, which is the image onto which the material properties will be transferred, is processed to understand its structure and lighting conditions. The structure is understood by estimating the depth of different parts of the image.

3. Inpainting: The final step is where the extracted material properties are applied to the input image. This is done in a way that takes into account the structure and lighting conditions of the input image, resulting in a new image where the material properties from the material image have been realistically transferred.

How to use Material Transfer

1. Subject image: Start by selecting an image that you want to modify. This is the image where the new material will be applied. This should ideally be of a single material type.

2. Material Image: Next, choose an image that represents the type of material you want to transfer. This could be any material like rock, marble, glass, etc. The properties of this material will be extracted and applied to the input image.

3. Prompt: provide a text prompt of what you want the final output to look like. For example, if you’re transferring the properties of marble onto an image of a bird, your prompt might be “marble sculpture.”

4. Material Strength: Adjust this parameter to control the strength of the material transfer. You can choose between ‘strong’ and ‘medium’ options. ‘Strong’ will result in a more pronounced material effect, while ‘medium’ will be more subtle.