A Designer's Tutorial in using Image Based Illumination and High Dynamic Range Imagery
Brock J. Stearn
Department of Industrial, Interior and Visual Communication Design
The Ohio State University
Introduction
This tutorial follows a pipeline using Image Based Illumination and High Dynamic Range Imagery to more effectively composite a CG mock-up into a live action photograph. For example the mock-up camera seen here is composited into a lobby. On the left is the typical way a designer will composite a mock-up into a photograph by lighting the model with basic lighting and then using Photoshop to place it into the scene. On the right is the same camera composited into the same photograph using this pipeline.
Figure 1
Comparison of a result from the exisiting pipeline (left) to the result of this proposed pipeline (right)
|
This pipeline is specific to Alias Maya 5.0 or higher, but could be adapted with some effort into other software packages. Also, there will be references to my MFA thesis paper on this pipeline and related work throughout this tutorial. The entire paper is available here in PDF format. The page references listed in this tutorial are to that document.
For this tutorial there are certain available pieces of software, plug-ins, and scripts that I will be referencing. Below are links to download them and directions as to where to install them.
Note: This tutorial includes instructions for Adobe Photoshop 7.01 and Alias Maya 5.0 (which includes Mental Ray) which are not available for free.
The first step in this process is to find a location to take photographs. This location should ideally be where you could see your CG mock-up existing realistically. Once this location has been found you will need to take some photographic reference of this area. The best way to do this is with a digital camera and a reflective sphere, such as a gazing ball.
Camera Setup
Place the reflective sphere where the computer generated mock-up will later be composited in. Find the desired angle of the camera and mark this spot (pp 36). Now rotate from that spot in 45-50° each direction and these two spots will be where you will take the photographs of the reflective sphere (pp 37-39). The reason we are shooting two different angles is so that they can be composited together to make one whole image without any loss of environmental information, or any objects in the images, such as the camera and camera person (pp 42).
Framing the Sphere
It is important to get as much information from the reflective sphere as possible without allowing the camera, and you, to take up a large part of the sphere. The best way to do this is to fully zoom in on the reflective sphere and then manually walk the camera towards the sphere so that the sphere takes up 30-50% of the image (pp 37-38).
Figure 2
|
|
Photograph showing Sphere taking up between 30-50% of the image
|
Figure 3
 
|
|
The environment in which the reflective sphere is placed
|
Range of Images
In order to use High Dynamic Range Imagery (HDRI) we want to take a full exposure range of images of the sphere. So adjust the camera's shutter speed setting and take one photo at each setting from the fastest shutter speed, such as 1/2000, to the slowest, such as 4. All of these exposure images will be later compiled into one image that more accurately captures all of the lighting information of the environment (pp 20). Below is a cropped example of photographs of the reflective sphere taken through a range of exposures.
|
Figure 4
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1/2000
|
1/1000
|
1/500
|
1/250
|
1/125
|
1/60
|
1/30
|
1/15
|
1/8
|
1/4
|
1/2
|
1
|
2
|
4
|
The full exposure range of photographs of the reflective sphere in the lobby
|
Matching the Angles
As both angles' range of images are going to be compiled together it is very important that each angle's images were taken from the same distance from the sphere with the same settings. A very simple way to do this it to tie a string to the base of the sphere. Pull this string back to where your first angle is and when it reaches the base of the tripod either mark or cut the string there. When you are ready to shoot your second angle use the string to pivot around the base and set the tripod at the same point on the string at the other angle. Make sure you take a few photos at the center line which you can use later for compositing (pp 36).
Figure 5
|
|
Illustration of Shooting two angles while pivoting from a center line with a string tied to the base of the Gazing Ball
|
Now that all the photographs are taken we need to assemble them so that we can use them in our 3D and Rendering software. To do this we need two pieces of software; Adobe Photoshop (or something similar such as Gimp) and HDRShop available free for noncommercial use.
Cropping and Masking the images
We only need the sphere for our information so we need to crop the rest out and mask everything but the sphere. To do this drag a circle marquee around the sphere in the photograph and select the inverse and fill that in with total black. All of the photographs should have the sphere in an identical position in the image. Given this, it is possible to simply copy the black filled area to each image file. Once the sphere is cropped out in every image, save the images in a name you will recognize, i.e. angle1_1_2000.jpg, in JPEG format.
Figure 6
|
|
Image after sphere isolated from background. Blue lines indicate area to be cropped
|
Converting from Angular to Latitude-Longitude
A perfect sphere's image can be transformed into several other formats since we know the equations of a perfect sphere. The most convenient way to do this is in HDRShop. We are going to convert angle one's images to Latitude-Longitude, or planar, format directly. The other set we are going to do the same process for angle 2 but offset by 90° so that we can composite the two together (pp 41).
For Angle 2:
- Import your Angle 2 images in HDRShop
- For each Angle 2 image click Image - Panorama - Panoramic Transformations
Figure 7
Panoramic Transformation in HDRShop
|
- Leave format as mirrored ball, change destination format to latitude/longitude
- In the 3d rotation section, click Arbitrary Rotation and then click Settings
- Set Y axis to 90, leave the others at 0
- Press OK on both windows
|
Figure 8
Your settings should look like those below:
|
|
|
|
Panoramic Transformation Settings for Angle 2
|
Figure 9
|
|
Arbitrary Rotation Settings for Angle 2
|
Compositing the two sets of images together.
This next step is perhaps the longest process but it is necessary to create a very nice lightprobe. I used Adobe Photoshop for this process but any similar program that supports multiple layers and layer locking should work.
Bring each exposure of angle 1 into Photoshop and copy each image into a new Photoshop file, labeling the layers accordingly. Repeat the same process for Angle 2 but places those images in a seperate Photoshop file. Bring in your complete image range for Angle 1 into a Photoshop file with a layer for each step in the range. Repeat the same for Angle 2.
Now look at the areas of Angle 1 that need to be removed. This is usually the camera, cameraman, whatever is holding up the reflective sphere, and any black areas (which is where you didn't capture any information in the photographs) You need to find those areas in Angle 2 and copy them to Angle 1 and transform them into the right location and rotation (Figure 10). The reason for doing this in Photoshop with multiple layers is that you can lock your layers together after you paste in each part. This allows everything you do to one layer to affect all the other linked layers. Below are the angles shot from the steps above and the final image composited together. Since this image will be just used for illumination purposes the images do not have to look precisely accurate as long as the local area of colors is still intact. For these images below you will notice the indoor window's edge in the middle of the image doesn't match properly but it will be enough for a IBI file. Once all the angles have been combined to create a better image, save out each exposure into its own JPEG file and once again name it accordingly (pp 40).
Figure 10
|
|
Angle 1 (top) and Angle 2 (bottom) after being converted to Lat/Long. Boxes indicate areas used from Angle 2 to fix and complete Angle 1.
|
|
Figure 11
|
|
Lat/Long image after compositing. There are still some minor differences in the corrected areas but since this will get blurred at rendering it isn't critical.
|
Creating a High Dynamic Range Image (HDRI)
The best application I have found for creating HDRI is HDRShop, but there are other utilities available such as MkHDR a command line utility, and HDRIE and Photogenics HDR which are both commercial applications with some minor image editing capabilities. For our purposes we will be continuing to use HDRShop (pp 41-42).
Creating an HDRI
- Launch HDRShop
- From the Create menu choose Assemble HDR from Image Sequence
- Click Load Images and load in your newly created range of fixed JPEG (.jpg) images.
- Your images should automatically be sorted correctly, if not then manually adjust their order so that the exposures are in the correct order.
- Click the Calculate button so it automatically adjusts the exposure difference
- Once that is complete click Generate Image and you will have a HDRI
- If your new HDR image seems dark then use the + or - to find what should be a new middle ground for your exposure range. Once this is found go to Image - Pixels - Scale to current Exposure and the HDR image will be adjusted.
- If needed you can now convert your image to the type of map you need by going to Image - Panorama - Panoramic Transformations and select the appropriate pull down menus. It is possible to create a Latitude/Longitude, Cubic, and Angular maps (Figure 12). For this tutorial we will be using a Latitude/Longitude map so create one of those.
- Save a Latitude/Longitude HDRI in HDR format and a compressed HDR Iin JPEG format.
|
Figure 12
|
 |
 |
 |
|
Latitude/Longitude Map of Composited image
|
Cubic Map of Composited image |
Angular Map of Composited image
|
|
How Image Based Illumination works
Principles
The basic principle behind Image Based Illumination is that a computer generated model is surrounded by another piece of geometry (i.e. sphere, cube), with an image mapped onto that geometry. To calculate the lighting of a particular point of the computer genereated model, IBI takes extends the rays, usually based on the vector of the normal (N), until it intersects with the surrounding surface. It then samples the color at that particular point on that surrounding surface's mapped image and uses that for the lighting color of the point where the vector started. Below is an example of this occuring with several rays.
Figure 13
|
|
Illustration of the principle behind Image Based Illumination
|
In Maya and Mental Ray
For Mental Ray, which comes bundled in with Maya 5.0, IBI is implemented through creating some surrounding geometry and using the Mental Ray Final Gathering technique. This is the most straight forward method and what will be used in this tutorial. There are some scripts I have written or modified for this tutorial which have been written for Maya 5.0 but should be useable in any later versions as long as Mental Ray is installed and the plug-in is loaded as shown below. The plug-in can be loaded by going through Windows -Settings - Plug-in Manager and checking load box for the Mayatomr.mll plugin.
Figure 14
|
|
Mental Ray Plug-in
|
Setting Up for IBI
More work in HDRShop
There are two more steps in HDRShop that we must complete before moving onto Maya. The first is to open your compiled HDR image and set the exposure to 0 by pressing 0. If the image appears dark this is probably due to a Gamma difference from the camera to the application. We will solve this through a script in Maya but it is important to note whether the image is bright or dark at 0 exposure. The second step is to use the Lightgen plug-in for HDRShop on our HDR image. Make sure the plug-in is in the HDRShop plug-in directory (pp 44-45).
Lightgen
- Launch HDRShop
- Open your composited HDR image
- Blur the image by going to Image - Filter - Guassian Blur
- The image will need to be shrinked for Lightgen so from the Image menu choose Arbitrary Resize
Figure 15
Arbitrary Resize in HDRShop
|
- Enter 128 and 64 for the new size
Figure 16
New Image Resize numbers for Lightgen
|
- Once the image is resized go to the Plugins menu choose Lightgen
Figure 17
Launching the Lightgen plugin
|
- Enter an Output filename (Browse to your desired location) and set the rest of the settings to the image below
Figure 18
Lightgen GUI plugin in HDRShop
|
- Click Execute and Lightgen will think for a short bit as it is creating the output
- If the plug-in was successful you should receive the following message
Figure 19
Successful output in Lightgen
|
- The final result should look like a bunch of colored dots in a black field
Figure 20
Visual output from Lightgen
|
- If you receive the following error it is most likely you forgot to resize your image before launching Lightgen
Figure 21
Lightgen plugin error in HDRshop
|
- Lightgen is also useable through the command line. Type Lightgen.exe to list the different flags
Making a MAP image
Mental Ray's native image file format is MAP. We are going to convert our HDR image to MAP to speed up render times in Mental Ray and to eliminate the Range Clamping that Maya does with HDR images.
hdr2map
- Open a command shell
- Navigate to where your HDR file is located
- Use my hdr2map batch file to convert your map
- Useage: hdr2map filename.hdr filename.map
- A MAP file will be created in that directory for you to use
Working in Maya
Basic Workflow
There are four basic steps in this pipeline for using Image Based Illumination and High Dynamic Range Imagery. The first is using the HDR image to provide the basic low level lighting, such as ambience and incandescence. If the object is reflective it can also be mapped into the specular channels. The second step is to use the HDR image to derive actual lights in Maya and to set them up so that they are convincing. The third step is specific to any special effects or elements the model may have. This may include Caustics, Fog, Sub-Surface Scattering and so on. The final step is to "dirty up" the image so it doesn't look so real and can blend back in with the live action image. The two scripts available in this tutorial greatly speed up this process. It should be noted that throughout this process it will be neccessary to move back and forth between the different stages to produce a great result.
Installing Scripts
Both scripts should be downloaded to your Maya's script directory. I recommend making shelf buttons for both scripts so that they can be easily accessed. In Maya open the Script editor and inside the script editor Open the brock_lightgen.mel script. The contents of the script will be displayed in the Script Editor window. Highlight the all of the text and drag, using the middle mouse button, to the Shelf and release. A new icon will be created that will launch Brock's Lightgen Control script. For Brock's HDRSetup Control script type brock_hdrsetup.mel into the command line and drag that text to the Shelf to create a new icon.
Figure 22
|
|
Placing brock_lightgenconrol.mel into the Shelf
|
Figure 23
|
|
Command Line entering of brock_hdrsetup.mel
|
Step 1: Using just HDR through IBI for general lighting
- Launch Maya
- Open a blank scene with just your model (or import your model into a new scene) and place the model as close to 0,0,0 as possible
- Load Brock's HDRSetup script
- A Create HDRSetup window will appear. Size the Sphere so that it will completely surround your object but not so that it is so far away. The closer it is the less the render time can be with Final Gathering. We also want a Full Sphere for our setup.
Figure 24
IBI Sphere setup
|
- After the Sphere is created you will be prompted with a new HDRI Setup menu
- Click the Browse button and locate your MAP file. I recommend placing this MAP file in your sourceimages directory of the active project in case you are using a distributed renderer.
Figure 25
Loading a MAP image
|
- After the file has been loaded you can now start affecting the MAP image through the HDRI Setup Controls panel
Figure 26
Brock's HDRI Setup Script
|
- Remember earlier when I mentioned to jot down what the gamma is of your image/camera. Enter that number in the Gamma Correct boxes. If needed you can skew the gamma to one channel or another since the correction is broken down to Red, Green, and Blue.
- The Expose Increaser and Expose Decreaser sections allow you to fake the exposure difference in the HDR image. This normally doesn't need to be touched unless you are trying to give a brighter or darker look then the original image.
- Reflection Energy controls how much the surrounding IBI geometry will continue to reflect back the image, the default value is usually fine.
- Before rendering the image should be blurred slightly since we are just using this image to generate the fill light for the object. In the Blur slider, a setting of1 results in no effect.
- If you want to resize the IBI sphere, use the Radius slider
- If you render now, you should receive a blank scene. This is because we need to setup Mental Ray accordingly.
- Click the Initialize Mental Ray Setup button
- Open up the Render Globals and click on the Mental Ray tab
- The Final Gather tab should already be opened
Figure 27
Final Gather settings in Mental Ray
|
- These values have just been initialized through the HDR Setup script. If you render now you will actually see your model being lit. However, the Max Radius needs to be tweaked to be just a bit larger then the radius of your surrounding sphere. The Final Gather Rays can also be increased to produce a better result, but that will also increase the render time.
- Render the image and continue to adjust these various settings until you have a good amount of fill lighting such as the image below
- In order to provide a visual example of where the placement of the image on the surrounding image is the MAP file must be replaced with a JPEG file. Launch the HDRSetup Control script again and open the JPEG file. Before rendering, the MAP file should be loaded back in.
|
Figure 28
|
|
|
|
First step in Maya (left) and the resulting rendered image (right)
|
Step 2: Using the HDR image to generate lights to create lights
At this stage we will be deriving actual Maya lights from the HDRI image through the use of the exported Lightgen file that was created earlier (pp 56).
- Load Brock's Lightgen Control script
- This will bring up a panel which allows you to import your lightgen outputted MEL file
Figure 29
Brock's Lightgen Control at load up
|
- From the File Menu choose Lightgen Script Open and open the MEL file you created in Lightgen.
Figure 30
Open the lightgen script
|
- After the script is opened you will now be able to control the directional lights created from the script.
Figure 31
Brock's Lightgen Control after MEL script loaded in
|
- It is also possible to adjust the Light Hue, Saturation, and Value for all the lights.
- Shadows may be enabled from the Upper, Lower, or all of the Lights in Dmap or Raytrace
- Once the lights are looking good it is important to break the Intensity so that each light may be tweaked individualy. Without breaking the intensity the values are locked to the dimmer
Figure 32
Locked intensities
|
- It is also possible to break all the other connections to the lights so that the shadows may be tweaked individually too.
Figure 33
Breaking connections
|
- Continue to modify the lights so that they provide good lighting for the model as shown below.
|
Figure 34
|
|
|
|
Second step in Maya (left) and the resulting rendered image (right)
|
Step 3: Creating scene specific elements
For some CG mock-ups there may be "special" lighting situations that need to be created. Examples include caustics and fog. If this step is not required for your CG mock-up, you may skip this step (pp 57).
- The Brock's Lightgen Control script also allows the input of an additional Lightgen MEL file
Figure 35
Generating additional lights
|
- Set the percentage first of what amount of lights to keep then click Generate and browse to the appropriate file.
- Once the top percentage of the lights have been selected then enable light specific items for those lights
- Below is an example image of an object that would require special lighting. On the left is the Maya setup and on the right is the image rendered in Maya with just a caustic pass.
|
Figure 36
|
|
|
|
Third step in Maya (left) and the resulting rendered image (right)
|
Advanced Use: Matching a specific angle and creating shadows
In order to make the final result look as convincing as possible there are two additional advanced used stages that can help blend a CG mock-up into the live action photograph.
- Any camera in Maya can have an image plane attached to it. By mapping in the desired background image into this image plane it is possible to interactively place the camera an image. This is valuable for making sure the camera's angle matches that of the real camera.
Figure 37
A scene in Maya with an the background image plane attached to the camera
|
- For shadows you need to create a ground plane that matches the one in the photograph. By renderning in seperate layers we can use the difference of the ground plane with the ground plane with shadows to extract just the shadow information. This can be done easily using the Difference layer in Photoshop.
Figure 38
A scene in Maya with an the background image plane attached to the camera and a ground plane to reveive shadows
|
By using these advanced steps you are able to create an even more realistic looking composite. Keep in mind at this point the lights have not been manipulated in position or placement at all from the default.
Figure 39
Rendered and composited image with all the stages of this pipeline plus shadows
|
Step 4: Dirtying up the image
- At this point you should be able to produce a pretty convincing high quality render. However, the image will still need to be composited into the background and "dirtied up" to make it look convincing. Above is how the image looks with all of the elements combined before dirtying up the image.
- Launch Photoshop and open your image(s)
- Blur and Noise work well for dirtying up the image
- Render out as much as possible in layers so that it can be composited separately
- Below is an example of this pipeline with the cg mock-up camera composited on top of the background image.
Figure 40
Rendered and composited image with all the stages of this pipeline plus shadows plus noise and blur
|
As with any lighting setup there needs to be constant tweaking in order to produce the best results. It should be clear that using just IBI alone is not a very effective solution, but by stepping though this pipeline it is possible to generate an accurate rendered image from using the HDRI file as the base for all of the different lighting stages.
Figure 41
Below are some examples of the camera being lit through different lightprobes (left) that are HDR images in angular format. The lighting is just using the first stage of this pipeline to visualize the different lighting aspects possible
|
|
|
