CMSC 15100 Lab 4

In this lab, we'll use what you've learned in class to this point to program an application that makes anaglyphs, images that appear three-dimensional when viewed through red/blue glasses. Unlike the labs we've done before, this lab brings together everything you've learned in class for the purpose of writing a single large program.

Part 1 An anaglyph is a pseudo-3D image that works by superimposing two flat images: an image representing what the left eye should see, and another representing what the right eye should see. If the left eye sees just the left image and the right eye sees just the right image, your brain will interpret the difference as being introduced by depth.

To superimpose the two images, anaglyphs employ a clever trick: if you put a filter over your left eye that lets only red light through and a filter over your right eye that does not let red light through at all, then when presented with an image your left eye will only see the red component of that image and your right eye will see only the blue and green components. So, given a left-eye image and a right-eye image for the same scene, you can combine them by making an image in which each pixel in the output comprises the red component of the left image and the blue and green components from the right image.

1a. Write a function combine-colors that takes two colors, one representing a pixel from a left-image and one representing the corresponding pixel from a right-image, and produces the color for the same pixel in the output image. For instance,
(combine-colors (make-color 255 13 160) (make-color 77 91 234))
should be (make-color 255 91 234).

1b. Using combine-colors, write combine-images, which takes two images of the same size and produces a combined image.

NASA's Mars Spirit rover uses pairs of cameras slightly offset from each other so that scientists can calculate the distances of various Martian objects, and NASA makes these images available to the public. Here are some picture pairs that come from that source:

(these all come from the NASA Spirit rover site).

Test your function on these smaller images:

Your result should look like:

Once your function works for the smaller images, try it on one of the larger ones. Once you have a larger image working, show it to me.

combine-images is great for making anaglyphs if you've already got a pair of suitable flat images, but what if you want to generate 3D scenes from scratch in the same way you can make flat images from scratch with circle, rectangle, and so on? In this part, we'll develop a program that does that.

To start, we need to decide how we'll represent these 3D scenes. Let's use this definition:

; a scene is either 
;   - empty,  or
;   - (cons slice scene)
; INVARIANT: within a scene all slices are sorted in ascending order of depth.
;
; a slice is (make-slice number image)
(define-struct slice (depth img))

For now, we'll interpret a slice as the image it contains at a distance that would require us to shift that image depth pixels to the right in the left eye and depth pixels to the left in the right eye to percieve it at the intended depth. The farther left an image is shifted in your left eye relative to where it appears to your right eye, they deeper it appears to be in the scene, and objects that appear in the same place to your left and right eyes are directly in focus. (This is a pretty strange way of representing depth since the numbers you type in don't directly correspond to distances, but it makes it easier to program, so it's good enough for a first draft.) A scene, in turn, will be the image you'd get by arranging all the slices so you were looking at them straight-on, with their pinholes all aligned (recall that every image has a pinhole associated with it and that when you overlay images with the overlay function their pinholes line up in the resulting image).

We can write a series of functions for scenes and slices that we can use to render them as anaglyphs.

Part 2 The first thing we'll need to know to render a scene to an anaglyph is how big our output image will need to be. To do that, write four functions: left, right, top, and bottom, each of which take a scene and produce a number. These functions should return the maximum number of pixels to the left, right, top, or bottom of the pinhole any slice's image extends. Remember to take into account each slice's depth when implementing left and right.

Part 3 Use those four functions to implement a function background that takes a scene and produces a solid white rectangle big enough to serve as the background for the entire scene when viewed either from the left eye or the right eye. Keep in mind that the result image must have enough pixels to the left and right of the pinhole, based on the left and right functions. Be especially careful to consider the case where left and right aren't the same.

Part 4 Write two functions, scene->left-image and scene->right-image, that take a scene and an image and produce an image representing how the left and right eyes, respectively, would view the scene. Assume that the given background image is big enough to serve as a background for the left and right images that your functions will produce. Again, remember to take each slice's depth into account when writing these functions.

Part 5 Write a function scene->image that uses the functions you've written so far to convert a scene to an anaglyph representing it.

Once you've finished part 5, make up some scenes and try your function out with them. Here's one:

(define t (text "Buttons!" 16 (make-color 250 250 250)))
(define buttons!-scene
  (list
    (make-slice 0 (move-pinhole t 
                                (- (/ (image-width t) 2) (pinhole-x t))
                                (- (/ (image-height t) 2) (pinhole-y t))))
    (make-slice 2 (move-pinhole (circle 10 'solid (make-color 240 240 240)) -40 -40))
    (make-slice 2 (move-pinhole (circle 10 'solid (make-color 240 240 240)) -40 40))
    (make-slice 2 (move-pinhole (circle 10 'solid (make-color 240 240 240)) 40 -40))
    (make-slice 2 (move-pinhole (circle 10 'solid (make-color 240 240 240)) 40 40))
    (make-slice 8 (circle 80 'solid (make-color 180 180 180)))))

(scene->image buttons!-scene)

should look like

Depth values bigger than around 10 or so will look bad, so it's best to keep those numbers small.

Once you've gotten to this point, you should be able to render some pretty cool scenes as anaglyphs. But there are still some refinements you could make. Here are a few examples of refinements you could choose to make if you wanted to.

Refinement 1 (easy) Constructing scenes from scratch is a pain because you've got to make sure each scene is in sorted order and shifting images horizontally or vertically requires directly manipulating image pinholes. Write a library of functions that eases this task:

; a slice-list is either
;   - empty, or
;   - (cons slice slice-list)
;
; slice-list->scene : slice-list -> scene
; converts an unsorted list of slices to a legal scene
(define (slice-list->scene ...)
  ...)
; 
; move-scene-horizontal : scene number -> scene
; produces a scene that's just like the input scene but
; with each image shifted the given number of pixels to
; the right (negative numbers mean shift left)
(define (move-slice-horizontal ...)
  ...)
;
; move-scene-back : scene number -> scene
; produces a scene that's just like the input scene but
; where each slice has depth the given amount greater than
; its depth in the original scene (negative numbers bring
; the slices forward)
(define (move-scene-back ...)
  ...)
;
; merge-scenes : scene scene -> scene
; produces the scene that has all slices from both input scenes
(define (merge-scenes ...)
  ...)

Refinement 2 (not so easy) Our way of representing the depth of a slice within a scene was easy to implement but totally nonsensical, since there's no direct connection how far away an object is from the viewer and the number of pixels you need to shift an image over.

To get that right, we can do a little trigonometry. A scene with an object in focus and one in the background at a known additional depth d and horizontal offset h is depicted in this figure:

Given this diagram, we need to solve for L Offset and R Offset given the eye separation distance (a constant representing how far apart your eyes are from each other), the focal length (the distance from your face to the object you're focusing on), distance behind the focal object the background image is, and the horizontal distance between the centerline and the background object (we won't worry about depth here).

First, recall from trigonometry that tan x = opposite/adjacent and thus the tangent of the angle formed between the line from right eye to focal object to center is (6.5 / 2) / Focal Length, and the angle, which we'll call theta, is the arctangent of that number.

Second, notice that the dashed hypotenuse line can be determined using the Pythagorean theorem (hypotenuse² = Object Depth² + Object Offset²) and the angle between the line from the background object to the focal object and the focal object straight forward is equal to arctangent(Object Offset / Object Depth), which we'll call theta', and the angle between the hypotenuse and the right eye line is theta - theta', which we'll call theta''. Using that, sin theta'' = R Offset / hypotenuse and R Offset = hypotenuse * sin theta'' (all these assuming the length of the hypotenuse is not 0, meaning the object we're investigating is not at the focal length).

Use that development and the equivalent development for R Offset to refine the data definition for a scene and slices. Then update your functions to reflect the updated data definition.

Something to think about (very hard) Even the development in refinement 2 isn't enough to make a general-purpose anaglyph library - for instance, there's no way to express that an an object is tilted with respect to the viewer. To really implement this the right way, you'll need to take a different approach entirely and represent objects within a scene as collections of points within a three-dimensional vector space rather than as image/depth pairs, and to render them you'll need to do a series of matrix transformations. Read this explanation of how 3D projections work and write a program that uses that technique to render left- and right-eye images, and then combine those images into anaglyphs using the function you wrote in part 1. (If you actually pursue this idea, I can help you with doing matrix math in Scheme.)