Should you normalize RGB values by 255 or 256?

Let’s say you’re writing an image processing program. The program takes in an image, converts it to floating point, does some processing and finally saves the modified pixels to disk as 8-bit colors. The question today concerns how exactly the integer-to-float conversion should be done. There are two approaches which, written in Python and NumPy, look like this:

Standard division by 255 Alternative division by 256 = img / 255.0 pixelsimg = process(pixels) resultprocess(pixels) = np.trunc(result * 255 + 0.5 ) outputnp.trunc(result = (img + 0.5 ) / 256.0 pixels(img = process(pixels) resultprocess(pixels) = np.trunc(result * 256 ) outputnp.trunc(result

I assume that in both cases the output values are clamped before the final typecast:

# Clamp and cast to 8 bits = output.clip( 0 , 255 ).astype(np.uint8) output_8bitoutput.clip().astype(np.uint8)

The standard approach maps the integer 0 to 0.0 and 255 to 1.0. It works perfectly fine and is how GPUs do it. The alternative adds a 0.5 bias and divides by 256 instead, so the integer 0 gets mapped to 0.5/256=0.001953125. This is inconvenient because your image processing code can’t detect black pixels, for example, without knowing the above constant. As a consequence, you tie your logic to 8-bit inputs even if you compute in floating point. With the standard approach, you can always assume black is 0.0.

But some programmers still feel a pull towards the alternative. What is going on? What do they see in it?

The case against 255.0

The standard approach does look quite strange when plotted on the number line. Below you can see an exaggerated version with 3-bit integers in the range [0..7] being mapped to [0,1]:

On the X-axis we’ve got a number line and the locations of brown circles on it represent the decoded floating-point values. The numbers inside are the integer inputs. Each integer has arrows pointing to it; these show a range of floating-point values that round to it. I’ll call these ranges “bins” in the rest of this article.

... continue reading