While there are more models than what is described here, the industry has largely standardized on the following two distortion models.

Image distortion is when the straight lines of an image appear to be deformed or curved unnaturally, creating different distortion types, including barrel, pincushion, and waveform. Distortion is often the result of the lens's geometrics and can significantly disrupt the image's quality.

Overall, Kannala-Brandt has been chosen in many applications for how well it performs on wide-angle lens types. It is one of the first distortion models to successfully displace the Brown-Conrady model after decades.

Camera distortiontest

Kannala-Brandt also aims to characterize other radial (such as asymmetric) and tangential distortions. This is done with the following additional parameter sets:

Where: i   =   a suffix representing each picture height; Ai, Bi, and V shall be represented by the number of pixels from the output image.

To relate this back to our image coordinate system (i.e. \(x\) and \(y\)), we usually need to do some basic trigonometry:

When measuring the local geometric distortion, we assume that the distortion close to the optical center is zero. Then you can calculate a regular grid based on the geometric positions of the nine structures (3x3) in the center of the image. This grid is expanded to the whole image and defines the nominal positions for each of the structures.

Camera distortionCorrection

Where: i is a suffix representing each picture width; αi, βi, and V shall be represented by the number of pixels in the output image.

Brown-Conrady distortion is probably what most think of as the "standard" radial and tangential distortion model. This model finds its roots in two | documents, authored by Brown and Conrady. The documents are quite old and date up to a century ago, but still form the foundation of many of the ideas around characterizing and modeling distortion today!

Specific lens systems (particularly small ones in mobile devices) will correct distortion at the maximum image height. These systems show the highest distortion level at lower distances from the optical center. The resulting type of distortion is often a mixture of barrel and pincushion distortion and is described with the term wave distortion.

It might also seem that with the Kannala-Brandt distortion model, we simplify the math by cancelling out \(f\) from our determination of \(\theta\). This is true, and it will make the math easier, and remove a focal length term from our determination of the Kannala-Brandt parameters. However, if one chooses a different distortion projection, e.g. orthographic for a fish-eye lens:

Tangential distortion, as characterized by the Brown-Conrady model, is often simplified into the following \(x\) and \(y\) components. We present these here first as they are probably what most are familiar with:

This actually derives from an even-power series much like the radial distortion is an odd-power series. The full formulation is a solution to the following:

Given that, why does this discrepancy exist? There are a few reasons this could be used. Historically, the balanced profile was used because distortion was measured manually through the use of mechanical stereoplotters. These stereoplotters had a maximum range in which they could move mechanically, so limiting the maximum value of distortion made practical sense.

Camera distortionfree

An astute reader of this article might be thinking right now: "Hey, doesn't use these exact models! That math seems off!" You would mostly be correct there: this math is a bit different from how many popular libraries will model distortion (e.g. OpenCV). Using Brown-Conrady as an example, you might see symmetric radial distortion formulated as so:

Image distortion occurs when the straight lines of an image appear to be deformed or curved unnaturally. There are three types of lens distortion called barrel, pincushion, and waveform (also known as mustache) distortion. It is important to note that distortion occurs differently depending on the lens system and whether the lens can or cannot be removed from the camera.

Despite these reasons, there is also one very important disadvantage. Notably: the balanced profile with Brown-Conrady distortions introduces a factor of \(( 1 + \frac{\Delta f}{f})\) into the determination of \(k_1\) through \(k_3\). This may not seem like much, but it means that we are introducing a correlation in the determination of these parameters with our focal length. If one chooses a model where \(f_x\) and \(f_y\) are used, then the choice in parameterization will make the entire calibration unstable, as small errors in the determination of any of these parameters will bleed into every other parameter. This is one kind of projective compensation, and is another reason for why our last article on the subject suggested not to use this parameterization.

This method is applicable when the vertical line Ai is located closer to the vertical line at the center of the image than to Bi, use formula 2: $$Dhi=\frac{\left(Bi-Ai\right)}{2V}\times100%$$

Camera distortionface

Modeling distortion is one important component of the calibration process. By picking a good model, we can reduce projective compensations between our focal length \(f\) and our distortion parameters, which leads to numerical stability in our calibration parameters.

$$\delta t = (m_1\theta + m_2 \theta^3 + m_3 \theta^5)(j_1\cos(\psi) + j_2\sin(\psi) + j_3 \cos(2\psi) + j_4 \sin(2\psi) + ...)$$

In Part I, we described in detail the variety of lens distortion types that impact vision system performance and calibration. But what do you do when you encounter lens distortion? You correct it with a distortion model. Despite the proliferation and prevalence of cameras and vision-enabled devices over the past century or so, there have been two primary distortion models that have gained widespread adoption to provide correction. We'll go over these, and dive into the math and approach to ground you in these techniques.

As can be seen, chasing simplicity in the mathematical representations is one way in which our choice of model can result in unstable calibrations, or nonsensical math. Given that we want to provide the most stable and precise calibrations possible, we lean towards favouring the Gaussian profile models where possible. It does mean some extra work to make sure the math is correct, but also means that by getting this right once, we can provide the most stable and precise calibrations ever after.

Above, we've shown the formula for \(\theta\) when using perspective projection, but the main advantage of the Kannala-Brandt model is that it can support different kinds of projection by swapping our formula for \(\theta\), which is what makes the distortion function smoother for wide-angle lenses. See the following figure, shared here from the original paper, for a better geometric description of \(\theta\):

Overall, this results in a 23 parameter model! This is admittedly overkill, and the original paper claims as much. These models, unlike the symmetric radial distortion, are an empirical model derived by fitting an N-term Fourier series to the data being calibrated. This is one way of characterizing it, but over-parameterizing our final model can lead to poor repeatability of our final estimated parameters. In practice, most systems will characterize Kannala-Brandt distortions purely in terms of the symmetric radial distortion, as that distortion is significantly larger in magnitude and will be the leading kind of distortion in wider-angle lenses.

Camera distortionmodels

Did we get too far into the weeds with this? Never fear, Tangram Vision has you covered. If you're worried about the best model for your camera and would rather do anything else, check out the Tangram Vision SDK! We're building perception tools today that lift the burden off your shoulders. ‍

This probably seems quite confusing, because all these formulae are a bit different from the papers presented in this article. The main difference here is that the distortions have been re-characterized using what is referred to in photogrammetry as the Balanced Distortion Profile. Up until now, we have been presenting distortions using what is referred to in photogrammetry as the Gaussian Distortion Profile.

Where \(P(r)\) is our de-centering distortion profile function, \(\psi\) is the polar angle of the image plane coordinate, and \(\psi_0\) is the angle to the axis of maximum tangential distortion (i.e. zero radial distortion). Expanding this into the general parameter set we use today is quite involved (read the original Brown paper!), however this will typically take the form:

Camera distortioneffect

We've explored camera distortions, some common models for camera distortions, and explored the ways in which these models have evolved and been implemented over the past century. The physics of optics has a rich history, and there's a lot of complexity to consider when picking a model.

Pincushiondistortionlens

In practice, only the \(k_1\) through \(k_3\) terms are typically used. For cameras with relatively simple lens assemblies (e.g. only contain one or two lenses in front of the CMOS/CCD sensor), it is often sufficient to just use the \(k_1\) and \(k_2\) terms.

Almost a century later (2006, from the original Conrady paper in 1919), Juho Kannala and Sami Brandt published their own paper on lens distortions. The main contribution of this paper adapts lens distortion modeling to be optimized for wide-angle, ultra wide-angle, and fish-eye lenses. Brown & Conrady's modeling was largely founded on the physics of Seidel aberrations, which were first formulated around 1867 for standard lens physics of the time, which did not include ultra wide and fish-eye lenses.

If deriving a single number from the local geometric distortion, the maximum distortion measured for any of the image's geometric structures is the one that is reported.

Fitting in with our previous camera model, we can formulate this in terms of the collinearity relationship (assuming Brown-Conrady distortions):

Pincushiondistortion

$$\begin{bmatrix} x_i \\ y_i \end{bmatrix} = \begin{bmatrix} f \cdot X_t / Z_t \\ f \cdot Y_t / Z_t \end{bmatrix} + \begin{bmatrix} a_1 x_i \\ 0 \end{bmatrix} + \begin{bmatrix} \frac{x_i}{r} (k_1 \theta + k_2\theta^3+k_3\theta^5 + k_4\theta^7) \\ \frac{y_i}{r} (k_1 \theta + k_2\theta^3+k_3\theta^5 + k_4\theta^7) \end{bmatrix}$$

For systems with a wave distortion, the measured picture height distortion can be zero or close to zero even if a strong distortion is visible in the image. We can break down the line geometric distortion method into three different methods to measure these systems' distortion as specified in ISO 17850.

$$\delta r_{other} = (l_1\theta + l_2\theta^3 + l_3\theta^5) (i_1 \cos(\psi) + i_2 \sin(\psi) + i_3 \cos(2\psi) + i_4 \sin(2\psi) + ...)$$

Use this method when the horizontal line αi is located closer to the horizontal line through the center of the image than to βi, use formula 4: $$Dvi=\frac{\left(\beta i-\alpha i\right)}{2V}\times100%$$

The local geometric distortion method is utilized when a single number for the distortion is not sufficient. In other words, we need a function for the distortion to correct the distortion in image processing. Keep in mind that when you need to address a specific image height, the local geometric distortion is more reliable.

Because tangential distortion is usually small, we tend to approximate it using only the first two terms. It is rare for de-centering to be so extreme that our tangential distortion requires higher order terms because that would mean that our lens is greatly de-centered relative to our image plane. In most cases, one might ask if their lens should simply be re-attached in a more appropriate manner.

It is essential to test and analyze lens distortion to ensure high image quality. We recommend following ISO 17850 and using proper test charts to capture various images using different lens functions, e.g., zoom. We then suggest analyzing the results using evaluation software (e.g., iQ-Analyzer-X) to see where you can improve the camera system.

The TV distortion method is essentially a system that shows a steadily increasing distortion from the image center to the corners. The bending of a straight line in the original image is quantified at the image's top edge (see figures below). The ratio of the bending over the height of the image multiplied by 100 is the percentage of picture height distortion. This method is compliant with the process described in EBU Tech 3249 32493.

But what does that mean for today? Nowadays with the abundance of computing resources it is atypical to use mechanical stereoplotters in favour of making digital measurements on digital imagery. There doesn't seem to be a paper trail for why this decision was made, so it could just be a historical artefact. However, doing the re-balancing has some advantages:

The total line distortion is calculated as: $$\left|D_{\mathrm{line}}i\right|=\sqrt{\mathrm{Dh}\mathrm{i}^2+\mathrm{Dv}\ \mathrm{i}^2}%$$

ISO 903911 is the standard that defines methods to measure a lens that is separated from a camera. Sometimes, however, you cannot remove the lens from a camera (such as in a mobile phone), and thus the time-consuming methods described in ISO 9039 will not be adequate. As a result, ISO 178502 was introduced to define methods to measure distortion using a camera lens combination.

The oldest method that uses the camera-lens combination is the TV distortion method, created to analyze TV camera systems. This method requires a test chart with a regular grid of geometric structures such as those shown below.

Then one will quickly notice that the limit of \(\theta_{Balanced}\)  is not well defined as \(r \rightarrow \infty\), and gives us a value of \((-i)\infty\). As we've tried to separate our focal length from our parameters, we've ended up wading into the territory of complex numbers! Since our max radius and focal length are often proportional, \(\theta_{Gaussian}\) does not suffer from the same breakdown in values at the extremes.

Note that this linear relationship is derived by similar triangles for Brown-Conrady distortions by the relationships shown in the following figure:

$$\begin{bmatrix} x_i \\ y_i \end{bmatrix} = \begin{bmatrix} f \cdot X_t / Z_t \\ f \cdot Y_t / Z_t \end{bmatrix} + \begin{bmatrix} a_1 x_i \\ 0 \end{bmatrix} + \begin{bmatrix} x_i (k_1 r^2 + k_2r^4+k_3r^6) \\ y_i(k_1r^2 + k_2r^4 + k_3r^6) \end{bmatrix} + \\\ \begin{bmatrix} p_1(r^2 + 2x_i^2) + 2 p_2 x_i y_i \\ p_2(r^2 + 2y_i^2) + 2 p_1 x_i y_i \end{bmatrix}$$

As can be seen in the figure above, the profile and maximum magnitude of distortion are fundamentally different in the above cases. More specifically, the balanced profile is one way to limit the maximum distortion of the final distortion profile, while still correcting for the same effects. So how does one go from one representation to the other? For Brown-Conrady, this is done by scaling the distortion profile by a linear correction:

The primary difference that most folks will notice using this model lies in symmetric radial distortion. Rather than characterizing radial distortion in terms of how far a point is from the image centre (the radius), Kannala-Brandt characterizes distortion as a function of the incidence angle of the light passing through the lens. This is done because the distortion function is smoother when parameterized with respect to this angle (\(\theta\)), which makes it easier to model as a power-series:

In general, rotationally symmetric optical systems function to form an image that is geometrically similar to the object. There are a few exceptions for some particular systems, such as fish-eye lenses, where geometric conditions are deliberately not maintained. Ideally, this function is accomplished according to the geometry of perspective projection. Deviations from the ideal image geometry are called distortion.

Distortion commonly occurs from aberrations near the edges of an image. Each type of distortion usually develops through different variables. Barrel distortion, for example, is often the result of a lens at full zoom, while pincushion distortion occurs most often from telephoto lenses. Waveform distortion results from a large angle camera in zoom mode, combining both barrel and pincushion distortion.