Shannon’s noisy channel coding theorem tells us that the capacity is the maximum rate we can transmit information reliably over a noisy channel in the class of memoryless channels. In general, computing the capacity is a difficult problem. As such, there has been extensive work on asymptotics.
In the case of the additive Gaussian noise channel, the capacity is well known. However, it is still interesting to characterize the behavior of the asymptotes (as the SNR tends to zero or infinity) for use in proofs or to provide simple design guidelines for real-world communication systems.
Despite the work on asymptotes, it is more difficult to characterize the behavior at medium SNR without using the exact expression for the capacity.
In this post, I look at the medium behavior of the Gaussian noise channel. It turns out the SNR of decibels is particularly special and suggests a way of obtaining simple capacity approximations at medium SNR for general classes of additive noise channels.
More precisely, I will look at the role of in the additive Gaussian noise channel , where is a standard Gaussian random variable (zero mean, variance of ) and is the signal-to-noise ratio (SNR). It is well known that in this case, the capacity (subject to a power constraint) is given by
In information and communication theory, it is common to work in decibels (dB) and we can write , where is the SNR in decibels. Here are three situations where having or dB is particularly special.
(1) The intercept of the asymptote of the capacity as is dB.
As the SNR in dB , we can write
which is the linear asymptote. Observe that setting means that the intercept dB, as claimed.
(2) The bend point of the capacity occurs at dB.
It is common to study the capacity as either the SNR tends to infinity or the SNR tends to zero. This is because, in general, it is difficult to obtain a simple characterization of the behavior at medium levels of SNR.
One approach to characterize the behavior of the capacity at medium SNR is to evaluate the bend point.
Definition: The bend point, , of the capacity is the SNR such that the second derivative of the capacity is maximized.
This might seem an odd definition, but there is a lot of intuition here. In particular, observe that the second derivative provides information about how the rate of change of the capacity is varying. By finding where the second derivative is maximized, we find where the curve is the most bent.
A figure helps to illustrate the situation. Observe in Fig. 1 that as , the slope of the capacity curve tends to zero. As the SNR is increased, the slope starts to increase until it reaches the high SNR asymptote, where it is once again constant. As such, the second derivative is zero as both and . In between, the second derivative is positive and the bend point its maximum.
Fig. 1: Plot of the Gaussian capacity and asymptotic capacity curves.
To find the bend point, it is not hard to show that the second derivative of the capacity has a unique maximum. As such, we can compute the third derivative and find where it is zero. After changing our units to nats (i.e., the logarithm is base ), we need to solve
Observe that the solution to this equation is dB, as claimed.
(3) The cross-over point between the capacity curve and the MMSE curve is dB.
Suppose we want to estimate the Gaussian input from the observation , where as usual is Gaussian noise. The error of an estimate can be measured in the mean-square sense
The choice of to minimize the mean-square error is achieved by
The minimum mean-square error (MMSE) is then given by
To find the cross-over point (illustrated in Fig. 2), we then need to solve
which has the solution , equivalent to dB.
Fig. 2: Plot of the capacity and MMSE curves.
We have seen that a signal-to-noise ratio or dB is special in the case of additive Gaussian noise channels. Although these facts may appear to be trivial, the fact that all three hold makes the compelling suggestion that there are medium SNR features of the capacity that are connected to the asymptotic behavior of the capacity and also the behavior of the MMSE.
For more on the bend point and an application to multiuser MIMO, see
- Egan, M., “Low-high SNR transition in multiuser MIMO,” Electronic Letters, vol. 51, no. 3, pp. 296-298, 2015.
For more on connections between the mutual information and MMSE in the Gaussian case, see
- Guo, D., Shamai, S. and Verdú, S., “Mutual information and minimum mean-square error in Gaussian channels,” IEEE Transactions on Information Theory, vol. 51, no. 4, pp. 1261-1282, 2005.
- Guo, D., Wu, Y., Shamai, S. and Verdú, S., “Estimation in Gaussian noise: properties of the minimum mean-square error,” IEEE Transactions on Information Theory, vol. 57, no. 4, pp. 2371-2385, 2011.
For connections in non-Gaussian noise, see
- Guo, D., Shamai, S. and Verdú, S., “Additive non-Gaussian noise channels: mutual information and conditional mean estimation,” in Proc. of the IEEE International Symposium on Information Theory, 2005.