What Is The Side Effect of Setting Tcp_Max_Tw_Buckets to a Very Small Value

what is the side effect of setting tcp_max_tw_buckets to a very small value?

As you can see in this Kernel source, that option prevents graceful termination of the socket. In terms of the socket state, you have reduced the time wait duration for this connection to zero.

So what happens next? First off, you'll see the error message on your server. The rest is then a race condition for subsequent connections from your clients. Section 2 of rfc 1337 then covers what you may see. In short, some connections may show the following symptoms.

1. Corruption of your data stream (because the socket accepts an old transmission).
2. Infinite ACK loops (due to an old duplicate ACK being picked up).
3. Dropped connections (due to old data turning up in the SYN-SENT state).

However, proving this may be hard. As noted in the same RFC:

The three hazards H1, H2, and H3 have been demonstrated on a stock Sun OS 4.1.1 TCP running in an simulated environment that massively duplicates segments. This environment is far more hazardous than most real TCP's must cope with, and the conditions were carefully tuned to create the necessary conditions for the failures.

TCP Window Scaling in Linux server - possible side effects

It seems to me that the Wikipedia article you cited considerably overstates the case. The Microsoft Knowledge Base article it links to only cites 5 devices with this problem. That isn't 'many'.

And you need to consider that the problem is caused by Windows Vista defaulting to an enormous window scale of 8, enough to describe a window of 64k << 8 = 16MB, a ridiculously large number. Linux may or may not trigger it at all: at present you have zero evidence about that.

TCP window scaling does not cause 'bad performance [of] many short connections'. It causes very good performance on long lived connections.

I would place a lot more reliance on RFCs and vendor statements than on arbitrary Web sources; even Wikipedia. I corrected a major error in a TCP article there just this month.

When is TCP option SO_LINGER (0) required?

The typical reason to set a SO_LINGER timeout of zero is to avoid large numbers of connections sitting in the TIME_WAIT state, tying up all the available resources on a server.

When a TCP connection is closed cleanly, the end that initiated the close ("active close") ends up with the connection sitting in TIME_WAIT for several minutes. So if your protocol is one where the server initiates the connection close, and involves very large numbers of short-lived connections, then it might be susceptible to this problem.

This isn't a good idea, though - TIME_WAIT exists for a reason (to ensure that stray packets from old connections don't interfere with new connections). It's a better idea to redesign your protocol to one where the client initiates the connection close, if possible.

Storing a bucket of numbers in an efficient data structure

If the buckets are contiguous and disjoint, as in your example, you need to store in a vector just the left bound of each bucket (i.e. 1, 5, 16, 22) plus, as the last element, the first number that doesn't fall in any bucket (35). (I assume, of course, that you are talking about integer numbers.)

Keep the vector sorted.
You can search the bucket in O(log n), with kind-of-binary search. To search which bucket does a number x belong to, just go for the only index i such that vector[i] <= x < vector[i+1]. If x is strictly less than vector[0], or if it is greater than or equal to the last element of vector, then no bucket contains it.

EDIT. Here is what I mean:

#include <stdio.h>

// ~ Binary search. Should be O(log n)
int findBucket(int aNumber, int *leftBounds, int left, int right)
{
    int middle;

    if(aNumber < leftBounds[left] || leftBounds[right] <= aNumber) // cannot find
        return -1;
    if(left + 1 == right) // found
        return left;

    middle = left + (right - left)/2;

    if( leftBounds[left] <= aNumber && aNumber < leftBounds[middle] )
        return findBucket(aNumber, leftBounds, left, middle);
    else
        return findBucket(aNumber, leftBounds, middle, right);
}

#define NBUCKETS 12
int main(void)
{
    int leftBounds[NBUCKETS+1] = {1, 4, 7, 15, 32, 36, 44, 55, 67, 68, 79, 99, 101};
    // The buckets are 1-3, 4-6, 7-14, 15-31, ...

    int aNumber;
    for(aNumber = -3; aNumber < 103; aNumber++)
    {
        int index = findBucket(aNumber, leftBounds, 0, NBUCKETS);
        if(index < 0)
            printf("%d: Bucket not found\n", aNumber);
        else
            printf("%d belongs to the bucket %d-%d\n", aNumber, leftBounds[index], leftBounds[index+1]-1);
    }   
    return 0;
}

---

Related Topics