How to Write a Scalable Tcp/Ip Based Server

How to write a scalable TCP/IP based server

I've written something similar to this in the past. From my research years ago showed that writing your own socket implementation was the best bet, using the asynchronous sockets. This meant that clients not really doing anything actually required relatively few resources. Anything that does occur is handled by the .NET thread pool.

I wrote it as a class that manages all connections for the servers.

I simply used a list to hold all the client connections, but if you need faster lookups for larger lists, you can write it however you want.

private List<xConnection> _sockets;

Also you need the socket actually listening for incoming connections.

private System.Net.Sockets.Socket _serverSocket;

The start method actually starts the server socket and begins listening for any incoming connections.

public bool Start()
{
  System.Net.IPHostEntry localhost = System.Net.Dns.GetHostEntry(System.Net.Dns.GetHostName());
  System.Net.IPEndPoint serverEndPoint;
  try
  {
     serverEndPoint = new System.Net.IPEndPoint(localhost.AddressList[0], _port);
  }
  catch (System.ArgumentOutOfRangeException e)
  {
    throw new ArgumentOutOfRangeException("Port number entered would seem to be invalid, should be between 1024 and 65000", e);
  }
  try
  {
    _serverSocket = new System.Net.Sockets.Socket(serverEndPoint.Address.AddressFamily, SocketType.Stream, ProtocolType.Tcp);
   }
   catch (System.Net.Sockets.SocketException e)
   {
      throw new ApplicationException("Could not create socket, check to make sure not duplicating port", e);
    }
    try
    {
      _serverSocket.Bind(serverEndPoint);
      _serverSocket.Listen(_backlog);
    }
    catch (Exception e)
    {
       throw new ApplicationException("An error occurred while binding socket. Check inner exception", e);
    }
    try
    {
       //warning, only call this once, this is a bug in .net 2.0 that breaks if
       // you're running multiple asynch accepts, this bug may be fixed, but
       // it was a major pain in the rear previously, so make sure there is only one
       //BeginAccept running
       _serverSocket.BeginAccept(new AsyncCallback(acceptCallback), _serverSocket);
    }
    catch (Exception e)
    {
       throw new ApplicationException("An error occurred starting listeners. Check inner exception", e);
    }
    return true;
 }

I'd just like to note the exception handling code looks bad, but the reason for it is I had exception suppression code in there so that any exceptions would be suppressed and return false if a configuration option was set, but I wanted to remove it for brevity sake.

The _serverSocket.BeginAccept(new AsyncCallback(acceptCallback)), _serverSocket) above essentially sets our server socket to call the acceptCallback method whenever a user connects. This method runs from the .NET threadpool, which automatically handles creating additional worker threads if you have many blocking operations. This should optimally handle any load on the server.

    private void acceptCallback(IAsyncResult result)
    {
       xConnection conn = new xConnection();
       try
       {
         //Finish accepting the connection
         System.Net.Sockets.Socket s = (System.Net.Sockets.Socket)result.AsyncState;
         conn = new xConnection();
         conn.socket = s.EndAccept(result);
         conn.buffer = new byte[_bufferSize];
         lock (_sockets)
         {
           _sockets.Add(conn);
         }
         //Queue receiving of data from the connection
         conn.socket.BeginReceive(conn.buffer, 0, conn.buffer.Length, SocketFlags.None, new AsyncCallback(ReceiveCallback), conn);
         //Queue the accept of the next incoming connection
         _serverSocket.BeginAccept(new AsyncCallback(acceptCallback), _serverSocket);
       }
       catch (SocketException e)
       {
         if (conn.socket != null)
         {
           conn.socket.Close();
           lock (_sockets)
           {
             _sockets.Remove(conn);
           }
         }
         //Queue the next accept, think this should be here, stop attacks based on killing the waiting listeners
         _serverSocket.BeginAccept(new AsyncCallback(acceptCallback), _serverSocket);
       }
       catch (Exception e)
       {
         if (conn.socket != null)
         {
           conn.socket.Close();
           lock (_sockets)
           {
             _sockets.Remove(conn);
           }
         }
         //Queue the next accept, think this should be here, stop attacks based on killing the waiting listeners
         _serverSocket.BeginAccept(new AsyncCallback(acceptCallback), _serverSocket);
       }
     }

The above code essentially just finished accepting the connection that comes in, queues BeginReceive which is a callback that will run when the client sends data, and then queues the next acceptCallback which will accept the next client connection that comes in.

The BeginReceive method call is what tells the socket what to do when it receives data from the client. For BeginReceive, you need to give it a byte array, which is where it will copy the data when the client sends data. The ReceiveCallback method will get called, which is how we handle receiving data.

private void ReceiveCallback(IAsyncResult result)
{
  //get our connection from the callback
  xConnection conn = (xConnection)result.AsyncState;
  //catch any errors, we'd better not have any
  try
  {
    //Grab our buffer and count the number of bytes receives
    int bytesRead = conn.socket.EndReceive(result);
    //make sure we've read something, if we haven't it supposadly means that the client disconnected
    if (bytesRead > 0)
    {
      //put whatever you want to do when you receive data here

      //Queue the next receive
      conn.socket.BeginReceive(conn.buffer, 0, conn.buffer.Length, SocketFlags.None, new AsyncCallback(ReceiveCallback), conn);
     }
     else
     {
       //Callback run but no data, close the connection
       //supposadly means a disconnect
       //and we still have to close the socket, even though we throw the event later
       conn.socket.Close();
       lock (_sockets)
       {
         _sockets.Remove(conn);
       }
     }
   }
   catch (SocketException e)
   {
     //Something went terribly wrong
     //which shouldn't have happened
     if (conn.socket != null)
     {
       conn.socket.Close();
       lock (_sockets)
       {
         _sockets.Remove(conn);
       }
     }
   }
 }

EDIT: In this pattern I forgot to mention that in this area of code:

//put whatever you want to do when you receive data here

//Queue the next receive
conn.socket.BeginReceive(conn.buffer, 0, conn.buffer.Length, SocketFlags.None, new AsyncCallback(ReceiveCallback), conn);

Generally, in the whatever you want code, I would do reassembly of packets into messages, and then create them as jobs on the thread pool. This way the BeginReceive of the next block from the client isn't delayed while whatever message processing code is running.

The accept callback finishes reading the data socket by calling end receive. This fills the buffer provided in the begin receive function. Once you do whatever you want where I left the comment, we call the next BeginReceive method which will run the callback again if the client sends any more data.

Now here's the really tricky part: When the client sends data, your receive callback might only be called with part of the message. Reassembly can become very very complicated. I used my own method and created a sort of proprietary protocol to do this. I left it out, but if you request, I can add it in. This handler was actually the most complicated piece of code I had ever written.

public bool Send(byte[] message, xConnection conn)
{
  if (conn != null && conn.socket.Connected)
  {
    lock (conn.socket)
    {
    //we use a blocking mode send, no async on the outgoing
    //since this is primarily a multithreaded application, shouldn't cause problems to send in blocking mode
       conn.socket.Send(bytes, bytes.Length, SocketFlags.None);
     }
   }
   else
     return false;
   return true;
 }

The above send method actually uses a synchronous Send call. For me that was fine due to the message sizes and the multithreaded nature of my application. If you want to send to every client, you simply need to loop through the _sockets List.

The xConnection class you see referenced above is basically a simple wrapper for a socket to include the byte buffer, and in my implementation some extras.

public class xConnection : xBase
{
  public byte[] buffer;
  public System.Net.Sockets.Socket socket;
}

Also for reference here are the usings I include since I always get annoyed when they aren't included.

using System.Net.Sockets;

I hope that's helpful. It may not be the cleanest code, but it works. There are also some nuances to the code which you should be weary about changing. For one, only have a single BeginAccept called at any one time. There used to be a very annoying .NET bug around this, which was years ago so I don't recall the details.

Also, in the ReceiveCallback code, we process anything received from the socket before we queue the next receive. This means that for a single socket, we're only actually ever in ReceiveCallback once at any point in time, and we don't need to use thread synchronization. However, if you reorder this to call the next receive immediately after pulling the data, which might be a little faster, you will need to make sure you properly synchronize the threads.

Also, I hacked out a lot of my code, but left the essence of what's happening in place. This should be a good start for you're design. Leave a comment if you have any more questions around this.

Server Side Language for Scalable TCP-socket application

I would use Java and Mina,

http://mina.apache.org

for performance and scalability.

Writing a highly scalable TCP/IP server in C# 5 with the async/await pattern?

What is the best way to use the new Async methods on Socket, TcpClient or TcpListener to create a scalable server in C#?

There aren't any new async methods on Socket; the methods named *Async on Socket are a special set of APIs to reduce memory usage. TcpClient and TcpListener did get some new async methods.

If you want the best scalability, you're probably best using Stephen Toub's custom awaiters for Socket. If you want the easiest to code, you're probably better off using TcpClient and TcpListener.

Do the new Async methods leverage I/O Completion Ports?

Yes, just like most of the other asynchronous APIs in the BCL. AFAIK, the Stream class is the only one that may possibly not use the IOCP; all other *Begin/*End/*Async methods use the IOCP.

Is rolling your own Socket listener more efficient, or are the TcpListener/TcpClient classes pretty good now?

The classes are pretty good as they are. Stephen Toub has a blog post that is a bit more efficient in terms of memory use.

How do I write a scalable socket server using C# 4.0?

I've been working on something similar for a week or two now so hopefully I'll be able to help you out a bit.

If your focus is on simple code, I'd recommend using the TcpClient and TcpListener classes. They both make sockets much easier to work with. While they have existed since .NET Framework 1.1 they have been updated and are still your best bet.

In terms of how to utilize the .NET Framework 4.0 in writing simplistic code, Tasks are the first thing that come to mind. They make writing asynchronous code much less painful and it will become much easier to migrate your code once C# 5 comes out (new async and await keywords). Here is an example of how Tasks can simplify your code:

Instead of using tcpListener.BeginAcceptTcpClient(AsyncCallback callback, object state); and providing a callback method which would call EndAcceptTcpClient(); and optionally cast your state object, C# 4 allows you to utilize closures, lambdas, and Tasks to make this process much more readable and scalable. Here is an example:

private void AcceptClient(TcpListener tcpListener)
{
    Task<TcpClient> acceptTcpClientTask = Task.Factory.FromAsync<TcpClient>(tcpListener.BeginAcceptTcpClient, tcpListener.EndAcceptTcpClient, tcpListener);

    // This allows us to accept another connection without a loop.
    // Because we are within the ThreadPool, this does not cause a stack overflow.
    acceptTcpClientTask.ContinueWith(task => { OnAcceptConnection(task.Result); AcceptClient(tcpListener); }, TaskContinuationOptions.OnlyOnRanToCompletion);
}

private void OnAcceptConnection(TcpClient tcpClient)
{
    string authority = tcpClient.Client.RemoteEndPoint.ToString(); // Format is: IP:PORT

    // Start a new Task to handle client-server communication
}

FromAsync is very useful as Microsoft has provided many overloads that can simplify common asynchronous operations. Here's another example:

private void Read(State state)
{
    // The int return value is the amount of bytes read accessible through the Task's Result property.
    Task<int> readTask = Task<int>.Factory.FromAsync(state.NetworkStream.BeginRead, state.NetworkStream.EndRead, state.Data, state.BytesRead, state.Data.Length - state.BytesRead, state, TaskCreationOptions.AttachedToParent);

    readTask.ContinueWith(ReadPacket, TaskContinuationOptions.OnlyOnRanToCompletion);
    readTask.ContinueWith(ReadPacketError, TaskContinuationOptions.OnlyOnFaulted);
}

State is just a user-defined class that usually just contains the TcpClient instance, the data (byte array), and perhaps the bytes read as well.

As you can see, ContinueWith can be used to replace a lot of cumbersome try-catches that until now were a necessary evil.

At the beginning of your post you mentioned not wanting to create a thread per connection or create very long running tasks and I thought I would address that at this point. Personally, I don't see the problem with creating a thread for each connection.

What you must be careful with, however, is using Tasks (an abstraction over the ThreadPool) for long-running operations. The ThreadPool is useful because the overhead of creating a new Thread is not negligible and for short tasks such as reading or writing data and handling a client connection, Tasks are preferred.

You must remember that the ThreadPool is a shared resource with a specialized function (avoiding the overhead of spending more time creating a thread than actually using it). Because it is shared, if you used a thread, another resource cannot and this can quickly lead to thread-pool starvation and deadlock scenarioes.

Efficient & Scalable connected TCP Windows Service using C# .Net 4.5

Efficient & Scalable

The very first thing you need to decide is how efficient you want to be. The socket APIs can get extremely complex if efficiency is your top priority. However, efficiency is almost never the top priority, even though a lot of people think it is. The problem is that complexity can increase exponentially with efficiency/scalability, and if you simply maximize efficiency/scalability, you'll end up with an almost unmaintainable system. So you'll need to decide where to draw the line on that scale.

Particularly if you have horizontal scaling, you probably don't need to use the extreme-efficiency socket APIs.

I am aware of the limitations on ports but still wonder why these limitations in this era of tech (we always have limits but why still the old ones?!).

Compatibility. Ports in particular are represented by a 16-bit value. The only way this would change is if a new standard came out, and everything upgraded. NICs, gateways, ISPs, and IoT devices. That's a high order, and will probably never happen.

Single thread per server for async-accept new connections (lifetime of server).

That's fine. If you have a large amount of connection turnover, you can have multiple accept threads, too. Just keep your backlog high (it should be high by default on Windows Server OSes).

One thread per client connection (loop) to hold client connection?

Er, no.

You'll want to use asynchronous I/O, for sure.

You should have continuous (asynchronous) reads going on all connected clients, and then do (asynchronous) writes as necessary. Also, if the protocol permits it, you should periodically write heartbeat messages to each connected client; otherwise, you'll need a timer for each client to drop the connection. Depending on the nature of your writes, you may need to have a queue of pending writes per client.

a Task for performing client operation (short term, intermittent
operations)

If you use asynchronous tasks, then all your actual code will just run on whatever threadpool thread is available. No need for dedicated tasks at all.

You may find my TCP/IP .NET Sockets FAQ helpful.

---

Related Topics