Efficient Reloading Data/Pushing Data from Server to Client

Efficient reloading data / pushing data from server to client

This question has been asked a number of times, but in a slightly different ways. Here are a few references that are worth a read:

What are Long-Polling, Websockets, Server-Sent Events (SSE) and Comet?
Using comet with PHP?
Apache with Comet Support
Server Scalability - HTML 5 websockets vs Comet
How to implement event listening in PHP

In summary: if you are looking at building your solution using PHP on Apache then holding open persistent connections (HTTP long-polling or streaming) is going to use up resources very quickly (is highly inefficient). So, you would be better using a hosted solution (*disclaimer - I work for a hosted solution).

HTTP-Long polling and HTTP Streaming are solutions which have been superseded by Server-Sent Events and WebSockets. So, where possible (where the web client provides support) you should use one of these solutions before falling back to an HTTP-based solution. A good realtime web technology will automatically handle this for you.

Since your diagram shows you are subscribing to multiple data streams you should also consider a Publish/Subscribe solution that naturally fits with this. Again, a good realtime web tech solution will provide you with this.

Also see the realtime web technology guide.

How to efficiently push new data to frontend without page reload or timer

Pulling data from server at a certain interval would still create delays and be unreliable, even if you do it every second. You should look into publish/subscribe service ably channels or pubnub. Those are the first 2 from the top of my head, I am sure there are more alternatives. Usually such services are paid, but some of them offer free plans for low volume customers.

If you decide to go publish/subscribe way, there are again a few ways you can accomplish our goal.

You can do everything in javascript, avoiding the need to connect to your php script altogether. In such case all the communication would be done through message service provider. You would have to make both front end apps publishers and subscribers at the same time. When the state changes on one of the clients, you publish a message to the channel, which is delivered to all other clients (subscribers of the channel). The clients change their state according to the contents of the message.
Alternatively, on each state change you can submit the form and send it to your php script, which can act like a central "hub" which maintains the current state of the image. In such case you would make PHP script the "publisher" and all javascript clients "subscirbers". The rest is very similar: php script sends a message to all channel subscribers, which change their state according to the message content.

The PHP solution would have an advantage of knowing the state during page load, which helps for the initial sync of all the clients.

You should be aware that even though publish/subscribe service works almost in "real time" (it will probably take around 30-50ms for the message to get delivered), there is still a possibility of race conditions. 2 clients can change their state (image touched) almost at the same time (say a microsecond difference). Client A might expect to go from "on" to "off" state, and client B might expect to go from "on" to "off" state, but since there are actually 2 events happening and neither of the clients is aware of that, they might both end up with the "on" state in the end (depends how you handle the messages). The good part is that in your scenario such conditions are quite easy to detect and you can decide what to do about it: force all clients to the latest state silently or throw an error explaining that their app went out of sync for a moment.

How to efficiently transfer large amount of data from DB to client user Spring REST

I (partly) managed to get this working using this answer: How To Stream Chunked Response With Spring Boot @RestController

The part where the browser directly begins downloading when the first entries are written into the output stream works - but the DB does not split the querying into batches but executes one large query. (Fetch size seems to be ignored)

Because of that the observed RAM consumption is still pretty high, while downloading the whole set of data takes now about 5 minutes instead of ~1 min before (no streaming) - which also makes sense, kind of.

Since the streaming part was the original question and it's working now, i'll mark this as the answer.

What's the most efficient architecture for this system? (push or pull)

Short answer: use notifications (option 3).

Long answer: this is a use case for some middle layer which propagates changes using a message-oriented middleware. This decouples the messaging logic from database metadata (triggers / stored procedures), can use peer-to-peer and publish/subscribe communication patterns, and more.

I have blogged a two-part article about this at

Firebird Database Events and Message-oriented Middleware (part 1)
Firebird Database Events and Message-oriented Middleware (part 2)

The article is about Firebird but the suggested solutions can be applied to any application / database.

In your scenarios, clients can also use the middleware message broker send messages to the system even if the database or the Delphi part is down. The messages will be queued in the broker until the other parts of the system are back online. This is an advantage if there are many clients and update installations or maintenance windows are required.

Similarly, if a member of the third group is viewing a report and a
member of either of the first two groups updates data, I wish to
reflect this in the report. What it the best way to do this?

If this is a real requirement (reports are usually a immutable 'snapshot' of data, but maybe you mean a view which needs to be updated while beeing watched, similar to a stock ticker) but it is easy to implement - a client just needs to 'subscribe' to an information channel which announces relevant data changes. This can be solved very flexible and resource-saving with existing message broker features like message selectors and destination wildcards. (Note that I am the author of some Delphi and Free Pascal client libraries for open source message brokers.)

SignalR push data Server to Client

Here is what I ended up writing, it is working very well.

It allows me to push data to clients (using the signalr .net client) by initiating the request from the web server business logic:

Imports Microsoft.AspNet.SignalR.Client
Imports Microsoft.AspNet.SignalR.Client.Hubs
Imports Microsoft.AspNet.SignalR.Client.Transports
Imports System.Threading.Tasks

''' <summary>
''' Methods and properties that relate to the Communication Hub.
''' Use to push data from the Server to the Client via SignalR.
''' </summary>
''' <remarks>
''' We are using shared variables, we can help prevent race conditions by initiating this class by calling the StartUp method from the application_start event
''' </remarks>
Public Class CommHub

    Private Shared Property Connection As HubConnection
    Private Shared Property Proxy As IHubProxy
    Private Shared ReadOnly _connectionLock As Object = New Object

    Private Sub New()
    End Sub

    ''' <summary>
    ''' Start up the connection to the realtime communication servers.
    ''' Call on application_start event.
    ''' </summary>
    ''' <remarks></remarks>
    Public Shared Sub Startup()
        Task.Run(Async Function()
                     'async must be run on a new thread:
                     Dim commHub As New CommHub
                     Await commHub.ConnectAsync()
                 End Function)
    End Sub

    ''' <summary>
    ''' Establishes an authenticated connection to the SignalR server.
    ''' </summary>
    ''' <remarks></remarks>
    Private Async Function ConnectAsync() As Tasks.Task

        'set connection to server:
        If IsNothing(Connection) Then
            SyncLock _connectionLock
                If IsNothing(Connection) Then

                    Dim realtimeUrl As String = "http://myserver/signalr"

                    Connection = New HubConnection(realtimeUrl)

                    'OPTIONAL:
                    'add custom headers:
                    'Connection.Headers.Add("headername", "headervalue")

                    'OPTIONAL:
                    'get realtime authentication cookie with permission to communicate with the realtime servers:
                    'Dim authCookie As Net.Cookie = Code_to_get_your_auth_cookie
                    'Connection.CookieContainer = New Net.CookieContainer
                    'Connection.CookieContainer.Add(authCookie)

                    'define hub proxy to connect to:
                    Proxy = Connection.CreateHubProxy("commHub")

                End If
            End SyncLock
        End If

        If Not IsNothing(Connection) Then
            Dim myConnection As HubConnection = Connection ' get pointer reference to shared object
            If myConnection.State = ConnectionState.Disconnected Then
                Try
                    'Two way communication is not required as we are only pushing data to the signalR server.
                    'Therefore we should use ServerSentEventsTransport as it is more efficient than websockets.
                    'Explicitly setting the transport type speeds up connection - otherwise connection transport is auto detected:
                    Await myConnection.Start(New ServerSentEventsTransport())

                Catch ex As Exception
                    'catch multiple connection attempts
                    'do nothing
                End Try
            End If
        End If

    End Function

    Public Enum CounterTypes As Integer
        Messages
        Requests
        Notifications
    End Enum

    ''' <summary>
    ''' Updates the counter on the specified client in realtime.
    ''' </summary>
    ''' <param name="mbrID">The unique id of the client.</param>
    ''' <param name="counterType">The type of counter.</param>
    ''' <param name="value">The value of the counter.</param>
    ''' <remarks></remarks>
    Friend Shared Async Sub UpdateCounterAsync(mbrID As Integer, counterType As CounterTypes, value As Integer)

        're-establish connection if it has been dropped 
        'otherwise reuses the existing connection 
        Dim commHub As New CommHub
        Await commHub.ConnectAsync()

        'call server side method:
        If Not IsNothing(Connection) Then
            Dim myConnection As HubConnection = Connection 'get pointer reference to shared object
            If myConnection.State = ConnectionState.Connected Then

                Dim myProxy As IHubProxy = Proxy 'get pointer reference to shared object
                If Not IsNothing(myProxy) Then

                    'invoke method on signalR server:
                    Await myProxy.Invoke("UpdateCounter", mbrID, counterType.ToString, value)

                End If

            End If
        End If

    End Sub

End Class

I can then push data to clients by simply calling methods from my BLL like so:

Await CommHub.UpdateCounterAsync(12731, CommHub.CounterTypes.Messages, 10)

Eg: user id 12731 has 10 new messages

Remember to call the the StartUp method from your application_start event like so:

CommHub.StartUp();

This will help prevent race conditions by establish the connection to the server on start up.

What is the most efficient way to reload a div without refreshing the whole page in django?

With standard Ajax, the client has to initiate the request. If you want the server to be able to "push" to the client without a request from the client, you'll want to use websockets. There's a package called Django Channels for implementing websockets in Django.

Is there some way to PUSH data from web server to browser?

Yes, what you're looking for is COMET http://en.wikipedia.org/wiki/Comet_(programming). Other good Google terms to search for are AJAX-push and reverse-ajax.

How can I efficiently send objects in my game with client-server architecture

Minimize the amount of data traversing the network. Only send the needed amount of data and provide your own verification through timestamps and validation.

Each actionId and AnimationId in the following should exhibit the minimal (power of 2) amount of bytes to uniquely describe all of the possible values.

Server -> Client:

Servers known client position and rotation. Acknowledgement of last actionId.
A list of all other clients positions, orientations(rotations), animationIds... within the field of view of the client +- maxRotationRate within a distance of MaxViewDistance +- maxMovementRate.

The client must average its current position with the servers position every step.

Process all inputs locally(mouse and keyboard), and only send the relevant data to the server:

Every update, the client will send the relevant data and update its own internal state and process the server state given, and increasing int value(call this a timestamp). Include parity or CRC here.

Client -> Server
Current position, rotation, actionId, animationId, and a monotonically increasing int value(call this a timestamp). Include parity or CRC here.

All rendering decisions, the Viewport, current animation/sprite/colour should be established on the client(the colour and initial position needs to be given to the client at logon).

Location data will be shared(local updates and server adjustments).

Location and rotation and velocity data(both linear and rotational) need to be sent to the server every update. Upon receipt at the server a validation process must occur to prevent cheating. On missed, out of order, or malformed updates, the server will assume the last position + velocity * (time difference from last timestamp + current server time).

You must use UDP for the communication(note that you are using your own timestamps and CRC to validate the data received, no more crashes on invalid data).

Only process packets where the parity/CRC checks pass and reject the packets that are received too short or fail the checks. These checks will eliminate the crash that is associated with bad packets.

TCP's retry mechanism and ordered delivery is detrimental to game performance. Each time a single packet is lost or delivered out of order, the time delta error(time the data is sent versus the time the server processes it) is increased by the RTT(Round Trip Time, i.e. Ping Time) due do the server retries and packet re-ordering. This will lead to multi-second delays that will only increase as the game goes on.

Network delays and dropped/malformed packets are normal occurrences for games. In the case of out of order packets based on timestamp values, drop all but the last, since this indicates the players latest wishes (for movement). With firing(spell casting in your case), keep sending the spell bits until acknowledged(ActionId). Ignore all but the first received spell on the server(send the Acknowledgement to the client), until the recharge period has expired.

While this answer excludes all points not brought up in the original question, mainly damage, a structure should be created that when an incidence of a damage inducing element is produced the on the server that distributes the damage according to the defined rules of damage distribution, that needs to be applied to each afflicted client.

Efficient Reloading Data/Pushing Data from Server to Client