Update a Data Frame in Shiny Server.R Without Restarting the App

Update a data frame in shiny server.R without restarting the App

If you just update regular variables (in the global environment, or otherwise) Shiny doesn't know to react to them. You need to use a reactiveValues object to store your variables instead. You create one using reactiveValues() and it works much like an environment or list--you can store objects by name in it. You can use either $foo or [['foo']] syntax for accessing values.

Once a reactive function reads a value from a reactiveValues object, if that value is overwritten by a different value in the future then the reactive function will know it needs to re-execute.

Here's an example (made more complicated by the fact that you are using load instead of something that returns a single value, like read.table):

values <- reactiveValues()
updateData <- function() {
  vars <- load(file = "my_data_frame.RData", envir = .GlobalEnv)
  for (var in vars)
    values[[var]] <- get(var, .GlobalEnv)
}
updateData()  # also call updateData() whenever you want to reload the data

output$foo <- reactivePlot(function() {
  # Assuming the .RData file contains a variable named mydata
  plot(values$mydata)
}

We should have better documentation on this stuff pretty soon. Thanks for bearing with us in the meantime.

How to save an edited dataframe and load it back when restarting an R Shiny app or query?

This is because you first save the data1 table with empty columns, then overwrite it after editing the table. When restarting the app, you again save the data1 with empty columns, overwriting the previously saved table (with edited columns).

If you need to display columns that you get from the sql query and the new columns created after the query, you need to save two dataframes and combine them to display them in the rendered DT.

First, save the results of the sql query right after the query.

data1 <- dbGetQuery(q_r,sql)
saveRDS(data1, 'data_query.rds')

Then you must create your combined dataframe with the added empty columns in the server. We can also retrieve data if the dataframe has already been edited and saved, by checking if a saved file exists in the app folder:
replace this:

d1 = readRDS('data.rds') 
d9 = d1

by this:

  if(!file.exists("data.rds")){
    d9 = data1
    d9['R/Y/G'] <- NA
    d9['R'] <- NA
    d9['Y'] <- NA
    d9['G'] <- NA
    d9['tcolor'] <- NA
  }
  else{
    cmp <- readRDS("data.rds")
    d9 = cbind(data1, cmp[,(ncol(cmp)-4):ncol(cmp)])
  }

̀d9 is now your combined dataframe that you display with renderDT() and can edit. Save this dataframe after edition as already described in your code (saveRDS(d9, 'data.rds')).

When starting the app again, if a data.rds file already exists in the folder, it contains all columns from data1 (from a previous query) and the additional columns (R, Y, G...). So we must cbind the new data1 created with the new query and the last 5th columns of the saved data.rds file (i.e. only the additional columns).

shiny server: what is the best practice to update data on the server

Let me try to reframe your question, positioning some of the paper / sample codes you are referring to.

AT very high level (i.e. without worrying so much about reactivity), R + shiny doesn't differ from a standard way to treat data as part of an ETL process (for example).

I.e. you can load into the shiny server one of the following types of external data:

1. Load data at rest, i.e. data residing in a file in the
   filesystem, or executing a RDBMS query. This is the standard case
   that covers most of the usage.
2. Load data in motion. This refers typically to a stream of data
   of some type that you are trying to analyse (i.e without persisting
   it into a file or a RDBMS table).

Lets talk about the different varieties of the first case first, data at rest:

server <- function(input, output, session) {
---
output$foo <- reactivePlot(function() {
  someQuery <- dbGetQuery(...)  # some query of a database
  plot(values$mydata)
}
---
}

The code above will run a query every time the reactive function is executed.

And this is where reactivity can help a big deal: for example with no other changes, the code above will be executed once for each user connecting to the application.

If the underlying data is getting updated frequently by an external process, the results for different users may be different.

Moreover, anything that cause the reactive construct to be re-execute, will re-execute the query as well (for example, just refreshing the browser the query will be re-executed, as each browser refresh generates a different session).

As you should know from any shiny training, the next steps could be to link the above reactive construct to some other UI element, for example an action button or a selectInput to filter the data.

server <- function(input, output, session) {
---
output$foo <- reactivePlot(function() {
if((length(input$actionbutton) ==0) | (length(input$selectData) == 0)) return()  
# the reactive now is connected to these two shiny inputs and executed every time they change

someQuery <- dbGetQuery(...)  # some query of a database, maybe with a *where* clause dependent on input$selectData
  plot(values$mydata)
}
---
}

Now the query will be executed every time the action button is pushed or a new selection is made.

Let's suppose that for your use case, as I'm sure you have either seen or implemented in ETL, your data is changing often. Suppose the file (or table) is continuously updated by an external process.

Please note that this use case is usually still considered at rest, even if updated frequently (you are processing the data through batches, or if the interval is really small, mini-batches).

It is here that your first example, where the different constructs of reactiveFileReader and reactivePoll enters into play.

If you have a file, for example a log file, updated very frequently by an external process, you can use reactiveFileReader.

If you have database table you can for example poll it every x seconds with reactivePoll.

Here your code can enjoy the full benefit of reactivity: automagically the code will be executed every x seconds for you and all the rest of your reactive code dependent on it will also be refreshed.

Now, lets assume you try to decrease the *batch size" (i.e. the window) over which shiny checks on data. How far can you go?

If I remember correctly a discussion with Joe Cheng a while back, he was confident that shiny would be able to handle up to 50,000 events per second (imagine to poll your database or read your file up to as many times per second).

Assuming that I remember this correctly, I would anyway consider 50,000 events a theoretical limit (you would have to discount the time taken to query your data in a RBMS, possibly over a LAN etc.), so for file access I would use something > 1 millisecond (i.e. <1,000 file read per second), and a much bigger time interval for a RDBMS.

It shouldn't therefore really surprise that the unit of time for the above functions is the millisecond.

I think that with the above constructs it is possible to implement using R + shiny very ambitious micro-batch pipelines.

It could be even possible imagine to use Apache Kafka to publish data to R + shiny (maybe serving Kafka using multiple instances of Shiny Server Pro with load balancing: yummy!)`

So, what about data in motion?

Well, if you get data from a firehouse at a rate manageable for R and shiny, you'd be OK (you may be in trouble to identify which R algorithms to use for this streaming use case, but this would deserve another question).

On the other hand if your process requires really low latency, much above and beyond what specified above, possibly you need to think to other types of tools and pipelines (e.g. using Apache Flink or consider ad hoc code).

Apologies for the very wordy explanation. Please let me know if it makes this complex topic any clearer.

How to update data in shiny app periodically?

Step one is to get yourself an auto-scheduler. I use taskscheduleR because I can't for the life of me figure out windows scheduler, and also the later package for some other tricky things.

Then you need to schedule your SQL download, and save that SQL as an R image.

Finally, just use deployApp("...", launch.browser = F, forceUpdate = T)

My Workflow

library(later)

shiny.auto <- function(interval = 2*60*60){ # 2 hours 60 minutes 60 seconds
     source("script source1")
     source("shinyappscript")
     later::later(shiny.auto, interval)
}

Script source1:

data<-sqlQuery(...)
save.image(...)

shinyappscript:

load(...)
deployApp("...", launch.browser = F, forceUpdate = T)

Hope this helps!

Immediately seeing changes in R shiny UI, in the browser, after making changes in code without having to restart the server?

You could use reactiveFileReader to regularly source the code you'd save in another file, and renderUI()/uiOutput to display that dynamic UI:

app.R

library(shiny)

ui <- uiOutput("ui")

server <- function(input, output, session) {
  ui <- reactiveFileReader(1000, session, "source.R", source)
  output$ui <- renderUI(ui())
}

shinyApp(ui = ui, server = server)

source.R

fluidPage(
  textInput("text", "text"),
  textInput("text2", "text")
  )

You still have to find out howto get rid of that "TRUE":

Shiny update data from selectInput

welcome to the site here. This is also a problem I have had recently -- it does not seem to be isolated to selectizeInput but pertains to any new data that you are trying to get your Shiny app to "see". Here are a couple things to try:

1. Delete any _cache directories in the root of your shiny server, then try to load again. If there is no difference,
2. touch the .csv files (or make an edit if on Windows, to get a new timestamp) and then try to load again. If there is no difference,
3. Restart the shiny server with sudo systemctl restart shiny-server where the last argument is the name of your shiny server (default is shiny-server)

This last solution is the one that worked most reliably for me -- it is #2 from this document (Shiny app does not reflect changes in update RData file) -- to restart the shiny server from the command line. This worked reliably and routinely. The other ideas in this solution did not work on my machine so must be dependent on your server or client specifications.

The second thing that worked for me was to use reactiveTimer (https://shiny.rstudio.com/reference/shiny/0.14/reactiveTimer.html).

In any case, if you have been beating your head against a wall with this, it is a tricky issue and is troublesome for many of us. Good luck with picking a best solution for your context.

How to trigger a data refresh in shiny?

You're looking for invalidateLater. Put this, with the appropriate interval, in the reactive expression that retrieves data from the the database.

Shiny app does not reflect changes in update RData file

Edit

There is actually a function called reactiveFileReader in the shiny package that does exactly what you are looking for: Periodically checking if the files "last modified" time or size changed and rereading accordingly. However, this function can only be used in the server context, so the file will be read at least once for each user that connects to your app. Options 3 and 4 in my Answer do not have these inefficiencies.

Original Answer from here on

First and foremost, shiny does not have a way to keep track of filechanges AFAIK. Your implementation reloads the .RData file whenever

1. shiny-server gets restarted via bash or
2. the global variables get reloaded because the app became idle at some point.

There is no way of telling, when the second condition is met. Therefore, I would advocate using one of the following four options. Sorted from easy to you better know your shiny!.

Option 1: Put the load statement in server

Here, the image is reloaded whenever a new user connects with the app. However, this might slow down your app if your .RData file is huge. If speed is not an issue, I would pick this solution since it is easy and clean.

# server.R
function(input, output, session) {
  load("working_dataset.RData")
  ...
}

The data will also be reread whenever a user refreshes the page (F5)

Option 2: Restart shiny-server whenever you want to re-import your data

(Also see @shosacos answer). This forces the .Rdata file to be reloaded.

$ sudo systemctl restart shiny-server

Again, this might slow-down your production process depending on the complecity of your app. One advantage of this approach is that you can also use the imported data to build the ui if you load the data in global.R. (I assume you don't given the code you gave).

Option 3: Import according to "last modified"

The idea here is to check whether the .RData has changed whenever a user connects to the app. To do this, you will have to use a "global" variable that contains a timestamp of the last imported version. The following code is untested, but should give you an idea on how to implement this feature.

# server.R
last_importet_timestamp <- reactiveVal("")

function(input,output,session){
  current_timestamp <- file.info(rdata_path)$mtime 

  if(last_importet_timestamp() != current_timestamp){
    # use parent.frame(2) to make data available in other sessions
    load(rdata_path, envir = parent.fame(2))
    # update last_importet_timestamp
    last_importet_timestamp(current_timestamp) 
  }

  ...
}

Speed-wise, this should be more efficient than the first two versions. The data is never imported more than once per timestamp (unless shiny server gets restarted or becomes idle).

Option 4: Import "reactvely"

Basically, the same as option 3 but the file will be checked for changes every 50ms. Here is a full working example of this approach. Note that the data is not loaded unless a change in "last modified" is detected, so the resulting overhead is not too bad.

library(shiny)

globalVars <- reactiveValues()

rdata_path = "working_dataset.RData"

server <- function(input, output, session){
  observe({
    text = input$text_in
    save(text = text, file = rdata_path, compress = TRUE)
  })
  observe({
    invalidateLater(50, session)
    req(file.exists(rdata_path))
    modified <- file.info(rdata_path)$mtime
    imported <- isolate(globalVars$last_imported)
    if(!identical(imported, modified)){
      tmpenv <- new.env()
      load(rdata_path, envir = tmpenv)
      globalVars$workspace <- tmpenv
      globalVars$last_imported <- modified
    }
  })
  output$text_out <- renderText({
    globalVars$workspace$text
  })
}

ui <- fluidPage(
  textInput("text_in", "enter some text to save in Rdata", "default text"),
  textOutput("text_out")
)

shinyApp(ui, server)

If you find it inconvenient to use globalVars$workspace$text, you can use with to access the contents of globalVars$workspace directly.

  output$text_out <- renderText({
    with(globalVars$workspace, {
      paste(text, "suffix")
    })
  })

---

Related Topics