Here We Go Again: Append an Element to a List in R

Here we go again: append an element to a list in R

Adding elements to a list is very slow when doing it one element at a time. See these two examples:

I'm keeping the Result variable in the global environment to avoid copies to evaluation frames and telling R where to look for it with .GlobalEnv$, to avoid a blind search with <<-:

Result <- list()

AddItemNaive <- function(item)
{
    .GlobalEnv$Result[[length(.GlobalEnv$Result)+1]] <- item
}

system.time(for(i in seq_len(2e4)) AddItemNaive(i))
#   user  system elapsed 
#  15.60    0.00   15.61 

Slow. Now let's try the second approach:

Result <- list()

AddItemNaive2 <- function(item)
{
    .GlobalEnv$Result <- c(.GlobalEnv$Result, item)
}

system.time(for(i in seq_len(2e4)) AddItemNaive2(i))
#   user  system elapsed 
#  13.85    0.00   13.89

Still slow.

Now let's try using an environment, and creating new variables within this environment instead of adding elements to a list. The issue here is that variables must be named, so I'll use the counter as a string to name each item "slot":

Counter <- 0
Result <- new.env()

AddItemEnvir <- function(item)
{
    .GlobalEnv$Counter <- .GlobalEnv$Counter + 1

    .GlobalEnv$Result[[as.character(.GlobalEnv$Counter)]] <- item
}

system.time(for(i in seq_len(2e4)) AddItemEnvir(i))
#   user  system elapsed 
#   0.36    0.00    0.38 

Whoa much faster. :-) It may be a little awkward to work with, but it works.

A final approach uses a list, but instead of augmenting its size one element at a time, it doubles the size each time the list is full. The list size is also kept in a dedicated variable, to avoid any slowdown using length:

Counter <- 0
Result <- list(NULL)
Size <- 1

AddItemDoubling <- function(item)
{
    if( .GlobalEnv$Counter == .GlobalEnv$Size )
    {
        length(.GlobalEnv$Result) <- .GlobalEnv$Size <- .GlobalEnv$Size * 2
    }

    .GlobalEnv$Counter <- .GlobalEnv$Counter + 1

    .GlobalEnv$Result[[.GlobalEnv$Counter]] <- item
}

system.time(for(i in seq_len(2e4)) AddItemDoubling(i))
#   user  system elapsed 
#   0.22    0.00    0.22

It's even faster. And as easy to a work as any list.

Let's try these last two solutions with more iterations:

Counter <- 0
Result <- new.env()

system.time(for(i in seq_len(1e5)) AddItemEnvir(i))
#   user  system elapsed 
#  27.72    0.06   27.83 

Counter <- 0
Result <- list(NULL)
Size <- 1

system.time(for(i in seq_len(1e5)) AddItemDoubling(i))
#   user  system elapsed 
#   9.26    0.00    9.32

Well, the last one is definetely the way to go.

Append an object to a list in R in amortized constant time, O(1)?

If it's a list of string, just use the c() function :

R> LL <- list(a="tom", b="dick")
R> c(LL, c="harry")
$a
[1] "tom"

$b
[1] "dick"

$c
[1] "harry"

R> class(LL)
[1] "list"
R> 

That works on vectors too, so do I get the bonus points?

Edit (2015-Feb-01): This post is coming up on its fifth birthday. Some kind readers keep repeating any shortcomings with it, so by all means also see some of the comments below. One suggestion for list types:

newlist <- list(oldlist, list(someobj))

In general, R types can make it hard to have one and just one idiom for all types and uses.

how to append an element to a list without keeping track of the index?

There is a function called append:

ans <- list()
for (i in 1992:1994){
n <- 1 #whatever the function is
ans <- append(ans, n)
}

  ans
## [[1]]
## [1] 1
## 
## [[2]]
## [1] 1
## 
## [[3]]
## [1] 1
## 

Note: Using apply functions instead of a for loop is better (not necessarily faster) but it depends on the actual purpose of your loop.

Answering OP's comment: About using ggplot2 and saving plots to a list, something like this would be more efficient:

plotlist <- lapply(seq(2,4), function(i) {
            require(ggplot2)
            dat <- mtcars[mtcars$cyl == 2 * i,]
            ggplot() + geom_point(data = dat ,aes(x=cyl,y=mpg))
})

Thanks to @Wen for sharing Comparison of c() and append() functions:

Concatenation (c) is pretty fast, but append is even faster and therefor preferable when concatenating just two vectors.

How to append all the elements of a list efficiently in R

We can use unlist with use.names=FALSE if the list elements are named

 unlist(x, use.names=FALSE)

Appending a list to a list of lists in R

Could it be this, what you want to have:

# Initial list:
myList <- list()

# Now the new experiments
for(i in 1:3){
  myList[[length(myList)+1]] <- list(sample(1:3))
}

myList

Does appending to a list in R result in copying?

I'm fairly confident the answer is "no". I used the following code to double check:

Rprof(tmp <- tempfile(), memory.profiling = TRUE)

x <- list()
for (i in 1:100) x[[i]] <- runif(10000)

Rprof()
summaryRprof(tmp, memory = "stats")
unlink(tmp)

The output:

# index: runif
#      vsize.small  max.vsize.small      vsize.large  max.vsize.large 
#            76411           381781           424523          1504387 
#            nodes        max.nodes     duplications tot.duplications 
#          2725878         13583136                0                0 
#          samples 
#                5 

The relevant part being duplications = 0.

Append value to empty vector in R?

Appending to an object in a for loop causes the entire object to be copied on every iteration, which causes a lot of people to say "R is slow", or "R loops should be avoided".

As BrodieG mentioned in the comments: it is much better to pre-allocate a vector of the desired length, then set the element values in the loop.

Here are several ways to append values to a vector. All of them are discouraged.

Appending to a vector in a loop

# one way
for (i in 1:length(values))
  vector[i] <- values[i]
# another way
for (i in 1:length(values))
  vector <- c(vector, values[i])
# yet another way?!?
for (v in values)
  vector <- c(vector, v)
# ... more ways

help("append") would have answered your question and saved the time it took you to write this question (but would have caused you to develop bad habits). ;-)

Note that vector <- c() isn't an empty vector; it's NULL. If you want an empty character vector, use vector <- character().

Pre-allocate the vector before looping

If you absolutely must use a for loop, you should pre-allocate the entire vector before the loop. This will be much faster than appending for larger vectors.

set.seed(21)
values <- sample(letters, 1e4, TRUE)
vector <- character(0)
# slow
system.time( for (i in 1:length(values)) vector[i] <- values[i] )
#   user  system elapsed 
#  0.340   0.000   0.343 
vector <- character(length(values))
# fast(er)
system.time( for (i in 1:length(values)) vector[i] <- values[i] )
#   user  system elapsed 
#  0.024   0.000   0.023 

Create a new vector by appending elements between them in R

Use purrr::accumulate with c:

library(purrr)
a_vec <- c("ant","bee","mosquito","fly")

accumulate(a_vec, c)

# [[1]]
# [1] "ant"
# 
# [[2]]
# [1] "ant" "bee"
# 
# [[3]]
# [1] "ant"      "bee"      "mosquito"
# 
# [[4]]
# [1] "ant"      "bee"      "mosquito"       "fly"

Or, in base R, Reduce with accumulate = T:

Reduce(c, a_vec, accumulate = T)

How to add elements to a list in R (loop)

You should not add to your list using c inside the loop, because that can result in very very slow code. Basically when you do c(l, new_element), the whole contents of the list are copied. Instead of that, you need to access the elements of the list by index. If you know how long your list is going to be, it's best to initialise it to this size using l <- vector("list", N). If you don't you can initialise it to have length equal to some large number (e.g if you have an upper bound on the number of iterations) and then just pick the non-NULL elements after the loop has finished. Anyway, the basic point is that you should have an index to keep track of the list element and add using that eg

i <- 1
while(...) {
    l[[i]] <- new_element
    i <- i + 1
}

For more info have a look at Patrick Burns' The R Inferno (Chapter 2).

---

Related Topics