Assigning by Reference into Loaded Package Datasets

assigning by reference into loaded package datasets

This has nothing to do with datasets or locking -- you can reproduce it simply using

DT<-unserialize(serialize(data.table(b = 1:5),NULL))
foo(DT)
DT

I suspect it has to do with the fact that data.table has to re-create the extptr inside the object on the first access on DT, but it's doing so on a copy so there is no way it can share the modification with the original in the global environment.

---

[From Matthew] Exactly.

DT<-unserialize(serialize(data.table(b = 1:3),NULL))
DT
   b
1: 1
2: 2
3: 3
DT[,newcol:=42]
DT                 # Ok. DT rebound to new shallow copy (when direct)
   b newcol
1: 1     42
2: 2     42
3: 3     42

DT<-unserialize(serialize(data.table(b = 1:3),NULL))
foo(DT)
   b a
1: 1 1
2: 2 1
3: 3 1
DT                 # but not ok when via function foo()
   b
1: 1
2: 2
3: 3

DT<-unserialize(serialize(data.table(b = 1:3),NULL))
alloc.col(DT)      # alloc.col needed first
   b
1: 1
2: 2
3: 3
foo(DT)
   b a
1: 1 1
2: 2 1
3: 3 1
DT                 # now it's ok
   b a
1: 1 1
2: 2 1
3: 3 1

Or, don't pass DT into the function, just refer to it directly. Use data.table like a database: a few fixed name tables in .GlobalEnv.

DT <- unserialize(serialize(data.table(b = 1:5),NULL))
foo <- function() {
   DT[, newcol := 7]
}
foo()
   b newcol
1: 1      7
2: 2      7
3: 3      7
4: 4      7
5: 5      7
DT              # Unserialized data.table now over-allocated and updated ok.
   b newcol
1: 1      7
2: 2      7
3: 3      7
4: 4      7
5: 5      7

Load dataset from R package using data(), assign it directly to a variable?

Using a helper function:

# define this function
getdata <- function(...)
{
    e <- new.env()
    name <- data(..., envir = e)[1]
    e[[name]]
}

# now load your data calling getdata()
x <- getdata("faithful")

Or using an anonymous function:

x <- (function(...)get(data(...,envir = new.env())))("faithful")

Lazy evaluation

You should also consider lazy loading your data adding LazyData: true in the DESCRIPTION file of your package.

If you use RStudio, after running data("faithful"), you'll see at the Environment panel that the "faithful" data.frame is called "promise" (another less common name is "thunk") and is greyed out. That means that it is lazily evaluated by R and not still loaded into memory. You can even lazy load the "x" variable with the delayedAssign() function:

data("faithful")              # lazy load "faithful"
delayedAssign("x", faithful)  # lazy assign "x" with a reference to "faithful"
rm(faithful)                  # remove "faithful"

Still nothing has been loaded into memory yet

summary(x)                    # now x has been loaded and evaluated

Learn more about lazy evaluation here.

data.table doesn't modify by reference if the object is freshly loaded from file?

A data.table needs to be over-allocated in memory for adding columns by reference to work. After loading it that's not the case:

load("ttt")
length(a)
#[1] 1
truelength(a)
#[1] 0

b <- data.table(x=1:2)
length(b)
#[1] 1
truelength(b)
#[1] 100

From help(truelength):

For tables loaded from disk however, truelength is 0 in R 2.14.0 and random in R <= 2.13.2; i.e., in both cases perhaps unexpected. data.table detects this state and over-allocates the loaded data.table when the next column addition or deletion occurs.

But it seems like if you pass a (freshly loaded) data.table to a function and then add by reference inside the function over-allocation happens but doesn't reach the symbol up in the global environment (only the local symbol inside the function). If you do it in the global environment directly or don't pass the data.table as function parameter, it works.

If the data.table is over-allocated already (as is normally the case, other than when freshly loaded from disk), then there are spare slots for the column to be added into by reference and no shallow copy (to achieve over-allocation) needs to be done by := inside the function.

This might be worth a bug report (but I haven't checked if there is already one).

R version 3.0.1 (2013-05-16)
Platform: x86_64-apple-darwin10.8.0 (64-bit)

locale:
[1] de_DE.UTF-8/de_DE.UTF-8/de_DE.UTF-8/C/de_DE.UTF-8/de_DE.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] microbenchmark_1.3-0 data.table_1.8.8    

loaded via a namespace (and not attached):
[1] tools_3.0.1

Understanding exactly when a data.table is a reference to (vs a copy of) another data.table

Yes, it's subassignment in R using <- (or = or ->) that makes a copy of the whole object. You can trace that using tracemem(DT) and .Internal(inspect(DT)), as below. The data.table features := and set() assign by reference to whatever object they are passed. So if that object was previously copied (by a subassigning <- or an explicit copy(DT)) then it's the copy that gets modified by reference.

DT <- data.table(a = c(1, 2), b = c(11, 12)) 
newDT <- DT 

.Internal(inspect(DT))
# @0000000003B7E2A0 19 VECSXP g0c7 [OBJ,NAM(2),ATT] (len=2, tl=100)
#   @00000000040C2288 14 REALSXP g0c2 [NAM(2)] (len=2, tl=0) 1,2
#   @00000000040C2250 14 REALSXP g0c2 [NAM(2)] (len=2, tl=0) 11,12
# ATTRIB:  # ..snip..

.Internal(inspect(newDT))   # precisely the same object at this point
# @0000000003B7E2A0 19 VECSXP g0c7 [OBJ,NAM(2),ATT] (len=2, tl=100)
#   @00000000040C2288 14 REALSXP g0c2 [NAM(2)] (len=2, tl=0) 1,2
#   @00000000040C2250 14 REALSXP g0c2 [NAM(2)] (len=2, tl=0) 11,12
# ATTRIB:  # ..snip..

tracemem(newDT)
# [1] "<0x0000000003b7e2a0"

newDT$b[2] <- 200
# tracemem[0000000003B7E2A0 -> 00000000040ED948]: 
# tracemem[00000000040ED948 -> 00000000040ED830]: .Call copy $<-.data.table $<- 

.Internal(inspect(DT))
# @0000000003B7E2A0 19 VECSXP g0c7 [OBJ,NAM(2),TR,ATT] (len=2, tl=100)
#   @00000000040C2288 14 REALSXP g0c2 [NAM(2)] (len=2, tl=0) 1,2
#   @00000000040C2250 14 REALSXP g0c2 [NAM(2)] (len=2, tl=0) 11,12
# ATTRIB:  # ..snip..

.Internal(inspect(newDT))
# @0000000003D97A58 19 VECSXP g0c7 [OBJ,NAM(2),ATT] (len=2, tl=100)
#   @00000000040ED7F8 14 REALSXP g0c2 [NAM(2)] (len=2, tl=0) 1,2
#   @00000000040ED8D8 14 REALSXP g0c2 [NAM(2)] (len=2, tl=0) 11,200
# ATTRIB:  # ..snip..

Notice how even the a vector was copied (different hex value indicates new copy of vector), even though a wasn't changed. Even the whole of b was copied, rather than just changing the elements that need to be changed. That's important to avoid for large data, and why := and set() were introduced to data.table.

Now, with our copied newDT we can modify it by reference :

newDT
#      a   b
# [1,] 1  11
# [2,] 2 200

newDT[2, b := 400]
#      a   b        # See FAQ 2.21 for why this prints newDT
# [1,] 1  11
# [2,] 2 400

.Internal(inspect(newDT))
# @0000000003D97A58 19 VECSXP g0c7 [OBJ,NAM(2),ATT] (len=2, tl=100)
#   @00000000040ED7F8 14 REALSXP g0c2 [NAM(2)] (len=2, tl=0) 1,2
#   @00000000040ED8D8 14 REALSXP g0c2 [NAM(2)] (len=2, tl=0) 11,400
# ATTRIB:  # ..snip ..

Notice that all 3 hex values (the vector of column points, and each of the 2 columns) remain unchanged. So it was truly modified by reference with no copies at all.

Or, we can modify the original DT by reference :

DT[2, b := 600]
#      a   b
# [1,] 1  11
# [2,] 2 600

.Internal(inspect(DT))
# @0000000003B7E2A0 19 VECSXP g0c7 [OBJ,NAM(2),ATT] (len=2, tl=100)
#   @00000000040C2288 14 REALSXP g0c2 [NAM(2)] (len=2, tl=0) 1,2
#   @00000000040C2250 14 REALSXP g0c2 [NAM(2)] (len=2, tl=0) 11,600
#   ATTRIB:  # ..snip..

Those hex values are the same as the original values we saw for DT above. Type example(copy) for more examples using tracemem and comparison to data.frame.

Btw, if you tracemem(DT) then DT[2,b:=600] you'll see one copy reported. That is a copy of the first 10 rows that the print method does. When wrapped with invisible() or when called within a function or script, the print method isn't called.

All this applies inside functions too; i.e., := and set() do not copy on write, even within functions. If you need to modify a local copy, then call x=copy(x) at the start of the function. But, remember data.table is for large data (as well as faster programming advantages for small data). We deliberately don't want to copy large objects (ever). As a result we don't need to allow for the usual 3* working memory factor rule of thumb. We try to only need working memory as large as one column (i.e. a working memory factor of 1/ncol rather than 3).

---

Related Topics