How Is J() Function Implemented in Data.Table

How is J() function implemented in data.table?

J() used to be exported before, but not since 1.8.8. Here's the note from 1.8.8:

o The J() alias is now removed outside DT[...], but will still work inside DT[...]; i.e., DT[J(...)] is fine. As warned in v1.8.2 (see below in this file) and deprecated with warning() in v1.8.4. This resolves the conflict with function J() in package XLConnect (#1747) and rJava (#2045). Please use data.table() directly instead of J(), outside DT[...].

Using R's lazy evaluation, J(.) is detected and simply replaced with list(.) using the (invisible) non-exported function .massagei.

That is, when you do:

require(data.table)
DT = data.table(x=rep(1:5, each=2L), y=1:10, key="x")
DT[J(1L)]

i (= J(1L)) is checked for its type and this line gets executed:

i = eval(.massagei(isub), x, parent.frame())

where isub = substitute(i) and .massagei is simply:

.massagei = function(x) {
    if (is.call(x) && as.character(x[[1L]]) %chin% c("J","."))
        x[[1L]] = quote(list)
    x
}

Basically, data.table:::.massagei(quote(J(1L))) gets executed which returns list(1L), which is then converted to data.table. And from there, it's clear that a join has to happen.

Using data.table i and j arguments in functions

Gavin and Josh are right. This answer is only to add more background. The idea is that not only can you pass variable column names into a function like that, but expressions of column names, using quote().

group = quote(car)
mtcars[, list(Total=length(mpg)), by=group][order(group)]
      group Total
        AMC     1
   Cadillac     1
     ...
     Toyota     2
    Valiant     1
      Volvo     1

Although, admitedly more difficult to start with, it can be more flexible. That's the idea, anyway. Inside functions you need substitute(), like this :

tableOrder = function(x,.expr) {
    .expr = substitute(.expr)
    ans = x[,list(Total=length(mpg)),by=.expr]
    setkeyv(ans, head(names(ans),-1))    # see below re feature request #1780
    ans
}

tableOrder(mtcars, car)
      .expr Total
        AMC     1
   Cadillac     1
     Camaro     1
      ...
     Toyota     2
    Valiant     1
      Volvo     1

tableOrder(mtcars, substring(car,1,1))  # an expression, not just a column name
      .expr Total
 [1,]     A     1
 [2,]     C     3
 [3,]     D     3
 ...
 [8,]     P     2
 [9,]     T     2
[10,]     V     2

tableOrder(mtcars, list(cyl,gear%%2))   # by two expressions, so head(,-1) above
     cyl gear Total
[1,]   4    0     8
[2,]   4    1     3
[3,]   6    0     4
[4,]   6    1     3
[5,]   8    1    14

A new argument keyby was added in v1.8.0 (July 2012) making it simpler :

tableOrder = function(x,.expr) {
    .expr = substitute(.expr)
    x[,list(Total=length(mpg)),keyby=.expr]
}

Comments and feedback in the area of i,j and by variable expressions are most welcome. The other thing you can do is have a table where a column contains expressions and then look up which expression to put in i, j or by from that table.

How to properly use ifelse() inside j of data.table?

You've gone down a rabbit hole. Here's the way up:

DT[, col5:= shift(col3, fill = -1) == 0 & col3 != 0]

# or
DT[, col5:= shift(col3, fill = -1) == 0 & col3 != 0, keyby = key(DT)]

implementation of dot function `.()` in data.table package

The data.table package accomplishes it with this bit of code

replace_dot_alias <- function(e) {
  # we don't just simply alias .=list because i) list is a primitive (faster to iterate) and ii) we test for use
  # of "list" in several places so it saves having to remember to write "." || "list" in those places
  if (is.call(e)) {
    # . alias also used within bquote, #1912
    if (e[[1L]] == 'bquote') return(e)
    if (e[[1L]] == ".") e[[1L]] = quote(list)
    for (i in seq_along(e)[-1L]) if (!is.null(e[[i]])) e[[i]] = replace_dot_alias(e[[i]])
  }
  e
}

found in R/data.table.R (currently at line 173). That's why you don't find data.table:::. anywhere, and how they accomplish the parsing you mention in your post.

Then in [.data.table" <- function (x, i, j,... they can do this sort of thing

if (!missing(j)) {
    jsub = replace_dot_alias(substitute(j))
    root = if (is.call(jsub)) as.character(jsub[[1L]])[1L] else ""

....

How to use R data.table column names with cube(..., j = ,...) within a function?

When reading through the code for data.table:::cube.data.table and data.table:::groupingsets.data.table, the j argument is already being evaluated using NSE. Hence, being unable to pass in as.name(var_work) to the environment argument of substitute, the function will fail.

As a workaround, you can use .SDcols:

library(data.table)    
DT <- data.table(mtcars)    
var_work <- "hp"
by_vars <- c("cyl", "carb")

my_fun <- function(table_work, var_w, by_v) {
    cube(table_work, j=as.list(quantile(.SD[[1L]])), by=by_v, .SDcols=var_w)
}

ans_2 <- my_fun(table_work = DT, var_w = var_work, by_v = by_vars)

How do I reference a function parameter inside inside a data.table with a column of the same name?

  1. One possible option is this:
myfunc <- function(dt, t){
  env <- environment()
  dt <- dt[t==get('t',env)]
  mean(dt$b)
}

  1. Another approach: while perhaps not strictly a "solution" to your current problem, you may find it of interest. Consider data.table version>= 1.14.3. In this case, we can use env param of DT[i,j,by,env,...], to indicate the datatable column as "t", and the function parameter as t. Notice that this will work on column t with function parameter t, even if dt contains columns named col and val
myfunc <- function(dt, t){
  dt <- dt[col==val, env=list(col="t", val=t)]
  mean(dt$b)
}

In both case, usage and output is as below:

Usage

myfunc(dt = foo, t = 3)

Output:

[1] 0.1292877

Input:

set.seed(123)
foo <- data.table(t = c(1,1,2,2,3), b = rnorm(5))

foo looks like this:

> foo
   t           b
1: 1 -0.56047565
2: 1 -0.23017749
3: 2  1.55870831
4: 2  0.07050839
5: 3  0.12928774

Why does the Datatable filter new data based on the last selected value

I solved the problem by sticking to the explanations offered here. In fact, I needed to change the code structure totally. The only thing, I needed to do so that the post here will fit my case was to add table.search( '' ).columns().search( '' );

As a result, the whole new code is the following:

function fetchData() {
        $.ajax({
            method: 'POST',
            url: 'someurl.php',
            data: {"acquire": true,
                    "term": $("#tval option:selected").val()
            },
            context: document.body
        }).done(function( data ) {
            var table = $('#tbl').DataTable();
            table.clear();
            table.search( '' ).columns().search( '' );
            table.rows.add(data);
            buildSelectLists();
            table.draw();
        });
    }

    function buildSelectLists() {

        var api = $('#tbl').dataTable().api();
        api.columns().each(function () {
            $('input', this.footer()).on('keyup change clear', function () {
                api.draw();
            });
        })
        api.columns([1,3,4,5,6]).every(function () {
            var column = this;
            var select = $('<select  class="form-control form-control-sm"><option value=""></option></select>')
                .appendTo($(column.footer()).empty())
                .on('change', function () {
                    var val = $.fn.dataTable.util.escapeRegex(
                        $(this).val()
                    );

                    column
                        .search(val ? '^' + val + '$' : '', true, false)
                        .draw();
                });

            column.data().unique().sort().each(function (d, j) {
                select.append('<option value="' + d + '">' + d + '</option>')
            });
        });
    }

    $(document).ready(function () {

        $('#tval').on('change', function() {
            fetchData();
        });

            $('#tbl').DataTable({
                columns: [
                    {"data": "title"},
                    {"data": "gender"},
                ],
                "bDestroy": true,
                "initComplete": function() {
                    fetchData();
                }
            });
});

How do I correctly use the env variable for data.tables within a function

It's because dots isn't a call, it's a list of calls. So when data.table evaluates j it's trying to insert that list into a new column.

To fix this you need to splice the list of calls into a single call. You can do this in a call to ':='() directly (Option 1 below), but you can also break this into multiple steps that mirrors what you were doing above by converting dots to be a call to list() (Option 2).

library(data.table)

data <- data.table::data.table(a = 1:5, b = 2:6)

# Option 1 - call to ':='
test <- function(data, ...) {
  dots <- eval(substitute(alist(...)))
  j <- bquote(':='(..(dots)), splice = TRUE)
  print(j)
  data[, j, env = list(j = j)][]
}

# # Option 2 - convert dots to a call to a list
# test <- function(data, ...) {
#   dots <- eval(substitute(alist(...)))
#   dots_names <- names(dots)
#   dots <- bquote(list(..(unname(dots))), splice = TRUE)
#   j <- call(":=", dots_names, dots)
#   print(j)
#   data[, j, env = list(j = j)][]
# }

test(data = data, c = a + 1, double_b = b * 2)
#> `:=`(c = a + 1, double_b = b * 2)
#>        a     b     c double_b
#>    <int> <int> <num>    <num>
#> 1:     1     2     2        4
#> 2:     2     3     3        6
#> 3:     3     4     4        8
#> 4:     4     5     5       10
#> 5:     5     6     6       12

Edit: You can also use test2() if you want to be able to edit the same column or use newly made columns.

test2 <- function(data, ...) {
  dots <- eval(substitute(alist(...)))
  dots_names <- names(dots)
  for (i in seq_along(dots)) {
    dot_name <- dots_names[[i]]
    dot <- dots[[i]]
    j <- call(":=", dot_name, dot)
    print(j)
    data[, j, env = list(j = j)]
  }
  data[]
}

How to add additional feature/function in DataTables?

You asked a very naive question. Hopefully this helps:

  1. You need to first add all dependencies of DataTables in your HTML file:
<link rel="stylesheet" type="text/css" href="https://cdn.datatables.net/1.10.16/css/jquery.dataTables.min.css">
<script type="text/javascript" language="javascript" src="//code.jquery.com/jquery-1.12.4.js"></script>
<script type="text/javascript" language="javascript" src="https://cdn.datatables.net/1.10.16/js/jquery.dataTables.min.js"></script>

  1. Create the table in HTML with all data inside <table id="YourIdOfTableTag"> tag or do some scripting to insert data into table as new rows.

  2. Once all data is present in table, you can your own custom script and add into the HTML:

<script type="text/javascript" language="javascript" src="urOwnScript.js"></script>

  1. urOwnScript.js can be written in two ways.
    * A. If you already have the table with data, then initialize the datatable once the page load is done.
    * B. If you are fetching data and editing html DOM with new rows, then once that operation is finished initialize datatable.

  2. For case A, contents of urOwnScript.js can be like this:

$(document).ready(function() { // Means this is run only on page load, which means <table> tag has all the data already.
    $('#YourIdOfTableTag').DataTable( {
        initComplete: function () {
            this.api().columns().every( function () {
                var column = this;
                var select = $('<select><option value=""></option></select>')
                    .appendTo( $(column.footer()).empty() )
                    .on( 'change', function () {
                        var val = $.fn.dataTable.util.escapeRegex(
                            $(this).val()
                        );

                        column
                            .search( val ? '^'+val+'$' : '', true, false )
                            .draw();
                    } );

                column.data().unique().sort().each( function ( d, j ) {
                    select.append( '<option value="'+d+'">'+d+'</option>' )
                } );
            } );
        }
    } );
} );

Related Topics

Plot a Jpg Image Using Base Graphics in R

Removing Particular Character in a Column in R

Is There a Limit for the Possible Number of Nested Ifelse Statements

R: Compare All the Columns Pairwise in Matrix

Sparkr Window Function

How to Find the Package Name in R for a Specific Function

Pass R Variable to Rodbc's SQLquery with Multiple Entries

How to Create Dummy Variables

R Subtract Value for the Same Id (From the First Id That Shows)

How to Get the Second Sub Element of Every Element in a List

Adding Multiple Columns in a Dplyr Mutate Call

How to Define Fill Colours in Ggplot Histogram

How to Change Name of Factor Levels

Change the Order of Stacked Fill Columns in Ggplot2

Rolling Join Grouped by a Second Variable in Data.Table

How to Retrieve the Client's Current Time and Time Zone When Using Shiny

Adding Counts of a Factor to a Dataframe

Find Overlapping Regions and Extract Respective Value


Leave a reply



Submit