Understand Op Registration and Kernel Linking in Tensorflow

Where is the bazel rule generating the `gen_io_ops.py` file when building TensorFlow from sources?

I finally got it.

---

There is indeed a call to tf_op_gen_wrapper_py but it's hidden in a call to tf_gen_op_wrapper_private_py:

def tf_gen_op_wrapper_private_py(name, out=None, deps=[],
                                 require_shape_functions=True,
                                 visibility=[]):
  if not name.endswith("_gen"):
    fail("name must end in _gen")
  [...]
  bare_op_name = name[:-4]
  tf_gen_op_wrapper_py(name=bare_op_name, ...

So the steps are the following.

In tensorflow/tensorflow/python/BUILD, there is this rule

tf_gen_op_wrapper_private_py(
    name = "io_ops_gen",
    [...]
)

And so, in this rule the _gen suffix will be removed (in tf_gen_op_wrapper_private_py) and a gen_ prefix will be added in tf_gen_op_wrapper_py and therefore the gen_io_ops.py module will be generated by this rule.

Meaning of TensorFlow operation ` IsExpensive()`?

Before answering your questions I think it is worth trying to understand how TensorFlow uses threads in order to get your work done. For this, I suggest you read this related and very good SO post.

You will find that TensorFlow uses a thread-pool in order to get you work done. The expensive Ops are being scheduled for execution on the thread-pool, whereas the cheap Ops are executed "inline" meaning by the same thread which schedules the tasks (Sidenote: from the source file you have linked you find only one exception, i.e. when the inline_ready queue is empty the thread can execute the last expensive Op by itself.).

With this in mind, let us try to answer your questions.

1. Is there any specific guideline when IsExpensive should be false, true?

I could not find a specific guideline in the TensorFlow manual, however, from the internals of what we discussed above a Op should be marked to be expensive, when the offset of scheduling a task to the thread pool is neglectable in comparison to the time the task needs to be executed.

2. What's the effect if an operation is flagged as expensive? So far I can only tell, that active profiling uses this just as a hint ? The only place querying this property is in the scheduler but without explaining what being inline means.

The effect is the following, everytime an Ops IsExpensive method returns false it might be pushed to the inline_ready queue and may block the thread from performing further tasks hence stalling your programm. In contrast, if the Ops IsExpensive method returns true, it will be scheduled for execution on the thread pool and the scheduling thread is free to continue doing its tasks in the process loop.

3. In conjunction with "1." should I care about it in my custom Ops?

I think you should care and try to reason as much as possible about the execution time of you Op. After that decide how you implement the IsExpensive method.

4. While it makes sense, that any AsyncOp (like RemoteFusedGraphExecuteOp) is expensive, MPIAllgatherOp seems to be defined as not expensive. Isn't this a contradiction?

No, it is not a contradiction. If you read the comment of MPIAllgatherOp you will find the following:

// Although this op is handled asynchronously, the ComputeAsync call is
// very inexpensive. It only sets up a CollectiveOpRecord and places it
// in the table for the background thread to handle. Thus, we do not need
// a TF pool thread to perform the op.

Which clearly states, that scheduling this task for the thread pool would almost only be overhead. Therefore performing it inline makes a lot of sense.

KeyError: 'NormalizeUTF8' loading a model to another jupyter notebook

I had this issue before. To solve it, you need to install tensorflow_text. You should try to :

>>> tf.__version__
2.8.2
>>>!pip install tensorflow-text==2.8.2

Tensorflow: Where is gen_logging_ops source code?

I had a similar question. gen_ files are files generated at build time by bazel and if you build TensorFlow yourself, they can be found in the bazel-genfiles/ directory.

The tf_op_gen_wrapper_py bazel rule is the rule generating the gen_ files. As explained, in this answer, this rule generates all the python wrapper of the ops defined in this library:

py_library(
    name = "logging_ops",
    srcs = ["ops/logging_ops.py"],
    srcs_version = "PY2AND3",
    deps = [
        ":framework_for_generated_wrappers",
        ":logging_ops_gen",
        ":util",
    ],
)

In the logging_ops case there is an indirection: the call to tf_op_gen_wrapper_py is hidden in tf_gen_op_wrapper_private_py:

 tf_gen_op_wrapper_private_py(
    name = "logging_ops_gen",
    visibility = [
        "//learning/brain/python/ops:__pkg__",
        "//tensorflow/python/kernel_tests:__pkg__",
    ],
)

---

Related Topics