c++ interface

python/pywrap_tensorflow.py:pywrap_tensorflow -> a swig converted c++ libarary compiled in python/BUILD, pywrap_tensorflow and pywrap_tensorflow_internal

how variable is stored

• variable_scope.py::variable_scope is activated per input using with
• _ModelVariable -> get the variable reference here
  • stored in vars_collections -> with some value in ops.GraphKeys
  • variable_scope.py::get_variable(…) -> call the following function
  • get_variable_scope().get_variable(_get_default_variable_store(),…) -> grab the variable here
    • get_variable_scope_store().current_scope -> get a VariableScope object
      • ops.py::get_collection(_VARSCOPESTORE_KEY) get previous scope_store
      • variable_scope.py::_VariableScopeStore generate empty store
      • The collections key is not listed in GraphKeys
    • _get_default_variable_store() -> get a VariableStore object
    • VariableStore.get_variable -> concrete function

how session is run

• python/client/session.py
  • run() -> _run() -> _do_run() -> _call_tf_sessionrun()
  • tf_session.TF_SessionRun_wrapper
  • from pywrap_tensorflow
  • client/tf_sessionrun_wrapper.i
• python/client/tf_session_helper.h:TF_SessionRun_wrapper
  • feed_dict -> TF_Output and ndarray
• TF_SessionRun_wrapper_helper
  • -> TF_Output and TF_Tensor
• c/c_api.cc:TF_SessionRun
  • -> string and Tensor
• TF_Run_Helper
  • session is created via SessionFactory
• DirectSession::Run
  • todo:

how memory is managed

• core/framework/tensor.h:Tensor
  • TensorShape and TensorBuffer*
• Buffer < BufferBase < TensorBuffer
• TypedAllocator::Allocate
• Allocator::AllocateRaw

allocator has many implementations. For example:

• increase allocator alignment, but keep actual data type

CPUAllocator

malloc:
posix_memalign

GPUcudaMallocAllocator

cuMemAlloc
-> stream_executor/cuda/cuda_stub.cc:LoadSymbol

when is tensor being transport between devices?

how kernels types are checked?

• core/framework/op_kernel.h
  • every kernel calls REGISTER_KERNEL_BUILDER
  • and implements OpKernel
• When Opkernel is constructed, it is taking a OpKernelConstruction
  • which contains the actual tensor type
  • the actual op will check against this tensor type before doing the calculation
  • the check is called MatchSignatureHelper

how does REGISTER_KERNEL_BUILDER work?

side note on swig type map

%typemap(method [, modifiers]) typelist code ;

1. typemap match function signature from different language
2. multiple patterns can be listed in typelist
3. each pattern can have a list of argument
4. the code will convert type from other languages to the interface’s langauge
5. many convert helper exist

method:

• in: code will map type from other language to the typelist language

tensorflow contrib removed

https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md

tensorflow session is not fork-safe

According to source, tensorflow session is not fork-safe. Thus, it is unusable under default multi-processing context.

BTW, the default context is fork. We need to change to spawn multiprocessing.set_start_method('spawn')