一文详解静态图和动态图中的自动求导机制

1、系统无法感知整个动态图模型的全局信息。有些变量可能后续不会再被引用了，可以释放内存，在动态图系统中由于无法感知到后续图的结构，因此就必须保留下来（除非工程师手动释放），导致显存占用一般会大于静态图（当然也并不一定）。

2、每次都需要重新建图，在计算效率上不如静态图，静态图是一次建图，后续永远都是在这个建图结果的基础上进行计算的。这个就类似于解释性语言（如python）和编译性语言（如C和C++）的区别。

3、由于动态图需要每次重新建图，导致其无法在嵌入式设备上进行部署（两种原因，1是效率问题，2是嵌入式设备通常不具有网站的建图运行时，只支持推理模式），通常需要其以某种形式（比如ONNX）转化为静态图的参数后，通过静态图部署。常见的部署方式包括TensorRT，Paddle Lite，TensorFlow Lite，TensorFlow Serving，NCNN（手机端居多）等等。

GEEK TALK

自动求导 AutoDiff

import tensorflow as tf
X1 = tf.placeholder(tf.float32, shape=(1,), name="X1")X2 = tf.placeholder(tf.float32, shape=(1,), name="X2")
h1 = tf.multiply(X1, X2)h2 = tf.add(h1, X1)output = tf.div(h2, X2)
grad = tf.gradients(output, [X1, X2])
feed_dict = {    "X1": 0.6, "X2": 0.2}sess = tf.Session()output_v = sess.run(output, feed_dict)grad_v = sess.run(grad, feed_dict)

if (X > 2) {    return X * X3} else {    return X4 - X}

GEEK TALK

静态图自动求导的实现示例

class Node(object):    def __init__(self):        self.inputs = []        self.op = None        self.const_attr = None        self.name = ""
    def __add__(self, other):        if isinstance(other, Node):            new_node = add_op(self, other)        else:            new_node = add_byconst_op(self, other)        return new_node
    def __mul__(self, other):        if isinstance(other, Node):            new_node = mul_op(self, other)        else:            new_node = mul_byconst_op(self, other)        return new_node
    def __truediv__(self, other):        raise ValueError('No implement div')
    # Allow left-hand-side add and multiply.    __radd__ = __add__    __rmul__ = __mul__
    def __str__(self):        return self.name
    __repr__ = __str__

class Op(object):    def __call__(self):        new_node = Node()        new_node.op = self        return new_node
    def compute(self, node, input_vals):        raise NotImplementedError
    def gradient(self, node, output_grad):        raise NotImplementedError

class MatMulOp(Op):    """Op to matrix multiply two nodes."""    def __call__(self, node_A, node_B, trans_A=False, trans_B=False):        new_node = Op.__call__(self)        new_node.matmul_attr_trans_A = trans_A        new_node.matmul_attr_trans_B = trans_B        new_node.inputs = [node_A, node_B]        new_node.name = "MatMul(%s,%s,%s,%s)" % (node_A.name, node_B.name, str(trans_A), str(trans_B))        return new_node
    def compute(self, node, input_vals):        assert len(input_vals) == 2        assert type(input_vals[0]) == np.ndarray and type(input_vals[1]) == np.ndarray        return np.dot(input_vals[0], input_vals[1])
    def gradient(self, node, output_grad):        """    if Y=AB, then dA=dY B^T, dB=A^T dY        """        return [matmul_op(output_grad, transpose_op(node.inputs[1])), matmul_op(transpose_op(node.inputs[0]), output_grad)]

def gradients(output_node, node_list):    """Take gradient of output node with respect to each node in node_list.    Parameters    ----------    output_node: output node that we are taking derivative of.    node_list: list of nodes that we are taking derivative wrt.    Returns    -------    A list of gradient values, one for each node in node_list respectively.    Something wrong, should be the backward graph of the gradients    """    node_to_output_grads_list = {}    node_to_output_grads_list[output_node] = [oneslike_op(output_node)]    reverse_topo_order = reversed(find_topo_sort([output_node]))
    for ind, each in enumerate(reverse_topo_order):        if ind == 0:            gg = each.op.gradient(each, oneslike_op(output_node))        else:            gg = each.op.gradient(each, node_to_output_grads_list[each])
        if gg is None:            continue        for indv, eachv in enumerate(gg):            if each.inputs[indv] in node_to_output_grads_list.keys():                node_to_output_grads_list[each.inputs[indv]] += gg[indv]            else:                node_to_output_grads_list[each.inputs[indv]] = gg[indv]
        node_to_output_grad[each] = each    grad_node_list = [node_to_output_grads_list[node] for node in node_list]    return grad_node_list
def find_topo_sort(node_list):    """Given a list of nodes, return a topological sort list of nodes ending in them.
    A simple algorithm is to do a post-order DFS traversal on the given nodes,     going backwards based on input edges. Since a node is added to the ordering    after all its predecessors are traversed due to post-order DFS, we get a topological    sort.    """    visited = set()    topo_order = []    for node in node_list:        topo_sort_dfs(node, visited, topo_order)    return topo_order
def topo_sort_dfs(node, visited, topo_order):    """Post-order DFS"""    if node in visited:        return    visited.add(node)    for n in node.inputs:        topo_sort_dfs(n, visited, topo_order)    topo_order.append(node)

class Executor:    """Executor computes values for a given subset of nodes in a computation graph."""    def __init__(self, eval_node_list):        self.eval_node_list = eval_node_list
    def run(self, feed_dict):        node_to_val_map = dict(feed_dict)        # Traverse graph in topological sort order and compute values for all nodes.        topo_order = find_topo_sort(self.eval_node_list)        for each in topo_order:            if each.inputs:                input_vals = []                for each_input in each.inputs:                    input_vals += [node_to_val_map[each_input]]                node_to_val_map[each]  = each.op.compute(node=each, input_vals=input_vals)        node_val_results = [node_to_val_map[node] for node in self.eval_node_list]        return node_val_results

参考资料：

[1]. https://blog.csdn.net/LoseInVain/article/details/88557173, 《AutoDiff理解》 之第一篇， 自动求导技术在深度学习中的应用

[2]. https://openmlsys.github.io/chapter_preface/index.html, OPEN MLSYS

[3]. https://github.com/FesianXu/ToyAutoDiff

推荐阅读：

千万级高性能长连接Go服务架构实践

百度搜索Push个性化：新的突破

数据交付变革：研发到产运自助化的转型之路

百度搜索exgraph图执行引擎设计与实践

百度搜索&金融：构建高时效、高可用的分布式数据传输系统