Content Overview

Below is a non-exhaustive list of changes that result from turning on the new type promotion.

More consistent and predictable promotion results

Using a lattice-based system allows the new type promotion to produce consistent and predictable type promotion results.

Old Type Promotion

Changing the order of operations produces inconsistent results using old type promotion.

# Setup
tnp.experimental_enable_numpy_behavior(dtype_conversion_mode="legacy")
a = np.array(1, dtype=np.int8)
b = tf.constant(1)
c = np.array(1, dtype=np.float16)


# (a + b) + c throws an InvalidArgumentError.
try:
  tf.add(tf.add(a, b), c)
except tf.errors.InvalidArgumentError as e:
  print(f'{type(e)}: {e}')  # InvalidArgumentError


<class 'tensorflow.python.framework.errors_impl.InvalidArgumentError'>: cannot compute AddV2 as input #1(zero-based) was expected to be a int8 tensor but is a int32 tensor [Op:AddV2] name:


# (b + a) + c returns an i32 result.
tf.add(tf.add(b, a), c)  # <tf.Tensor: shape=(), dtype=int32, numpy=3>


<tf.Tensor: shape=(), dtype=int32, numpy=3>

New Type Promotion

New type promotion produces consistent results regardless of the order.

tnp.experimental_enable_numpy_behavior(dtype_conversion_mode="all")
a = np.array(1, dtype=np.int8)
b = tf.constant(1)
c = np.array(1, dtype=np.float16)


WARNING:tensorflow:UserWarning: enabling the new type promotion must happen at the beginning of the program. Please ensure no TF APIs have been used yet.


# (a + b) + c returns a f16 result.
tf.add(tf.add(a, b), c)  # <tf.Tensor: shape=(), dtype=float16, numpy=3.0>


<tf.Tensor: shape=(), dtype=float16, numpy=3.0>


# (b + a) + c also returns a f16 result.
tf.add(tf.add(b, a), c)  # <tf.Tensor: shape=(), dtype=float16, numpy=3.0>


<tf.Tensor: shape=(), dtype=float16, numpy=3.0>

Reduced risk of bit-widening

Old Type Promotion

Old type promotion often resulted in 64-bit results.

tnp.experimental_enable_numpy_behavior(dtype_conversion_mode="legacy")


np.array(3.2, np.float16) + tf.constant(1, tf.int8) + tf.constant(50)  # <tf.Tensor: shape=(), dtype=float64, numpy=54.19921875>


<tf.Tensor: shape=(), dtype=float64, numpy=54.19921875>

New Type Promotion

New type promotion returns results with minimal number of bits necessary.

tnp.experimental_enable_numpy_behavior(dtype_conversion_mode="all")


WARNING:tensorflow:UserWarning: enabling the new type promotion must happen at the beginning of the program. Please ensure no TF APIs have been used yet.


np.array(3.2, np.float16) + tf.constant(1, tf.int8) + tf.constant(50)  # <tf.Tensor: shape=(), dtype=float16, numpy=54.2>


<tf.Tensor: shape=(), dtype=float16, numpy=54.1875>

tf.Tensor mathematical dunder methods

All tf.Tensor mathematical dunder methods will follow the new type promotion.

-tf.constant(5)  # <tf.Tensor: shape=(), dtype=int32, numpy=-5, weak=True>


<tf.Tensor: shape=(), dtype=int32, numpy=-5, weak=True>


tf.constant(5, tf.int16) - tf.constant(1, tf.float32)  # <tf.Tensor: shape=(), dtype=float32, numpy=4.0>


<tf.Tensor: shape=(), dtype=float32, numpy=4.0>

tf.Variable in-place ops

Implicit conversions will be allowed in tf.Variable in-place ops.

Note: Any promotion that results in a dtype that is different from the variable's original dtype will be not allowed. This is because tf.Variable cannot change its dtype.

tnp.experimental_enable_numpy_behavior(dtype_conversion_mode="all")
a = tf.Variable(10, tf.int32)
a.assign_add(tf.constant(5, tf.int16))  # <tf.Variable shape=() dtype=int32, numpy=15>


WARNING:tensorflow:UserWarning: enabling the new type promotion must happen at the beginning of the program. Please ensure no TF APIs have been used yet.
<tf.Variable 'UnreadVariable' shape=() dtype=int32, numpy=15>

tf.constant implicit conversions

In the old type promotion, tf.constant required an input Tensor to have the same dtype as the dtype argument. However, in the new type promotion, we implicitly convert Tensor to the specified dtype.

tnp.experimental_enable_numpy_behavior(dtype_conversion_mode="all")
a = tf.constant(10, tf.int16)
tf.constant(a, tf.float32)  # <tf.Tensor: shape=(), dtype=float32, numpy=10.0>


WARNING:tensorflow:UserWarning: enabling the new type promotion must happen at the beginning of the program. Please ensure no TF APIs have been used yet.
<tf.Tensor: shape=(), dtype=float32, numpy=10.0>

TF-NumPy Array

tnp.array defaults to i32* and f32* for python inputs using the new type promotion.

tnp.array(1)  # <tf.Tensor: shape=(), dtype=int32, numpy=1, weak=True>


<tf.Tensor: shape=(), dtype=int32, numpy=1, weak=True>


tnp.array(1.0)  # <tf.Tensor: shape=(), dtype=int32, numpy=1, weak=True>


<tf.Tensor: shape=(), dtype=float32, numpy=1.0, weak=True>

Input Type Inference

This is how different inputs' types are inferred in the new type promotion.

Further Reading

The new type promotion closely resembles JAX-NumPy's type promotion. If you want to know more details about the new type promotion and the design choices, check out the resources below.

WeakTensor-supporting APIs

Below is a list of APIs that supports WeakTensor.

For an unary op, this means that if an input with no user-specified type is passed in, it will return a WeakTensor.

For a binary op, it will follow the promotion table here. It may or may not return a WeakTensor depending on the promotion result of the two inputs.

Note: All mathematical operations (+, -, *, ...) are supported.

Originally published on the TensorFlow website, this article appears here under a new headline and is licensed under CC BY 4.0. Code samples shared under the Apache 2.0 License.