pg.typing¶
Symbolic typing.
Overview¶
To enable symbolic programmability on classes and functions, PyGlove intercepts the assignment operation on the attributes of symbolic objects, to achieve two goals:
Enable automatic type checking, value validation, conversion and transformation on attribute values. See Runtime typing for more details.
Allow symbolic attributes to be placeheld by special symbols, in order to represent abstract concepts such as a space of objects. E.g. hyper primitives like
pg.oneof
placeholds a symbolic attribute to create a space of values for that attribute. See Symbolic placeholding for more details.
Runtime typing¶
Symbolic objects are intended to be manipulated after creation. Without a runtime typing system, things can go wrong easily. For instance, an int attribute which was mistakenly modified at early program stages can be very difficut to debug at later stages. PyGlove introduces a runtime type system that automatically validates symbolic objects upon creation and modification, minimizing boilerplated code for input validation, so the developer can focus on the main business logic.
Understanding Schema¶
PyGlove’s runtime type system is based on the concept of Schema
(
class pg.Schema
), which defines what symbolic attributes are held
by a symbolic type (e.g. a symbolic dict, a symbolic list or a symbolic class)
and what values each attribute accepts. A Schema
object consists of a list
of Field
(class pg.Field
), which define the acceptable
keys (class pg.KeySpec
) and their values (class
pg.ValueSpec
) for these types. A Schema
object is usually
created automatically and associated with a symbolic type upon its declaration,
through decorators such as pg.members
, pg.symbolize
or
pg.functor
. For example:
@pg.members([
('x', pg.typing.Int(default=1)),
('y', pg.typing.Float().noneable())
])
class A(pg.Object):
pass
print(A.__schema__)
@pg.symbolize([
('a', pg.typing.Int()),
('b', pg.typing.Float())
])
def foo(a, b):
return a + b
print(foo.__schema__)
The first argument of all the decorators takes a list of field definitions, with each described by a tuple of 4 items:
(key specification, value specification, doc string, field metadata)
The key specification and value specification are required, while the doc string and the field metadata are optional. The key specification defines acceptable identifiers for this field, and the value specification defines the attribute’s value type, its default value, validation rules. The doc string will serve as the description for the field, and the field metadata can be used for field-based code generation.
The following code snippet illustrates all supported KeySpec
and
ValueSpec
subclasses and their usage with a manually created schema:
schema = pg.typing.create_schema([
# Primitive types.
('a', pg.typing.Bool(default=True).noneable()),
('b', True), # Equivalent to ('b', pg.typing.Bool(default=True)).
('c', pg.typing.Int()),
('d', 0), # Equivalent to ('d', pg.typing.Int(default=0)).
('e', pg.typing.Int(
min_value=0,
max_value=10).noneable()),
('f', pg.typing.Float()),
('g', 1.0), # Equivalent to ('g', pg.typing.Float(default=0.0)).
('h', pg.typing.Str()),
('i', 'foo'), # Equivalent to ('i', pg.typing.Str(default='foo').
('j', pg.typing.Str(regex='foo.*')),
# Enum type.
('l', pg.typing.Enum('foo', ['foo', 'bar', 0, 1]))
# List type.
('m', pg.typing.List(pg.typing.Int(), size=2, default=[])),
('n', pg.typing.List(pg.typing.Dict([
('n1', pg.typing.List(pg.typing.Int())),
('n2', pg.typing.Str().noneable())
]), min_size=1, max_size=10, default=[])),
# Dict type.
('o', pg.typing.Dict([
('o1', pg.typing.Int()),
('o2', pg.typing.List(pg.typing.Dict([
('o21', 1),
('o22', 1.0),
]))),
('o3', pg.typing.Dict([
# Use of regex key,
(pg.typing.StrKey('n3.*'), pg.typing.Int())
]))
]))
# Tuple type.
('p', pg.typing.Tuple([
('p1', pg.typing.Int()),
('p2', pg.typing.Str())
]))
# Object type.
('q', pg.typing.Object(A, default=A()))
# Type type.
('r', pg.typing.Type(int))
# Callable type.
('s', pg.typing.Callable([pg.typing.Int(), pg.typing.Int()],
kw=[('a', pg.typing.Str())])),
# Functor type (same as Callable, but only for symbolic.Functor).
('t', pg.typing.Functor([pg.typing.Str()],
kwargs=[('a', pg.typing.Str())]))
# Union type.
('u', pg.typing.Union([
pg.typing.Int(),
pg.typing.Str()
], default=1),
# Any type.
('v', pg.typing.Any(default=1))
])
Schema inheritance¶
Symbolic attributes can be inherited during subclassing. Accordingly, the
schema that defines a symbolic class’ attributes can be inherited too by its
subclasses. The fields from the bases’ schema will be carried over into the
subclasses’ schema, while the subclass can override, by redefining that field
with the same key. The subclass cannot override its base classes’ field with
arbitrary value specs, it must be overriding non-frozen fields with more
restrictive validation rules of the same type, or change their default values.
See pg.ValueSpec
for more details.
The code snippet below illustrates schema inheritance during subclassing:
@pg.members([
('x', pg.typing.Int(min_value=1)),
('y', pg.typing.Float()),
])
class A(pg.Object):
pass
@pg.members([
# Further restrict inherited 'x' by specifying the max value, as well
# as providing a default value.
('x', pg.typing.Int(max_value=5, default=2)),
('z', pg.typing.Str('foo').freeze())
])
class B(A):
pass
assert B.__schema__.fields.keys() == ['x', 'y', 'z']
@pg.members([
# Raises: 'z' is frozen in class B and cannot be extended further.
('z', pg.typing.Str())
])
class C(B):
pass
Automatic type conversions¶
Type conversion is another mechanism to extend the typing system, which allows the user to register converters between types. So when a value is assigned to a attribute whose value specification does not match with the input value type, a conversion will take place automatically if there is a converter from the input value type to the type required by the value specification. For example:
class A:
def __init__(self, str):
self._str = str
def __str__(self):
return self._str
def __eq__(self, other):
return isinstance(other, self.__class__) and self._str == other._str
pg.typing.register_converter(A, str, str)
pg.typing.register_converter(str, A, A)
assert pg.typing.Str().accept(A('abc')) == 'abc'
assert pg.typing.Object(A).accept('abc') == A('abc')
See pg.register_converter
for more details.
Symbolic placeholding¶
Symbolic placeholding enables scenarios that requires an abstract (non-material) representation of objects - for example - a search space or a partially bound object. Such objects with placeheld attributes can only be used symbolically, which means they can be symbolically queried or manipulated, but not yet materialized to execute the business logic. For example:
@pg.functor()
def foo(x, y):
return x + y
f = f(x=pg.oneof(range(5)), y=pg.floatv(0.0, 2.0))
# Error: `x` and `y` are not materialized, thus the functor cannot be
# executed.
f()
# But we can symbolically manipulated it:
f.rebind(x=1, y=2)
# Returns 3
f()
Symbolic placeholding is achieved by the pg.CustomTyping
interface, which was intended as a mechanism to extend the typing system
horizontally without modifying the pg.ValueSpec
of existing
classes’ attributes.
This is done by allowing the CustomTyping subclasses to take over value
validation and transformation from an attribute’s original ValueSpec
via pg.CustomTyping
method.
Following is an example of using CustomTyping to extend the schema system:
class FloatTensor(pg.typing.CustomTyping):
def __init__(self, tensor):
self._tensor = tensor
def custom_apply(
self, path, value_spec, allow_partial, child_transform):
if isinstane(value_spec, pg.typing.Float):
# Validate initial tensor, we can also add an TF operator to guard
# tensor value to go beyond value_spec.min_value and max_value.
value_spec.apply(self._tensor.numpy())
# Shortcircuit .apply and returns object itself as final value.
return (False, self)
else:
raise ValueError('FloatTensor can only be applied to float type.')
@pg.members([
('x', pg.typing.Float())
])
class Foo(pg.Object):
pass
# FloatTensor can be accepted for all symbolic attributes
# with Float value spec.
f = Foo(x=FloatTensor(tf.constant(1.0)))
Objects¶
Classes¶
- Annotated
- Any
- Argument
- Bool
- Callable
- CallableType
- CallableWithOptionalKeywordArgs
- ConstStrKey
- CustomTyping
- Dict
- Enum
- Field
- Float
- ForwardRef
- Functor
- Generic
- GenericMeta
- GenericTypeAlias
- Int
- KeySpec
- List
- ListKey
- MissingValue
- NonConstKey
- Number
- Object
- Optional
- PrimitiveType
- Schema
- Sequence
- Signature
- Str
- StrKey
- Tuple
- TupleKey
- Type
- Union
- ValueSpec