A low-level explanation of Perl 6 containers

This article started as a conversation on IRC explaining the difference between the Array and the List type in Perl 6. It explains the levels of indirection involved in dealing with variables and container elements.

What is a variable?

Some people like to say "everything is an object", but in fact a variable is not a user-exposed object in Perl 6.

When the compiler encounters a variable declaration like my $x, it registers it in some internal symbol table. This internal symbol table is used to detect undeclared variables and to tie the code generation for the variable to the correct scope.

At run time, a variable appears as an entry in a lexical pad, or lexpad for short. This is a per-scope data structure that stores a pointer for each variable.

In the case of my $x, the lexpad entry for the variable $x is a pointer to an object of type Scalar, usually just called the container.

Scalar containers

Although objects of type Scalar are everywhere in Perl 6, you rarely see them directly as objects, because most operations decontainerize, which means they act on the Scalar container's contents instead of the container itself.

In code like

my $x = 42;
say $x;

the assignment $x = 42 stores a pointer to the Int object 42 in the scalar container to which the lexpad entry for $x points.

The assignment operator asks the container on the left to store the value on its right. What exactly that means is up to the container type. For Scalar it means "replace the previously stored value with the new one".

Note that subroutine signatures allow passing around of containers:

sub f($a is rw{
    $a = 23;
my $x = 42;
say $x;         # OUTPUT: «23␤» 

Inside the subroutine, the lexpad entry for $a points to the same container that $x points to outside the subroutine. Which is why assignment to $a also modifies the contents of $x.

Likewise a routine can return a container if it is marked as is rw:

my $x = 23;
sub f() is rw { $x };
f() = 42;
say $x;         # OUTPUT: «42␤» 

For explicit returns, return-rw instead of return must be used.

Returning a container is how is rw attribute accessors work. So

class A {
    has $.attr is rw;

is equivalent to

class A {
    has $!attr;
    method attr() is rw { $!attr }

Scalar containers are transparent to type checks and most kinds of read-only accesses. A .VAR makes them visible:

my $x = 42;
say $x.^name;       # OUTPUT: «Int␤» 
say $x.VAR.^name;   # OUTPUT: «Scalar␤» 

And is rw on a parameter requires the presence of a writable Scalar container:

sub f($x is rw{ say $x };
f 42;
CATCH { default { say .^name''.Str } };
# OUTPUT: «X::Parameter::RW: Parameter '$x' expected a writable container, but got Int value␤» 

Callable containers

Callable containers provide a bridge between the syntax of a Routine call and the actual call of the method CALL-ME of the object that is stored in the container. The sigil & is required when declaring the container and has to be omitted when executing the Callable. The default type constraint is Callable.

my &c = -> Int $a { say "$a is whole" }

The sigil has to be provided when referring to the value stored in the container. This in turn allows Routines to be uses as arguments to calls.

sub f() {}
my &g = sub {}
sub caller(&c1&c2){ c1c2 }


Next to assignment, Perl 6 also supports binding with the := operator. When binding a value or a container to a variable, the lexpad entry of the variable is modified (and not just the container it points to). If you write

my $x := 42;

then the lexpad entry for $x directly points to the Int 42. Which means that you cannot assign to it anymore:

my $x := 42;
$x = 23;
CATCH { default { say .^name''.Str } };
# OUTPUT: «X::AdHoc: Cannot assign to an immutable value␤» 

You can also bind variables to other variables:

my $a = 0;
my $b = 0;
$a := $b;
$b = 42;
say $a;         # OUTPUT: «42␤» 

Here, after the initial binding, the lexpad entries for $a and $b both point to the same scalar container, so assigning to one variable also changes the contents of the other.

You've seen this situation before: it is exactly what happened with the signature parameter marked as is rw.

Sigilless variables also bind by default and so do parameters with the trait is raw.

my $a = 42;
my \b = $a;
say $a;         # OUTPUT: «43␤» 
sub f($c is raw{ $c++ }
say $a;         # OUTPUT: «44␤» 

Scalar containers and listy things

There are a number of positional container types with slightly different semantics in Perl 6. The most basic one is List It is created by the comma operator.

say (123).^name;    # OUTPUT: «List␤» 

A list is immutable, which means you cannot change the number of elements in a list. But if one of the elements happens to be a scalar container, you can still assign to it:

my $x = 42;
($x12)[0= 23;
say $x;                 # OUTPUT: «23␤» 
($x12)[1= 23;     # Cannot modify an immutable value 
CATCH { default { say .^name''.Str } };
# OUTPUT: «X::Assignment::RO: Cannot modify an immutable Int␤» 

So the list doesn't care about whether its elements are values or containers, they just store and retrieve whatever was given to them.

Lists can also be lazy, so elements at the end are generated on demand from an iterator.

An Array is just like a list, except that it forces all its elements to be containers, which means that you can always assign to elements:

my @a = 123;
@a[0= 42;
say @a;         # OUTPUT: «[42 2 3]␤» 

@a actually stores three scalar containers. @a[0] returns one of them, and the assignment operator replaces the integer value stored in that container with the new one, 42.

Assigning and binding to array variables

Assigning to a scalar variable and to an array variable both do basically the same thing: discard the old value(s), and enter some new value(s).

Nevertheless, it's easy to observe how different they are:

my $x = 42say $x.^name;   # OUTPUT: «Int␤» 
my @a = 42say @a.^name;   # OUTPUT: «Array␤» 

This is because the Scalar container type hides itself well, but Array makes no such effort. Also assignment to an array variable is coercive, so you can assign a non-array value to an array variable.

To place a non-Array into an array variable, binding works:

my @a := (123);
say @a.^name;               # OUTPUT: «List␤» 

Binding to array elements

As a curious side note, Perl 6 supports binding to array elements:

my @a = (123);
@a[0:= my $x;
$x = 42;
say @a;                     # OUTPUT: «[42 2 3]␤» 

If you've read and understood the previous explanations, it is now time to wonder how this can possibly work. After all, binding to a variable requires a lexpad entry for that variable, and while there is one for an array, there aren't lexpad entries for each array element (you cannot expand the lexpad at run time).

The answer is that binding to array elements is recognized at the syntax level and instead of emitting code for a normal binding operation, a special method (called BIND-KEY) is called on the array. This method handles binding to array elements.

Note that, while supported, one should generally avoid directly binding uncontainerized things into array elements. Doing so may produce counter-intuitive results when the array is used later.

my @a = (123);
@a[0:= 42;         # This is not recommended, use assignment instead. 
my $b := 42;
@a[1:= $b;         # Nor is this. 
@a[2= $b;          # ...but this is fine. 
@a[12:= 12;    # runtime error: X::Bind::Slice 
CATCH { default { say .^name''.Str } };
# OUTPUT: «X::Bind::Slice: Cannot bind to Array slice␤» 

Operations that mix Lists and Arrays generally protect against such a thing happening accidentally.

Flattening, items and containers

The % and @ sigils in Perl 6 generally indicate multiple values to an iteration construct, whereas the $ sigil indicates only one value.

my @a = 123;
for @a { };         # 3 iterations 
my $a = (123);
for $a { };         # 1 iteration 

@-sigiled variables do not flatten in list context:

my @a = 123;
my @b = @a45;
say @b.elems;               # OUTPUT: «3␤» 

There are operations that flatten out sublists that are not inside a scalar container: slurpy parameters (*@a) and explicit calls to flat:

my @a = 123;
say (flat @a45).elems;  # OUTPUT: «5␤» 
sub f(*@x{ @x.elems };
say f @a45;             # OUTPUT: «5␤» 

As hinted above, scalar containers prevent that flattening:

sub f(*@x{ @x.elems };
my @a = 123;
say f $@a45;            # OUTPUT: «3␤» 

The @ character can also be used as a prefix to remove a scalar container:

my $x = (123);
.say for @$x;               # 3 iterations 

Methods generally don't care whether their invocant is in a scalar, so

my $x = (123);
$*.say);              # 3 iterations 

maps over a list of three elements, not of one.

Self-Referential Data

Containers types, including Array and Hash, allow you to create self-referential structures.

my @a;
@a[0= @a;
put @a.perl;
# OUTPUT: «((my @Array_75093712) = [@Array_75093712,])␤» 

Perl 6 does not prevent you from creating and using self-referential data; You may end up in a loop trying to dump the data; as a last resort, you can use Promises to handle timeouts.

Type Constraints

Any container can have a type constraint in the form of a type object or a subset. Both can be placed between a declarator and the variable name or after the trait of. The constraint is a property of the container, not the variable. Any (re-)binding may change the type constraint or remove the constraint altogether if bound to a value instead of a container. Introspection of type constraints on containers is provided by .VAR.of.

EVAL my Int $i = 42; $i := "forty plus two";;
CATCH { default { say .^name' '.Str } }
# OUTPUT: «Type check failed in binding; expected Int but got Str ("forty plus two")␤…» 

The default type constraint of a Scalar container is Any. For &-sigiled containers it's Callable and @-sigiled containers are of type Array. Please note that binding can change the type constraint.

Custom containers

To provide custom containers Perl 6 provides the class Proxy. It takes two methods that are called when values are stored or fetched from the container. Type checks are not done by the container itself and other restrictions like readonlyness can be broken. The returned value must therefore be of the same type as the type of the variable it is bound to. We can use type captures to work with types in Perl 6.

sub lucky(::T $type{
    my T $c-value# closure variable 
        FETCH => method () { $c-value },
        STORE => method (T $new-value{
   => 'number'got => '13'range => '-∞..12, 14..∞').throw
                if $new-value == 13;
            $c-value = $new-value;
my Int $a := lucky(Int);
say $a = 12;    # OUTPUT: «12␤» 
say $a = 'FOO'# X::TypeCheck::Binding 
say $a = 13;    # X::OutOfRange 
CATCH { default { say .^name''.Str } };