Group of named regexes that form a formal grammar

Grammars are a powerful tool used to destructure text and often to return data structures that have been created by interpreting that text.

For example, Perl 6 is parsed and executed using a Perl 6-style grammar.

An example that's more practical to the common Perl 6 user is the JSON::Tiny module, which can deserialize any valid JSON file, however the deserializing code is written in less than 100 lines of simple, extensible code.

If you didn't like grammar in school, don't let that scare you off grammars. Grammars allow you to group regexes, just as classes allow you to group methods of regular code.

Named Regexes

The main ingredient of grammars is named regexes. While the syntax of Perl 6 Regexes is outside the scope of this document, named regexes have a special syntax, similar to subroutine definitions:[1]

my regex number { \d+ [ \. \d+ ]? }

In this case, we have to specify that the regex is lexically scoped using the my keyword, because named regexes are normally used within grammars.

Being named gives us the advantage of being able to easily reuse the regex elsewhere:

say "32.51" ~~ &number;
say "15 + 4.5" ~~ / <number> \s* '+' \s* <number> /

regex isn't the only declarator for named regexes -- in fact, it's the least common. Most of the time, the token or rule declarators are used. These are both ratcheting, which means that the match engine won't back up and try again if it fails to match something. This will usually do what you want, but isn't appropriate for all cases:

my regex works-but-slow { .+ q }
my token fails-but-fast { .+ q }
my $s = 'Tokens won\'t backtrack, which makes them fail quicker!';
say so $s ~~ &works-but-slow; # True
say so $s ~~ &fails-but-fast; # False, the entire string get taken by the .+

The only difference between the token and rule declarators is that the rule declarator causes :sigspace to go into effect for the Regex:

my token non-space-y { 'once' 'upon' 'a' 'time' }
my rule space-y { 'once' 'upon' 'a' 'time' }
say 'onceuponatime'    ~~ &non-space-y;
say 'once upon a time' ~~ &space-y;

Creating Grammars

Grammar is the superclass that classes automatically get when they are declared with the grammar keyword instead of class. Grammars should only be used to parse text; if you wish to extract complex data, an action object is recommended to be used in conjunction with the grammar.

Action Objects

A successful grammar match gives you a parse tree of Match objects, and the deeper that match tree gets, and the more branches in the grammar are, the harder it becomes to navigate the match tree to get the information you are actually interested in.

To avoid the need for diving deep into a match tree, you can supply an actions object. After each successful parse of a named rule in your grammar, it tries to call a method of the same name as the grammar rule, giving it the newly created Match object as a positional argument. If no such method exists, it is skipped.

Here is a contrived example of a grammar and actions in action:

use v6;

grammar TestGrammar {
    token TOP { ^ \d+ $ }

class TestActions {
    method TOP($/) {
        $/.make(2 + $/);

my $actions =;
my $match = TestGrammar.parse('40', :$actions);
say $match;         # 「40」
say $match.made;    # 42

An instance of TestActions is passed as named argument actions to the parse call, and when token TOP has matched successfully, it automatically calls method TOP, passing the match object as an argument.

To make it clear that the argument is a match object, the example uses $/ as a parameter name to the action method, though that's just a handy convention, nothing intrinsic. $match would have worked too. (Though using $/ does give the advantage of providing $<capture> as a shortcut for $/<capture> ).

A slightly more involved example follows:

use v6;

grammar KeyValuePairs {
    token TOP {
        [<pair> \n+]*
    token ws { \h* }

    rule pair {
        <key=.identifier> '=' <value=.identifier>
    token identifier {

class KeyValuePairsActions {
    method identifier($/) { $/.make: ~$/                          }
    method pair      ($/) { $/.make: $<key>.made => $<value>.made }
    method TOP       ($/) { $/.make: $<pair>».made                }

my  $res = KeyValuePairs.parse(q:to/EOI/, :actions(KeyValuePairsActions)).made;

for @$res -> $p {
    say "Key: $p.key()\tValue: $p.value()";

This produces the following output:

Key: second     Value: b
Key: hits       Value: 42
Key: perl       Value: 6

Rule pair, which parsed a pair separated by an equals sign, aliases the two calls to token identifier to separate capture names to make them available more easily and intuitively. The corresponding action method constructs a Pair object, and uses the .made property of the sub match objects. So it (like the action method TOP too) exploits the fact that action methods for submatches are called before those of the calling/outer regex. So action methods are called in post-order.

The action method TOP simply collects all the objects that were .made by the multiple matches of the pair rule, and returns them in a list.

Also note that KeyValuePairsActions was passed as a type object to method parse, which was possible because none of the action methods use attributes (which would only be available in an instance).

In other cases, action methods might want to keep state in attributes. Then of course you must pass an instance to method parse.

Note that token ws is special: when :sigspace is enabled (and it is when we are using rule), it replaces certain whitespace sequences. This is why the spaces around the equals sign in rule pair work just fine and why the whitespace before closing } does not gobble up the newlines looked for in token TOP.