Native Calling Interface

Call into dynamic libraries that follow the C calling convention

Getting Started

The simplest imaginable use of NativeCall would look something like this:

use NativeCall;
sub some_argless_function() is native('something'{ * }
some_argless_function();

The first line imports various traits and types. The next line looks like a relatively ordinary Perl 6 sub declaration—with a twist. We use the "native" trait in order to specify that the sub is actually defined in a native library. The platform-specific extension (e.g., '.so' or '.dll'), as well as any customary prefixes (e.g., 'lib') will be added for you.

The first time you call "some_argless_function", the "libsomething" will be loaded and the "some_argless_function" will be located in it. A call will then be made. Subsequent calls will be faster, since the symbol handle is retained.

Of course, most functions take arguments or return values—but everything else that you can do is just adding to this simple pattern of declaring a Perl 6 sub, naming it after the symbol you want to call and marking it with the "native" trait.

Changing names

Sometimes you want the name of your Perl subroutine to be different from the name used in the library you're loading. Maybe the name is long or has different casing or is otherwise cumbersome within the context of the module you are trying to create.

NativeCall provides a "symbol" trait for you to specify the name of the native routine in your library that may be different from your Perl subroutine name.

    unit module Foo;
    use NativeCall;
    our sub init() is native('foo'is symbol('FOO_INIT'{ * }

Inside of "libfoo" there is a routine called "FOO_INIT" but, since we're creating a module called Foo and we'd rather call the routine as Foo::init, we use the "symbol" trait to specify the name of the symbol in "libfoo" and call the subroutine whatever we want ("init" in this case).

Passing and Returning Values

Normal Perl 6 signatures and the returns trait are used in order to convey the type of arguments a native function expects and what it returns. Here is an example.

use NativeCall;
sub add(int32int32returns int32 is native("calculator"{ * }

Here, we have declared that the function takes two 32-bit integers and returns a 32-bit integer. Here are some of the other types that you may pass (this will likely grow with time).

int8 (int8_t in C, also used for char)
int16 (int16_t in C, also used for short)
int32 (int32_t in C, also used for int)
int64 (int64_t in C)
uint8 (uint8_t in C, also used for unsigned char)
uint16 (uint16_t in C, also used for unsigned short)
uint32 (uint32_t in C, also used for unsigned int)
uint64 (uint64_t in C)
long (long in C)
longlong (long long in C, at least 64-bit)
ulong (unsigned long in C)
ulonglong (unsigned long long in C, at least 64-bit)
num32 (float in C)
num64 (double in C)
Str (C string)
CArray[int32] (int* in C, an array of ints)
Pointer[void] (void* in C, can point to all other types)
bool (bool from C99)
size_t (size_t in C)
ssize_t (ssize_t in C)

Don't use Perl 6 native types like int or num, as they don't have to correspond to the local C equivalent (e.g., Perl 6's int can be 8 bytes but C's int is only 4 bytes).

Note that the lack of a returns trait is used to indicate void return type. Do not use the 'void' type anywhere except in the Pointer parameterization.

For strings, there is an additional "encoded" trait to give some extra hints on how to do the marshaling.

use NativeCall;
sub message_box(Str is encoded('utf8')) is native('gui'{ * }

To specify how to marshal string return types, just apply this trait to the routine itself.

use NativeCall;
sub input_box() returns Str is encoded('utf8'is native('gui'{ * }

Note that a null string can be passed by passing the Str type object; a null return will also be represented by the type object.

If the C function requires the lifetime of a string to exceed the function call, the argument must be manually encoded and passed as CArray[uint8]:

use NativeCall;
# C prototype is void set_foo(const char *) 
sub set_foo(CArray[uint8]) is native('foo'{ * }
# C prototype is void use_foo(void) 
sub use_foo() is native('foo'{ * } # will use pointer stored by set_foo() 
 
my $string = "FOO";
# The lifetime of this variable must be equal to the required lifetime of 
# the data passed to the C function. 
my $array = CArray[uint8].new($string.encode.list);
 
set_foo($array);
# ... 
use_foo();
# It's fine if $array goes out of scope starting from here. 

Basic use of Pointers

When the signature of your native function needs a pointer to some native type (int32, uint32, etc.) all you need to do is declare the argument is rw :

use NativeCall;
# C prototype is void my_version(int *major, int *minor) 
sub my_version(int32 is rwint32 is rwis native('foo'{ * }
my_version(my int32 $majormy int32 $minor); # Pass a pointer to 

Sometimes you need to get a pointer (for example, a library handle) back from a C library. You don't care about what it points to - you just need to keep hold of it. The Pointer type provides for this.

use NativeCall;
sub Foo_init() returns Pointer is native("foo"{ * }
sub Foo_free(Pointeris native("foo"{ * }

This works out OK, but you may fancy working with a type named something better than Pointer. It turns out that any class with the representation "CPointer" can serve this role. This means you can expose libraries that work on handles by writing a class like this:

    use NativeCall;
 
    class FooHandle is repr('CPointer'{
        # Here are the actual NativeCall functions. 
        sub Foo_init() returns FooHandle is native("foo"{ * }
        sub Foo_free(FooHandleis native("foo"{ * }
        sub Foo_query(FooHandleStrreturns int8 is native("foo"{ * }
        sub Foo_close(FooHandlereturns int8 is native("foo"{ * }
 
        # Here are the methods we use to expose it to the outside world. 
        method new {
            Foo_init();
        }
 
        method query(Str $stmt{
            Foo_query(self$stmt);
        }
 
        method close {
            Foo_close(self);
        }
 
        # Free data when the object is garbage collected. 
        submethod DESTROY {
            Foo_free(self);
        }
    }

Note that the CPointer representation can do nothing more than hold a C pointer. This means that your class cannot have extra attributes. However, for simple libraries this may be a neat way to expose an object oriented interface to it.

Of course, you can always have an empty class:

class DoorHandle is repr('CPointer'{ }

And just use the class as you would use Pointer, but with potential for better type safety and more readable code.

Once again, type objects are used to represent nulls.

Function Pointers

C libraries can expose pointers to C functions as return values of functions and as members of Structures like, e.g., structs and unions.

Example of invoking a function pointer "$fptr" returned by a function "f", using a signature defining the desired function parameters and return value:

    sub f() returns Pointer is native('mylib'{ * }
 
    my $fptr    = f();
    my &newfunc = nativecast(:(Strsize_t --> int32), $fptr);
 
    say newfunc("test"4);

Arrays

NativeCall has some support for arrays. It is constrained to work with machine-size integers, doubles and strings, sized numeric types, arrays of pointers, arrays of structs, and arrays of arrays.

Perl 6 arrays, which support amongst other things laziness, are laid out in memory in a radically different way to C arrays. Therefore, the NativeCall library offers a much more primitive CArray type, which you must use if working with C arrays.

Here is an example of passing a C array.

    sub RenderBarChart(Strint32CArray[Str], CArray[num64]) is native("chart"{ * }
    my @titles := CArray[Str].new;
    @titles[0]  = 'Me';
    @titles[1]  = 'You';
    @titles[2]  = 'Hagrid';
    my @values := CArray[num64].new;
    @values[0]  = 59.5e0;
    @values[1]  = 61.2e0;
    @values[2]  = 180.7e0;
    RenderBarChart('Weights (kg)'3@titles@values);

Note that binding was used to @titles, not assignment! If you assign, you are putting the values into a Perl 6 array, and it will not work out. If this all freaks you out, forget you ever knew anything about the @ sigil and just use $ all the way when using NativeCall. :-)

use NativeCall;
my $titles = CArray[Str].new;
$titles[0= 'Me';
$titles[1= 'You';
$titles[2= 'Hagrid';

Getting return values for arrays works out just the same.

Some library APIs may take an array as a buffer that will be populated by the C function and, for instance, return the actual number of items populated:

use NativeCall;
sub get_n_ints(CArray[int32], int32returns int32 is native('ints'{ * }

In these cases it is important that the CArray has at least the number of elements that are going to be populated before passing it to the native subroutine, otherwise the C function may stomp all over Perl's memory leading to possibly unpredictable behaviour:

    my $ints = CArray[int32].new;
    my $number_of_ints = 10;
    $ints[$number_of_ints - 1= 0# extend the array to 10 items 
 
    my $n = get_n_ints($ints$number_of_ints);

The memory management of arrays is important to understand. When you create an array yourself, then you can add elements to it as you wish and it will be expanded for you as required. However, this may result in it being moved in memory (assignments to existing elements will never cause this, however). This means you'd best know what you're doing if you twiddle with an array after passing it to a C library.

By contrast, when a C library returns an array to you, then the memory can not be managed by NativeCall, and it doesn't know where the array ends. Presumably, something in the library API tells you this (for example, you know that when you see a null element, you should read no further). Note that NativeCall can offer you no protection whatsoever here - do the wrong thing, and you will get a segfault or cause memory corruption. This isn't a shortcoming of NativeCall, it's the way the big bad native world works. Scared? Here, have a hug. Good luck! :-)

Structs

Thanks to representation polymorphism, it's possible to declare a normal looking Perl 6 class that, under the hood, stores its attributes in the same way a C compiler would lay them out in a similar struct definition. All it takes is a quick use of the "repr" trait:

class Point is repr('CStruct'{
    has num64 $.x;
    has num64 $.y;
}

The attributes can only be of the types that NativeCall knows how to marshal into struct fields. Currently, structs can contain machine-sized integers, doubles, strings, and other NativeCall objects (CArrays, and those using the CPointer and CStruct reprs). Other than that, you can do the usual set of things you would with a class; you could even have some of the attributes come from roles or have them inherited from another class. Of course, methods are completely fine too. Go wild!

CStruct objects are passed to native functions by reference and native functions must also return CStruct objects by reference. The memory management rules for these references are very much like the rules for arrays, though simpler since a struct is never resized. When you create a struct, the memory is managed for you and when the variable(s) pointing to the instance of a CStruct go away, the memory will be freed when the GC gets to it. When a CStruct-based type is used as the return type of a native function, the memory is not managed for you by the GC.

NativeCall currently doesn't put object members in containers, so assigning new values to them (with =) doesn't work. Instead, you have to bind new values to the private members:

    class MyStruct is repr('CStruct'{
        has CArray[num64$!arr;
        has Str $!str;
        has Point $!point# Point is a user-defined class 
 
        submethod TWEAK {
            my $arr := CArray[num64].new;
            $arr[0= 0.9e0;
            $arr[1= 0.2e0;
            $!arr := $arr;
            $!str := 'Perl 6 is fun';
            $!point := Point.new;
        }
    }

As you may have predicted by now, a null is represented by the type object of the struct type.

CUnions

Likewise, it is possible to declare a Perl 6 class that stores its attributes the same way a C compiler would lay them out in a similar union definition; using the CUnion representation:

    use NativeCall;
 
    class MyUnion is repr('CUnion'{
        has int32 $.flags32;
        has int64 $.flags64;
    }
 
    say nativesizeof(MyUnion.new);  # 8, ie. max(sizeof(MyUnion.flags32), sizeof(MyUnion.flags64)) 

Embedding CStructs and CUnions

CStructs and CUnions can be in turn referenced by—or embedded into—a surrounding CStruct and CUnion. To say the former we use has as usual, and to do the latter we use the HAS declarator instead:

    class MyStruct is repr('CStruct'{
        has Point $.point;  # referenced 
        has int32 $.flags;
    }
 
    say nativesizeof(MyStruct.new);  # 16, ie. sizeof(struct Point *) + sizeof(int32_t) 
 
    class MyStruct2 is repr('CStruct'{
        HAS Point $.point;  # embedded 
        has int32 $.flags;
    }
 
    say nativesizeof(MyStruct2.new);  # 24, ie. sizeof(struct Point) + sizeof(int32_t) 

Typed Pointers

TBD more

You can type your Pointer by passing the type as a parameter. It works with the native type but also with CArray and CStruct defined types. NativeCall will not implicitly allocate the memory for it even when calling new on them. It's mostly useful in the case of a C routine returning a pointer, or if it's a pointer embedded in a CStruct.

You have to call .deref on it to access the embedded type.

    my Pointer[int32$p#For a pointer on int32; 
    my Pointer[MyCstruct$p2 = some_c_routine();
    my MyCstruct $mc = $p2.deref;
    say $mc.field1;

It's quite common for a native function to return a pointer to an array of elements. Typed pointers can be dereferenced as an array to obtain individual elements.

    my $n = 5;
    # returns a pointer to an array of length $n 
    my Pointer[Point$plot = some_other_c_routine($n);
    # display the 5 elements in the array 
    for 1 .. $n -> $i {
        my $x = $plot[$i - 1].x;
        my $y = $plot[$i - 1].y;
        say "$i: ($x$y)";
    }

Pointers can also be updated to reference successive elements in the array:

    my Pointer[Point$elem = $plot;
    # show differences between successive points 
    for 1 ..^ $n {
        my Point $lo = $elem.deref;
        ++$elem# equivalent to $elem = $elem.add(1); 
        my Point $hi = (++$elem).deref;
        my $dx = $hi.x = $lo.x;
        my $dy = $hi.y = $lo.y;
        say "$_: delta ($dx$dy)";
    }

Buffers and Blobs

TBD

Function arguments

NativeCall also supports native functions that take functions as arguments. One example of this is using function pointers as callbacks in an event-driven system. When binding these functions via NativeCall, one need only provide the equivalent signature as a constraint on the code parameter:

use NativeCall;
# void SetCallback(int (*callback)(const char *)) 
my sub SetCallback(&callback (Str --> int32)) is native('mylib'{ * }

Note: the native code is responsible for memory management of values passed to Perl 6 callbacks this way. In other words, NativeCall will not free() strings passed to callbacks.

Library Paths and Names

TBD more

The native trait accepts the library name or the full path.

use NativeCall;
constant LIBMYSQL = 'mysqlclient';
constant LIBFOO = '/usr/lib/libfoo.so.1';
# and later 
sub mysql_affected_rows returns int32 is native(LIBMYSQL{*};
sub bar is native(LIBFOO{*}

You can also put an incomplete path like './foo' and NativeCall will automatically put the right extension according to the platform specification.

BE CAREFUL: the native trait and constant are evaluated at compile time. Don't write a constant that depends on a dynamic variable like:

# WRONG: 
constant LIBMYSQL = %*ENV<P6LIB_MYSQLCLIENT> || 'mysqlclient';

This will keep the value given at compile time. A module will be precompiled and LIBMYSQL will keep the value it acquires when the module gets precompiled.

ABI/API Version

If you write native('foo') NativeCall will search libfoo.so under Unix like system (libfoo.dynlib on OS X, foo.dll on win32). In most modern system it will require you or the user of your module to install the development package because it's recommended to always provide an API/ABI version to a shared library, so libfoo.so ends often being a symbolic link provided only by a development package.

To avoid that, the native trait allows you to specify the API/ABI version. It can be a full version or just a part of it. (Try to stick to Major version, some BSD code does not care for Minor.)

use NativeCall;
sub foo1 is native('foo'v1{*} # Will try to load libfoo.so.1 
sub foo2 is native('foo'v1.2.3) {*} # Will try to load libfoo.so.1.2.3 
 
my List $lib = ('foo''v1');
sub foo3 is native($lib{*}

Routine

The native trait also accepts a Callable as argument, allowing you to provide your own way to handle the way it will find the library file to load.

use NativeCall;
sub foo is native(sub {'libfoo.so.42'}{*}

It will only be called at the first invocation of the sub.

Calling into the standard library

If you want to call a C function that's already loaded, either from the standard library or from your own program, you can pass the Str type object as the argument to is native, so is native(Str).

For example on a UNIX-like operating system, you could use the following code to print the home directory of the current user:

use NativeCall;
my class PwStruct is repr('CStruct'{
    has Str $.pw_name;
    has Str $.pw_passwd;
    has uint32 $.pw_uid;
    has uint32 $.pw_gid;
    has Str $.pw_gecos;
    has Str $.pw_dir;
    has Str $.pw_shell;
}
sub getuid()              returns uint32   is native(Str{ * };
sub getpwuid(uint32 $uidreturns PwStruct is native(Str{ * };
 
say getpwuid(getuid()).pw_dir;

Though of course $*HOME is a much easier way :-)

Exported variables

Variables exported by a library – also named "global" or "extern" variables – can be accessed using cglobal. For example:

    my $var := cglobal('libc.so.6''errno'int32)

This code binds to $var a new Proxy object that redirects all its accesses to the integer variable named "errno" as exported by the "libc.so.6" library.

C++ Support

NativeCall offers support to use classes and methods from C++ as shown in https://github.com/rakudo/rakudo/blob/nom/t/04-nativecall/13-cpp-mangling.t (and its associated C++ file). Note that at the moment it's not as tested and developed as C support.

Helper Functions

The NativeCall library exports several subroutines to help you work with data from native libraries.

sub nativecast

sub nativecast($target-type$sourceis export(:DEFAULT)

This will cast the Pointer $source to an object of $target-type. The source pointer will typically have been obtained from a call to a native subroutine that returns a pointer or as a member of a struct, this may be specified as void * in the C library definition for instance, but you may also cast from a pointer to a less specific type to a more specific one.

As a special case, if a Signature is supplied as $target-type then a subroutine will be returned which will call the native function pointed to by $source in the same way as a subroutine declared with the native trait. This is described in Function Pointers.

sub cglobal

sub cglobal($libname$symbol$target-typeis export is rw

This returns a Proxy object that provides access to the extern named $symbol that is exposed by the specified library. The library can be specified in the same ways that they can be to the native trait.

sub nativesizeof

sub nativesizeof($objis export(:DEFAULT)

This returns the size in bytes of the supplied object, it can be thought of as being equivalent to sizeof in C. The object can be a builtin native type such as int64 or num64, a CArray or a class with the repr CStruct, CUnion or CPointer.

Examples

Some examples can be found in the DBIsh repository.

MySQL

You'll need to install MySQL server locally; on Debian-esque systems it can be installed with something like:

    sudo apt-get install mysql-server

Prepare your system along these lines before trying out the examples:

    $ mysql -u root -p
    UPDATE mysql.user SET password=password('sa'WHERE user = 'root';
    CREATE DATABASE test;

Microsoft Windows

Here is an example of a Windows API call:

    use NativeCall;
 
    sub MessageBoxA(int32StrStrint32)
        returns int32
        is native('user32')
        { * }
 
    MessageBoxA(0"We have NativeCall""ohai"64);