Interfacing to C, Fortran, and Python

Questions

Why Julia interfacing with other languages?
How interfacing Julia with compiled languages (C and Fortran)?
How interfacing Julia with Python and vice versa?

Instructor note

20 min teaching
20 min exercises

Why Julia interfacing with other languages?

One of the most significant advantages of Julia is its speed. As we have shown in the Episode Motivation, Julia is fast out-of-box without the necessity to do any additional steps. As such, Julia solves the so-called two-language problem.

Since Julia is fast enough, most of the libraries are written in pure Julia, and there is no need to use C or Fortran for performance. However, there are many high-quality, mature libraries for numerical computing already written in C and Fortran. It would be resource-wasting if it is not possible to use them in Julia.

In fact, to allow easy use of existing C and Fortran code, Julia has native support for calling C and Fortran functions. Julia has a no boilerplate philosophy: functions can be called directly from Julia without any glue code generation or compilation – even from the interactive prompt.

In this episode, we will show examples of Julia interfacing with C and Fortran. Extensive description of all provided functionality can be found in the official manual.

Interfacing Julia with C

The interfacing of Julia with C and Fortran is accomplished by making an appropriate call with the ccall syntax, which looks like an ordinary function call.

The C and Fortran code to be called must be available as a shared library. Most C and Fortran libraries ship compiled as shared libraries already. However, if you want to compile the code yourself using GCC (or Clang), you will need to use the -shared and -fPIC options. The machine instructions generated by Julia’s JIT are the same as a native C call would be, so the resulting overhead is the same as calling a library function from C code.

By default, Fortran compilers generate mangled names (for example, converting function names to lowercase or uppercase, often appending an underscore), and so to call a Fortran function you must pass the mangled identifier corresponding to the rule followed by your Fortran compiler. When calling a Fortran function, all inputs must be passed as pointers to allocated values on the heap or stack. This applies not only to arrays and other mutable objects which are normally heap-allocated, but also to scalar values such as integers and floats which are normally stack-allocated and commonly passed in registers when using C or Julia calling conventions.

When calling C and Fortran functions, the name of the function and the library it lives in are passed as a tuple in the first argument, followed by the return type of the function and the types of the function arguments, and finally the argument themselves. It’s a bit klunky, but it works!

Here is one example to calculate the square root of a number (herein, it is 64.0).

ccall((:sqrt, "libm"), Float64, (Float64,), 64.0)

It also makes sense to wrap a call like that in a native Julia function.

csqrt(x) = ccall((:sqrt, "libm"), Float64, (Float64,), x);

csqrt(81.0)

The following example is adapted from Calling C from Julia by The Craft of Coding. Let’s conside the following C function which computes the mean from an array of 64-bit integer values. We will name the file as mean.c.

double mean(long *arr, long n)
{
    long i, sum=0;
    double mean;
    for (i=0; i<n; i=i+1)
        sum = sum + arr[i];
    mean = sum / (double)n;
    return mean;
}

Next, we need to compile the code in mean.c into a shared object named as mean.so. We use the GNU C compiler (GCC) with the flags -Wall to enable warnings, -fpic to make the shared object relocatable and -shared to produce a shared object. A collection of shared objects is usually referred to as a library.

gcc -Wall -fpic -shared -o mean.so mean.c

Now, we can call the shared object from Julia using the ccall function as follows:

# Define the array in Julia
arr = [1,2,3,4,5]

# Length of the array
n = length(arr)

# We need to convert the inputs because Julia integer type can be 32 or 64-bit
# depending on the system.
arr_c = convert(Vector{Clong}, arr)
n_c = convert(Clong, length(arr))

# Call the shared library
ccall((:mean, "./mean.so"), Cdouble, (Ptr{Clong}, Clong), arr_c, n_c)

We can also create a wrapper function for convenient access to the function as follows:

function mean(arr::Vector{Int64}, n::Int64)
    ccall((:mean, "./mean.so"), Cdouble, (Ptr{Clong}, Clong), arr, n)
end

function mean(arr::Vector{Integer})
    mean(convert(Vector{Clong}, arr), convert(Clong, length(arr)))
end

Interfacing Julia with Fortran

During the compilation, the Fortran compilers usually generate mangled names by appending an underscore to the lowercased/uppercased function names. Therefore, if you want to call a Fortran function using Julia, you must pass the mangled identifier corresponding to the rule followed by your Fortran compiler. In addition, all inputs must be passed by reference when calling a Fortran function.

Below we provide an example for interfacing Julia with Fortran.

# fortran_julia.f90

module fortran_julia
   implicit none
   public
   contains

   real(8) function add(a, b)
      implicit none
      real(8), intent(in)  :: a, b
      add = a + b
      return
   end function add

   subroutine addsub(x, y, a, b)
      implicit none
      real(8), intent(out) :: x, y
      real(8), intent(in)  :: a, b
      x = a + b
      y = a - b
      return
   end subroutine addsub

   subroutine concatenate(x, a, b)
      implicit none
      character(*), intent(out) :: x
      character(*), intent(in)  :: a, b
      x = a // b
      return
   end subroutine concatenate

   subroutine add_array(x, a, b, n)
      implicit none
      integer, intent(in)  :: n
      real(8), intent(out) :: x(n)
      real(8), intent(in)  :: a(n), b(n)
      x = a + b
      return
   end subroutine add_array

end module fortran_julia

Then we compile the code fortran_julia.f90 into a shared object named as fortran_julia.so.

gfortran fortran_julia.f90 -O3 -shared -fPIC -o fortran_julia.so

Next we can call the shared object from Julia using the ccall function:

ccall((:__fortran_julia_MOD_add, "fortran_julia.so"), Float64, (Ref{Float64}, Ref{Float64}), 1.1, 3.5)
# 4.6

In addition, the add function in the Fortran module can be further wrapped in the following Julia function to simplify the calling convention.

function add(a::Float64, b::Float64)
    ccall((:__fortran_julia_MOD_add, "fortran_julia.so"), Float64, (Ref{Float64}, Ref{Float64}), a, b)
end
# add (generic function with 1 method)

add(6.7, 3.9)
# 10.6

Calling a Fortran subroutine is similar to calling a Fortran function. In fact, the subroutine in Fortran can be regarded as a special function, and its return value is void (corresponding to the Nothing type in Julia). Here is another Fortran wrapper example.

function addsub(a::Float64, b::Float64)
    x = Ref{Float64}()
    y = Ref{Float64}()
    ccall((:__fortran_julia_MOD_addsub, "fortran_julia.so"), Nothing, (Ref{Float64}, Ref{Float64}, Ref{Float64}, Ref{Float64}), x, y, a, b)
    x[], y[]
end
# addsub (generic function with 1 method)

addsub(5.9, 1.5)
# (7.4, 4.4)

The Fortran subroutine can pass the calculation results to the caller via modifying the values of input parameters. In this example, x and y are the output results to the caller. Therefore two pointers should be defined using Ref{Float64}()` and then passed to the Fortran subroutine. After calling the Fortran subroutine, we will use x[] and y[] to extract the results from the addresses the pre-defined pointers pointing to. The rest of the this process is similar as calling the Fortran function.

Here is another example to concatenate two strings via calling a Fortran subroutine.

function concatenate(a::String, b::String)
    x = Vector{UInt8}(undef, sizeof(a) + sizeof(b))
    ccall((:__fortran_julia_MOD_concatenate, "fortran_julia.so"), Nothing, (Ref{UInt8}, Ref{UInt8}, Ptr{UInt8}, UInt, UInt, UInt), x, Vector{UInt8}(a), b, sizeof(x), sizeof(a), sizeof(b))
    String(x)
end
# concatenate (generic function with 1 method)

concatenate("Hello ", "Julia!!!")
# "Hello Julia!!!"

Finally, we have the sample to passing to and fetching an output array from the Fortran subroutine.

function add_array(a::Array{Float64,1}, b::Array{Float64,1})
    x = Array{Float64,1}(undef, length(a))
    ccall((:__fortran_julia_MOD_add_array, "fortran_julia.so"), Nothing, (Ref{Float64}, Ref{Float64}, Ref{Float64}, Ref{UInt32}), x, a, b, length(x))
    x
end
# add_array (generic function with 1 method)

add_array([0.2, 1.3, 1.6, 4.6], [-1.8, -0.3, 1.1, 2.4])
# 4-element Vector{Float64}:
# -1.6
#  1.0
#  2.7
#  7.0

The fortran_julia.f90 file and a Jupyter notebook file (fortran_julia.ipynb) containing the above examples for interfacing Julia with Fortran are provided in the github repository.

Interfacing Julia with Python

Besides interfacing Julia with compiled languages like C and Fortran, it is also possible for Julia to have intensive interactions with interpreted languages, such as Python, which provide a powerful procedure to leverage the strengths of both languages.

Actually we have came to the interfacing of Julia with Python at the Setup section in the ENCCS lesson of Introduction to programming in Julia. We have demonstrated the creation of Jupyter notebooks in Julia using the IJulia package. The Jupyter notebooks support multiple languages, including Julia and Python. You can write Julia code in one cell and Python code in another, allowing seamless integration.

For specific interactions between Julia and Python, there are two formats, that is, you can call Python from Julia, and you can also call Julia from Python.

Calling Python from Julia

The “standard” way to call Python code in Julia is to use the PyCall package, which has nice features including:

It can automatically download and install a local copy of Python, private to Julia, in order to avoid messing with version dependency from the “main” Python installation and provide a consistent environment within Linux, Windows, and MacOS.
It imports a Python module and provides Julia wrappers for all functions and constants including automatic conversion of types between Julia and Python.
Type conversions are automatically performed for numeric, boolean, string, and I/O streams plus all tuples, arrays, and dictionaries of these types. Other types are converted to the generic PyObject type.

Before calling Python code from Julia, make sure you have PyCall installed in Julia

using Pkg
Pkg.add("PyCall")

Then you can use PyCall to import and call Python functions:

using PyCall
math = pyimport("math")
println(math.sin(math.pi / 4))

Embedding Python code in a Julia program is similar to what we saw with C and Fortran, except that you don’t need (for the most part) to worry about transforming data. You define and call the Python functions using py-strings (py"..."), and, in the function call, you can use your Julia data directly. Note that the py-strings are not part of the Julia itself: they are defined by the PyCall module.

py"""
def sumMyArgs(a,b):
    return a+b
def getNElement(n):
    c = [0,1,2,3,4,5]
    return c[n]
"""

py"sumMyArgs"(3,4)
# 7

py"sumMyArgs"([3,4],[5,6])
# 2-element Vector{Int64}:
#  8
# 10

py"sumMyArgs"([3,4],7)
# 2-element Vector{Int64}:
# 10
# 11

py"getNElement"(1)
# 1

It is noted that

you don’t need to convert complex data like arrays, and the results are automatically converted to Julia types
in the last line of the example that PyCall doesn’t attempt index conversion (Python arrays are zero-based while Julia arrays are one-based).

Calling the Python getNElement() function with 1 being the argument will retrieve what in Python is the first element of the array.

It is very easy to mix Julia and Python code. So if you like a developed module in Python, you can directly use it in Julia.

np = pyimport("numpy")
# PyObject <module 'numpy' from '/Users/XXX/.julia/conda/3/aarch64/lib/python3.10/site-packages/numpy/__init__.py'>

a = np.random.rand(2, 3)
# 2×3 Matrix{Float64}:
# 0.0558569  0.631385  0.109421
# 0.220353   0.547723  0.962298

exp_a = np.exp(a)
# 2×3 Matrix{Float64}:
# 1.05745  1.88021  1.11563
# 1.24652  1.72931  2.6177

(Optional) Calling Julia from Python

The other way around, embedding Julia code in a Python script or terminal, is equally of importance, as in many cases it provides substantial performance gains for Python programmers, and it may be easier than embedding C or Fortran code.

This is achieved using the PyJulia Python package, which is a Python interface to Julia (similar to PyCall being the Julia interface to Python).

Before installing PyJulia, be sure that the PyCall module is installed in Julia and that it is using the same Python version as the one from which you want to embed the Julia code.

Note

It should be noted that the name of the package in pip is julia, not PyJulia.

$ python3 -m pip install julia
$ python3
>>> import julia
>>> julia.install()
>>> jl = julia.Julia(compiled_modules=False)

Note

If you have multiple Julia versions, you can specify the one to use in Python by passing julia=”/path/to/julia/binary/executable” (e.g., julia = “/home/myUser/lib/julia-1.1.0/bin/julia”) to the julia.install() function.

Now you can now access to Julia in multiple ways. For example, you can define all your functions in a Julia script and “include” it. Herein we have a Julia script named as julia_for_python.jl, which contains the following Julia code:

function helloWorld()
   println("Hello World!")
end

function sumMyArgs(a, b)
   return a+b
end

function getNElement(n)
   c = [0,1,2,3,4,5,6,7,8,9]
   return c[n]
end

You can access these defined functions in Python with:

>>> jl = julia.Julia(compiled_modules=False)

>>> jl.include("julia_for_python.jl")
<PyCall.jlwrap getNElement>

>>> jl.helloWorld()
Hello World!

>>> jl.sumMyArgs([1, 2, 3], [4, 5, 6])
array([5, 7, 9], dtype=int64)

>>> jl.getNElement(1)
0

You can otherwise embed Julia code directly into Python using the Julia eval() function

jl.eval("""
function func_prod(is, js)
   prod = 0
   for i in 1:is
      for j in 1:js
         prod += 1
      end
   end
   return prod
end
""")

Then you can call this function in Python as

>>> jl.func_prod(2, 3)
6

It should be noted that if you want to run the function in broadcasted mode, i.e., apply the function for each element of a given array. In Julia, you could use the dot notation, e.g., func_prod.([2,3],[4,5]). But herein you will get an error as this is not a valid Python syntax. In cases like this, when you can’t simply calling a Julia function using Python syntax, you can still rely to the same Julia eval() function you used to define the Python function to call it:

>>> jl.eval("func_prod.([2,3],[4,5])")
array([ 8, 15], dtype=int64)

Finally, you can also access any module available in Julia with from julia import ModuleName, and in particular you can set and access global Julia variables using the Main module.

Interfacing Julia with other languages

In addition, it is also possible interfacing Julia with other programming languages using third-party packages. The following table shows an overview of those packages.

Language	Calling from Julia	Calling Julia
R	RCall.jl	JuliaCall
MATLAB	MATLAB.jl	Mex.jl
Java	JavaCall.jl	JuliaCaller

Moreover, other Julia packages provide Julia interface for some well-known libraries from other languages. As an example, we can mention ScikitLearn.jl, which provides an interface for the scikit-learn library from Python or the RDatasets.jl that provides an easy way to load famous R datasets.