Making/compiling C functions for use in Julia on Windows
I often find it harder to do development work on Windows as it doesn't come with essential developer tools as out-of-box standard (an example is R's excellent Rtools bundle which helps alleviate the problem in the R world). In fact, I struggled with getting Julia's ccall
to work on Windows for a while, so I have decided to write the steps that I followed to make it work.
This post is meant to be a guide for complete beginners like I was at the beginning of this process.
Calling C is supposed to be easy
Julia was designed to be able to call C function easily. Indeed the official manual states that
To allow easy use of this existing code, Julia makes it simple and efficient to call C and Fortran functions. Julia has a "no boilerplate" philosophy: functions can be called directly from Julia without any "glue" code, code generation, or compilation – even from the interactive prompt. This is accomplished just by making an appropriate call with ccall syntax, which looks like an ordinary function call.
I found that this is true: but only after jumping through a few undocumented hoops in Windows. To be fair, this is not something that Julia should document, as it's probably "common knowledge" to Windows software developers and is not specific to Julia.
Choice of compiler
Julia's official manual mentions gcc
as the C compiler to use and that's what I have set up. The compiler gcc
is an open source C compiler (it can compile other languages as well). The Microsoft VC compiler is also available. However, I am not familiar with any of these tools, so I will detail how I made it work for me, and it's by no means the only way.
What you need to do to compile C functions on Windows
- Set up environment variable
LD_LIBRARY_PATH
(google: how to set up environment variable) and set it to a path that you have access to, e.g.d:/c_lib
.- This is where Julia will look for compiled C libraries (
clibname
). - As a number of Julians have pointed out, the
LD_LIBRARY_PATH
may contain multiple paths, sojoinpath(env_path, clibname)
may not work; and you may have to manually extract the appropriate path you need, e.g. use regex. - The other way is to have the .dll file in the same folder as julia.exe or provide the full path to the variable in the form of a
const
string in place ofclibname
in theccall
call
- This is where Julia will look for compiled C libraries (
- Install MinGW e.g from this link
- I have tried some other MinGW installations and some of them didn't work, the above link contains a MinGW distribution that I have tested to work on Windows 10 Pro 64 bit machine
- In the folder of where MinGW is installed you will find a
bin
folder. Add the path to thebin
folder to your PATH environment variable
Now you are ready to write some C code and compile it in Julia, see example below. The trick is something I learned from Julia Discourse member @Liso
# write the C code inside a raw string
C_code = raw"""
#include <stdlib.h> //size_t is defined in stdlib.h now
int add2(size_t a, size_t b){
return a+b;
}
"""
# obtain the environment variable
env_path = ENV["LD_LIBRARY_PATH"]
# your compiled C code must have a library name
# it MUST be declared as a `const` for `ccall` to work
const clibname = "addc"
# the path to store the C library file
const Clib = joinpath(env_path,clibname)
# compile the C code into a shared library
open(`gcc -fPIC -O3 -msse3 -xc -shared -o $(Clib * "." * Libdl.dlext) -`, "w") do f
print(f, C_code)
end
# define a Julia function to call the C library
add2jl(x, y) = ccall(
(:add2, clibname), # ("function_name", library)
Int, # return type from the C library
(Cint, Cint), # input parameter type to the C function
x, y # actual values to pass to C function
)
# call C function via Julia
add2jl(UInt(80), UInt(8))
add2jl(80,8)
Things to watch out for
- The
Clib
must be declared as a constant or it won't work, see this SO post - Once you have used the function
add2jl
once, the addc.dll file that contains the compiled C library is now "locked" so you can't recompile or delete it. The only way I found so far to "free" the file is to close Julia.
Side note: why do you need C? Doesn't Julia solve the two language problem?
Sometimes you want to benchmark a C implementation vs. a pure Julia one.
For my use case, I found the C implementation of string sort to be more performant and so I can call a quick C function instead as it has efficient sorting algorithms already coded up. In this case, Julia didn't solve the two-language problem as I wanted more performance than what I can get out of Julia at the moment, and there is an efficient C implementation already. I guess that my use-case is one of the reasons why the Julia devs made calling C functions is so painless in Julia.
Be aware that the
LD_LIBRARY_PATH
is almost always a list of directories (on Linux), separated by:
.Really hoping that those that has multiple paths in LD_LIBRARY_PATH don’t need to read this. It’s more for people where it is empty, like it was for me.
I thought from the thread (https://discourse.julialang.org/t/how-to-make-a-c-function-compiled-by-myself-available-to-ccall/7972/39) your Julia
radixsort!
was eventually the most efficient?only for strings of fixed size 8 or less. i will post my findings soon.
I see. Have you done a conversion of the code to Julia yet? Or is it just a complicated function?
“simple” function. just a quicksort actually. i think Julia has slowish access to underlying bytes and slow string reaasignments. Radixsort based method in Julia is already faster than data.table for integers. String-sort is the hard case.
I see. That’s one of the things Scott has been asking about with the memory buffer implementation. I know nothing about strings so I’ll just say I hope you work it out.