Build Swift Executable with Metal Library

June 18, 2022

When you build a Swift Package Xcode will compile all your .swift files to single executable file and process all other resources which includes Assets catalogue, Core Data files, Storyboards and many more including all .metal files. All these resources will be put in package Bundle and you have access to them through Bundle.module if you add them to resources array in package’s target.

But it will work only for library product type. If you want to build a Swift executable product resources won’t be processed or copied to Bundle because there is no actual Bundle there. So Swift executable is limited to only .swift files so you can’t build executable which will run you Metal shaders. Let’s try to fix it!

Metal allows offline library compilation and library initialisation from URL with MTLDevice instance method

func makeLibrary(URL url: URL) throws -> MTLLibrary

This method requires a valid URL to .metallib file so we need a way to build Metal Library from our .metal files, put it somewhere near our executable file and create library on the fly from it.

Let’s do it using shell script! You can do it manually on you Mac or run shell script on your CI assuming that CI is also a Mac server.

You can get the whole script here or go step by step with me.

Create intermediate representation from Metal files

First of all we need to locate all our .metal files and create .air file from each of them

#!/bin/bash

METAL_FILES=`ls /path/to/metal/files/*.metal`

for path in $METAL_FILES
do
  FILE_NAME=${path%%.*}
  xcrun -sdk macosx metal -c $path -o ${FILE_NAME}.air
done

FILE_NAME here will be the name of Metal file but without .metal extension. xcrun command will build .air file from each .metal file and put in the same folder where all your Metal files exist. We will remove them after we compile the whole library.

If you want to put .air files somewhere else just update -o flag to /path/to/air/files/${FILE_NAME}.air

Also if you have only one Metal file (assume it is named Shaders.metal) you can omit for-loop and run only

xcrun -sdk macosx metal -c Shaders.metal -o Shaders.air

Archive AIR files to single MetalAR file

The next step will be to archive all .air files to single .metalar file which will be used to compile the .metallib file later on. You can also omit this step if you have a single .metal file which you already compiled to single .air file. The Metal Library can be created from .air or .metalar file it doesn’t matter.

This step will look pretty like the previous but we will iterate through .air files this time

AIR_FILES=`ls /path/to/air/files/*.air`

for path in $AIR_FILES
do
  xcrun metal-ar r default.metalar $path
done

I used default.metalar name here like Xcode do when compiles Metal Library during package or app building. But you can use the name more appropriate for you of course. And you can put it anywhere you want if you replace default.metalar with /path/to/metalar/file/default.metalar.

metal-ar is the name of the tool which will archive .air files and r command will add the second file ($path in our case) to the first one and create it if needed.

So now we have a single .metalar file with all our Metal shaders in intermediate representation. Let’s build a Metal Library file from it.

Build Metal Library from MetalAR archive

This step is pretty straightforward. Again we need to run xcrun command to do the thing

xcrun -sdk macosx metallib default.metalar -o default.metallib

So now we have a .metallib file which is a good candidate to create MTLLibrary from. But the last but not least step is to be good citizens and clean up the shaders folder where we created all our intermediate files.

Clean up

rm default.metalar

for path in $AIR_FILES
do
  rm $path
done

Create MTLLibrary from .metallib file

When you archive you Swift executable Xcode will export it as a folder with your single executable file in Products/usr/local/bin folder inside. You can move it anywhere you want and run it from shell using path to executable file or if you add the executable location to $PATH you can run it from anywhere in your system. But we need a way to tell the executable file where our .metallib is located.

There are two ways to do it. If you create Swift executable you are already familiar with Apple’s Swift Argument Parser framework which is must-have dependency for Swift executables. So you can create an @Option for Metal Library file location which will be required for you script or if you build an executable for internal usage you can assume that .metallib file will be placed near your executable file and create library from it.

There is one thing you need to notice if you use the second approach. If you use

FileManager.default.currentDirectoryPath

in your Swift code it will return you not the executable’s location but the current location from where you run your executable. So if you open the Terminal and run

/path/to/your/swift/executable

your current directory will be your user’s home directory and FileManager won’t be able to locate .metallib file which is placed near your executable but not in your home folder.

To get access to executable’s location you need to get process info from inside your run() method in Swift file. So the valid approach to locate .metallib file which is placed near executable will be

func run() throws {
    guard let execPath = ProcessInfo.processInfo.arguments.first,
          let device = MTLCreateSystemDefaultDevice()
    else { return }

    let execURL = URL(fileURLWithPath: execPath)
    let execRootURL = execURL.deletingLastPathComponent()
    let libraryURL = execRootURL.appendingPathComponent("default.metallib")
    let library = try device.makeLibrary(URL: libraryURL)
    // PROFIT!
}

execPath and execURL here will be the full path to executable itself so we need to remove its name from the URL to get the path where it is located and where we put our .metallib file.

Conclusion

Now you’re not limited by Swift only files inside Swift executable and can build an executable which can run Metal Shaders right from the command line.

I personally used this flow to create the Swift Package with two products. The first one is a plain old Swift library which I can add as dependency to iOS apps and the second one is executable which has the first library as dependency and can be used to run the whole library functionality from command line. We use it to simplify library tests and to make the library available to run from Python scripts for example.

Here is the gist with all the previous shell code. I hope it will help you in creating awesome Swift packages with CLI availability.

If you have any questions feel free to shoot me an email to contact@mtldoc.com or connect via Twitter @petertretyakov.