Building a System Root Directory

pyqtdeploy-sysroot is used to create a target-specific system root directory (sysroot) containing the target Python installation and any third-party components required by the application. Its use is optional but highly recommended.

pyqtdeploy-sysroot is actually a wrapper around a number of component plugins. A plugin, written in Python, is responsible for building and/or installing (in the sysroot) an individual component. It would often be a Python package or extension module but could just as easily be a supporting library.

A sysroot is defined by a JSON specification file. This contains an object for each component to build and/or install. The attributes of an object determine how the component is configured. Component and attribute names may be scoped in the same way as qmake variables (described in Adding Other Extension Modules) so that components can be included and configured on a target by target basis.

The components are built and/or installed in the order in which their objects appear in the specification file.

An API is provided to allow you to develop your own component plugins. If you develop a plugin for a commonly used component then please consider contributing it so that it can be included in a future release of pyqtdeploy.

Standard Component Plugins

The following component plugins are included as standard with pyqtdeploy.

openssl
This builds the OpenSSL libraries for v1.0.* and v1.1.* on Android (dynamically), macOS (statically) and Windows (statically). It requires perl to be installed on PATH.
pip
This is a meta-component which will install any number of components using pip.
pyqt3d
This builds a static version of the PyQt3D extension modules for all target architectures. It must be built after PyQt5.
pyqt5
This builds a static version of the PyQt5 extension modules for all target architectures. It must be built after sip and Qt5.
pyqtchart
This builds a static version of the PyQtChart extension module for all target architectures. It must be built after PyQt5.
pyqtdatavisualization
This builds a static version of the PyQtDataVisualization extension module for all target architectures. It must be built after PyQt5.
pyqtpurchasing
This builds a static version of the PyQtPurchasing extension module for all target architectures. It must be built after PyQt5.
pyqtwebengine
This builds a static version of the PyQtWebEngine extension module for all target architectures. It must be built after PyQt5.
python
This will build Python from source, or install it into sysroot from an existing installation, for both the host and target architectures. Building the host version from source is not supported on Windows. Installing the host version from an existing installation is not supported on Android or iOS. The target version of the Python library and extension modules built from source will be built statically. Installing the target version from an existing installation is only supported on Windows. If building the target version from source and SSL support is required then OpenSSL must be built first. If building Python v3.7.0 or later and zlib support is needed then zlib must be built first.
qscintilla
This builds a static version of the QScintilla library and Python extension module for all target architectures. It must be built after PyQt5.
qt5
This will build a static version of Qt5 from source (but not for the Android and iOS targets). It will also install Qt5 into the sysroot from an existing installation created by the standard installer. When building from source on Windows it requires Python v2.7 to be installed (but it does not need to be on PATH). If building from source and SSL support using OpenSSL is required then OpenSSL must be built first.
sip
This builds a static version of the sip extension module for all target architectures. It also builds the sip code generator for the host platform. It must be built after Python.
zlib
This builds a static version of the zlib library for all target architectures.

Creating a Sysroot Specification File

The following specification file contains an object for each of the standard component plugins. (You can also download a copy of the file from here). No configuration options have been set for any component.

{
    "Description": "A skeleton sysroot specification.",

    "openssl": {
    },

    "zlib": {
    },

    "qt5": {
    },

    "python": {
    },

    "sip": {
    },

    "pyqt5": {
    },

    "pyqt3d": {
    },

    "pyqtchart": {
    },

    "pyqtdatavisualization": {
    },

    "pyqtpurchasing": {
    },

    "pyqtwebengine": {
    },

    "qscintilla": {
    },

    "pip": {
    }
}

The first object, called Description, is simple a way of including a comment in the specification and is otherwise ignored.

Using this file, run the following command:

pyqtdeploy-sysroot --options skeleton.json

You will then see a description of each component’s configuration options, the type of value expected and whether or not a value is required. You can then add options as attributes to the appropriate objects to meet your requirements.

If your application does not require all of the standard components then simply remove the corresponding objects from the specification file. If your application requires additional components then you need to create appropriate component plugins and add corresponding objects to the specification file.

To build a native sysroot (i.e. for the host architecture) from a fully configured specification file, run:

pyqtdeploy-sysroot skeleton.json

The pyqt-demo Sysroot

In this section we walk through the sysroot specification file for pyqt-demo, component by component.

openssl

"android|macos|win#openssl": {
    "android#source":   "openssl-1.0.2r.tar.gz",
    "macos|win#source": "openssl-1.1.0j.tar.gz",
    "win#no_asm":       true
},

The first thing to notice is that the object name is scoped so that the component is only built for Android, macOS and Windows. On iOS we choose to not support SSL from Python and use Qt’s SSL support instead (which will use Apple’s Secure Transport). On Linux we will use the system versions of the OpenSSL libraries.

On Android we use OpenSSL v1.0 because that is the version used by the pre-built binaries provided by the Qt installer.

On macOS and Windows we choose to use OpenSSL v1.1.

On Windows we have disabled the used of assember optimisations. They can be enabled (with is the default) but the nasm assember must be installed on PATH.

zlib

"ios|linux|macos|win#zlib": {
    "source":               "zlib-1.2.11.tar.gz",
    "static_msvc_runtime":  true
},

On Android we are using the zlib library provided on the device. On other architectures we choose to use a static version of the library. On Windows we choose to link to static versions of the MSVC runtime libraries.

qt5

"qt5": {
    "android-32#qt_dir":        "android_armv7",
    "android-64#qt_dir":        "android_arm64_v8a",
    "ios#qt_dir":               "ios",

    "linux|macos|win#source":   "qt-everywhere-src-5.12.2.tar.xz",
    "edition":                  "opensource",

    "android|linux#ssl":        "openssl-runtime",
    "ios#ssl":                  "securetransport",
    "macos|win#ssl":            "openssl-linked",

    "configure_options":        [
            "-opengl", "desktop", "-no-dbus", "-qt-pcre"
    ],
    "skip":                     [
            "qtactiveqt", "qtconnectivity", "qtdoc", "qtgamepad",
            "qtlocation", "qtmultimedia", "qtnetworkauth",
            "qtquickcontrols", "qtquickcontrols2", "qtremoteobjects",
            "qtscript", "qtscxml", "qtsensors", "qtserialbus",
            "qtserialport", "qtspeech", "qtsvg", "qttools",
            "qttranslations", "qtwayland", "qtwebchannel", "qtwebengine",
            "qtwebsockets", "qtwebview", "qtxmlpatterns"
    ],

    "static_msvc_runtime":      true
},

The Qt5 component plugin will install Qt into the sysroot from an existing installation. We have chosen to do this for Android and iOS by specifying the qt_dir attribute. In the context of the demo this is defined by the --installed-qt-dir option of the build-demo.py script.

The plugin will build Qt from source if the source attribute is specified. We have chosen to do this for Linux, macOS and Windows.

The ssl attribute specifies how Qt’s SSL support is to be implemented.

For Android and Linux we have chosen to dynamically load external OpenSSL libraries at runtime. On Android the external libraries are those built by the openssl component plugin and are bundled automaticallly with the application executable. On Linux the external libraries are the system versions.

For iOS Qt is dynamically linked to the Secure Transport libraries.

For macOS and Windows we have chosen to link against the static OpenSSL libraries built by the openssl component plugin.

Next we use the configure_options and skip attributes to tailor the Qt build in order to reduce the time taken to do the build.

Finally we have specified that (on Windows) we will link to static versions of the MSVC runtime libraries.

python

"python": {
    "build_host_from_source":   false,
    "build_target_from_source": true,
    "source":                   "Python-3.7.2.tar.xz"
},

The Python component plugin handles installation for both host and target architectures. For the host we choose to use an existing Python installation rather than build from source. On Windows the registry is searched for the location of the existing installation. On Linux and macOS the Python interpreter must be on PATH. For all target architecures we choose to build Python from source.

Note that the version number of Python must match the version specified in the pyqt-deploy.pdy project file.

pyqt-demo is a very simple application that does not need to dynamically load extension modules. If this was needed then the dynamic_loading attribute would be set to true.

sip

"sip": {
    "module_name":  "PyQt5.sip",
    "source":       "sip-4.19.15.tar.gz"
},

As well as the name of the source archive, it is also necessary to specify the name of the sip module.

pyqt5

"pyqt5": {
    "android#disabled_features":    [
            "PyQt_Desktop_OpenGL", "PyQt_Printer", "PyQt_PrintDialog",
            "PyQt_PrintPreviewDialog", "PyQt_PrintPreviewWidget"
    ],
    "android#modules":              [
            "QtCore", "QtGui", "QtNetwork", "QtPrintSupport", "QtWidgets",
            "QtAndroidExtras"
    ],

    "ios#disabled_features":        [
            "PyQt_Desktop_OpenGL", "PyQt_MacOSXOnly",
            "PyQt_MacCocoaViewContainer", "PyQt_Printer",
            "PyQt_PrintDialog", "PyQt_PrintPreviewDialog",
            "PyQt_PrintPreviewWidget", "PyQt_Process",
            "PyQt_NotBootstrapped"
    ],
    "ios|macos#modules":            [
            "QtCore", "QtGui", "QtNetwork", "QtPrintSupport", "QtWidgets",
            "QtMacExtras"
    ],

    "linux#modules":                [
            "QtCore", "QtGui", "QtNetwork", "QtPrintSupport", "QtWidgets",
            "QtX11Extras"
    ],

    "win#disabled_features":        ["PyQt_Desktop_OpenGL"],
    "win#modules":                  [
            "QtCore", "QtGui", "QtNetwork", "QtPrintSupport", "QtWidgets",
            "QtWinExtras"
    ],

    "source":                   "PyQt5_*-5.12.1.tar.gz"
},

The two attributes used to tailor the build of PyQt5 are disabled_features and modules.

Unfortunately the list of features that can be disabled is not properly documented and relate to how Qt5 was configured. However how disabled_features is set in the above will be appropriate for most cases.

The modules attribute is used to specify the names of the individual PyQt extension modules to be built. We choose to build only those extension modules needed by pyqt-demo.

pyqt3D

"pyqt3d": {
    "source":   "PyQt3D_*-5.12.tar.gz"
},

It is only necessary to specifiy the name of the source archive.

pyqtchart

"pyqtchart": {
    "source":   "PyQtChart_*-5.12.tar.gz"
},

It is only necessary to specifiy the name of the source archive.

pyqtdatavisualization

"pyqtdatavisualization": {
    "source":   "PyQtDataVisualization_*-5.12.tar.gz"
},

It is only necessary to specifiy the name of the source archive.

pyqtpurchasing

"pyqtpurchasing": {
    "source":   "PyQtPurchasing_*-5.12.tar.gz"
},

It is only necessary to specifiy the name of the source archive.

qscintilla

"qscintilla": {
    "source":   "QScintilla_*-2.11.1.tar.gz"
}

It is only necessary to specifiy the name of the source archive.

The Command Line

The full set of command line options is:

-h, --help

This will display a summary of the command line options.

--component COMPONENT

COMPONENT is the name of the component (specified in the JSON file) that will be built. It may be used more than once to build multiple components. If the option is not specified then all components specified in the JSON file will be built.

--no-clean

A temporary build directory (called build in the sysroot) is created in order to build the required components. Normally this is removed automatically after all components have been built. Specifying this option leaves the build directory in place to make debugging component plugins easier.

--options

This causes the configurable options of each component specified in the JSON file to be displayed on stdout. The program will then terminate.

--plugin-dir DIR

DIR is added to the list of directories that are searched for component plugins. It may be used more than once to search multiple directories. All directories specified in this way will be searched before those directories (internal to pyqtdeploy-sysroot) that are searched by default.

--source-dir DIR

DIR is the name of a directory containing the source archives used to build the components specified in the JSON file. It may be specified any number of times and each directory will be searched in turn. If it is omitted then the current directory is searched.

--sysroot DIR

DIR is the name of the system root directory. The default value is sysroot- followed by a target-specific suffix. Unless the --component option is specified any existing sysroot will first be removed and re-created.

--target TARGET

TARGET is the target architecture. By default the host architecture is used. On Windows the default is determined by the target architecture of the currently configured compiler.

--quiet

This specifies that progress messages should be disabled.

--verbose

This specifies that additional progress messages should be enabled.

-V, --version

This specifies that the version number should be displayed on stdout. The program will then terminate.

specification

specification is the name of a JSON specification file that defines each component to be included in the sysroot and how each is to be configured.

Writing a Component Plugin

A component plugin is a Python module that defines a sub-class of pyqtdeploy.ComponentBase. The sub-class must re-implement the build() method and may also re-implement the configure() method. It should also include a class attribute called options which is a sequence of pyqtdeploy.ComponentOption instances that describe each of the component’s configurable options. It does not matter what the name of the class is.

class pyqtdeploy.ComponentBase

This is the base class of all component plugins.

options

This class attribute is a sequence of ComponentOption instances describing the component’s configurable options.

build(sysroot)

This abstract method is re-implemented to build the component.

Parameters:sysroot (Sysroot) – the sysroot being built.
configure(sysroot)

This method is re-implemented to configure the component. A component will always be configured even if it does not get built. The plugin should check that everything is available (e.g. source code, external tools) for a successful build.

Parameters:sysroot (Sysroot) – the sysroot being configured.
class pyqtdeploy.ComponentOption(name, type=str, required=False, default=None, values=None, help='')

This class implements an option used to configure the component. An option can be specified as an attribute of the component’s object in the sysroot specification file. An instance of the component plugin will contain an attribute for each option whose value is that specified in the sysroot specification file (or an appropriate default if it was omitted).

Parameters:
  • name (str) – the name of the option.
  • type (bool, int, list or str) – the type of a value of the option.
  • required (bool) – True if a value for the option is required.
  • default – the default value of the option.
  • values – the possible values of the option.
  • help (str) – the help text displayed by the --options option of pyqtdeploy-sysroot.
class pyqtdeploy.Sysroot

This class encapsulates a sysroot as seen by a component plugin. Instances are only created by pyqtdeploy-sysroot and are passed to the plugin when required.

add_to_path(name)

The name of a directory is added to the start of PATH if it isn’t already present.

Parameters:name (str) – is the name of the directory.
Returns:the original value of PATH.
android_api

The numerical Android API level to use.

android_ndk_sysroot

The path of the Android NDK’s sysroot directory.

android_ndk_version

New in version 2.4.

The 3-tuple version number of the Android NDK.

android_sdk_version

New in version 2.4.

The 3-tuple version number of the Android SDK.

android_toolchain_bin

The path of the Android toolchain’s bin directory.

android_toolchain_cc

New in version 2.4.

The name of the Android toolchain’s C compiler.

android_toolchain_cflags

New in version 2.4.

The list of the Android toolchain’s C compiler’s recommended flags.

android_toolchain_prefix

The name of the Android toolchain’s prefix.

apple_sdk

The Apple SDK to use.

building_for_target

This is set to True by the component plugin to configure building (i.e. compiling and linking) for the target (rather than the host) architecture. The default value is True.

components

The sequence of component names in the sysroot specification.

copy_file(src, dst)

A file is copied. Any errors are handled automatically.

Parameters:
  • src (str) – is the name of the source file.
  • dst (str) – is the name of the destination file.
copy_dir(src, dst, ignore=None)

A directory is copied, optionally excluding file and sub-directories that match a number of glob patterns. If the destination directory already exists then it is first removed. Any errors are handled automatically.

Parameters:
  • src (str) – is the name of the source directory.
  • dst (str) – is the name of the destination directory.
  • ignore (list[str]) – is an optional sequence of glob patterns that specify files and sub-directories that should be ignored.
create_file(name)

A new text file is created and its file object returned. Any errors are handled automatically.

Parameters:name (str) – is the name of the file.
Returns:the file object of the created file.
create_dir(name, empty=False)

A new directory is created if it does not already exist. If it does already exist then it is optionally emptied. Any errors are handled automatically.

Parameters:
  • name (str) – is the name of the directory.
  • empty (bool) – True if an existing directory should be emptied.
decode_version_nr(version_nr)

An encoded version number is decoded to a 3-tuple of major version, minor version and patch version.

Parameters:version_nr (int) – is the encoded version number.
Returns:the decoded 3-tuple.
delete_dir(name)

A directory and any contents are deleted. Any errors are handled automatically.

Parameters:name (str) – is the name of the directory.
error(message, detail='')

An error message is displayed to the user and the program immediately terminates.

Parameters:
  • message (str) – is the message.
  • detail (str) – is additional detail displayed if the --verbose option was specified.
extract_version_nr(name)

An encoded version number is extracted from the name of file or directory based on common naming standards.

Parameters:name (str) – is the name of the file or directory.
Returns:the encoded version number.
find_component(name, required=True)

The ComponentBase instance for a component is returned.

Parameters:
  • name (str) – is the name of the component.
  • required (bool) – True if the component must exist.
Returns:

the component instance.

find_exe(name)

The absolute path name of an executable located on PATH is returned. Any errors are handled automatically.

Parameters:name (str) – is the generic executable name.
Returns:the absolute path name of the executable.
find_file(name, required=True)

The absolute path name of a file or directory is returned. If the name is relative then it is assumed t be relative to the directory specified by the --source-dir option. If this option has not been specified then the directory containing the JSON specification file is used. The name may be a glob pattern but must only identify a single file or directory.

Parameters:
  • name (str) – is the name of the file or directory.
  • required (bool) – True if the file or directory must exist.
Returns:

the absolute path name of the file or directory.

format_version_nr(version_nr)

An encoded version number is converted to a string.

Parameters:version_nr (int) – is the encoded version number.
Returns:the string conversion.
get_python_install_path(version_nr=None)

The name of the directory containing the root of a Python installation on Windows is returned. If an encoded version number is not given then target_py_version_nr is used. It must only be called by a Windows host.

Parameters:version_nr (int) – is the encoded version number.
Returns:the absolute path of the installation directory.
host_arch_name

The name of the host architecture.

host_bin_dir

The name of the directory where executables built for the host architecture should be installed.

host_dir

The name of the root directory where components built for the host architecture should be installed.

host_exe(name)

A generic executable name is converted to a host-specific version.

Parameters:name (str) – is the generic name.
Returns:the host-specific name.
host_make

The name of the host make executable.

host_platform_name

The name of the host platform.

host_pip

The full path name of the host pip executable.

host_python

The full path name of the host python executable.

host_qmake

The full path name of the host qmake executable.

host_sip

The full path name of the host sip executable.

A symbolic link is made between source and destination files. (Note that, on Windows, a copy of the source file is made.)

Parameters:
  • src (str) – is the name of the source file.
  • dst (str) – is the name of the destination link that is created.
open_file(name)

An existing text file is opened and its file object returned. Any errors are handled automatically.

Parameters:name (str) – is the name of the file.
Returns:the file object of the opened file.
parse_version_nr(version_str)

Convert a string in the form [M[.m[.p]]] to an encoded version number.

Parameters:version_str (str) – is the version number to parse.
Returns:an encoded version number.
pip_install(package)

Install a package using host_pip to target_sitepackages_dir. The package may refer either to a local file or to one to be downloaded from PyPI.

Parameters:package (str) – is the name of the package.
progress(message)

A progress message is displayed to the user. It will be suppressed if the --quiet option was specified.

Parameters:message (str) – is the message.
run(*args, capture=False)

An external command is run. The command’s stdout can be optionally captured.

Parameters:
  • *args – are the name of the command and its arguments.
  • capture (bool) – True if the command’s stdout should be captured and returned.
Returns:

the stdout of the command if requested, otherwise None.

target_arch_name

The name of the target architecture.

target_include_dir

The name of the directory where header files built for the target architecture should be installed.

target_lib_dir

The name of the directory where libraries built for the target architecture should be installed.

target_platform_name

The name of the target platform.

target_py_include_dir

The name of the directory where Python.h built for the target architecture can be found.

target_py_lib

The name of Python library built for the target architecture.

target_py_stdlib_dir

The name of the directory where the Python standard library built for the target architecture can be found.

target_py_version_nr

The version of Python being targeted.

target_pyqt_platform

The name of the target platform as known by PyQt’s configure.py.

target_sip_dir

The name of the directory where .sip files built for the target architecture can be found.

target_sitepackages_dir

The name of the site-packages directory for the target architecture.

target_src_dir

The name of the directory where source files can be found. Note that these are sources left by components for the use of other components and not the sources used to build a component.

unpack_archive(archive, chdir=True)

An archive (e.g. a .tar.gz or .zip file) is unpacked in the current directory.

Parameters:
  • archive (str) – the name of the archive.
  • chdir (bool) – True if the top level directory of the extracted archive should become the new current directory.
Returns:

the name of the top level directory of the extracted archive excluding any path.

verbose(message)

A verbose progress message is displayed to the user. It will be suppressed unless the --verbose option was specified.

Parameters:message (str) – is the message.
verbose_enabled

This is set if the --verbose option was specified.

verify_source(name)

New in version 2.4.

Verify that a source file or directory exists and return the encoded version number embedded in its name. See find_file() for how the name is interpreted.

Parameters:name (str) – is the name of the source file or directory.
Returns:the encoded version number.