mozbuild package

Subpackages

Submodules

mozbuild.android_version_code module

mozbuild.android_version_code.android_version_code(buildid, *args, **kwargs)
mozbuild.android_version_code.android_version_code_v0(buildid, cpu_arch=None, min_sdk=0, max_sdk=0)
mozbuild.android_version_code.android_version_code_v1(buildid, cpu_arch=None, min_sdk=0, max_sdk=0)

Generate a v1 android:versionCode.

The important consideration is that version codes be monotonically increasing (per Android package name) for all published builds. The input build IDs are based on timestamps and hence are always monotonically increasing.

The generated v1 version codes look like (in binary):

0111 1000 0010 tttt tttt tttt tttt txpg

The 17 bits labelled ‘t’ represent the number of hours since midnight on September 1, 2015. (2015090100 in YYYYMMMDDHH format.) This yields a little under 15 years worth of hourly build identifiers, since 2**17 / (366 * 24) =~ 14.92.

The bits labelled ‘x’, ‘p’, and ‘g’ are feature flags.

The bit labelled ‘x’ is 1 if the build is for an x86 architecture and 0 otherwise, which means the build is for an ARM architecture. (Fennec no longer supports ARMv6, so ARM is equivalent to ARMv7 and above.)

The bit labelled ‘p’ is a placeholder that is always 0 (for now).

Firefox no longer supports API 14 or earlier.

This version code computation allows for a split on API levels that allowed us to ship builds specifically for Gingerbread (API 9-10); we preserve that functionality for sanity’s sake, and to allow us to reintroduce a split in the future.

At present, the bit labelled ‘g’ is 1 if the build is an ARM build targeting API 15+, which will always be the case.

We throw an explanatory exception when we are within one calendar year of running out of build events. This gives lots of time to update the version scheme. The responsible individual should then bump the range (to allow builds to continue) and use the time remaining to update the version scheme via the reserved high order bits.

N.B.: the reserved 0 bit to the left of the highest order ‘t’ bit can, sometimes, be used to bump the version scheme. In addition, by reducing the granularity of the build identifiers (for example, moving to identifying builds every 2 or 4 hours), the version scheme may be adjusted further still without losing a (valuable) high order bit.

mozbuild.android_version_code.main(argv)

mozbuild.artifacts module

mozbuild.base module

exception mozbuild.base.BadEnvironmentException

Bases: exceptions.Exception

Base class for errors raised when the build environment is not sane.

exception mozbuild.base.BuildEnvironmentNotFoundException

Bases: mozbuild.base.BadEnvironmentException

Raised when we could not find a build environment.

class mozbuild.base.ExecutionSummary(summary_format, **data)

Bases: dict

Helper for execution summaries.

extend(summary_format, **data)
class mozbuild.base.MachCommandBase(context)

Bases: mozbuild.base.MozbuildObject

Base class for mach command providers that wish to be MozbuildObjects.

This provides a level of indirection so MozbuildObject can be refactored without having to change everything that inherits from it.

class mozbuild.base.MachCommandConditions

Bases: object

A series of commonly used condition functions which can be applied to mach commands with providers deriving from MachCommandBase.

static is_android()

Must have an Android build.

static is_b2g()

Must have a B2G build.

static is_b2g_desktop()

Must have a B2G desktop build.

static is_emulator()

Must have a B2G build with an emulator configured.

static is_firefox()

Must have a Firefox build.

static is_git()

Must have a git source checkout.

static is_hg()

Must have a mercurial source checkout.

static is_mulet()

Must have a Mulet build.

class mozbuild.base.MozbuildObject(topsrcdir, settings, log_manager, topobjdir=None, mozconfig=<object object>)

Bases: mach.mixin.process.ProcessExecutionMixin

Base class providing basic functionality useful to many modules.

Modules in this package typically require common functionality such as accessing the current config, getting the location of the source directory, running processes, etc. This classes provides that functionality. Other modules can inherit from this class to obtain this functionality easily.

bindir
config_environment

Returns the ConfigEnvironment for the current build configuration.

This property is only available once configure has executed.

If configure’s output is not available, this will raise.

defines
distdir
classmethod from_environment(cwd=None, detect_virtualenv_mozinfo=True)

Create a MozbuildObject by detecting the proper one from the env.

This examines environment state like the current working directory and creates a MozbuildObject from the found source directory, mozconfig, etc.

The role of this function is to identify a topsrcdir, topobjdir, and mozconfig file.

If the current working directory is inside a known objdir, we always use the topsrcdir and mozconfig associated with that objdir.

If the current working directory is inside a known srcdir, we use that topsrcdir and look for mozconfigs using the default mechanism, which looks inside environment variables.

If the current Python interpreter is running from a virtualenv inside an objdir, we use that as our objdir.

If we’re not inside a srcdir or objdir, an exception is raised.

detect_virtualenv_mozinfo determines whether we should look for a mozinfo.json file relative to the virtualenv directory. This was added to facilitate testing. Callers likely shouldn’t change the default.

get_binary_path(what=u'app', validate_exists=True, where=u'default')

Obtain the path to a compiled binary for this build configuration.

The what argument is the program or tool being sought after. See the code implementation for supported values.

If validate_exists is True (the default), we will ensure the found path exists before returning, raising an exception if it doesn’t.

If where is ‘staged-package’, we will return the path to the binary in the package staging directory.

If no arguments are specified, we will return the main binary for the configured XUL application.

have_winrm()
includedir
is_clobber_needed()
mozconfig

Returns information about the current mozconfig file.

This a dict as returned by MozconfigLoader.read_mozconfig()

non_global_defines
notify(msg)

Show a desktop notification with the supplied message

On Linux and Mac, this will show a desktop notification with the message, but on Windows we can only flash the screen.

remove_objdir()

Remove the entire object directory.

static resolve_config_guess(mozconfig, topsrcdir)
static resolve_mozconfig_topobjdir(topsrcdir, mozconfig, default=None)
statedir
substs
topobjdir
virtualenv_manager
exception mozbuild.base.ObjdirMismatchException(objdir1, objdir2)

Bases: mozbuild.base.BadEnvironmentException

Raised when the current dir is an objdir and doesn’t match the mozconfig.

class mozbuild.base.PathArgument(arg, topsrcdir, topobjdir, cwd=None)

Bases: object

Parse a filesystem path argument and transform it in various ways.

objdir_path()
relpath()

Return a path relative to the topsrcdir or topobjdir.

If the argument is a path to a location in one of the base directories (topsrcdir or topobjdir), then strip off the base directory part and just return the path within the base directory.

srcdir_path()
mozbuild.base.ancestors(path)

Emit the parent directories of a path.

mozbuild.base.samepath(path1, path2)

mozbuild.config_status module

mozbuild.config_status.config_status(topobjdir='.', topsrcdir='.', defines=None, non_global_defines=None, substs=None, source=None, mozconfig=None)

Main function, providing config.status functionality.

Contrary to config.status, it doesn’t use CONFIG_FILES or CONFIG_HEADERS variables.

Without the -n option, this program acts as config.status and considers the current directory as the top object directory, even when config.status is in a different directory. It will, however, treat the directory containing config.status as the top object directory with the -n option.

The options to this function are passed when creating the ConfigEnvironment. These lists, as well as the actual wrapper script around this function, are meant to be generated by configure. See build/autoconf/config.status.m4.

mozbuild.doctor module

class mozbuild.doctor.Doctor(srcdir, objdir, fix)

Bases: object

check_all()
check_disk_8dot3(path, disk)
check_mount_lastaccess(mount)
cpu
fs_8dot3
fs_lastaccess
getmount(path)
memory
mozillabuild
platform
prompt_bool(prompt, limit=5)

Prompts the user with prompt and requires a boolean value.

report(results)
storage_freespace

mozbuild.dotproperties module

class mozbuild.dotproperties.DotProperties(file=None)

A thin representation of a key=value .properties file.

get(key, default=None)
get_dict(prefix, required_keys=[])

Turns {‘foo.title’:’title’, ...} into {‘title’:’title’, ...}.

If |required_keys| is present, it must be an iterable of required key names. If a required key is not present, ValueError is thrown.

Returns {} to indicate an empty or missing dict.

get_list(prefix)

Turns {‘list.0’:’foo’, ‘list.1’:’bar’} into [‘foo’, ‘bar’].

Returns [] to indicate an empty or missing list.

update(file)

Updates properties from a file name or file-like object.

Ignores empty lines and comment lines.

mozbuild.html_build_viewer module

class mozbuild.html_build_viewer.BuildViewerServer(address=u'localhost', port=0)

Bases: object

add_resource_json_file(key, path)

Register a resource JSON file with the server.

The file will be made available under the name/key specified.

add_resource_json_url(key, url)

Register a resource JSON file at a URL.

run()
url
class mozbuild.html_build_viewer.HTTPHandler(request, client_address, server)

Bases: BaseHTTPServer.BaseHTTPRequestHandler

do_GET()
do_POST()
serve_docroot(root, path)

mozbuild.jar module

jarmaker.py provides a python class to package up chrome content by processing jar.mn files.

See the documentation for jar.mn on MDC for further details on the format.

class mozbuild.jar.JarMaker(outputFormat='flat', useJarfileManifest=True, useChromeManifest=False)

Bases: object

JarMaker reads jar.mn files and process those into jar files or flat directories, along with chrome.manifest files.

class OutputHelper_flat(basepath)

Bases: object

Provide getDestModTime and getOutput for a given flat output directory. The helper method ensureDirFor is used by the symlink subclass.

ensureDirFor(name)
getDestModTime(aPath)
getOutput(name)
class JarMaker.OutputHelper_jar(jarfile)

Bases: object

Provide getDestModTime and getOutput for a given jarfile.

getDestModTime(aPath)
getOutput(name)

Bases: mozbuild.jar.OutputHelper_flat

Subclass of OutputHelper_flat that provides a helper for creating a symlink including creating the parent directories.

JarMaker.finalizeJar(jardir, jarbase, jarname, chromebasepath, register, doZip=True)
Helper method to write out the chrome registration entries to
jarfile.manifest or chrome.manifest, or both.

The actual file processing is done in updateManifest.

JarMaker.generateLocaleDirs(relativesrcdir)
JarMaker.getCommandLineParser()

Get a optparse.OptionParser for jarmaker.

This OptionParser has the options for jarmaker as well as the options for the inner PreProcessor.

JarMaker.makeJar(infile, jardir)

makeJar is the main entry point to JarMaker.

It takes the input file, the output directory, the source dirs and the top source dir as argument, and optionally the l10n dirs.

JarMaker.processJarSection(jarinfo, jardir)

Internal method called by makeJar to actually process a section of a jar.mn file.

JarMaker.updateManifest(manifestPath, chromebasepath, register)

updateManifest replaces the % in the chrome registration entries with the given chrome base path, and updates the given manifest file.

mozbuild.mach_commands module

mozbuild.makeutil module

class mozbuild.makeutil.Makefile

Bases: object

Provides an interface for writing simple makefiles

Instances of this class are created, populated with rules, then written.

add_statement(statement)

Add a raw statement in the makefile. Meant to be used for simple variable assignments.

create_rule(targets=[])

Create a new rule in the makefile for the given targets. Returns the corresponding Rule instance.

dump(fh, removal_guard=True)

Dump all the rules to the given file handle. Optionally (and by default), add guard rules for file removals (empty rules for other rules’ dependencies)

class mozbuild.makeutil.Rule(targets=[])

Bases: object

Class handling simple rules in the form: target1 target2 ... : dep1 dep2 ...

command1 command2 ...
add_commands(commands)

Add commands to the rule.

add_dependencies(deps)

Add dependencies to the rule.

add_targets(targets)

Add additional targets to the rule.

commands()

Return an iterator on the rule commands.

dependencies()

Return an iterator on the rule dependencies.

dump(fh)

Dump the rule to the given file handle.

targets()

Return an iterator on the rule targets.

mozbuild.makeutil.read_dep_makefile(fh)

Read the file handler containing a dep makefile (simple makefile only containing dependencies) and returns an iterator of the corresponding Rules it contains. Ignores removal guard rules.

mozbuild.makeutil.write_dep_makefile(fh, target, deps)

Write a Makefile containing only target’s dependencies to the file handle specified.

mozbuild.milestone module

mozbuild.milestone.get_milestone_ab_with_num(milestone)

Returns the alpha and beta tag with its number (a1, a2, b3, ...).

mozbuild.milestone.get_milestone_major(milestone)

Returns the major (first) part of the milestone.

mozbuild.milestone.get_official_milestone(path)

Returns the contents of the first line in path that starts with a digit.

mozbuild.milestone.main(args)

mozbuild.mozconfig module

exception mozbuild.mozconfig.MozconfigFindException

Bases: exceptions.Exception

Raised when a mozconfig location is not defined properly.

exception mozbuild.mozconfig.MozconfigLoadException(path, message, output=None)

Bases: exceptions.Exception

Raised when a mozconfig could not be loaded properly.

This typically indicates a malformed or misbehaving mozconfig file.

class mozbuild.mozconfig.MozconfigLoader(topsrcdir)

Bases: object

Handles loading and parsing of mozconfig files.

AUTODETECT = <object object>
DEFAULT_TOPSRCDIR_PATHS = (u'.mozconfig', u'mozconfig')
DEPRECATED_HOME_PATHS = (u'.mozconfig', u'.mozconfig.sh', u'.mozmyconfig.sh')
DEPRECATED_TOPSRCDIR_PATHS = (u'mozconfig.sh', u'myconfig.sh')
ENVIRONMENT_VARIABLES = set([u'LDFLAGS', u'CXX', u'CXXFLAGS', u'CC', u'CFLAGS', u'MOZ_OBJDIR'])
IGNORE_SHELL_VARIABLES = set([u'_'])
RE_MAKE_VARIABLE = <_sre.SRE_Pattern object>
find_mozconfig(env={'LANG': 'C.UTF-8', 'READTHEDOCS_PROJECT': 'gfritzsche-demo', 'READTHEDOCS': 'True', 'APPDIR': '/app', 'DEBIAN_FRONTEND': 'noninteractive', 'OLDPWD': '/', 'HOSTNAME': 'build-4258433-project-55928-gfritzsche-demo', u'SHELL': u'/bin/bash', 'PWD': '/home/docs/checkouts/readthedocs.org/user_builds/gfritzsche-demo/checkouts/latest/tools/docs', 'BIN_PATH': '/home/docs/checkouts/readthedocs.org/user_builds/gfritzsche-demo/envs/latest/bin', 'READTHEDOCS_VERSION': 'latest', 'PATH': '/home/docs/checkouts/readthedocs.org/user_builds/gfritzsche-demo/checkouts/latest/tools/docs/_build/html/_venv/bin:/home/docs/checkouts/readthedocs.org/user_builds/gfritzsche-demo/envs/latest/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'HOME': '/home/docs'})

Find the active mozconfig file for the current environment.

This emulates the logic in mozconfig-find.

  1. If ENV[MOZCONFIG] is set, use that
  2. If $TOPSRCDIR/mozconfig or $TOPSRCDIR/.mozconfig exists, use it.
  3. If both exist or if there are legacy locations detected, error out.

The absolute path to the found mozconfig will be returned on success. None will be returned if no mozconfig could be found. A MozconfigFindException will be raised if there is a bad state, including conditions from #3 above.

read_mozconfig(path=None, moz_build_app=None)

Read the contents of a mozconfig into a data structure.

This takes the path to a mozconfig to load. If the given path is AUTODETECT, will try to find a mozconfig from the environment using find_mozconfig().

mozconfig files are shell scripts. So, we can’t just parse them. Instead, we run the shell script in a wrapper which allows us to record state from execution. Thus, the output from a mozconfig is a friendly static data structure.

mozbuild.mozinfo module

mozbuild.mozinfo.build_dict(config, env={'LANG': 'C.UTF-8', 'READTHEDOCS_PROJECT': 'gfritzsche-demo', 'READTHEDOCS': 'True', 'APPDIR': '/app', 'DEBIAN_FRONTEND': 'noninteractive', 'OLDPWD': '/', 'HOSTNAME': 'build-4258433-project-55928-gfritzsche-demo', u'SHELL': u'/bin/bash', 'PWD': '/home/docs/checkouts/readthedocs.org/user_builds/gfritzsche-demo/checkouts/latest/tools/docs', 'BIN_PATH': '/home/docs/checkouts/readthedocs.org/user_builds/gfritzsche-demo/envs/latest/bin', 'READTHEDOCS_VERSION': 'latest', 'PATH': '/home/docs/checkouts/readthedocs.org/user_builds/gfritzsche-demo/checkouts/latest/tools/docs/_build/html/_venv/bin:/home/docs/checkouts/readthedocs.org/user_builds/gfritzsche-demo/envs/latest/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'HOME': '/home/docs'})

Build a dict containing data about the build configuration from the environment.

mozbuild.mozinfo.write_mozinfo(file, config, env={'LANG': 'C.UTF-8', 'READTHEDOCS_PROJECT': 'gfritzsche-demo', 'READTHEDOCS': 'True', 'APPDIR': '/app', 'DEBIAN_FRONTEND': 'noninteractive', 'OLDPWD': '/', 'HOSTNAME': 'build-4258433-project-55928-gfritzsche-demo', u'SHELL': u'/bin/bash', 'PWD': '/home/docs/checkouts/readthedocs.org/user_builds/gfritzsche-demo/checkouts/latest/tools/docs', 'BIN_PATH': '/home/docs/checkouts/readthedocs.org/user_builds/gfritzsche-demo/envs/latest/bin', 'READTHEDOCS_VERSION': 'latest', 'PATH': '/home/docs/checkouts/readthedocs.org/user_builds/gfritzsche-demo/checkouts/latest/tools/docs/_build/html/_venv/bin:/home/docs/checkouts/readthedocs.org/user_builds/gfritzsche-demo/envs/latest/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'HOME': '/home/docs'})

Write JSON data about the configuration specified in config and an environment variable dict to |file|, which may be a filename or file-like object. See build_dict for information about what environment variables are used, and what keys are produced.

mozbuild.preprocessor module

This is a very primitive line based preprocessor, for times when using a C preprocessor isn’t an option.

It currently supports the following grammar for expressions, whitespace is ignored:

expression :
and_cond ( ‘||’ expression ) ? ;
and_cond:
test ( ‘&&’ and_cond ) ? ;
test:
unary ( ( ‘==’ | ‘!=’ ) unary ) ? ;
unary :
‘!’? value ;
value :
[0-9]+ # integer | ‘defined(‘ w+ ‘)’ | w+ # string identifier or value;
class mozbuild.preprocessor.Context

Bases: dict

This class holds variable values by subclassing dict, and while it truthfully reports True and False on

name in context

it returns the variable name itself on

context[“name”]

to reflect the ambiguity between string literals and preprocessor variables.

class mozbuild.preprocessor.Expression(expression_string)
exception ParseError(expression)

Bases: exceptions.StandardError

Error raised when parsing fails. It has two members, offset and content, which give the offset of the error and the offending content.

Expression.evaluate(context)

Evaluate the expression with the given context

class mozbuild.preprocessor.Preprocessor(defines=None, marker='#')

Class for preprocessing text files.

exception Error(cpp, MSG, context)

Bases: exceptions.RuntimeError

Preprocessor.addDefines(defines)

Adds the specified defines to the preprocessor. defines may be a dictionary object or an iterable of key/value pairs (as tuples or other iterables of length two)

Preprocessor.applyFilters(aLine)
Preprocessor.clone()

Create a clone of the current processor, including line ending settings, marker, variable definitions, output stream.

Preprocessor.computeDependencies(input)

Reads the input stream, and computes the dependencies for that input.

Preprocessor.do_define(args)
Preprocessor.do_elif(args)
Preprocessor.do_elifdef(args)
Preprocessor.do_elifndef(args)
Preprocessor.do_else(args, ifState=2)
Preprocessor.do_endif(args)
Preprocessor.do_error(args)
Preprocessor.do_expand(args)
Preprocessor.do_filter(args)
Preprocessor.do_if(args, replace=False)
Preprocessor.do_ifdef(args, replace=False)
Preprocessor.do_ifndef(args, replace=False)
Preprocessor.do_include(args, filters=True)

Preprocess a given file. args can either be a file name, or a file-like object. Files should be opened, and will be closed after processing.

Preprocessor.do_includesubst(args)
Preprocessor.do_literal(args)
Preprocessor.do_undef(args)
Preprocessor.do_unfilter(args)
Preprocessor.ensure_not_else()
Preprocessor.failUnused(file)
Preprocessor.filter_attemptSubstitution(aLine)
Preprocessor.filter_emptyLines(aLine)
Preprocessor.filter_slashslash(aLine)
Preprocessor.filter_spaces(aLine)
Preprocessor.filter_substitution(aLine, fatal=True)
Preprocessor.getCommandLineParser(unescapeDefines=False)
Preprocessor.handleCommandLine(args, defaultToStdin=False)

Parse a commandline into this parser. Uses OptionParser internally, no args mean sys.argv[1:].

Preprocessor.handleLine(aLine)

Handle a single line of input (internal).

Preprocessor.noteLineInfo()
Preprocessor.processFile(input, output, depfile=None)

Preprocesses the contents of the input stream and writes the result to the output stream. If depfile is set, the dependencies of output file are written to depfile in Makefile format.

Preprocessor.setMarker(aMarker)

Set the marker to be used for processing directives. Used for handling CSS files, with pp.setMarker(‘%’), for example. The given marker may be None, in which case no markers are processed.

Preprocessor.setSilenceDirectiveWarnings(value)

Sets whether missing directive warnings are silenced, according to value. The default behavior of the preprocessor is to emit such warnings.

Preprocessor.write(aLine)

Internal method for handling output.

mozbuild.preprocessor.preprocess(includes=[<open file '<stdin>', mode 'r' at 0x7ff5094850c0>], defines={}, output=<open file '<stdout>', mode 'w'>, marker='#')

mozbuild.pythonutil module

mozbuild.pythonutil.iter_modules_in_path(*paths)

mozbuild.shellutil module

exception mozbuild.shellutil.MetaCharacterException(char)

Bases: exceptions.Exception

mozbuild.shellutil.split(cline)

Split the given command line string.

mozbuild.shellutil.quote(*strings)

Given one or more strings, returns a quoted string that can be used literally on a shell command line.

>>> quote('a', 'b')
"a b"
>>> quote('a b', 'c')
"'a b' c"

mozbuild.sphinx module

class mozbuild.sphinx.MozbuildSymbols(name, arguments, options, content, lineno, content_offset, block_text, state, state_machine)

Bases: docutils.parsers.rst.Directive

Directive to insert mozbuild sandbox symbol information.

required_arguments = 1
run()
mozbuild.sphinx.format_module(m)
mozbuild.sphinx.function_reference(f, attr, args, doc)
mozbuild.sphinx.setup(app)
mozbuild.sphinx.special_reference(v, func, typ, doc)
mozbuild.sphinx.variable_reference(v, st_type, in_type, doc)

mozbuild.testing module

class mozbuild.testing.SupportFilesConverter

Bases: object

Processes a “support-files” entry from a test object, either from a parsed object from a test manifests or its representation in moz.build and returns the installs to perform for this test object.

Processing the same support files multiple times will not have any further effect, and the structure of the parsed objects from manifests will have a lot of repeated entries, so this class takes care of memoizing.

convert_support_files(test, install_root, manifest_dir, out_dir)
class mozbuild.testing.TestInstallInfo

Bases: object

class mozbuild.testing.TestMetadata(filename=None)

Bases: object

Holds information about tests.

This class provides an API to query tests active in the build configuration.

resolve_tests(paths=None, flavor=None, subsuite=None, under_path=None, tags=None)

Resolve tests from an identifier.

This is a generator of dicts describing each test.

paths can be an iterable of values to use to identify tests to run. If an entry is a known test file, tests associated with that file are returned (there may be multiple configurations for a single file). If an entry is a directory, or a prefix of a directory containing tests, all tests in that directory are returned. If the string appears in a known test file, that test file is considered. If the path contains a wildcard pattern, tests matching that pattern are returned.

If under_path is a string, it will be used to filter out tests that aren’t in the specified path prefix relative to topsrcdir or the test’s installed dir.

If flavor is a string, it will be used to filter returned tests to only be the flavor specified. A flavor is something like xpcshell.

If subsuite is a string, it will be used to filter returned tests to only be in the subsuite specified.

If tags are specified, they will be used to filter returned tests to only those with a matching tag.

tests_with_flavor(flavor)

Obtain all tests having the specified flavor.

This is a generator of dicts describing each test.

class mozbuild.testing.TestResolver(*args, **kwargs)

Bases: mozbuild.base.MozbuildObject

Helper to resolve tests from the current environment to test files.

resolve_tests(cwd=None, **kwargs)

Resolve tests in the context of the current environment.

This is a more intelligent version of TestMetadata.resolve_tests().

This function provides additional massaging and filtering of low-level results.

Paths in returned tests are automatically translated to the paths in the _tests directory under the object directory.

If cwd is defined, we will limit our results to tests under the directory specified. The directory should be defined as an absolute path under topsrcdir or topobjdir for it to work properly.

mozbuild.testing.all_test_flavors()
mozbuild.testing.install_test_files(topsrcdir, topobjdir, tests_root, test_objs)

Installs the requested test files to the objdir. This is invoked by test runners to avoid installing tens of thousands of test files when only a few tests need to be run.

mozbuild.testing.read_manifestparser_manifest(context, manifest_path)
mozbuild.testing.read_reftest_manifest(context, manifest_path)
mozbuild.testing.read_wpt_manifest(context, paths)
mozbuild.testing.rewrite_test_base(test, new_base, honor_install_to_subdir=False)

Rewrite paths in a test to be under a new base path.

This is useful for running tests from a separate location from where they were defined.

honor_install_to_subdir and the underlying install-to-subdir field are a giant hack intended to work around the restriction where the mochitest runner can’t handle single test files with multiple configurations. This argument should be removed once the mochitest runner talks manifests (bug 984670).

mozbuild.util module

class mozbuild.util.DefinesAction(option_strings, dest, nargs=None, const=None, default=None, type=None, choices=None, required=False, help=None, metavar=None)

Bases: argparse.Action

An ArgumentParser action to handle -Dvar[=value] type of arguments.

class mozbuild.util.EmptyValue

Bases: unicode

A dummy type that behaves like an empty string and sequence.

This type exists in order to support mozbuild.frontend.reader.EmptyConfig. It should likely not be used elsewhere.

class mozbuild.util.EnumString(value)

Bases: unicode

A string type that only can have a limited set of values, similarly to an Enum, and can only be compared against that set of values.

The class is meant to be subclassed, where the subclass defines POSSIBLE_VALUES. The subclass method is a helper to create such subclasses.

POSSIBLE_VALUES = ()
static subclass(*possible_values)
exception mozbuild.util.EnumStringComparisonError

Bases: exceptions.Exception

class mozbuild.util.FileAvoidWrite(filename, capture_diff=False, dry_run=False, mode=u'rU')

Bases: _io.BytesIO

File-like object that buffers output and only writes if content changed.

We create an instance from an existing filename. New content is written to it. When we close the file object, if the content in the in-memory buffer differs from what is on disk, then we write out the new content. Otherwise, the original file is untouched.

Instances can optionally capture diffs of file changes. This feature is not enabled by default because it a) doesn’t make sense for binary files b) could add unwanted overhead to calls.

Additionally, there is dry run mode where the file is not actually written out, but reports whether the file was existing and would have been updated still occur, as well as diff capture if requested.

close()

Stop accepting writes, compare file contents, and rewrite if needed.

Returns a tuple of bools indicating what action was performed:

(file existed, file updated)

If capture_diff was specified at construction time and the underlying file was changed, .diff will be populated with the diff of the result.

write(buf)
mozbuild.util.FlagsFactory(flags)

Returns a class which holds optional flags for an item in a list.

The flags are defined in the dict given as argument, where keys are the flag names, and values the type used for the value of that flag.

The resulting class is used by the various <TypeName>WithFlagsFactory functions below.

class mozbuild.util.HierarchicalStringList

Bases: object

A hierarchy of lists of strings.

Each instance of this object contains a list of strings, which can be set or appended to. A sub-level of the hierarchy is also an instance of this class, can be added by appending to an attribute instead.

For example, the moz.build variable EXPORTS is an instance of this class. We can do:

EXPORTS += [‘foo.h’] EXPORTS.mozilla.dom += [‘bar.h’]

In this case, we have 3 instances (EXPORTS, EXPORTS.mozilla, and EXPORTS.mozilla.dom), and the first and last each have one element in their list.

class StringListAdaptor(hsl)

Bases: _abcoll.Sequence

HierarchicalStringList.walk()

Walk over all HierarchicalStringLists in the hierarchy.

This is a generator of (path, sequence).

The path is ‘’ for the root level and ‘/’-delimited strings for any descendants. The sequence is a read-only sequence of the strings contained at that level.

class mozbuild.util.KeyedDefaultDict(default_factory, *args, **kwargs)

Bases: dict

Like a defaultdict, but the default_factory function takes the key as argument

class mozbuild.util.List(iterable=None, **kwargs)

Bases: mozbuild.util.ListMixin, list

A list specialized for moz.build environments.

We overload the assignment and append operations to require that the appended thing is a list. This avoids bad surprises coming from appending a string to a list, which would just add each letter of the string.

class mozbuild.util.ListMixin(iterable=None, **kwargs)

Bases: object

extend(l)
class mozbuild.util.ListWithAction(iterable=None, **kwargs)

Bases: mozbuild.util.ListMixin, mozbuild.util.ListWithActionMixin, list

A list that accepts a callable to be applied to each item.

A callable (action) may optionally be passed to the constructor to run on each item of input. The result of calling the callable on each item will be stored in place of the original input.

class mozbuild.util.ListWithActionMixin(iterable=None, action=None)

Bases: object

Mixin to create lists with pre-processing. See ListWithAction.

extend(l)
class mozbuild.util.LockFile(lockfile)

Bases: object

LockFile is used by the lock_file method to hold the lock.

This object should not be used directly, but only through the lock_file method below.

exception mozbuild.util.MozbuildDeletionError

Bases: exceptions.Exception

class mozbuild.util.OrderedDefaultDict(default_factory, *args, **kwargs)

Bases: collections.OrderedDict

A combination of OrderedDict and defaultdict.

class mozbuild.util.ReadOnlyDefaultDict(default_factory, *args, **kwargs)

Bases: mozbuild.util.ReadOnlyDict

A read-only dictionary that supports default values on retrieval.

class mozbuild.util.ReadOnlyDict(*args, **kwargs)

Bases: dict

A read-only dictionary.

update(*args, **kwargs)
class mozbuild.util.ReadOnlyKeyedDefaultDict(default_factory, *args, **kwargs)

Bases: mozbuild.util.KeyedDefaultDict, mozbuild.util.ReadOnlyDict

Like KeyedDefaultDict, but read-only.

class mozbuild.util.ReadOnlyNamespace(**kwargs)

Bases: object

A class for objects with immutable attributes set at initialization.

class mozbuild.util.StrictOrderingOnAppendList(iterable=None, **kwargs)

Bases: mozbuild.util.ListMixin, mozbuild.util.StrictOrderingOnAppendListMixin, list

A list specialized for moz.build environments.

We overload the assignment and append operations to require that incoming elements be ordered. This enforces cleaner style in moz.build files.

class mozbuild.util.StrictOrderingOnAppendListMixin(iterable=None, **kwargs)

Bases: object

static ensure_sorted(l)
extend(l)
class mozbuild.util.StrictOrderingOnAppendListWithAction(iterable=None, **kwargs)

Bases: mozbuild.util.StrictOrderingOnAppendListMixin, mozbuild.util.ListMixin, mozbuild.util.ListWithActionMixin, list

An ordered list that accepts a callable to be applied to each item.

A callable (action) passed to the constructor is run on each item of input. The result of running the callable on each item will be stored in place of the original input, but the original item must be used to enforce sortedness. Note that the order of superclasses is therefore significant.

class mozbuild.util.StrictOrderingOnAppendListWithFlags(iterable=None, **kwargs)

Bases: mozbuild.util.StrictOrderingOnAppendList

A list with flags specialized for moz.build environments.

Each subclass has a set of typed flags; this class lets us use isinstance for natural testing.

mozbuild.util.StrictOrderingOnAppendListWithFlagsFactory(flags)

Returns a StrictOrderingOnAppendList-like object, with optional flags on each item.

The flags are defined in the dict given as argument, where keys are the flag names, and values the type used for the value of that flag.

Example:
FooList = StrictOrderingOnAppendListWithFlagsFactory({
‘foo’: bool, ‘bar’: unicode

}) foo = FooList([‘a’, ‘b’, ‘c’]) foo[‘a’].foo = True foo[‘b’].bar = ‘bar’

mozbuild.util.TypedList

A list with type coercion.

The given type is what list elements are being coerced to. It may do strict validation, throwing ValueError exceptions.

A base_class type can be given for more specific uses than a List. For example, a Typed StrictOrderingOnAppendList can be created with:

TypedList(unicode, StrictOrderingOnAppendList)
class mozbuild.util.TypedListMixin(iterable=None, **kwargs)

Bases: object

Mixin for a list with type coercion. See TypedList.

append(other)
extend(l)
mozbuild.util.TypedNamedTuple(name, fields)

Factory for named tuple types with strong typing.

Arguments are an iterable of 2-tuples. The first member is the the field name. The second member is a type the field will be validated to be.

Construction of instances varies from collections.namedtuple.

First, if a single tuple argument is given to the constructor, this is treated as the equivalent of passing each tuple value as a separate argument into __init__. e.g.:

t = (1, 2)
TypedTuple(t) == TypedTuple(1, 2)

This behavior is meant for moz.build files, so vanilla tuples are automatically cast to typed tuple instances.

Second, fields in the tuple are validated to be instances of the specified type. This is done via an isinstance() check. To allow multiple types, pass a tuple as the allowed types field.

exception mozbuild.util.UnsortedError(srtd, original)

Bases: exceptions.Exception

mozbuild.util.ensureParentDir(path)

Ensures the directory parent to the given file exists.

mozbuild.util.exec_(object, globals=None, locals=None)

Wrapper around the exec statement to avoid bogus errors like:

SyntaxError: unqualified exec is not allowed in function ... it is a nested function.

or

SyntaxError: unqualified exec is not allowed in function ... it contains a nested function with free variable

which happen with older versions of python 2.7.

mozbuild.util.expand_variables(s, variables)

Given a string with $(var) variable references, replace those references with the corresponding entries from the given variables dict.

If a variable value is not a string, it is iterated and its items are joined with a whitespace.

mozbuild.util.group_unified_files(files, unified_prefix, unified_suffix, files_per_unified_file)

Return an iterator of (unified_filename, source_filenames) tuples.

We compile most C and C++ files in “unified mode”; instead of compiling a.cpp, b.cpp, and c.cpp separately, we compile a single file that looks approximately like:

#include "a.cpp"
#include "b.cpp"
#include "c.cpp"

This function handles the details of generating names for the unified files, and determining which original source files go in which unified file.

mozbuild.util.hash_file(path, hasher=None)

Hashes a file specified by the path given and returns the hex digest.

mozbuild.util.lock_file(lockfile, max_wait=600)

Create and hold a lockfile of the given name, with the given timeout.

To release the lock, delete the returned object.

class mozbuild.util.memoize(func)

Bases: dict

A decorator to memoize the results of function calls depending on its arguments. Both functions and instance methods are handled, although in the instance method case, the results are cache in the instance itself.

method_call(instance, *args)
class mozbuild.util.memoized_property(func)

Bases: object

A specialized version of the memoize decorator that works for class instance properties.

mozbuild.util.mkdir(path, not_indexed=False)

Ensure a directory exists.

If not_indexed is True, an attribute is set that disables content indexing on the directory.

mozbuild.util.pair(iterable)

Given an iterable, returns an iterable pairing its items.

For example,
list(pair([1,2,3,4,5,6]))
returns
[(1,2), (3,4), (5,6)]
mozbuild.util.resolve_target_to_make(topobjdir, target)

Resolve target (a target, directory, or file) to a make target.

topobjdir is the object directory; all make targets will be rooted at or below the top-level Makefile in this directory.

Returns a pair (reldir, target) where reldir is a directory relative to topobjdir containing a Makefile and target is a make target (possibly None).

A directory resolves to the nearest directory at or above containing a Makefile, and target None.

A regular (non-Makefile) file resolves to the nearest directory at or above the file containing a Makefile, and an appropriate target.

A Makefile resolves to the nearest parent strictly above the Makefile containing a different Makefile, and an appropriate target.

mozbuild.util.simple_diff(filename, old_lines, new_lines)

Returns the diff between old_lines and new_lines, in unified diff form, as a list of lines.

old_lines and new_lines are lists of non-newline terminated lines to compare. old_lines can be None, indicating a file creation. new_lines can be None, indicating a file deletion.

class mozbuild.util.undefined_default

Bases: object

Represents an undefined argument value that isn’t None.

mozbuild.virtualenv module

class mozbuild.virtualenv.VirtualenvManager(topsrcdir, topobjdir, virtualenv_path, log_handle, manifest_path)

Bases: object

Contains logic for managing virtualenvs for building the tree.

activate()

Activate the virtualenv in this Python context.

If you run a random Python script and wish to “activate” the virtualenv, you can simply instantiate an instance of this class and call .ensure() and .activate() to make the virtualenv active.

activate_path
bin_path
build(python='/home/docs/checkouts/readthedocs.org/user_builds/gfritzsche-demo/envs/latest/bin/python')

Build a virtualenv per tree conventions.

This returns the path of the created virtualenv.

call_setup(directory, arguments)

Calls setup.py in a directory.

create(python='/home/docs/checkouts/readthedocs.org/user_builds/gfritzsche-demo/envs/latest/bin/python')

Create a new, empty virtualenv.

Receives the path to virtualenv’s virtualenv.py script (which will be called out to), the path to create the virtualenv in, and a handle to write output to.

ensure(python='/home/docs/checkouts/readthedocs.org/user_builds/gfritzsche-demo/envs/latest/bin/python')

Ensure the virtualenv is present and up to date.

If the virtualenv is up to date, this does nothing. Otherwise, it creates and populates the virtualenv as necessary.

This should be the main API used from this class as it is the highest-level.

get_exe_info()

Returns the version and file size of the python executable that was in use when this virutalenv was created.

install_pip_package(package)

Install a package via pip.

The supplied package is specified using a pip requirement specifier. e.g. ‘foo’ or ‘foo==1.0’.

If the package is already installed, this is a no-op.

packages()
populate()

Populate the virtualenv.

The manifest file consists of colon-delimited fields. The first field specifies the action. The remaining fields are arguments to that action. The following actions are supported:

setup.py – Invoke setup.py for a package. Expects the arguments:
  1. relative path directory containing setup.py.
  2. argument(s) to setup.py. e.g. “develop”. Each program argument is delimited by a colon. Arguments with colons are not yet supported.
filename.pth – Adds the path given as argument to filename.pth under
the virtualenv site packages directory.
optional – This denotes the action as optional. The requested action
is attempted. If it fails, we issue a warning and go on. The initial “optional” field is stripped then the remaining line is processed like normal. e.g. “optional:setup.py:python/foo:built_ext:-i”
copy – Copies the given file in the virtualenv site packages
directory.
packages.txt – Denotes that the specified path is a child manifest. It
will be read and processed as if its contents were concatenated into the manifest being read.
objdir – Denotes a relative path in the object directory to add to the
search path. e.g. “objdir:build” will add $topobjdir/build to the search path.

Note that the Python interpreter running this function should be the one from the virtualenv. If it is the system Python or if the environment is not configured properly, packages could be installed into the wrong place. This is how virtualenv’s work.

python_path
up_to_date(python='/home/docs/checkouts/readthedocs.org/user_builds/gfritzsche-demo/envs/latest/bin/python')

Returns whether the virtualenv is present and up to date.

virtualenv_script_path

Path to virtualenv’s own populator script.

write_exe_info(python)

Records the the version of the python executable that was in use when this virutalenv was created. We record this explicitly because on OS X our python path may end up being a different or modified executable.

mozbuild.virtualenv.verify_python_version(log_handle)

Ensure the current version of Python is sufficient.

Module contents