scriptworker package

Submodules

scriptworker.client module

Scripts running in scriptworker will use functions in this file.

This module should be largely standalone. This should only depend on scriptworker.exceptions and scriptworker.constants, or other standalone modules, to avoid circular imports.

scriptworker.client.log

logging.Logger – the log object for the module

scriptworker.client.get_task(config)

Read the task.json from work_dir.

Parameters:config (dict) – the running config, to find work_dir.
Returns:the contents of task.json
Return type:dict
Raises:ScriptWorkerTaskException – on error.
scriptworker.client.sync_main(async_main, config_path=None, default_config=None, should_validate_task=True, loop_function=<function get_event_loop>)

Entry point for scripts using scriptworker.

This function sets up the basic needs for a script to run. More specifically:
  • it creates the scriptworker context and initializes it with the provided config
  • the path to the config file is either taken from config_path or from sys.argv[1].
  • it verifies sys.argv doesn’t have more arguments than the config path.
  • it creates the asyncio event loop so that async_main can run
Parameters:
  • async_main (function) – The function to call once everything is set up
  • config_path (str, optional) – The path to the file to load the config from. Loads from sys.argv[1] if None. Defaults to None.
  • default_config (dict, optional) – the default config to use for _init_context. defaults to None.
  • should_validate_task (bool, optional) – whether we should validate the task schema. Defaults to True.
  • loop_function (function, optional) – the function to call to get the event loop; here for testing purposes. Defaults to asyncio.get_event_loop.
scriptworker.client.validate_artifact_url(valid_artifact_rules, valid_artifact_task_ids, url)

Ensure a URL fits in given scheme, netloc, and path restrictions.

If we fail any checks, raise a ScriptWorkerTaskException with malformed-payload.

Parameters:
  • valid_artifact_rules (tuple) – the tests to run, with schemas, netlocs, and path_regexes.
  • valid_artifact_task_ids (list) – the list of valid task IDs to download from.
  • url (str) – the url of the artifact.
Returns:

the filepath of the path regex.

Return type:

str

Raises:

ScriptWorkerTaskException – on failure to validate.

scriptworker.client.validate_json_schema(data, schema, name='task')

Given data and a jsonschema, let’s validate it.

This happens for tasks and chain of trust artifacts.

Parameters:
  • data (dict) – the json to validate.
  • schema (dict) – the jsonschema to validate against.
  • name (str, optional) – the name of the json, for exception messages. Defaults to “task”.
Raises:

ScriptWorkerTaskException – on failure

scriptworker.client.validate_task_schema(context, schema_key='schema_file')

Validate the task definition.

Parameters:
  • context (scriptworker.context.Context) – the scriptworker context. It must contain a task and the config pointing to the schema file
  • schema_key – the key in context.config where the path to the schema file is. Key can contain dots (e.g.: ‘schema_files.file_a’), in which case
Raises:

TaskVerificationError – if the task doesn’t match the schema

scriptworker.config module

Config for scriptworker.

scriptworker.config.log

logging.Logger – the log object for the module.

scriptworker.config.CREDS_FILES

tuple – an ordered list of files to look for taskcluster credentials, if they aren’t in the config file or environment.

scriptworker.config.apply_product_config(config)

Apply config values that are keyed by cot_product.

This modifies the passed in configuration.

Parameters:dict (config) – the config to apply cot_product keying too

Returns: dict

scriptworker.config.check_config(config, path)

Validate the config against DEFAULT_CONFIG.

Any unknown keys or wrong types will add error messages.

Parameters:
  • config (dict) – the running config.
  • path (str) – the path to the config file, used in error messages.
Returns:

the error messages found when validating the config.

Return type:

list

scriptworker.config.create_config(config_path='scriptworker.yaml')

Create a config from DEFAULT_CONFIG, arguments, and config file.

Then validate it and freeze it.

Parameters:config_path (str, optional) – the path to the config file. Defaults to “scriptworker.yaml”
Returns:(config frozendict, credentials dict)
Return type:tuple
Raises:SystemExit – on failure
scriptworker.config.get_context_from_cmdln(args, desc='Run scriptworker')

Create a Context object from args.

This was originally part of main(), but we use it in scriptworker.gpg.rebuild_gpg_homedirs too.

Parameters:args (list) – the commandline args. Generally sys.argv
Returns:
scriptworker.context.Context with populated config, and
credentials frozendict
Return type:tuple
scriptworker.config.get_frozen_copy(values)

Convert values‘s list values into tuples, and dicts into frozendicts.

A recursive function(bottom-up conversion)

Parameters:values (dict/list) – the values/list to be modified in-place.
scriptworker.config.get_unfrozen_copy(values)

Recursively convert value‘s tuple values into lists, and frozendicts into dicts.

Parameters:values (frozendict/tuple) – the frozendict/tuple.
Returns:values – the unfrozen copy.
Return type:dict/list
scriptworker.config.read_worker_creds(key='credentials')

Get credentials from CREDS_FILES or the environment.

This looks at the CREDS_FILES in order, and falls back to the environment.

Parameters:key (str, optional) – each CREDS_FILE is a json dict. This key’s value contains the credentials. Defaults to ‘credentials’.
Returns:the credentials found. None if no credentials found.
Return type:dict

scriptworker.context module

scriptworker context.

Most functions need access to a similar set of objects. Rather than having to pass them all around individually or create a monolithic ‘self’ object, let’s point to them from a single context object.

scriptworker.context.log

logging.Logger – the log object for the module.

class scriptworker.context.Context

Bases: object

Basic config holding object.

Avoids putting everything in single monolithic object, but allows for passing around config and easier overriding in tests.

config

dict – the running config. In production this will be a FrozenDict.

credentials_timestamp

int – the unix timestamp when we last updated our credentials.

proc

asyncio.subprocess.Process – when launching the script, this is the process object.

queue

taskcluster.aio.Queue – the taskcluster Queue object containing the scriptworker credentials.

session

aiohttp.ClientSession – the default aiohttp session

task

dict – the task definition for the current task.

temp_queue

taskcluster.aio.Queue – the taskcluster Queue object containing the task-specific temporary credentials.

claim_task

dict – The current or most recent claimTask definition json from the queue.

This contains the task definition, as well as other task-specific info.

When setting claim_task, we also set self.task and self.temp_credentials, zero out self.reclaim_task and self.proc, then write a task.json to disk.

config = None
create_queue(credentials)

Create a taskcluster queue.

Parameters:credentials (dict) – taskcluster credentials.
credentials

dict – The current scriptworker credentials.

These come from the config or CREDS_FILES or environment.

When setting credentials, also create a new self.queue and update self.credentials_timestamp.

credentials_timestamp = None
event_loop

asyncio.BaseEventLoop – the running event loop.

This fixture mainly exists to allow for overrides during unit tests.

populate_projects(force=False)

Download the projects.yml file and populate self.projects.

This only sets it once, unless force is set.

Parameters:force (bool, optional) – Re-run the download, even if self.projects is already defined. Defaults to False.
proc = None
projects

dict – The current contents of projects.yml, which defines CI configuration.

I’d love to auto-populate this; currently we need to set this from the config’s project_configuration_url.

queue = None
reclaim_task

dict – The most recent reclaimTask definition.

This contains the newest expiration time and the newest temp credentials.

When setting reclaim_task, we also set self.temp_credentials.

reclaim_task will be None if there hasn’t been a claimed task yet, or if a task has been claimed more recently than the most recent reclaimTask call.

session = None
task = None
temp_credentials

dict – The latest temp credentials, or None if we haven’t claimed a task yet.

When setting, create self.temp_queue from the temp taskcluster creds.

temp_queue = None
write_json(path, contents, message)

Write json to disk.

Parameters:
  • path (str) – the path to write to
  • contents (dict) – the contents of the json blob
  • message (str) – the message to log

scriptworker.exceptions module

scriptworker exceptions.

exception scriptworker.exceptions.CoTError(msg)

Bases: scriptworker.exceptions.ScriptWorkerTaskException, KeyError

Failure in Chain of Trust verification.

exit_code

int – this is set to 3 (malformed-payload).

exception scriptworker.exceptions.ConfigError

Bases: scriptworker.exceptions.ScriptWorkerException

Invalid configuration provided to scriptworker.

exit_code

int – this is set to 5 (internal-error).

exception scriptworker.exceptions.DownloadError(msg)

Bases: scriptworker.exceptions.ScriptWorkerTaskException

Failure in scriptworker.utils.download_file.

exit_code

int – this is set to 4 (resource-unavailable).

exception scriptworker.exceptions.ScriptWorkerException

Bases: Exception

The base exception in scriptworker.

When raised inside of the run_loop loop, set the taskcluster task status to at least self.exit_code.

exit_code

int – this is set to 5 (internal-error).

exit_code = 5
exception scriptworker.exceptions.ScriptWorkerGPGException

Bases: scriptworker.exceptions.ScriptWorkerException

Scriptworker GPG error.

exit_code

int – this is set to 5 (internal-error).

exit_code = 5
exception scriptworker.exceptions.ScriptWorkerRetryException

Bases: scriptworker.exceptions.ScriptWorkerException

Scriptworker retry error.

exit_code

int – this is set to 4 (resource-unavailable)

exit_code = 4
exception scriptworker.exceptions.ScriptWorkerTaskException(*args, *, exit_code=1, **kwargs)

Bases: scriptworker.exceptions.ScriptWorkerException

Scriptworker task error.

To use:

import sys
try:
    ...
except ScriptWorkerTaskException as exc:
    log.exception("log message")
    sys.exit(exc.exit_code)
exit_code

int – this is 1 by default (failure)

exception scriptworker.exceptions.TaskVerificationError(msg)

Bases: scriptworker.exceptions.ScriptWorkerTaskException

Verification error on a Taskcluster task.

Use it when your script fails to validate any input from the task definition

scriptworker.gpg module

GPG support.

These currently assume gpg 2.0.x

These GPG functions expose considerable functionality over gpg key management, data signatures, and validation, but by no means are they intended to cover all gnupg functionality. They are intended for automated key management and validation for scriptworker.

scriptworker.gpg.log

logging.Logger – the log object for this module.

scriptworker.gpg.GPG_CONFIG_MAPPING

dict – This maps the scriptworker config key names to the python-gnupg names.

scriptworker.gpg.GPG(context, gpg_home=None)

Get a python-gnupg GPG instance based on the settings in context.

Parameters:
  • context (scriptworker.context.Context) – the scriptworker context.
  • gpg_home (str, optional) – override context.config['gpg_home'] if desired. Defaults to None.
Returns:

the GPG instance with the appropriate configs.

Return type:

gnupg.GPG

scriptworker.gpg.build_gpg_homedirs_from_repo(context, tag, basedir=None, verify_function=<function verify_signed_tag>, flat_function=<function rebuild_gpg_home_flat>, signed_function=<function rebuild_gpg_home_signed>)

Build gpg homedirs in basedir, from the context-defined git repo.

Parameters:
  • context (scriptworker.context.Context) – the scriptworker context.
  • tag (str) – the tag name to verify
  • basedir (str, optional) – the path to the base directory to create the gpg homedirs in. This directory will be wiped if it exists. If None, use context.config['base_gpg_home_dir']. Defaults to None.
Returns:

on success.

Return type:

str

Raises:

ScriptWorkerGPGException – on rebuild exception.

scriptworker.gpg.check_ownertrust(context, gpg_home=None)

In theory, this will repair a broken trustdb.

Rebuild the trustdb via –import-ownertrust if not.

Parameters:
  • context (scriptworker.context.Context) – the scriptworker context.
  • gpg_home (str, optional) – override the gpg_home with a different gnupg home directory here. Defaults to None.
scriptworker.gpg.consume_valid_keys(context, keydir=None, ignore_suffixes=(), gpg_home=None)

Given a keydir, traverse the keydir, and import all gpg public keys.

Parameters:
  • context (scriptworker.context.Context) – the scriptworker context.
  • keydir (str, optional) – the path of the directory to traverse. If None, this function is noop. Default is None.
  • ignore_suffixes (list, optional) – file suffixes to ignore. Default is ().
  • gpg_home (str, optional) – override the gpg_home dir. Default is None.
Returns:

fingerprints

Return type:

list

Raises:

ScriptworkerGPGException – on error.

scriptworker.gpg.create_gpg_conf(gpg_home, keyserver=None, my_fingerprint=None)

Create a gpg.conf with Mozilla infosec guidelines.

Parameters:
  • gpg_home (str) – the homedir for this keyring.
  • keyserver (str, optional) – The gpg keyserver to specify, e.g. hkp://gpg.mozilla.org or hkp://keys.gnupg.net. If set, we also enable auto-key-retrieve. Defaults to None.
  • my_fingerprint (str, optional) – the fingerprint of the default key. Once set, gpg will use it by default, unless a different key is specified. Defaults to None.
scriptworker.gpg.create_lockfile(context, message='locked')

Create the lockfile.

Parameters:context (scriptworker.context.Context) – the scriptworker context
scriptworker.gpg.export_key(gpg, fingerprint, private=False)

Return the ascii armored key identified by fingerprint.

Parameters:
  • gpg (gnupg.GPG) – the GPG instance.
  • fingerprint (str) – the fingerprint of the key to export.
  • private (bool, optional) – If True, return the private key instead of the public key. Defaults to False.
Returns:

the ascii armored key identified by fingerprint.

Return type:

str

Raises:

ScriptworkerGPGException – if the key isn’t found.

scriptworker.gpg.fingerprint_to_keyid(gpg, fingerprint, private=False)

Return the keyid of the key that corresponds to fingerprint.

Keyids should default to long keyids; this will happen once create_gpg_conf() is called.

Parameters:
  • gpg (gnupg.GPG) – gpg object for the appropriate gpg_home / keyring
  • fingerpint (str) – the fingerprint of the key we’re searching for.
  • private (bool, optional) – If True, search the private keyring instead of the public keyring. Defaults to False.
Returns:

keyid – the keyid of the key with fingerprint fingerprint

Return type:

str

Raises:

ScriptworkerGPGException – if we can’t find fingerprint in this keyring.

scriptworker.gpg.generate_key(gpg, name, comment, email, key_length=4096, expiration=None)

Generate a gpg keypair.

Parameters:
  • gpg (gnupg.GPG) – the GPG instance.
  • name (str) – the name attached to the key. 1/3 of the key user id.
  • comment (str) – the comment attached to the key. 1/3 of the key user id.
  • email (str) – the email attached to the key. 1/3 of the key user id.
  • key_length (int, optional) – the key length in bits. Defaults to 4096.
  • expiration (str, optional) – The expiration of the key. This can take the forms “2009-12-31”, “365d”, “3m”, “6w”, “5y”, “seconds=<epoch>”, or 0 for no expiry. Defaults to None.
Returns:

fingerprint – the fingerprint of the key just generated.

Return type:

str

scriptworker.gpg.get_body(gpg, signed_data, gpg_home=None, verify_sig=True, **kwargs)

Verify the signature, then return the unsigned data from signed_data.

Parameters:
  • gpg (gnupg.GPG) – the GPG instance.
  • signed_data (str) – The ascii armored signed data.
  • gpg_home (str, optional) – override the gpg_home with a different gnupg home directory here. Defaults to None.
  • verify_sig (bool, optional) – verify the signature before decrypting. Defaults to True.
  • kwargs (dict, optional) – These are passed directly to gpg.decrypt(). Defaults to {}. https://pythonhosted.org/python-gnupg/#decryption
Returns:

unsigned contents on success.

Return type:

str

Raises:

ScriptWorkerGPGException – on signature verification failure.

scriptworker.gpg.get_git_revision(path, ref='HEAD', exec_function=<function create_subprocess_exec>)

Get the git revision of path.

Parameters:
  • path (str) – the path to run git log -n1 --format=format:%H REF in.
  • ref (str, optional) – the ref to find the revision for. Defaults to “HEAD”
Returns:

the revision found.

Return type:

str

Raises:

ScriptWorkerRetryException – on failure.

scriptworker.gpg.get_last_good_git_revision(context)

Return the contents of the config[‘last_good_git_revision_file’], if it exists.

Parameters:context (scriptworker.context.Context) – the scriptworker context.
Returns:the latest good git revision, if the file exists None: if the file doesn’t exist
Return type:str
scriptworker.gpg.get_latest_tag(path, exec_function=<function create_subprocess_exec>)

Get the latest tag in path.

Parameters:path (str) – the path to run git describe --abbrev=0 in.
Returns:the tag name found.
Return type:str
Raises:ScriptWorkerRetryException – on failure.
scriptworker.gpg.get_list_sigs_output(context, key_fingerprint, gpg_home=None, validate=True, expected=None)

Get output from gpg –list-sigs.

This will be machine parsable output, for gpg 2.0.x.

Parameters:
  • context (scriptworker.context.Context) – the scriptworker context.
  • key_fingerprint (str) – the fingerprint of the key we want to get signature information about.
  • gpg_home (str, optional) – override the gpg_home with a different gnupg home directory here. Defaults to None.
  • validate (bool, optional) – Validate the output via parse_list_sigs_output() Defaults to True.
  • expected (dict, optional) – This is passed on to parse_list_sigs_output() if validate is True. Defaults to None.
Returns:

the output from gpg –list-sigs, if validate is False dict: the output from parse_list_sigs_output, if validate is True

Return type:

str

Raises:

ScriptWorkerGPGException – if there is an issue with the key.

scriptworker.gpg.get_tmp_base_gpg_home_dir(context)

Return the base_gpg_home_dir with a .tmp at the end.

This function is really only here so we don’t have to duplicate this logic.

Parameters:context (scriptworker.context.Context) – the scriptworker context.
Returns:the base_gpg_home_dir with .tmp at the end.
Return type:str
scriptworker.gpg.gpg_default_args(gpg_home)

For commandline gpg calls, use these args by default.

Parameters:gpg_home (str) – The path to the gpg homedir. gpg will look for the gpg.conf, trustdb.gpg, and keyring files in here.
Returns:the list of default commandline arguments to add to the gpg call.
Return type:list
scriptworker.gpg.guess_gpg_home(obj, gpg_home=None)

Guess gpg_home. If gpg_home is specified, return that.

Parameters:
  • obj (object) – If gpg_home is set, return that. Otherwise, if obj is a context object and context.config['gpg_home'] is not None, return that. If obj is a GPG object and obj.gnupghome is not None, return that. Otherwise look in ~/.gnupg.
  • gpg_home (str, optional) – The path to the gpg homedir. gpg will look for the gpg.conf, trustdb.gpg, and keyring files in here. Defaults to None.
Returns:

the path to the guessed gpg homedir.

Return type:

str

Raises:

ScriptWorkerGPGException – if obj doesn’t contain the gpg home info and os.environ[‘HOME’] isn’t set.

scriptworker.gpg.guess_gpg_path(context)

Guess gpg_path.

Parameters:context (scriptworker.context.Context) – the scriptworker context.
Returns:either context.config['gpg_path'] or ‘gpg’ if that’s not defined.
Return type:str
scriptworker.gpg.has_suffix(path, suffixes)

Given a list of suffixes, return True if path ends with one of them.

Parameters:
  • path (str) – the file path to check
  • suffixes (list) – the suffixes to check for
scriptworker.gpg.import_key(gpg, key_data, return_type='fingerprints')

Import ascii key_data.

In theory this can be multiple keys. However, jenkins is barfing on multiple key import tests, although multiple key import tests are working locally. Until we identify what the problem is (likely gpg version?) we should only import 1 key at a time.

Parameters:
  • gpg (gnupg.GPG) – the GPG instance.
  • key_data (str) – ascii armored key data
  • return_type (str, optional) – if ‘fingerprints’, return the fingerprints only. Otherwise return the result list.
Returns:

if return_type is ‘fingerprints’, return the fingerprints of the

imported keys. Otherwise return the results list. https://pythonhosted.org/python-gnupg/#importing-and-receiving-keys

Return type:

list

scriptworker.gpg.is_lockfile_present(context, name, level=30)

Check for the lockfile.

Parameters:
  • context (scriptworker.context.Context) – the scriptworker context
  • name (str) – the name of the calling function
  • level (int, optional) – the level to log to. Defaults to logging.WARNING
Returns:

“locked” on r/w lock; “ready” if ready to copy. None: if lockfile is not present

Return type:

str

scriptworker.gpg.keyid_to_fingerprint(gpg, keyid, private=False)

Return the fingerprint of the key that corresponds to keyid.

Keyids should default to long keyids; this will happen once create_gpg_conf() is called.

Parameters:
  • gpg (gnupg.GPG) – gpg object for the appropriate gpg_home / keyring
  • keyid (str) – the long keyid that represents the key we’re searching for.
  • private (bool, optional) – If True, search the private keyring instead of the public keyring. Defaults to False.
Returns:

fingerprint – the fingerprint of the key with keyid keyid

Return type:

str

Raises:

ScriptworkerGPGException – if we can’t find keyid in this keyring.

scriptworker.gpg.overwrite_gpg_home(tmp_gpg_home, real_gpg_home)

Take the contents of tmp_gpg_home and copy them to real_gpg_home.

Parameters:
  • tmp_gpg_home (str) – path to the rebuilt gpg_home with the new keychains+ trust models
  • real_gpg_home (str) – path to the old gpg_home to overwrite
scriptworker.gpg.parse_list_sigs_output(output, desc, expected=None)

Parse the output from –list-sigs; validate.

NOTE: This doesn’t work with complex key/subkeys; this is only written for the keys generated through the functions in this module.

  1. Field: Type of record
    pub = public key crt = X.509 certificate crs = X.509 certificate and private key available sub = subkey (secondary key) sec = secret key ssb = secret subkey (secondary key) uid = user id (only field 10 is used). uat = user attribute (same as user id except for field 10). sig = signature rev = revocation signature fpr = fingerprint: (fingerprint is in field 10) pkd = public key data (special field format, see below) grp = keygrip rvk = revocation key tru = trust database information spk = signature subpacket

There are also ‘gpg’ lines like

gpg: checking the trustdb gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model gpg: depth: 0 valid: 3 signed: 0 trust: 0-, 0q, 0n, 0m, 0f, 3u

This is a description of the web of trust. I’m currently not parsing these; per [1] and [2] I would need to read the source for full parsing.

[1] http://security.stackexchange.com/a/41209

[2] http://gnupg.10057.n7.nabble.com/placing-trust-in-imported-keys-td30124.html#a30125

Parameters:
  • output (str) – the output from get_list_sigs_output()
  • desc (str) – a description of the key being tested, for exception message purposes.
  • expected (dict, optional) – expected outputs. If specified and the expected doesn’t match the real, raise an exception. Expected takes keyid, fingerprint, uid, sig_keyids (list), and sig_uids (list), all optional. Defaults to None.
Returns:

real

the real values from the key. This specifies

keyid, fingerprint, uid, sig_keyids, and sig_uids.

Return type:

dict

Raises:

ScriptWorkerGPGException – on mismatched expectations, or if we found revocation markers or the like that make for a bad key.

scriptworker.gpg.rebuild_gpg_home(context, tmp_gpg_home, my_pub_key_path, my_priv_key_path)

Import my key and create gpg.conf and trustdb.gpg.

Parameters:
  • gpg (gnupg.GPG) – the GPG instance.
  • tmp_gpg_home (str) – the path to the tmp gpg_home. This should already exist.
  • my_pub_key_path (str) – the ascii public key file we want to import as the primary key
  • my_priv_key_path (str) – the ascii private key file we want to import as the primary key
Returns:

my fingerprint

Return type:

str

scriptworker.gpg.rebuild_gpg_home_flat(context, real_gpg_home, my_pub_key_path, my_priv_key_path, consume_path, ignore_suffixes=(), consume_function=<function consume_valid_keys>)

Rebuild real_gpg_home with new trustdb, pub+secrings, gpg.conf.

In this ‘flat’ model, import all the pubkeys in consume_path and sign them directly. This makes them valid but not trusted.

Parameters:
  • context (scriptworker.context.Context) – the scriptworker context.
  • real_gpg_home (str) – the gpg_home path we want to rebuild
  • my_pub_key_path (str) – the ascii public key file we want to import as the primary key
  • my_priv_key_path (str) – the ascii private key file we want to import as the primary key
  • consume_path (str) – the path to the directory tree to import pubkeys from
  • ignore_suffixes (list, optional) – the suffixes to ignore in consume_path. Defaults to ()
scriptworker.gpg.rebuild_gpg_home_signed(context, real_gpg_home, my_pub_key_path, my_priv_key_path, trusted_path, untrusted_path=None, ignore_suffixes=(), consume_function=<function consume_valid_keys>)

Rebuild real_gpg_home with new trustdb, pub+secrings, gpg.conf.

In this ‘signed’ model, import all the pubkeys in trusted_path, sign them directly, and trust them. Then import all the pubkeys in untrusted_path with no signing. The intention is that one of the keys in trusted_path has already signed the keys in untrusted_path, making them valid.

Parameters:
  • context (scriptworker.context.Context) – the scriptworker context.
  • real_gpg_home (str) – the gpg_home path we want to rebuild
  • my_pub_key_path (str) – the ascii public key file we want to import as the primary key
  • my_priv_key_path (str) – the ascii private key file we want to import as the primary key
  • trusted_path (str) – the path to the directory tree to import trusted pubkeys from
  • untrusted_path (str, optional) – the path to the directory tree to import untrusted but valid pubkeys from
  • ignore_suffixes (list, optional) – the suffixes to ignore in consume_path. Defaults to ()
scriptworker.gpg.rebuild_gpg_homedirs(event_loop=None)

Rebuild the gpg homedirs in the background.

This is an entry point, and should be called before scriptworker is run.

Parameters:event_loop (asyncio.BaseEventLoop, optional) – the event loop to use. If None, use asyncio.get_event_loop(). Defaults to None.
Raises:SystemExit – on failure.
scriptworker.gpg.rm_lockfile(context)

Remove the lockfile.

Parameters:context (scriptworker.context.Context) – the scriptworker context
scriptworker.gpg.sign(gpg, data, **kwargs)

Sign data with the key kwargs['keyid'], or the default key if not specified.

Parameters:
Returns:

the ascii armored signed data.

Return type:

str

scriptworker.gpg.sign_key(context, target_fingerprint, signing_key=None, exportable=False, gpg_home=None)

Sign the target_fingerprint key with signing_key or default key.

This signs the target key with the signing key, which adds to the web of trust.

Due to pexpect async issues, this function is once more synchronous.

Parameters:
  • context (scriptworker.context.Context) – the scriptworker context.
  • target_fingerprint (str) – the fingerprint of the key to sign.
  • signing_key (str, optional) – the fingerprint of the signing key to sign with. If not set, this defaults to the default-key in the gpg.conf. Defaults to None.
  • exportable (bool, optional) – whether the signature should be exportable. Defaults to False.
  • gpg_home (str, optional) – override the gpg_home with a different gnupg home directory here. Defaults to None.
Raises:

ScriptWorkerGPGException – on a failed signature.

scriptworker.gpg.update_ownertrust(context, my_fingerprint, trusted_fingerprints=None, gpg_home=None)

Trust my key ultimately; trusted_fingerprints fully.

Parameters:
  • context (scriptworker.context.Context) – the scriptworker context.
  • my_fingerprint (str) – the fingerprint of the key we want to specify as ultimately trusted.
  • trusted_fingerprints (list, optional) – the list of fingerprints that we want to mark as fully trusted. These need to be signed by the my_fingerprint key before they are trusted.
  • gpg_home (str, optional) – override the gpg_home with a different gnupg home directory here. Defaults to None.
Raises:

ScriptWorkerGPGException – if there is an error.

scriptworker.gpg.update_signed_git_repo(context, repo='origin', ref='master', exec_function=<function create_subprocess_exec>, log_function=<function pipe_to_log>)

Update a git repo with signed git commits, and verify the signature.

This function updates the repo.

Parameters:
  • context (scriptworker.context.Context) – the scriptworker context.
  • repo (str, optional) – the repo to update from. Defaults to ‘origin’.
  • ref (str, optional) – the ref to update to. Defaults to ‘master’.
Returns:

tuple – the current git revision, and the latest tag name.

Return type:

str, str

Raises:
  • ScriptWorkerGPGException – on signature validation failure.
  • ScriptWorkerRetryException – on git pull failure.
scriptworker.gpg.verify_ownertrust(context, my_fingerprint, trusted_fingerprints=None, gpg_home=None)

Verify the ownertrust is exactly as expected.

Parameters:
  • context (scriptworker.context.Context) – the scriptworker context.
  • my_fingerprint (str) – the fingerprint of the key we specified as ultimately trusted.
  • trusted_fingerprints (list, optional) – the list of fingerprints that we marked as fully trusted.
  • gpg_home (str, optional) – override the gpg_home with a different gnupg home directory here. Defaults to None.
Raises:

ScriptWorkerGPGException – if there is an error.

scriptworker.gpg.verify_signature(gpg, signed_data, **kwargs)

Verify signed_data with the key kwargs['keyid'], or the default key if not specified.

Parameters:
Returns:

on success.

Return type:

gnupg.Verify

Raises:

ScriptWorkerGPGException – on failure.

scriptworker.gpg.verify_signed_tag(context, tag, exec_function=<function check_call>)

Verify git_key_repo_dir is at the valid signed tag.

Parameters:
Raises:

ScriptWorkerGPGException – if we’re not updated to tag

scriptworker.gpg.write_last_good_git_revision(context, revision)

Write revision to config[‘last_good_git_revision_file’].

Parameters:

scriptworker.log module

Scriptworker logging.

scriptworker.log.log

logging.Logger – the log object for this module.

scriptworker.log.contextual_log_handler(context, path, log_obj=None, level=10, formatter=None)

Add a short-lived log with a contextmanager for cleanup.

Parameters:
  • context (scriptworker.context.Context) – the scriptworker context
  • path (str) – the path to the log file to create
  • log_obj (logging.Logger) – the log object to modify. If None, use scriptworker.log.log. Defaults to None.
  • level (int, optional) – the logging level. Defaults to logging.DEBUG.
  • formatter (logging.Formatter, optional) – the logging formatter. If None, defaults to logging.Formatter(fmt=fmt). Default is None.
Yields:

None – but cleans up the handler afterwards.

scriptworker.log.get_log_filehandle(context)

Open the log and error filehandles.

Parameters:context (scriptworker.context.Context) – the scriptworker context.
Yields:log filehandle
scriptworker.log.get_log_filename(context)

Get the task log/error file paths.

Parameters:context (scriptworker.context.Context) – the scriptworker context.
Returns:log file path
Return type:string
scriptworker.log.pipe_to_log(pipe, filehandles=(), level=20)

Log from a subprocess PIPE.

Parameters:
  • pipe (filehandle) – subprocess process STDOUT or STDERR
  • filehandles (list of filehandles, optional) – the filehandle(s) to write to. If empty, don’t write to a separate file. Defaults to ().
  • level (int, optional) – the level to log to. Defaults to logging.INFO.
scriptworker.log.update_logging_config(context, log_name=None, file_name='worker.log')

Update python logging settings from config.

By default, this sets the scriptworker log settings, but this will change if some other package calls this function or specifies the log_name.

  • Use formatting from config settings.
  • Log to screen if verbose
  • Add a rotating logfile from config settings.
Parameters:
  • context (scriptworker.context.Context) – the scriptworker context.
  • log_name (str, optional) – the name of the Logger to modify. If None, use the top level module (‘scriptworker’). Defaults to None.

scriptworker.task module

Scriptworker task execution.

scriptworker.task.KNOWN_TASKS_FOR

tuple – the known reasons for creating decision/action tasks.

scriptworker.task.REPO_SCOPE_REGEX

regex – the regex for the repo_scope of a task

scriptworker.task.log

logging.Logger – the log object for the module

scriptworker.task.claim_work(context)

Find and claim the next pending task in the queue, if any.

Parameters:context (scriptworker.context.Context) – the scriptworker context.
Returns:a dict containing a list of the task definitions of the tasks claimed.
Return type:dict
scriptworker.task.complete_task(context, result)

Mark the task as completed in the queue.

Decide whether to call reportCompleted, reportFailed, or reportException based on the exit status of the script.

If the task has expired or been cancelled, we’ll get a 409 status.

Parameters:context (scriptworker.context.Context) – the scriptworker context.
Raises:taskcluster.exceptions.TaskclusterRestFailure – on non-409 error.
scriptworker.task.get_action_name(task)

Get the name of an action task.

Parameters:obj (ChainOfTrust or LinkOfTrust) – the trust object to inspect
Returns:the name.
Return type:str
scriptworker.task.get_and_check_project(valid_vcs_rules, source_url)

Given vcs rules and a source_url, return the project.

The project is in the path, but is the repo name. releases/mozilla-beta is the path; mozilla-beta is the project.

Parameters:
  • valid_vcs_rules (tuple of frozendicts) – the valid vcs rules, per match_url_regex.
  • source_url (str) – the source url to find the project for.
Raises:

RuntimeError – on failure to find the project.

Returns:

the project.

Return type:

str

scriptworker.task.get_and_check_tasks_for(task, msg_prefix='')

Given a parent task, return the reason the parent task was spawned.

.taskcluster.yml uses this to know whether to spawn an action, cron, or decision task definition. The current known tasks_for are in KNOWN_TASKS_FOR.

Parameters:
  • task (dict) – the task definition.
  • msg_prefix (str) – the string prefix to use for an exception.
Raises:

(KeyError, ValueError) – on failure to find a valid tasks_for.

Returns:

the tasks_for

Return type:

str

scriptworker.task.get_commit_message(task)

Get the commit message for a task.

Parameters:obj (ChainOfTrust or LinkOfTrust) – the trust object to inspect
Returns:the commit message.
Return type:str
scriptworker.task.get_decision_task_id(task)

Given a task dict, return the decision taskId.

By convention, the decision task of the taskId is the task’s taskGroupId.

Parameters:task (dict) – the task dict.
Returns:the taskId of the decision task.
Return type:str
scriptworker.task.get_parent_task_id(task)

Given a task dict, return the parent taskId.

The parent taskId could be a decision taskId, or an action taskId. The parent is the task that created this task; it should have a task-graph.json containing this task’s definition as an artifact.

Parameters:task (dict) – the task dict
Returns:the taskId of the parent.
Return type:str
scriptworker.task.get_repo(task, source_env_prefix)

Get the repo for a task.

Parameters:
  • task (ChainOfTrust or LinkOfTrust) – the trust object to inspect
  • source_env_prefix (str) – The environment variable prefix that is used to get repository information.
Returns:

the source url. None: if not defined for this task.

Return type:

str

scriptworker.task.get_repo_scope(task, name)

Given a parent task, return the repo scope for the task.

Background in https://bugzilla.mozilla.org/show_bug.cgi?id=1459705#c3

Parameters:task (dict) – the task definition.
Raises:ValueError – on too many `repo_scope`s (we allow for 1 or 0).
Returns:the repo_scope None: if no repo_scope is found
Return type:str
scriptworker.task.get_revision(task, source_env_prefix)

Get the revision for a task.

Parameters:
  • obj (ChainOfTrust or LinkOfTrust) – the trust object to inspect
  • source_env_prefix (str) – The environment variable prefix that is used to get repository information.
Returns:

the revision. None: if not defined for this task.

Return type:

str

scriptworker.task.get_run_id(claim_task)

Given a claim_task json dict, return the runId.

Parameters:claim_task (dict) – the claim_task dict.
Returns:the runId.
Return type:int
scriptworker.task.get_task_id(claim_task)

Given a claim_task json dict, return the taskId.

Parameters:claim_task (dict) – the claim_task dict.
Returns:the taskId.
Return type:str
scriptworker.task.get_worker_type(task)

Given a task dict, return the workerType.

Parameters:task (dict) – the task dict.
Returns:the workerType.
Return type:str
scriptworker.task.is_action(task)

Determine if a task is an action task.

Trusted decision and action tasks are important in that they can generate other valid tasks. The verification of decision and action tasks is slightly different, so we need to be able to tell them apart.

This checks for the following things:

* ``task.payload.env.ACTION_CALLBACK`` exists
* ``task.extra.action`` exists
Parameters:task (dict) – the task definition to check
Returns:True if it’s an action
Return type:bool
scriptworker.task.is_try(task, source_env_prefix)

Determine if a task is a ‘try’ task (restricted privs).

This goes further than get_repo. We may or may not want to keep this.

This checks for the following things:

* ``task.payload.env.GECKO_HEAD_REPOSITORY`` == "https://hg.mozilla.org/try/"
* ``task.payload.env.MH_BRANCH`` == "try"
* ``task.metadata.source`` == "https://hg.mozilla.org/try/..."
* ``task.schedulerId`` in ("gecko-level-1", )
Parameters:
  • task (dict) – the task definition to check
  • source_env_prefix (str) – The environment variable prefix that is used to get repository information.
Returns:

True if it’s try

Return type:

bool

scriptworker.task.kill_pid(pid, sleep_time=1)

Kill pid with various signals.

Parameters:
  • pid (int) – the process id to kill.
  • sleep_time (int, optional) – how long to sleep between killing the pid and checking if the pid is still running.
scriptworker.task.kill_proc(proc, message, exit_code)

Make sure the proc pid’s process and process group are killed.

First, terminate the process, then kill the process group (-pid), then kill the process.

Then raise a ScriptWorkerTaskException with exit_code.

Parameters:
  • proc (asyncio.subprocess.Process) – the process to kill
  • message (str) – the error message
  • exit_code (int) – the exit code to specify in ScriptWorkerTaskException (should be part of STATUSES)
Raises:

ScriptWorkerTaskException – after killing the task.

Returns:

if no proc to kill

Return type:

None

scriptworker.task.prepare_to_run_task(context, claim_task)

Given a claim_task json dict, prepare the context and work_dir.

Set context.claim_task, and write a work_dir/current_task_info.json

Parameters:
Returns:

the contents of current_task_info.json

Return type:

dict

scriptworker.task.reclaim_task(context, task)

Try to reclaim a task from the queue.

This is a keepalive / heartbeat. Without it the job will expire and potentially be re-queued. Since this is run async from the task, the task may complete before we run, in which case we’ll get a 409 the next time we reclaim.

Parameters:context (scriptworker.context.Context) – the scriptworker context
Raises:taskcluster.exceptions.TaskclusterRestFailure – on non-409 status_code from taskcluster.aio.Queue.reclaimTask()
scriptworker.task.run_task(context)

Run the task, sending stdout+stderr to files.

https://github.com/python/asyncio/blob/master/examples/subprocess_shell.py

Parameters:context (scriptworker.context.Context) – the scriptworker context.
Returns:exit code
Return type:int
scriptworker.task.worst_level(level1, level2)

Given two int levels, return the larger.

Parameters:
  • level1 (int) – exit code 1.
  • level2 (int) – exit code 2.
Returns:

the larger of the two levels.

Return type:

int

scriptworker.utils module

Generic utils for scriptworker.

scriptworker.utils.log

logging.Logger – the log object for the module

scriptworker.utils.add_enumerable_item_to_dict(dict_, key, item)

Add an item to a list contained in a dict.

For example: If the dict is {'some_key': ['an_item']}, then calling this function will alter the dict to {'some_key': ['an_item', 'another_item']}.

If the key doesn’t exist yet, the function initializes it with a list containing the item.

List-like items are allowed. In this case, the existing list will be extended.

Parameters:
  • dict (dict) – the dict to modify
  • key (str) – the key to add the item to
  • item (whatever) – The item to add to the list associated to the key
scriptworker.utils.calculate_sleep_time(attempt, delay_factor=5.0, randomization_factor=0.5, max_delay=120)

Calculate the sleep time between retries, in seconds.

Based off of taskcluster.utils.calculateSleepTime, but with kwargs instead of constant delay_factor/randomization_factor/max_delay. The taskcluster function generally slept for less than a second, which didn’t always get past server issues.

Parameters:
  • attempt (int) – the retry attempt number
  • delay_factor (float, optional) – a multiplier for the delay time. Defaults to 5.
  • randomization_factor (float, optional) – a randomization multiplier for the delay time. Defaults to .5.
  • max_delay (float, optional) – the max delay to sleep. Defaults to 120 (seconds).
Returns:

the time to sleep, in seconds.

Return type:

float

scriptworker.utils.cleanup(context)

Clean up the work_dir and artifact_dir between task runs, then recreate.

Parameters:context (scriptworker.context.Context) – the scriptworker context.
scriptworker.utils.create_temp_creds(client_id, access_token, start=None, expires=None, scopes=None, name=None)

Request temp TC creds with our permanent creds.

Parameters:
  • client_id (str) – the taskcluster client_id to use
  • access_token (str) – the taskcluster access_token to use
  • start (str, optional) – the datetime string when the credentials will start to be valid. Defaults to 10 minutes ago, for clock skew.
  • expires (str, optional) – the datetime string when the credentials will expire. Defaults to 31 days after 10 minutes ago.
  • scopes (list, optional) – The list of scopes to request for the temp creds. Defaults to [‘assume:project:taskcluster:worker-test-scopes’, ]
  • name (str, optional) – the name to associate with the creds.
Returns:

the temporary taskcluster credentials.

Return type:

dict

scriptworker.utils.datestring_to_timestamp(datestring)

Create a timetamp from a taskcluster datestring.

Parameters:datestring (str) – the datestring to convert. isoformat, like “2016-04-16T03:46:24.958Z”
Returns:the corresponding timestamp.
Return type:int
scriptworker.utils.download_file(context, url, abs_filename, session=None, chunk_size=128)

Download a file, async.

Parameters:
  • context (scriptworker.context.Context) – the scriptworker context.
  • url (str) – the url to download
  • abs_filename (str) – the path to download to
  • session (aiohttp.ClientSession, optional) – the session to use. If None, use context.session. Defaults to None.
  • chunk_size (int, optional) – the chunk size to read from the response at a time. Default is 128.
scriptworker.utils.filepaths_in_dir(path)

Find all files in a directory, and return the relative paths to those files.

Parameters:path (str) – the directory path to walk
Returns:
the list of relative paths to all files inside of path or its
subdirectories.
Return type:list
scriptworker.utils.format_json(data)

Format json as a sorted string (indents of 2).

Parameters:data (dict) – the json to format.
Returns:the formatted json.
Return type:str
scriptworker.utils.get_hash(path, hash_alg='sha256')

Get the hash of the file at path.

I’d love to make this async, but evidently file i/o is always ready

Parameters:
  • path (str) – the path to the file to hash.
  • hash_alg (str, optional) – the algorithm to use. Defaults to ‘sha256’.
Returns:

the hexdigest of the hash.

Return type:

str

scriptworker.utils.get_loggable_url(url)

Strip out secrets from taskcluster urls.

Parameters:url (str) – the url to strip
Returns:the loggable url
Return type:str
scriptworker.utils.get_results_and_future_exceptions(tasks)

Given a list of futures, await them, then return results and exceptions.

This is similar to raise_future_exceptions, except that it doesn’t raise any exception. They are returned instead. This allows some tasks to optionally fail. Please consider that no exception will be raised when calling this function. You must verify the content of the second item of the tuple. It contains all exceptions raised by the futures.

Parameters:tasks (list) – the list of futures to await and check for exceptions.
Returns:the list of results from the futures, then the list of exceptions.
Return type:tuple
scriptworker.utils.get_single_item_from_sequence(sequence, condition, ErrorClass=<class 'ValueError'>, no_item_error_message='No item matched condition', too_many_item_error_message='Too many items matched condition', append_sequence_to_error_message=True)

Return an item from a python sequence based on the given condition.

Parameters:
  • sequence (sequence) – The sequence to filter
  • condition – A function that serves to filter items from sequence. Function must have one argument (a single item from the sequence) and return a boolean.
  • ErrorClass (Exception) – The error type raised in case the item isn’t unique
  • no_item_error_message (str) – The message raised when no item matched the condtion
  • too_many_item_error_message (str) – The message raised when more than one item matched the condition
  • append_sequence_to_error_message (bool) – Show or hide what was the tested sequence in the error message. Hiding it may prevent sensitive data (such as password) to be exposed to public logs
Returns:

The only item in the sequence which matched the condition

scriptworker.utils.load_json_or_yaml(string, is_path=False, file_type='json', exception=<class 'scriptworker.exceptions.ScriptWorkerTaskException'>, message='Failed to load %(file_type)s: %(exc)s')

Load json or yaml from a filehandle or string, and raise a custom exception on failure.

Parameters:
  • string (str) – json/yaml body or a path to open
  • is_path (bool, optional) – if string is a path. Defaults to False.
  • file_type (str, optional) – either “json” or “yaml”. Defaults to “json”.
  • exception (exception, optional) – the exception to raise on failure. If None, don’t raise an exception. Defaults to ScriptWorkerTaskException.
  • message (str, optional) – the message to use for the exception. Defaults to “Failed to load %(file_type)s: %(exc)s”
Returns:

the data from the string.

Return type:

dict

Raises:

Exception – as specified, on failure

scriptworker.utils.load_json_or_yaml_from_url(context, url, path, overwrite=True)

Retry a json/yaml file download, load it, then return its data.

Parameters:
  • context (scriptworker.context.Context) – the scriptworker context.
  • url (str) – the url to download
  • path (str) – the path to download to
  • overwrite (bool, optional) – if False and path exists, don’t download. Defaults to True.
Returns:

the url data.

Return type:

dict

Raises:

Exception – as specified, on failure

scriptworker.utils.makedirs(path)

Equivalent to mkdir -p.

Parameters:path (str) – the path to mkdir -p
Raises:ScriptWorkerException – if path exists already and the realpath is not a dir.
scriptworker.utils.match_url_path_callback(match)

Return the path, as a match_url_regex callback.

Parameters:match (re.match) – the regex match object from match_url_regex
Returns:the path matched in the regex.
Return type:string
scriptworker.utils.match_url_regex(rules, url, callback)

Given rules and a callback, find the rule that matches the url.

Rules look like:

(
    {
        'schemes': ['https', 'ssh'],
        'netlocs': ['hg.mozilla.org'],
        'path_regexes': [
            "^(?P<path>/mozilla-(central|unified))(/|$)",
        ]
    },
    ...
)
Parameters:
  • rules (list) – a list of dictionaries specifying lists of schemes, netlocs, and path_regexes.
  • url (str) – the url to test
  • callback (function) – a callback that takes an re.MatchObject. If it returns None, continue searching. Otherwise, return the value from the callback.
Returns:

the value from the callback, or None if no match.

Return type:

value

scriptworker.utils.noop_sync(*args, **kwargs)

Do nothing.

scriptworker.utils.raise_future_exceptions(tasks)

Given a list of futures, await them, then raise their exceptions if any.

Without something like this, a bare:

await asyncio.wait(tasks)

will swallow exceptions.

Parameters:tasks (list) – the list of futures to await and check for exceptions.
Returns:the list of results from the futures.
Return type:list
Raises:Exception – any exceptions in task.exception()
scriptworker.utils.remove_empty_keys(values, remove=({}, None, []'null'))

Recursively remove key/value pairs where the value is in remove.

This is targeted at comparing json-e rebuilt task definitions, since json-e drops key/value pairs with empty values.

Parameters:values (dict/list) – the dict or list to remove empty keys from.
Returns:values – a dict or list copy, with empty keys removed.
Return type:dict/list
scriptworker.utils.request(context, url, timeout=60, method='get', good=(200, ), retry=(500, 501, 502, 503, 504, 505, 506, 507, 508, 509, 510, 511), return_type='text', **kwargs)

Async aiohttp request wrapper.

Parameters:
  • context (scriptworker.context.Context) – the scriptworker context.
  • url (str) – the url to request
  • timeout (int, optional) – timeout after this many seconds. Default is 60.
  • method (str, optional) – The request method to use. Default is ‘get’.
  • good (list, optional) – the set of good status codes. Default is (200, )
  • retry (list, optional) – the set of status codes that result in a retry. Default is tuple(range(500, 512)).
  • return_type (str, optional) – The type of value to return. Takes ‘json’ or ‘text’; other values will return the response object. Default is text.
  • **kwargs – the kwargs to send to the aiohttp request function.
Returns:

the response text() if return_type is ‘text’; the response

json() if return_type is ‘json’; the aiohttp request response object otherwise.

Return type:

object

Raises:
  • ScriptWorkerRetryException – if the status code is in the retry list.
  • ScriptWorkerException – if the status code is not in the retry list or good list.
scriptworker.utils.retry_async(func, attempts=5, sleeptime_callback=<function calculate_sleep_time>, retry_exceptions=<class 'Exception'>, args=(), kwargs=None, sleeptime_kwargs=None)

Retry func, where func is an awaitable.

Parameters:
  • func (function) – an awaitable function.
  • attempts (int, optional) – the number of attempts to make. Default is 5.
  • sleeptime_callback (function, optional) – the function to use to determine how long to sleep after each attempt. Defaults to calculateSleepTime.
  • retry_exceptions (list or exception, optional) – the exception(s) to retry on. Defaults to Exception.
  • args (list, optional) – the args to pass to function. Defaults to ()
  • kwargs (dict, optional) – the kwargs to pass to function. Defaults to {}.
  • sleeptime_kwargs (dict, optional) – the kwargs to pass to sleeptime_callback. If None, use {}. Defaults to None.
Returns:

the value from a successful function call

Return type:

object

Raises:

Exception – the exception from a failed function call, either outside of the retry_exceptions, or one of those if we pass the max attempts.

scriptworker.utils.retry_request(*args, *, retry_exceptions=(<class 'concurrent.futures._base.TimeoutError'>, <class 'scriptworker.exceptions.ScriptWorkerRetryException'>), retry_async_kwargs=None, **kwargs)

Retry the request function.

Parameters:
  • *args – the args to send to request() through retry_async().
  • retry_exceptions (list, optional) – the exceptions to retry on. Defaults to (ScriptWorkerRetryException, ).
  • retry_async_kwargs (dict, optional) – the kwargs for retry_async. If None, use {}. Defaults to None.
  • **kwargs – the kwargs to send to request() through retry_async().
Returns:

the value from request().

Return type:

object

scriptworker.utils.rm(path)

Equivalent to rm -rf.

Make sure path doesn’t exist after this call. If it’s a dir, shutil.rmtree(); if it’s a file, os.remove(); if it doesn’t exist, ignore.

Parameters:path (str) – the path to nuke.
scriptworker.utils.to_unicode(line)

Avoid b'line' type messages in the logs.

Parameters:line (str) – The bytecode or unicode string.
Returns:
the unicode-decoded string, if line was a bytecode string.
Otherwise return line unmodified.
Return type:str

scriptworker.worker module

Scriptworker worker functions.

scriptworker.worker.log

logging.Logger – the log object for the module.

scriptworker.worker.async_main(context, credentials)

Set up and run tasks for this iteration.

http://docs.taskcluster.net/queue/worker-interaction/

Parameters:context (scriptworker.context.Context) – the scriptworker context.
scriptworker.worker.do_run_task(context)

Run the task logic.

Returns the integer status of the task.

Parameters:context (scriptworker.context.Context) – the scriptworker context.
Raises:Exception – on unexpected exception.
Returns:exit status
Return type:int
scriptworker.worker.do_upload(context)

Upload artifacts and return status.

Returns the integer status of the upload.

Parameters:context (scriptworker.context.Context) – the scriptworker context.
Raises:Exception – on unexpected exception.
Returns:exit status
Return type:int
scriptworker.worker.main(event_loop=None)

Scriptworker entry point: get everything set up, then enter the main loop.

Parameters:event_loop (asyncio.BaseEventLoop, optional) – the event loop to use. If None, use asyncio.get_event_loop(). Defaults to None.
scriptworker.worker.run_tasks(context, creds_key='credentials')

Run any tasks returned by claimWork.

Returns the integer status of the task that was run, or None if no task was run.

Parameters:
  • context (scriptworker.context.Context) – the scriptworker context.
  • creds_key (str, optional) – when reading the creds file, this dict key corresponds to the credentials value we want to use. Defaults to “credentials”.
Raises:

Exception – on unexpected exception.

Returns:

exit status None: if no task run.

Return type:

int

Module contents

Scriptworker.