Scriptworker Readme

Scriptworker implements the TaskCluster worker model, then launches a pre-defined script.

This worker was designed for Releng processes that need specific, limited, and pre-defined capabilities.

Free software: MPL2 License

Usage

• Create a config file. By default scriptworker will look in ./scriptworker.yaml, but this config path can be specified as the first and only commandline argument. There is an example config file, and all config items are specified in scriptworker.constants.DEFAULT_CONFIG.

Credentials can live in ./scriptworker.yaml, ./secrets.json, ~/.scriptworker.

• Launch: scriptworker [config_path]

Testing

Without integration tests install tox, then

NO_CREDENTIALS_TESTS=1 tox -e py36

Without any tests connecting to the net, then NO_TESTS_OVER_WIRE=1 tox -e py36

With integration tests, first create a client in the Taskcluster UI with the scopes:

queue:cancel-task:test-dummy-scheduler/*
queue:claim-work:test-dummy-provisioner/dummy-worker-*
queue:create-task:lowest:test-dummy-provisioner/dummy-worker-*
queue:define-task:test-dummy-provisioner/dummy-worker-*
queue:get-artifact:SampleArtifacts/_/X.txt
queue:scheduler-id:test-dummy-scheduler
queue:schedule-task:test-dummy-scheduler/*
queue:task-group-id:test-dummy-scheduler/*
queue:worker-id:test-dummy-workers/dummy-worker-*

Then generate a no priviledge personal access token in Github for the scriptworker_github_token (to avoid rate limiting) and create a ./secrets.json or ~/.scriptworker that looks like:

{
    "integration_credentials": {
        "clientId": "...",
        "accessToken": "...",
    }
    "scriptworker_github_token": "..."
}

then to run all tests: tox

Table of Contents

• Flow of Control
• Maintenance
  • New docker shas
  • Chain of Trust settings
  • Ed25519 keys
• Scriptworker Releases
  • Code changes
  • Tests and test coverage
  • Versioning
  • Changelog
  • Release files
    • Versioning
  • Create a Release
  • Rollout
• Adding new scriptworker instance types
  • Is scriptworker the right tool?
    • Chain of Trust considerations
  • Creating a new scriptworker instance type
    • Write a script
  • Deployment considerations
    • Graph
    • GCP
• Adding new scriptworker instances of an existing type
• Chain of Trust
  • Overview
    • High level view
  • Chain of Trust Key Management
    • verifying new ed25519 keys
    • Rotating the FirefoxCI CoT keys
  • Chain of Trust Artifact Generation
    • Embedded ed25519 keys
    • Chain of Trust artifacts
  • Chain of Trust Verification
    • Decision Task
    • Ed25519 key management
    • Building the chain
    • Verifying the chain
  • Chain of Trust Testing / debugging
    • Create the virtualenv
    • Set up the test env
    • Find a task to test
    • Run the test
• Scriptworker Readme
  • Usage
  • Testing
• scriptworker package
  • Submodules
  • scriptworker.artifacts module
    • assert_is_parent()
    • compress_artifact_if_supported()
    • create_artifact()
    • download_artifacts()
    • get_and_check_single_upstream_artifact_full_path()
    • get_artifact_url()
    • get_artifacts_matching_glob()
    • get_expiration_arrow()
    • get_optional_artifacts_per_task_id()
    • get_single_upstream_artifact_full_path()
    • get_upstream_artifacts_full_paths_per_task_id()
    • guess_content_type_and_encoding()
    • list_latest_artifacts()
    • retry_create_artifact()
    • retry_list_latest_artifacts()
    • upload_artifacts()
  • scriptworker.client module
    • log
    • get_task()
    • sync_main()
    • validate_artifact_url()
    • validate_json_schema()
    • validate_task_schema()
  • scriptworker.config module
    • log
    • CREDS_FILES
    • apply_product_config()
    • check_config()
    • create_config()
    • get_context_from_cmdln()
    • get_frozen_copy()
    • get_unfrozen_copy()
    • read_worker_creds()
  • scriptworker.constants module
    • DEFAULT_CONFIG
    • STATUSES
    • get_reversed_statuses()
  • scriptworker.context module
    • log
    • DEFAULT_MAX_CONCURRENT_DOWNLOADS
    • Context
  • scriptworker.cot.generate module
    • log
    • generate_cot()
    • generate_cot_body()
    • get_cot_artifacts()
    • get_cot_environment()
  • scriptworker.cot.verify module
    • DECISION_TASK_TYPES
    • PARENT_TASK_TYPES
    • log
    • AuditLogFormatter
    • ChainOfTrust
    • LinkOfTrust
    • build_link()
    • build_task_dependencies()
    • build_taskcluster_yml_url()
    • check_and_update_action_task_group_id()
    • check_interactive_docker_worker()
    • check_interactive_generic_worker()
    • compare_jsone_task_definition()
    • create_test_workdir()
    • download_cot()
    • download_cot_artifact()
    • download_cot_artifacts()
    • find_sorted_task_dependencies()
    • get_action_context_and_template()
    • get_all_artifacts_per_task_id()
    • get_in_tree_template()
    • get_jsone_context_and_template()
    • get_pushlog_info()
    • get_scm_level()
    • get_source_url()
    • get_valid_task_types()
    • get_valid_worker_impls()
    • guess_task_type()
    • guess_worker_impl()
    • is_artifact_optional()
    • populate_jsone_context()
    • raise_on_errors()
    • trace_back_to_tree()
    • verify_build_task()
    • verify_chain_of_trust()
    • verify_cot_cmdln()
    • verify_cot_signatures()
    • verify_docker_image_sha()
    • verify_docker_image_task()
    • verify_docker_worker_task()
    • verify_generic_worker_task()
    • verify_link_ed25519_cot_signature()
    • verify_link_in_task_graph()
    • verify_parent_task()
    • verify_parent_task_definition()
    • verify_partials_task()
    • verify_repo_matches_url()
    • verify_scriptworker_task()
    • verify_task_in_task_graph()
    • verify_task_types()
    • verify_worker_impls()
  • scriptworker.ed25519 module
    • log
    • ed25519_private_key_from_file()
    • ed25519_private_key_from_string()
    • ed25519_private_key_to_string()
    • ed25519_public_key_from_file()
    • ed25519_public_key_from_string()
    • ed25519_public_key_to_string()
    • verify_ed25519_signature()
    • verify_ed25519_signature_cmdln()
  • scriptworker.github module
    • GitHubRepository
    • extract_github_repo_and_revision_from_source_url()
    • extract_github_repo_full_name()
    • extract_github_repo_owner_and_name()
    • extract_github_repo_ssh_url()
    • is_github_repo_owner_the_official_one()
    • is_github_url()
  • scriptworker.exceptions module
    • BaseDownloadError
    • CoTError
    • ConfigError
    • Download404
    • DownloadError
    • ScriptWorkerEd25519Error
    • ScriptWorkerException
    • ScriptWorkerRetryException
    • ScriptWorkerTaskException
    • TaskVerificationError
    • WorkerShutdownDuringTask
  • scriptworker.log module
    • log
    • contextual_log_handler()
    • get_log_filehandle()
    • get_log_filename()
    • pipe_to_log()
    • update_logging_config()
  • scriptworker.task module
    • REPO_SCOPE_REGEX
    • log
    • claim_work()
    • complete_task()
    • get_action_callback_name()
    • get_and_check_tasks_for()
    • get_base_branch()
    • get_base_revision()
    • get_branch()
    • get_commit_message()
    • get_decision_task_id()
    • get_head_revision()
    • get_parent_task_id()
    • get_project()
    • get_provisioner_id()
    • get_pull_request_number()
    • get_push_date_time()
    • get_repo()
    • get_repo_scope()
    • get_revision()
    • get_run_id()
    • get_task_definition()
    • get_task_id()
    • get_triggered_by()
    • get_worker_pool_id()
    • get_worker_type()
    • is_action()
    • is_github_task()
    • is_pull_request()
    • is_try()
    • is_try_or_pull_request()
    • prepare_to_run_task()
    • reclaim_task()
    • retry_get_task_definition()
    • run_task()
    • worst_level()
  • scriptworker.utils module
    • log
    • add_enumerable_item_to_dict()
    • add_projectid()
    • add_taskqueueid()
    • calculate_sleep_time()
    • cleanup()
    • create_temp_creds()
    • datestring_to_timestamp()
    • download_file()
    • filepaths_in_dir()
    • format_json()
    • get_hash()
    • get_loggable_url()
    • get_parts_of_url_path()
    • get_results_and_future_exceptions()
    • get_single_item_from_sequence()
    • load_json_or_yaml()
    • load_json_or_yaml_from_url()
    • makedirs()
    • match_url_path_callback()
    • match_url_regex()
    • raise_future_exceptions()
    • read_from_file()
    • remove_empty_keys()
    • request()
    • retry_async()
    • retry_async_decorator()
    • retry_request()
    • retry_sync()
    • rm()
    • scriptworker_session()
    • semaphore_wrapper()
    • to_unicode()
    • write_to_file()
  • scriptworker.worker module
    • log
    • RunTasks
    • async_main()
    • do_run_task()
    • do_upload()
    • main()
    • run_tasks()
  • Module contents

Indices and tables

• Index

• Module Index

• Search Page