Welcome to Wellcome AWS Utils’s documentation!¶
This package is a collection of utilities written at Wellcome for interacting with AWS.
Some of these utilities are very specific to Wellcome projects, others are more generic and should be generally useful.
The best place to start is the API reference, which describes all the utilities the package provides. It’s a bit sparse at the moment, but hopefully we’ll expand it soon!
API reference¶
Deployment utilities¶
Shared library to help surface ECS deployment information.
-
class
wellcome_aws_utils.deployment_utils.
Deployment
(deployment_key, deployment_status, color, created_at, task_definition)¶ -
color
¶ Alias for field number 2
-
created_at
¶ Alias for field number 3
-
deployment_key
¶ Alias for field number 0
-
deployment_status
¶ Alias for field number 1
-
task_definition
¶ Alias for field number 4
-
DynamoDB utilities¶
ECS¶
-
wellcome_aws_utils.ecs_utils.
clone_task_definition
(client, task_definition)[source]¶ Given a task definition ARN, clone the associated task.
Returns the new task definition ARN.
-
wellcome_aws_utils.ecs_utils.
describe_cluster
(ecs_client, cluster_arn)[source]¶ Given a cluster ARN attempts to find a matching cluster description.
Returns a cluster description.
-
wellcome_aws_utils.ecs_utils.
describe_service
(ecs_client, cluster_arn, service_arn)[source]¶ Given a cluster ARN and service ARN, attempts to find a matching service description.
Returns a service description.
-
wellcome_aws_utils.ecs_utils.
get_cluster_arns
(ecs_client)[source]¶ Extract the list of cluster ARNs in this account.
Returns a list of cluster ARNs.
-
wellcome_aws_utils.ecs_utils.
get_latest_task_definition
(client, cluster, service)[source]¶ Given the name of a cluster and a service, return the ARN for its latest task definition.
-
wellcome_aws_utils.ecs_utils.
get_service_arns
(ecs_client, cluster_arn)[source]¶ Given a cluster ARN, extracts the associated service ARNs.
Returns a list of service ARNS.
S3¶
-
wellcome_aws_utils.s3_utils.
is_object
(bucket, key)[source]¶ Checks if an object exists in S3. Returns True/False.
Parameters: - bucket – Bucket of the object to check.
- key – Key of the object to check.
-
wellcome_aws_utils.s3_utils.
copy_object
(src_bucket, src_key, dst_bucket, dst_key, lazy=False)[source]¶ Copy an object from one S3 bucket to another.
Parameters: - src_bucket – Bucket of the source object.
- src_key – Key of the source object.
- dst_bucket – Bucket of the destination object.
- dst_key – Key of the destination object.
- lazy – Do a lazy copy. This means that the object will only be copied if the destination object does not exist, or exists but has a different ETag from the source object.
-
wellcome_aws_utils.s3_utils.
parse_s3_record
(event)[source]¶ Extracts a simple subset of an S3 update event.
-
wellcome_aws_utils.s3_utils.
write_objects_to_s3
(bucket, key, objects)[source]¶ Given an iterable of objects that can be serialised as JSON, serialise them as JSON, and write them to a file in S3, one per line.
Parameters: - bucket – S3 bucket to upload the new file to.
- key – S3 key to upload the new file to.
- objects – An iterable of objects that can be serialised as JSON.
SNS¶
-
class
wellcome_aws_utils.sns_utils.
EnhancedJSONEncoder
(*, skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, sort_keys=False, indent=None, separators=None, default=None)[source]¶ -
default
(obj)[source]¶ Implement this method in a subclass such that it returns a serializable object for
o
, or calls the base implementation (to raise aTypeError
).For example, to support arbitrary iterators, you could implement default like this:
def default(self, o): try: iterable = iter(o) except TypeError: pass else: return list(iterable) # Let the base class default method raise the TypeError return JSONEncoder.default(self, o)
-
-
class
wellcome_aws_utils.sns_utils.
SNSEvent
(subject, message)¶ -
message
¶ Alias for field number 1
-
subject
¶ Alias for field number 0
-
-
wellcome_aws_utils.sns_utils.
extract_json_message
(event)[source]¶ Extracts a JSON message from an SNS event sent to a lambda
Deprecated in favour of extract_sns_messages_from_lambda_event
Changelog¶
This is a record of all releases of wellcome_aws_utils.
3.2.0 - 2020-07-15¶
Also handle SQS messages
3.1.0 - 2020-06-29¶
The adapters now only send an ID / Version, and we need to look that up in Dynamo to fetch the S3 object in the reporting code.
3.0.0 - 2020-06-25¶
The schema of VHS data has changed meaning the reporting_utils are currently broken. This updates to the latest VHS data format.
2.3.3 - 2019-07-26¶
Makes sure that the Elasticsearch doc is sent over as a string.
2.3.2 - 2019-05-21¶
Adds ability to switch AWS roles when fetching elasticsearch credentials
2.3.1 - 2019-05-21¶
Now with fixed Travis credentials.
2.3.0 - 2019-05-21¶
This release modifies the way that secrets are handled by lambdas in the reporting pipleine. Previously, secrets were passed to lambdas as environment variables, defined in terraform. We now fetch secrets from AWS secretsmanager as records move through the pipeline.
2.2.1 - 2018-11-23¶
A large number of records in the Sierra VHS contain a reindexShard
parameter which is not expected when initialising a HybridRecord()
object. attrs
can’t handle data it doesn’t expect, and the records with reindexShard
parameters therefore fail to pass through the pipeline.
We now throw away any unnecessary data in the received message, allowing originally dirty messages to pass through without issue.
2.2.0 - 2018-11-08¶
This release adds utils for the reporting pipeline.
The functions under reporting_utils.py
describe a basic ETL pipeline from VHS to Elasticsearch, without a transformation specified. In this way, the shape of the pipeline remains independent of both the data within it and the transforms being applied.
As further data sources are added to the reporting pipeline and more Lambda functions are created, we keep repeated code to a minimum. In a new Lambda function, the user should specify a set of data-source-specific transformations in a transform.py
file. The Lambda’s main
can then remain minimal and generic:
2.1.3 - 2018-08-17¶
This fixes a bug in the @log_on_error
decorator where the return value
of the original function would be replaced by None
. This decorator now
preserves the original return value.
2.1.2 - 2018-06-26¶
Previously sending a message with sns_utils.publish_sns_message
would
print a message upon success.
Now this message is only logged at debug level.
2.1.1 - 2018-06-04¶
Now @log_on_error
can be used to decorate functions with arbitrary arguments/keyword arguments.
2.1.0 - 2018-06-04¶
This adds a new method: lambda_utils.log_on_error
. This can be used to
decorate the main function for a Lambda, and logs the event/context if the
Lambda throws an unexpected exception.
For example, running the following snippet:
@log_on_error
def handler(event, context=None):
if event == {1: '1', 2: '2'}:
raise ValueError
handler(event={'foo': 'bar'})
handler(event='99 green bottles' * 99)
handler(event={1: '1', 2: '2'})
gives the following output:
This makes it easier to debug failed Lambdas, but without the expense of logging every event that a Lambda receives.
2.0.2 - 2018-06-04¶
Previously sending a message with sns_utils.publish_sns_message
would
log the entire SNS response.
Now the response is only logged if the SNS message is unsuccessful.
2.0.1 - 2018-01-12¶
This fixes a bug in s3_utils.parse_s3_record
. If the key of a changed
file included a character which is usually quoted in URLs (e.g. +
),
a parsed record from the S3 event stream would use the URL-quoted form
of the object key.
For example, a change to s3://example/foo+bar
would become foo%2Bbar
.
This version unquotes the key when parsing the event.
2.0.0 - 2017-11-29¶
Replacing the DynamoImageFactory and DynamoImage classes with DynamoEventFactory and DynamoEvent
- Perform quite a bit of sanity checking on event object received
- DynamoEvent can: - return old and new images (if available) - return modified keys only - return deserialized or otherwise images and keys based on params
1.1.0 - 2017-11-15¶
Deprecates sns_utils.extract_json_message
in favour of sns_utils.extract_sns_messages_from_lambda_event
.
extract_sns_messages_from_lambda_event provides: - better error reporting if the event is malformed - loops over all available records from event not just the first - returns subject along with the json decoded message
This release also adds UnWellcomeException
which will be used as the base exception for new errors.
1.0.0 - 2017-11-07¶
First production release!