Amazon S3

With the @fabr-build/aws plugin, an object in an S3 bucket is a build input like a file in your project, and a sync target can publish files to a bucket. The plugin works with AWS’s S3 and with S3-compatible stores such as MinIO, LocalStack and ministack.

plugin @fabr-build/aws;

generate site {
  srcs = ./src/** @aws:s3:acme-assets:fonts/**;
  …
}

sync release {
  @aws:s3:acme-site:v3/ = site;
}

fabr build site       # downloads the fonts it needs
fabr sync release     # uploads the site to s3://acme-site/v3/

@aws is the AWS account the requests are signed with; see Accounts and credentials for choosing a profile, using another account, or pointing at an S3-compatible store.

Coming from the AWS CLI

With the AWS CLI	In fabr
aws s3 cp s3://acme/v3/model.bin .	A reference to the object, @aws:s3:acme:v3/model.bin, as a build input; or fabr cp @aws:s3:acme:v3/model.bin out.
aws s3 cp --recursive s3://acme/fonts/ .	@aws:s3:acme:fonts/**.
aws s3 ls	fabr ls @aws:s3.
aws s3 ls s3://acme/	fabr ls @aws:s3:acme.
aws s3 sync ./site s3://acme-site/v3/	sync release { @aws:s3:acme-site:v3/ = ./site:**; }, then fabr sync release.
aws s3 sync --dryrun	fabr ls release, which lists what would be stored.
aws s3 sync --delete	Not supported: publishing never deletes objects.
--content-type, --cache-control	Content-Type is chosen by extension; other headers can’t be set yet.

Reading objects

Naming objects

A reference to S3 is the account, the service, the bucket, then the object’s key, separated by :. The object is named by its key, as a file in your project is named by its path:

@aws:s3:acme-models:v3/model.bin       # the file v3/model.bin
@aws:s3:acme-models:v3:model.bin       # the file model.bin
@aws:s3:acme-models/v3/model.bin       # the file acme-models/v3/model.bin

The projection rules for any reference apply: a : in the key strips what comes before it from the name, a / after the bucket keeps the bucket in it, and a -> name renames what is selected. A key containing a space or another character a name can’t hold is written quoted (@aws:s3:acme:'odd/a b.txt') or escaped (odd/a\ b.txt).

A pattern selects every object it matches:

@aws:s3:acme-site:assets/**/*.png      # every .png under assets/, named by key
@aws:s3:acme-site:assets:**            # everything under assets/, named from there

fabr lists the bucket from the pattern’s literal start (assets/), so a pattern starting with literal text lists only that part of the bucket. Using a pattern needs permission to list the bucket; a literal key doesn’t.

A literal name selects one object, never a directory: @aws:s3:acme-site:assets selects the object whose key is assets, if there is one. To select a directory’s contents, write assets:** or assets/**.

Files inside an archive

An object that is an archive (.tar.gz, .tgz, .zip, .tar.xz) can be read into as a directory, as an archive in your project can; see Archives:

@aws:s3:acme-dist:tools/linter.tgz:*:bin/lint     # bin/lint inside the archive's one top directory

Listing buckets and objects

fabr ls lists one level at a time:

fabr ls @aws:s3                  # the account's buckets
fabr ls @aws:s3:acme-site        # the objects and directories at the top of the bucket
fabr ls '@aws:s3:acme-site:assets/**'   # everything under assets/

A directory is listed with a trailing /. Listing the account’s buckets needs the s3:ListAllMyBuckets permission, which a read-only build role often lacks; a bucket can still be read by name without it.

How objects are cached

An object’s key can be given new content at any time, so fabr checks an object it has downloaded again once its copy is a day old, with a conditional request that S3 answers in a few bytes when the object hasn’t changed (HTTP 304 Not Modified). An object stored with a longer Cache-Control: max-age=… is trusted for that long without checking. A build that uses the object rebuilds only if its content changed. A pattern’s listing of the bucket is kept and checked again the same way.

To pick up a change sooner, run with --refresh, which checks every object and listing the run reads, however recently fabr last checked it:

fabr build --refresh site

Publishing always reads what the bucket holds now; see What is uploaded.

If S3 can’t be reached when an object is checked, fabr uses the copy it already has, with a warning:

warn:cannot refresh https://acme-models.s3.eu-west-2.amazonaws.com/v3/model.bin (…); serving the cached copy

A missing object or a refused request (404, 403) is never answered from the cache.

In watch mode, nothing tells fabr that an object in S3 changed; restart with --refresh to pick the change up.

Publishing to S3

A sync target publishes files to S3. Each entry names a place in a bucket and the target or files that go there:

sync release {
  @aws:s3:acme-site:v3/ = site;                 # every file of `site`, stored under v3/
  @aws:s3:acme-dist:tools/linter.tgz = linter;  # one object, holding linter's one file
}

• A key ending in / is a prefix: every file of the entry’s content is stored under it, at its own path. site’s css/a.css is stored as v3/css/a.css.
• Any other key is one object, and the entry’s content must be exactly one file.

The key is written literally, with / between its parts: no patterns, and no empty, . or .. parts.

Checking a release before publishing

fabr build release produces what would be stored without uploading anything, and fabr ls lists it, under each entry’s coordinate:

fabr ls release
# @aws/s3/acme-site/v3/css/a.css
# @aws/s3/acme-site/v3/index.html
# @aws/s3/acme-dist/tools/linter.tgz/linter.tgz

What is uploaded

fabr sync release compares each file with what the bucket holds at its key, and uploads only the files that differ. Run where there is no terminal, as in CI, it logs:

info:Publishing @aws:s3:acme-site:v3/
info:✓ Publishing @aws:s3:acme-site:v3/ (1.2s)
info:Publishing @aws:s3:acme-dist:tools/linter.tgz
info:✓ Publishing @aws:s3:acme-dist:tools/linter.tgz (8ms)
info:@aws:s3:acme-dist:tools/linter.tgz is already synced

An entry whose files are all there already is reported as already synced. On a terminal, each entry being published shows a progress bar for the bytes it is uploading, in place of the start line.

• Comparing: a file is already there if the object’s ETag is the file’s MD5, which is how S3 tags an object uploaded in one request. In a bucket encrypted with AWS KMS, an ETag is no MD5, so fabr compares the object’s stored SHA-256 checksum instead. fabr stores that checksum with every object it uploads, so it can tell; an object uploaded by another tool without one is uploaded again, once.
• Integrity: each upload is signed with the file’s SHA-256 and carries it as the object’s x-amz-checksum-sha256 checksum, so S3 refuses an upload that arrives damaged, and other tools can read the checksum later.
• Content-Type: chosen by the key’s extension (index.html is text/html; charset=utf-8), else from the file’s content.
• Order: within one entry, files are uploaded in no particular order, as with aws s3 sync, so a reader of a site mid-publish may see a page before the assets it uses. Publishing each release to a new prefix avoids that.
• Nothing is deleted: a file you removed from site stays in the bucket under v3/. Publish to a new prefix to start clean.
• Symbolic links can’t be stored in S3, and an entry containing one is refused.

Permissions

The account’s credentials need permission for each thing the build does with S3. Start from this policy, keep the statements for what your build does, and replace acme with your bucket’s name:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ReadObjects",
      "Effect": "Allow",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::acme/*"
    },
    {
      "Sid": "ListBucket",
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::acme"
    },
    {
      "Sid": "PublishObjects",
      "Effect": "Allow",
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::acme/*"
    },
    {
      "Sid": "ListAccountBuckets",
      "Effect": "Allow",
      "Action": "s3:ListAllMyBuckets",
      "Resource": "*"
    },
    {
      "Sid": "UseBucketKey",
      "Effect": "Allow",
      "Action": ["kms:Decrypt", "kms:GenerateDataKey"],
      "Resource": "arn:aws:kms:eu-west-2:123456789012:key/<key-id>"
    }
  ]
}

What each statement is for:

• ReadObjects: reading objects, as in @aws:s3:acme:v3/model.bin.
• ListBucket: reading with a pattern (@aws:s3:acme:assets/*.png), listing a bucket with fabr ls, and publishing to a prefix.
• PublishObjects: publishing with sync. Keep ReadObjects too: fabr reads what is already stored, to skip what hasn’t changed.
• ListAccountBuckets: only for fabr ls @aws:s3, which lists the account’s buckets. A build doesn’t need it.
• UseBucketKey: only for a bucket encrypted with your own KMS key; use that key’s ARN. kms:Decrypt is for reading, kms:GenerateDataKey for publishing.

A read-only build needs only ReadObjects, and ListBucket if it uses patterns. fabr finds a bucket’s region itself, which needs no permission.

Troubleshooting

'@aws:s3:acme-models:v3/model.bni' matched no files: no object has that key. Check the key with fabr ls @aws:s3:acme-models. A name for a directory also matches nothing: write v3:**.

https://acme-models.s3.eu-west-2.amazonaws.com/v3/model.bin was not found, or is not readable with @aws's credentials: S3 refused the request. Without permission to list a bucket, S3 answers 403 for a missing key as well as for an object you can’t read, so check the key, then the account’s permissions.

S3 has no bucket 'acme-modles': no bucket of that name exists. With an endpoint, the message names the store: http://localhost:4566 has no bucket 'acme-modles'.

'@aws:s3:Acme:x' names 'Acme', which is not an S3 bucket name: a bucket’s name is 3 to 63 lowercase letters, digits, . and -.

cannot publish @aws:s3:acme-dist:tools/linter.tgz: an object's content must be one file, and it holds 2: an entry for one object was given several files. Publish to a prefix instead (@aws:s3:acme-dist:tools/), or narrow the content to the one file.

'@aws:s3:acme-site:v3/*.html' is not an S3 address to publish to: it is written as a pattern, a rename or with a marker: a sync entry’s key is where files go, so it is written literally. Publish to a prefix (@aws:s3:acme-site:v3/) and narrow the content instead.

Failed to publish @aws:s3:acme-site:v3/: cannot store s3://acme-site/v3/index.html (403): AccessDenied: Access Denied: the credentials may not store objects there; see Permissions.

Not supported

• An object larger than 5 GB: each object is uploaded in one request, which S3 limits to 5 GB.
• Deleting what a prefix no longer holds, or setting headers such as Cache-Control, when publishing.
• Publishing to a bucket’s top level as a prefix.
• A key containing : can’t be written in a reference, and neither can one whose path a URL would shorten (a/../b).

Known issues

• An entry for one object is listed by fabr ls release under the key and then the object’s name (tools/linter.tgz/linter.tgz).