Hosting build artifacts on AWS

Thu, Feb 25, 2021 9-minute read

With JCenter closing down folks are looking at alternative places to host their build artifacts, and AWS can easily meet that need. AWS isn’t the only platform capable of hosting Maven artifacts, any hosting provide can do it, but AWS has the ability to scale to high levels and offers a free tier which means that you can get started very cheaply and, if your libraries are really successful, you don’t need to revisit the set-up unless you want to.

The first part of this post will cover setting up AWS to serve your libraries, if you have another hosting provider you’re happy with you can skip to the section which describes where to put your files so that build tools will be able to find the files they need.

Create an AWS Account

If you don’t already have one you will need to create an AWS Account.

Decide on whether you’ll use a custom domain name

The reason you need to think about this now is because it will affect the name of the S3 bucket you create below. Personally I would recommend using your own domain, this allows you to move the hosting around if you want to without all of your libraries users needing to update their configuration. It’s not absolutely necessary though, so if you’re running through this post to see how things work you shouldn’t feel the need to go and register a domain.

If you already have a DNS domain registered you will need to be able to create a DNS CNAME record. This is something your domain registrar should have instructions for, and the instructions will vary between domain name registrars. If you want to keep everything in AWS you can use AWS Route 53 which will do the job.

Setting up the Simple Storage Service (S3)

Go to the S3 control panel and create a bucket with either the domain name you want to serve your artifacts from, or, if you don’t want to use a custom domain name, a short name which will be the start of the URL you provide your libraries from.

The AWS region you want to use is entirely up to you. I’m in the UK, so I use “EU (London) eu-west-2” out of habit. If you think your library will appeal to folk in a certain area you might want to use a region close to where your users are (e.g. for a library focused on supporting the Hindi language, you might want to use “Asia Pacific (Mumbai) ap-south-1” so downloads are slightly faster for users in India).

You should make sure you turn off “Block Public Access settings for bucket”. You want your artifacts to be available publicly, and this setting prevents that, so it needs to be switched off. You will need to click the acknowledgement box to confirm that you want to be able to make things uploaded to your bucket available to everyone.

S3 Creations with Public Access Off

You can then scroll to the bottom of the page and click “Create Bucket” to create the bucket.

Turn on “Static website hosting”

To serve files from the bucket to your users you’ll need to turn on “Static website hosting”. You can do this by clicking on the bucket in the bucket list in the S3 control console, then clicking on the “Properties” option;

S3 Bucket Properties Screenshot

Then scroll to the bottom of the page where you’ll see the “Static website hosting” option.

S3 Bucket Hosting Properties Screenshot

If it says “Disabled” you will need to click the “Edit” button, change the value to “Enabled”, leave the “Hosting Type” option set to “Host a static website”, enter index.html in the box titled “Index Document”, and then click “Save Changes”

S3 Static Hosting Screenshot

Once you’ve done this you can scroll to the bottom of the bucket properties to confirm that static hosting is now turned on for the bucket. This is also where you’ll find the URL which S3’s web serving service has associated with your bucket (for this example it’s http://test-maven.alsutton.com.s3-website.eu-west-2.amazonaws.com).

S3 Static Hosting On Screenshot

If you’ve decided not to use a custom domain this is the URL you’ll give folk who want to use your library.

Custom Domain Update

If you’ve chosen to use a custom domain, this is where you need to go to your DNS registrar and create the CNAME mentioned earlier. Your CNAME will map the domain you want to make public to the s3 web serving domain that you saw at the end of the last section.

For the example domain above (test-maven.alsutton.com) I would create a CNAME record which points to test-maven.alsutton.com.s3-website.eu-west-2.amazonaws.com.

CNAME Creation Screenshot

Warning: The Domain Name System (DNS) caches both successful and failed look-ups. When you create your CNAME it may be some time before everyone in the world can see it, and, similarly, if you update the CNAME the change make take a while to propagate, so don’t be surprised if you can’t use your custom domain immediately. It’s a good idea to allow a day or two between setting everything up and letting folk know about the domain.

A quick test

We can test to see if our S3 backed web serving is working by uploading a simple HTML index page and trying to access it. You can create an index.html file which contains <html><body><H1>IT WORKED!!!!!!!!</H1></body></html>, and use the S3 control panel to upload it to your bucket (Click on the bucket name, then click on the “upload” button and follow the instructions).

You now need to make the file available to everyone. The S3 control panel is a bit clumsy when it comes to editing permissions, so you’ll need to click on the index.html file you’ve just uploaded, then select “Make Public” from the “Object Actions” menu, then confirm your choice by clicking the “Make Public” button.

S3 Make Public Screenshot

If the DNS CNAME has propagated to your local DNS servers you should be able to access your test page using the URL http://{your domain}/index.html. If you get an error saying the domain does not exist the CNAME creation may not have reached your servers yet, and if you get an error starting 40 (usually 404 or 403) it means that you should check you made the index.html public in the S3 control panel.

Folder Layout

Ensuring your library files are in the right place is the most important thing to get right. If you get it wrong, your library artifacts won’t be found by any build tool.

Fortunately it’s easy to translate an artefact name into the correct layout; You need to create a nested set of folders with one for each section of the “group” which has a dot (.) between them, then another folder for the artefact name, then a final folder for the version, which is where you place your release files which we’ll create below.

For example, for com.alsutton:java-cryptography-wrapper:1.0 all of the relevant files go in com/alsutton/java-cryptography-wrapper/1.0/. For com.alsutton:java-cryptography-wrapper:1.1 they would go in com/alsutton/java-cryptography-wrapper/1.1/ and so on.

Inside Amazons S3 control panel you can create each folder using the “Create Folder” button, and when you’ve created the right structure you should see something like this;

S3 Console Screenshot

Creating the files to upload

If you’re using Gradle you can make use of the Maven Publishing Plugin to create the files you need to make available. If you’re unsure of how to configure the plugin you can use the configuration from one of my projects as a template.

With the Gradle Maven Publishing Plugin configured you should now be able to create the maven artifacts locally by running the following command;

$ gradlew publishToMavenLocal 

This will create a set of files in a local folder which is usually inside “{your_home_directory}/.m2/repository/” and, from there, follows the same folder layout we created in the section above (e.g. com/alsutton/java-cryptography-wrapper/1.0).

Uploading the files

Now you’ve found your files you will need to upload them to your S3 bucket and make them public in the same way we uploaded the index.html file for our earlier test. You will need to make sure that these files (.module, .pom, etc.) are uploaded and public for other people to be able to find your library. If any of these are missing, or aren’t public, users will see errors such as being told there was a “403 error” fetching your library.

To make things easier for users you should also upload and make public the maven metadata file that’s available one folder above the other files. When you publish locally you will get a folder called maven-metadata-local.xml, you will need to rename this maven-metadata.xml before uploading it to the same location that it’s in on your local machine (so in the artefact_name folder, not in artefact_name/1.0).

Using your new repository

Congratulations, your library is now available!

If you’re targeting folk who use gradle they will need to add in your repository to their builds. They can do this by adding the following to their gradle build files;

repositories {
    maven {
        url = uri("http://{your_domain}/")
    }
}

They can now reference your library using gradle’s normal dependency declaration (e.g., for the artefact in the “Folder Layout” section, they would add implementation("com.alsutton:java-cryptography-wrapper:1.0")).

(optional) Going Next Level - Content Delivery Network and HTTPS

So far what we have will only serve artifacts via http, and so could be susceptible to a Supply Chain Attack, and it’s from an S3 bucket in a single location, meaning some folk will see slower downloads because they’re geographically far away from where the S3 bucket is being served from.

Fortunately AWS has a solution to both of these problems, it’s called CloudFront. CloudFront is part of the AWS free tier, and they have some simple documentation which covers using CloudFront to provide distributed, HTTPS secured, serving for data which is coming from an S3 bucket, which will take you through how to set everything up.

If you’re providing libraries to others I would recommend setting up CloudFront. You’re not only giving them a better experience through using a CDN to provide your library faster than the S3 bucket would, you’re also giving them confidence that your library has not been altered by a third party at some point between you and them

That’s all folks

Hopefully you’ve found this useful. You’ve now reduced the risk of building up a following for a library using a third party repository, and then being hit by that repository closing, causing a lot of work for your users, and you’ve also given your users an easy way to find the authoritative source for the latest versions of your library.

If you have questions you can find me on Mastodon, GitHub, and Twitter.