https://breweriesnearme.theclouddevopslearningblog.com

Here’s exactly how I set it up.

1. Creating a New S3 Bucket for the Subdomain

The first step was to create a dedicated S3 bucket to hold the SPA build. Following AWS best practices:

• Bucket name: breweriesnearme.theclouddevopslearningblog.com
• Block all public access: ✅ ON
• Static website hosting: ❌ Disabled (we’ll use CloudFront instead)

I uploaded the build artifacts (index.html, main.js, assets/, etc.) directly into the bucket root.

💡 Tip: Make sure index.html is at the root of the bucket, not in a subfolder like dist/, unless you plan to set a CloudFront origin path.

2. Requesting an SSL/TLS Certificate in ACM

CloudFront requires certificates to be in the us-east-1 region, so I switched to N. Virginia and requested a new cert for:

breweriesnearme.theclouddevopslearningblog.com

I used DNS validation and, because the domain is managed in Route 53, ACM automatically created the necessary CNAME record. Once validation succeeded, the certificate was ready to attach.

3. Setting Up the CloudFront Distribution

Next, I created a new CloudFront distribution to serve the SPA.

Key settings:

• Origin Domain: S3 regional endpoint (not the website endpoint)
• Origin Access Control (OAC): Enabled, to keep the S3 bucket private
• Viewer Protocol Policy: Redirect HTTP → HTTPS
• Alternate Domain Name (CNAME): breweriesnearme.theclouddevopslearningblog.com
• SSL Certificate: Custom ACM certificate from step 2
• Default Root Object: index.html (⚠️ no leading slash)
• Compression: Enabled

🔁 SPA-Friendly Error Pages

Because SPAs handle routing client-side, I needed to configure CloudFront to serve index.html even when a 403 or 404 occurs:

• 403 → 200 → /index.html
• 404 → 200 → /index.html

This ensures deep links like /brewery/42 work correctly.

4. Bucket Policy for CloudFront Access

With OAC enabled, I updated the S3 bucket policy to allow CloudFront to read objects:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowCloudFrontOACRead",
      "Effect": "Allow",
      "Principal": { "Service": "cloudfront.amazonaws.com" },
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::breweriesnearme.theclouddevopslearningblog.com/*",
      "Condition": {
        "StringEquals": {
          "AWS:SourceArn": "arn:aws:cloudfront::<ACCOUNT_ID>:distribution/<DISTRIBUTION_ID>"
        }
      }
    }
  ]
}

5. Adding the Subdomain in Route 53

Finally, I added a new DNS record in Route 53:

• Record type: A (Alias)
• Name: breweriesnearme
• Target: My CloudFront distribution

Once propagation completed, the subdomain pointed to CloudFront and the SPA became publicly accessible.

6. Common Gotchas (and How I Fixed Them)

• AccessDenied at root: I initially saw an AccessDenied error when visiting /. The fix was making sure Default Root Object was set to index.html (without a slash).
• Certificate validation not showing: The first time I requested the cert, I did it in the wrong region (ap-southeast-2). Certificates for CloudFront must be in us-east-1.

🧰 CI/CD Deployment

I set up a GitHub Actions pipeline to automatically build and deploy the SPA to S3 and invalidate the CloudFront cache:

name: Deploy Breweries Near Me SPA

on:
  push:
    branches: [ master ]

permissions:
  contents: read

jobs:
  deploy:
    name: Build and Deploy SPA
    runs-on: ubuntu-latest

    steps:
      - name: Checkout repository
        uses: actions/checkout@v4

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: 20

      - name: Install dependencies
        run: npm ci

      - name: Build application
        run: npm run build

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          aws-access-key-id: $
          aws-secret-access-key: $
          aws-region: ap-southeast-2

      # Upload static assets from dist/ (immutable cache)
      - name: Upload dist/ to S3
        run: |
          aws s3 sync dist/ s3://$/ \
            --delete \
            --cache-control "public,max-age=31536000,immutable"

      # Upload images from img/ (long cache but not immutable)
      - name: Upload img/ to S3
        run: |
          aws s3 sync src/img/ s3://$/img/ \
            --delete \
            --cache-control "public,max-age=31536000"

      # Upload index.html separately with no-cache
      - name: Upload index.html
        run: |
          aws s3 cp src/index.html s3://$/index.html \
            --cache-control "no-store" \
            --content-type "text/html"

      - name: Invalidate CloudFront cache
        run: |
          aws cloudfront create-invalidation \
            --distribution-id $ \
            --paths "/*"

7. Application Changes: Updating the App and Migrating to OpenBreweryDB

I wrote BreweriesNearMe as a standalone app (was previously deployed and hosted on a raspberry pi at home), as part of some study years ago (I was doing a Functional Programming in JavaScript course using the Rambda library). The code for this app is pretty cool, as it used the hyperscript-helpers library, which allows you to assign the css class for each element where it is being coded via helper functions (making the styling much more readable, than normal CSS, and for my purposes, was definitely sufficient), e.g:

function fieldSet(labelText, inputValue, oninput) {
    return div({ className: 'w-80'},
    [
        label({ className: 'db mb1 mw-80' }, labelText),
        input({ 
            className: 'pa2 input-reset ba w-100 mb2 br3',
            id: 'addressSearch',
            type: 'text',
            value: inputValue,
            oninput
        }),
    ]);
}

Switching from the proprietary BreweryDB API to the open, community-driven OpenBreweryDB required several code and data handling changes. In my previous blog, I wrote how I scraped, enriched and then added data to this project, so I won’t go over it again today. However, while cutting the source over was straight-forward, there were a few things that I had to update to get everything working smoothly:

🔄 API Endpoint & Data Model Changes

• API URL: Updated all fetch calls to use the OpenBreweryDB REST endpoints instead of the old BreweryDB URLs.
• Field Names: OpenBreweryDB uses different field names (e.g., name, street, city, state, postal_code, website_url, latitude, longitude). I refactored the code to map and display these new fields.
• No API Key Needed: Removed all authentication logic and API key handling, since OpenBreweryDB is public. In fact, my previous authenticated calls through to BreweryDB.com were proxied via an AWS Lambda function to abstract the API auth side of things altogether.

🗺️ Address & Location Handling

• Address Formatting: Adjusted the address formatting logic to handle OpenBreweryDB’s fields, which sometimes differ from BreweryDB (e.g., address_1 vs street, state_province vs state).
• Geolocation: Ensured that the app gracefully handles missing or partial location data, since not all breweries in OpenBreweryDB have latitude/longitude.

📏 Distance Calculation

• Distance Calculation: Since OpenBreweryDB doesn’t provide distance-from-user, I implemented a Haversine formula in the frontend to calculate the distance between the user’s search location and each brewery’s coordinates.
• Unit Selection: Preserved support for both kilometers and miles.
• Filtering by Distance: Updated the logic so that breweries are filtered by the selected radius before rendering, and the “No breweries to display within the selected distance.” message is shown if none match. The distance calculation is now performed for each brewery before filtering and deduplication, ensuring the UI is always accurate and user-friendly.

🧹 Deduplication

• Duplicate Results: OpenBreweryDB sometimes returns duplicate or near-duplicate breweries (with slight name or address variations - kind of my bad given I created the initial dataset 💀). I added a deduplication step in the frontend, matching on address, website, and distance, to ensure only unique breweries are shown.

🖥️ UI & Table Rendering

• Table Columns: Updated the UI to show the new fields, and ensured links (like website and Google Maps) use the correct data. Sadly, I had to remove the Image field, as this information isn’t supported in the OpenBreweryDB API (perhaps room to add improvements later!).
• Error Handling: Improved error handling for missing data and empty results.

🧪 Testing & Edge Cases

• Deep Links: Verified that SPA routing still works for direct links to brewery detail pages.
• No Results: Ensured the app displays a friendly message if no breweries are found for a given search or if none are within the selected distance.

With these changes, the app now works seamlessly with OpenBreweryDB, is easier to maintain, and is free from API key or quota restrictions.

Final Thoughts

This project was a great reminder of how flexible AWS’s static hosting model is. With just a few services — S3, CloudFront, ACM, and Route 53 — I was able to stand up a completely separate React application under the same domain as my Jekyll blog, with full HTTPS support, CDN caching, and SPA-friendly routing.

I recently tried to get one of my old projects, BreweriesNearMe working again. After multiple issues getting the site running again (details will be in another post soon), I realised the API that I was using, brewerydb.com, no longer exists. I found OpenBreweryDB pretty soon, and had it hooked up and running. However, I found that there was no local data for Australia (it’s an open/crowd sourced data source). I then thought, “How hard can that be to get?”

When I started this task, my goal was deceptively simple: collect a comprehensive list of breweries in Australia and format them for ingestion into the OpenBreweryDB schema. The catch? There’s no single authoritative source — and every source that does exist presents its own challenges.

This post is a deep dive into how I designed and built a fully automated data pipeline to tackle that problem. It’s not a tutorial, but a breakdown of the engineering decisions, mistakes, and solutions that got us from “let’s scrape some websites” to “production-ready, enriched, validated data.”

Architecture Overview

At a high level, the scraper evolved into a modular system with three major components:

1. Data Extraction Layer – Scrape and parse multiple upstream sources.
2. Normalization & Cleaning Layer – Standardize the fields into a unified schema.
3. Enrichment & Post-Processing Layer – Use the Google Places API to add structured location data, validate results, and filter noise.

The guiding principle was “merge many imperfect sources into one high-quality dataset.”

1. Data Extraction Layer

The first major step was building scrapers for four key sources:

• Craft Cartel: A long-form page with a single block of comma-separated brewery names.
• Independent Brewers Association (IBA): Paginated cards with names and rough locations.
• Wikipedia: Tables split into “major company owned” and “microbreweries.”
• Untappd: Initially attempted, but abandoned due to authentication requirements and closed API.

Each of these posed different challenges.

Craft Cartel: Parsing Unstructured Text

The simplest-looking source turned out to be tricky. The brewery names weren’t in HTML tables or lists — they were buried inside a block of text. Once extracted, each needed to be split and cleaned. That got us names, but no addresses, phones, or coordinates — we’d deal with that later.

IBA: Paginated and Noisy Data

IBA was the richest source but introduced pagination (/page/2/, /page/3/, etc.). Handling this meant writing a loop that crawled until no more pages existed. I also had to filter out non-brewery blocks like “Brewery Members” and “Want to be a member?” that appeared on each page.

Wikipedia: Tables That Weren’t Really Tables

Wikipedia’s structure was the most brittle. The tables I needed were buried after specific <h2> headings, but weren’t direct siblings. My first attempt returned nothing. The fix was to walk forward through the DOM until the next <table class="wikitable"> appeared, then flatten it — accounting for rowspan and column misalignment.

In the end, Wikipedia provided high-value signals: ownership classification and historical data that wasn’t available elsewhere.

2. Normalization & Cleaning

With data now flowing from three sources, the next challenge was making it consistent. OpenBreweryDB expects a schema like:

id, name, brewery_type, address_1, address_2, city, state_province, postal_code, country, phone, website_url, longitude, latitude

This meant handling dozens of edge cases:

• Splitting free-form location strings like "340 Melton Rd, Northgate QLD 4013, Australia" into structured fields.
• Dropping breweries with no Australian presence.
• Deduplicating records by normalizing names and fuzzy matching city/state combinations.
• Removing metadata rows from Wikipedia that weren’t breweries at all.

Normalization turned out to be where most of the real engineering time went — 70% of the work was spent here.

3. Enrichment with Google Places API

The raw scraped data was still thin — we had names and maybe a rough suburb. The next step was to enrich it.

For each brewery, I constructed a text query like:

<name> brewery Australia

and passed it to the Google Places API. From the response, I extracted:

• ✅ Full structured address
• ✅ Phone number
• ✅ Official website
• ✅ Latitude & longitude

The enrichment step transformed the dataset from “interesting” to “useful.”

Filtering Non-Australian Results

Many names overlapped with breweries overseas. Adding a strict "country: Australia" filter and rejecting any result not geocoded inside Australia cleaned up the data dramatically.

4. Reliability, Error Handling & Backoff

The first full run failed halfway through: ConnectionResetError: [Errno 54] Connection reset by peer. That turned out to be a network-level reset during Places API calls — likely due to too many requests too quickly.

The solution was threefold:

• ✅ Exponential backoff + jitter on all API calls.
• ✅ A global requests.Session adapter with automatic retries and respect for Retry-After.
• ✅ A configurable --places-rate flag to control throughput (e.g. 1.5s between calls).

With these improvements, I could run enrichment across 500+ breweries without a single crash.

5. Lessons Learned & Future Work

This project ended up being far more complex than a “simple web scraper.” Along the way, I learned a few key lessons:

• Scraping is the easy part. Normalization and enrichment are where the real complexity lies.
• DOM structures are fragile. Wikipedia’s layout changes broke three early attempts — defensive parsing is essential.
• Error handling isn’t optional. A 500-request enrichment job will hit transient network issues. Plan for them.
• Multiple imperfect sources > one perfect one. The final dataset was only possible by merging three sources and using Places data to fill gaps.

Future improvements include:

• Adding brewery size classification (micro, regional, large) automatically.
• Building a scheduling layer to re-run the scraper periodically.
• Adding CI tests to validate schema integrity and geolocation coverage.

Conclusion

What started as a simple scraper evolved into a production-ready data pipeline — one that fetches, normalizes, enriches, and validates hundreds of Australian breweries. The final result is a clean, geocoded dataset that can slot directly into OpenBreweryDB, providing far richer data than any single source on its own.

If you’re building something similar, my advice is simple: treat scraping as just the first step. The real value lies in how you clean, enrich, and harden that data for downstream use.

Final note: If you want, check out the source of the scraper here: https://github.com/simonmackinnon/breweriesnearme/blob/master/data/scrape_au_breweries.py

This walkthrough builds on some great work others have shared — in particular, this excellent guide from PagerTree, which I used as my starting point. I’ve adapted and expanded on it here to match my own workflow and highlight some issues I ran into along the way.

🧰 Why Automate Your Jekyll Deployments?

Every time you push to your main branch, you can have GitHub automatically:

• Install Ruby and your Jekyll dependencies
• Build your static site
• Upload the generated files to your S3 bucket
• Invalidate your CloudFront cache so changes go live immediately

This turns deployment from a manual multi-step process into a simple git push.

🛠️ Setting Up the Workflow

The workflow file lives at .github/workflows/deploy.yml in your repo. Here’s a trimmed-down version of mine:

name: CI / CD

on:
  push:
    branches: [ master ]
  workflow_dispatch:

jobs:
  build:
    runs-on: ubuntu-latest
    defaults:
      run:
        working-directory: jekyll-clouddevopslearningblog

    steps:
      - uses: actions/checkout@v4

      - name: Set up Ruby
        uses: ruby/setup-ruby@v1
        with:
          ruby-version: "3.2"
          bundler-cache: true

      - name: Ensure Linux platform support
        run: bundle lock --add-platform x86_64-linux

      - name: Build the site
        env:
          JEKYLL_ENV: production
        run: bundle exec jekyll build --trace

      - name: Deploy to S3
        run: aws s3 sync ./_site/ s3://$ --delete --acl public-read --cache-control max-age=604800

      - name: Invalidate CloudFront cache
        run: aws cloudfront create-invalidation --distribution-id $ --paths "/*"

🔐 Configuring Secrets

You’ll need to store a few secrets in your GitHub repository settings (Settings → Secrets and variables → Actions):

• AWS_ACCESS_KEY_ID – your AWS access key
• AWS_SECRET_ACCESS_KEY – your AWS secret key
• AWS_S3_BUCKET_NAME – the name of your bucket
• AWS_CLOUDFRONT_DISTRIBUTION_ID – the ID of your CloudFront distribution

These are injected into the workflow automatically and keep sensitive data out of your repo.

🧩 Common Gotchas (And How I Fixed Them)

I ran into a few issues that are worth mentioning:

1. Could not locate Gemfile or .bundle/ directory

This happens when the workflow runs in the wrong directory. If your Jekyll site is inside a subfolder (like jekyll-clouddevopslearningblog), make sure to set:

defaults:
  run:
    working-directory: jekyll-clouddevopslearningblog

2. bundler: command not found: jekyll

This one confused me at first — it means Bundler installed your gems, but jekyll wasn’t among them. Usually the cause is that you’re not running commands with bundle exec, or that Jekyll isn’t listed in your Gemfile.

✅ Fix: Make sure your Gemfile includes:

gem "jekyll", "~> 4.3"

And build the site like this:

- run: bundle exec jekyll build --trace

3. You must add the platform x86_64-linux to your lockfile

If you created your Gemfile.lock on macOS, the Linux runner on GitHub Actions won’t install some gems. You can fix this by adding a step before installation:

- name: Ensure Linux platform support
  run: bundle lock --add-platform x86_64-linux

Commit the updated lockfile once and you can remove this step.

4. You must copy the vendor posts.html to the repo if you've added custom code

I added some custom code in the post.html of the bundle directly on my machine. So when I used github actions to build the site and deploy, it didn’t have these changes. By copying this to the repo version at _includes/post.html, this meant these changes could be added when built remotely

🚀 Going Further

Some ideas for future improvements:

• Use OIDC and aws-actions/configure-aws-credentials instead of storing long-lived AWS keys.
• Add a paths: filter so the workflow only runs if Jekyll files change.
• Automate CloudFront invalidations conditionally to save API calls.

🎉 Final Thoughts

This setup has completely changed my workflow: now I can just commit and push, and within a minute or two, the live site updates automatically. It’s one of those small bits of DevOps automation that pays off quickly — especially if you’re constantly tweaking content or adding new posts.

If you’re still deploying manually, give this a try. Once you see how easy it is, you’ll never want to go back.

Have questions or ran into different errors? Drop them in the comments below — I’d love to hear how you’ve set up your own Jekyll CI/CD pipeline!

• A Jekyll site stored in an S3 bucket
• Served through CloudFront as a CDN and to support HTTPS
• A free SSL certificate from AWS Certificate Manager (ACM)

It worked perfectly… until one day, my site suddenly started showing security warnings, and browsers said the connection was “Not Secure.”

Here’s how I diagnosed the problem and fixed it — and how you can do the same if it happens to you.

Step 1: Spot the Problem

The first sign something was wrong was when I tried to run a simple test:

curl -I https://theclouddevopslearningblog.com

This gave me:

curl: (60) SSL certificate problem: certificate has expired

This error means the certificate that encrypts traffic to my site was no longer valid — so HTTPS wasn’t working.

Step 2: Check What Certificate Is Being Used

To see exactly what certificate CloudFront was serving, I used openssl:

openssl s_client -servername theclouddevopslearningblog.com   -connect theclouddevopslearningblog.com:443 -showcerts </dev/null 2>/dev/null   | openssl x509 -noout -issuer -subject -dates -ext subjectAltName

The output showed this:

issuer=C=US, O=Amazon, CN=Amazon RSA 2048 M02
subject=CN=theclouddevopslearningblog.com
notBefore=Mar 10 00:00:00 2024 GMT
notAfter=Apr  8 23:59:59 2025 GMT

The important part is notAfter — the certificate expired on April 8, 2025. Mystery solved!

Step 3: Why Didn’t It Renew Automatically?

Certificates from ACM usually renew automatically, but only if the DNS validation records are still in place.
When I checked the certificate in the ACM console (in us-east-1), I saw that:

• The certificate had expired
• Renewal failed because the DNS validation records had been deleted

This is a very common mistake — if you remove the validation CNAME records after issuing the certificate, AWS can’t confirm you still own the domain. And without that, it won’t renew.

Step 4: Request a New Certificate

The fix was simple:

1. Go to ACM → Request a public certificate
2. Add both:
   • theclouddevopslearningblog.com
   • *.theclouddevopslearningblog.com
3. Choose DNS validation
4. Add the CNAME records ACM gives you into your DNS (for example, in Route 53)
5. Wait until the certificate status changes to “Issued”

⚠️ Tip: Leave those DNS CNAMEs in place forever. They’re needed for future renewals too.

Step 5: Attach the New Certificate in CloudFront

Just creating the new certificate isn’t enough — CloudFront still needs to know to use it.

Here’s how:

• Go to CloudFront → Distributions → [your distribution]
• Under Alternate domain names (CNAMEs), make sure your domain is listed
• Under Viewer certificate, choose “Custom SSL certificate” and select the new one
• Save your changes

This will trigger a new deployment, which usually takes a few minutes.

Step 6: Test Again

After the update finished, I tested again:

curl -I https://theclouddevopslearningblog.com

Now I got:

HTTP/2 200
server: CloudFront

And openssl showed the new certificate expiry date was in 2026.

Lessons Learned

This was a small problem, but it taught me a few useful lessons that are worth sharing:

1. Leave DNS validation records in place. They’re essential for automatic renewal.
2. Set a renewal alert. You can use EventBridge + SNS to send yourself an email before a certificate expires.
3. Remember the region. CloudFront only works with ACM certificates in us-east-1.
4. Add monitoring. A simple curl script in a cron job can catch certificate problems before users do.

Final Thoughts

Static sites on S3 + CloudFront are incredibly powerful and cost-effective, but even “serverless” websites need a little maintenance. SSL certificates are one of those things you don’t want to ignore — and now I’ll never forget to keep an eye on mine!

Hopefully, this guide helps you fix an expired certificate quickly and with confidence.

Have you run into similar CloudFront issues? Let me know — I might write a follow-up post on automating SSL monitoring!

It’s been a while between drinks hey?

When I created this blog, I initially used the default support for Disqus as the blog commenting capability. After a few issues with the privacy, etc., I switched to a free trial of an alternative, HyvorTalk. This worked fine, but after the free trial, my comments got disabled, and I hadn’t focused on this, until the last few days.

I moved the comments back to Disqus. However, I haven’t maintained this blog for about 4-5 years, so making changes was problematic.

In fact, I’d hardly used my personal development machine in about that time too, so there were a few things that were broken:

• Jekyll wouldn’t run because my Ruby version was out of date
• Homebrew wouldn’t re-install Ruby because it was out of date

So after a complete re-install of Homebrew, Ruby and Jekyll, I was able to get Jekyll running. The next issue was that the gem bundler wouldn’t run with various issues. I was able to get it running by deleting the Gemfile.lock and changing the theme / gem back to the default. However, I was still getting errors with undefined methods, etc. when trying to serve the site. My solution was to re-initialise the Jekyll site, update the parameters needed for the theme to work, and copy the posts acrosss. This got it working fine.

At this point, I was able to get the Disqus comments section added again via the default support (I’d played around with the code for this to enable HyvorTalk previously, so had to wipe that code). I was able to build and run the site locally and get the comments section to load. However, after I built the site, I pushed it to my static site S3 bucket. After I loaded the posts again, I wasn’t getting it loading. After a while playing around with the site files, I realised I was being served different files than I uploaded. Having not looked at my site for several years, I’d completely forgotten that I had put a CDN (CloudFront) in front of the site (mainly to get https working). After pushing a ‘/*’ invalidation to the distribution, the site was serving up the right content (dull yay).

So, what’s next? I’m going to be trying to post a bit more of some of the stuff I’m working on and learning, and maybe a refresh of the site altogether at some point.

Background

• Goal: Build a Netflix Style Recommendation Engine with Amazon SageMaker
• Outcome: Gain real machine learning and AWS skills while getting hands-on with a real-world project to add to your portfolio https://acloudguru.com/blog/engineering/cloudguruchallenge-machine-learning-on-aws

TL;DR:

I built:

• Movie Recommendation Engine (K-Means clustering using AWS SageMaker)
• Serverless API and Website for users to view recommendations for selected movies (using API Gateway, Lambda, DynamoDB, S3 & CloudFront)
• Visit https://moviesforme.net/ to try out the recommendations!

Here’s the architecture I implemented:

Machine Learning in the Cloud

Steps

1. Determine use case and obtain data.

I thought about using GoodReads data to build a book recommendation engine, moreso because it would be a point of differentiation. However, my interests align with TV and film more than literature, so I decided movie datasets will be a better fit for me.

The first decision to make was to do with what data to use. In the brief, Kesha Williams made the example suggestion of movie datasets from IMDB.

The datasets are all available for download here: https://datasets.imdbws.com/

The recommended sets to use by Kesha were the title.akas (for grouping alternate titles’ info), title.basics (basic information about the titles) and title.ratings (rating information for the titles). These could all be merged on the “titleid” column. I used the ‘requests’ python library to download these, and then converted to Pandas dataframes for analysis / ML training.

A further dataset that was considered for use was name.basics. This data shows actors (and relevant info) and some titles (csv of titleid values) that the actor is known for. This information would be very useful.

Other information could be easily scraped from imdb itself, such as plot, reviews, etc. While this information is most likely going to improve the quality of the recommendations, the type of Machine Learning required to do this is beyond the scope of this exercise.

In general, there’s a few main main ways of grouping this type of data for recommendations: (a) Simple recommendations. Recommending the same items regardless of user, normally based on highest rating, or sales data. (b) Content based filtering. The first of these is finding commonalities about data attributes, e.g. movie genre’s, actors, plot, ratings. (c) Collaborative filtering. This is more user behaviour driven, grouping data based on interactions (simliar ratings for one title would group allow for recommending another one)

The data that I selected will allow for some fairly simple content-based filtering.

2. Create Jupyter hosted notebook

My experience with Jupyter is pretty minimal. I’d played with it very briefly when doing a AWS DeepRacer lab over a year ago now. I wanted to get a good understanding of how Jupyter works, so I did the following course on A Cloud Guru: https://learn.acloud.guru/course/introduction-to-jupyter-notebooks

I installed Jupyter on my local machine to begin with and to learn about how the notebooks work.

A really cool advantage of Jupyter notebooks is the reproducable nature of the runs, meaning anyone can run the same experiment, even when the underlying data changes.

The course was a very interesting introduction to some of the good data science tools that are available in Python, as well as how to use hosted notebooks in the cloud.

I highly recommend this course if you’re keen on learning how to use Jupyter.

While I’m on th subject, the real advantage of using Jupyter is that you can perform data science experiments and ML training using resources that you normally wouldn’t have access to, and only need to pay for the infrastructure as you use it!

Being able to see visualisations generated inline with the code is really adventageous as well, making the connection between the context, the code and information really straight-forward.

However, a downfall of running data science scripts on AWS hosted infrastructure is the cost. Pandas loads dataframes into memory (much like other statistics software) and more data means bigger instance type. To load the data I chose, I required an instance type of ml.t2.xlarge… not a cheap instance. Couple that with the cost of SageMaker instances, and costs can quickly add up, especially if you’re just doing this as a training exercise (excuse the pun!).

3. Inspect and visualize data

To understand what the data I got meant, I used Pandas and MatPlotLib Python libraries for analysing and visualising the data.

The real value of this is to see the relationship between different variables. A good example of this is to see number of titles in the data vs. year of realease for each movie.

   plt.bar(df_titles.year.unique(),
         df_titles.year.value_counts().sort_index())
   

Other good relationships to view is between number of votes per title vs. the average rating.

   plt.figure(figsize = (10,8))
   sns.scatterplot(x = df_titles['numvotes'], y = df_titles['averagerating'])
   plt.xlabel('number of votes')
   plt.ylabel('average rating of movie')
   

4. Prepare and transform data

As I was, previous to this challenge, unfamiliar with AWS SageMaker and K-Means Clustering, I used the following AWS provided example Jupyter Notebook as a guide on performing my own clustering: https://github.com/aws/amazon-sagemaker-examples/blob/master/introduction_to_applying_machine_learning/US-census_population_segmentation_PCA_Kmeans/sagemaker-countycensusclustering.ipynb

The information in the data is quite useful for classifying movies. Columns such as ‘genres’ allows us to see movies with th same genre, for instance, which is likely going to be a solid basis for grouping movies. However, K-Means clustering algorithms don’t work with descriptive data, so we need to transform the data.

The genres column contains CSV data of the genres for each movie. Each movie can have no, one, or several genres.

There’s many different permutations (1258 unique combinations) of genres that exist for the dataset:

   # we can see there's lots of unique values, as each genre can be combined with others
   df_titles.genres.unique()
   array(['Romance', 'Biography,Drama', '\\N', ..., 'Fantasy,History,War',
         'Documentary,Family,Sci-Fi', 'Horror,Musical,Thriller'],
         dtype=object)
   df_titles.genres.unique().shape
   (1258,)
   

To convert this to data that an ML-algorithm can use, we need to transform it. Firstly, I converted the CSV data to a list of strings:

   # let's convert the csv column to a pandas list object in a new column
   df_titles['genres_list'] = df_titles.genres.str.split(',').tolist()
   

And then used the Pandas ‘get_dummies()’ function to perform ‘One-Hot-Encoding’ on the individual genres

   # get the one hot encoded values for genre. 
   # (this table is relatively sparse)
   genres_one_hot_encoded = df_titles.genres_list.str.join('|').str.get_dummies().add_prefix('genre_')
   genres_one_hot_encoded.shape
   (254179, 29)
   genres_one_hot_encoded.head()
   

From this, we’ve binarised the data for each genre, where each a movie gets a ‘1’ if it has that as a genre, and ‘0’ if it’s absent. We can also see that the number of unique genres is actually only 29 elements long, which is a bit reduction from 1258! We can then join this data to the main dataframe, and drop the existing descriptive columns, as well as the one-hot-encoded columns for n/a or null values (in the IMDB dataset, these are represented by ‘\N’). This process can be repeated for any descriptive attribute. In my example data, I also included movie language, although this didn’t need string splitting first.

Now the data for the other numerical attributes (runtime, number of votes and average rating) can be scaled using MinMaxScaler. We need to standardise the scaling of the numerical columns in order to use any distance based analytical methods so that we can compare the relative distances between different feature columns.

   scaler=MinMaxScaler()
   df_titles_scaled=pd.DataFrame(scaler.fit_transform(df_titles))
   df_titles_scaled.columns=df_titles.columns
   df_titles_scaled.index=df_titles.index

   df_titles_scaled.describe()
   

The dimensionality of the data is then really large (95 columns!). I used principal component analysis (PCA) to reduce the dimensionality of the data.

   num_components=95

   pca_SM = PCA(role=role,
      instance_count=1,
      instance_type='ml.c4.xlarge',
      output_path='s3://'+ bucket_name +'/titles/',
      num_components=num_components)
   

I then used the PCA job output to transform the original data. Once transformed, it was ready for training!

When viewing what attributes make up the components found, it’s mostly Genre, with some variation based on release year, language and popularity.

5. Train

Once the data is transformed, I was able to call through to the Python Sagemaker library to perform segmentation using unsupervised clustering, like this:

   import sagemaker
   from sagemaker import KMeans

   num_clusters = 40
   kmeans = KMeans(role=role,
                  instance_count=1,
                  instance_type='ml.c4.xlarge',
                  output_path='s3://'+ bucket_name +'/titles/',              
                  k=num_clusters)
   

After this has been run, the original data can have the cluster label mapped back to it. The distrbution of the clusters looks like so:

6. Recommend

I wanted to present the outcome of the recommendation engine to real users. For this, I needed a website, or at a minimum, an API.

The basic flow of this information is:

• Sagemaker Notebook writes trained model to CSV file in S3 Bucket
• Scheduled Lambda loads CSV data into DynamoDB table
• API Gateway using Lambda proxy queries the data to find titles and return a sample of titles in the same cluster as the chosen title.
• Static React JS website (hosted in S3, served up via CloudFront) allows users to search for movies and request recommendations based on this. Don’t judge on the styling!

7. Source control

You can view all the code for the training notebook, app infrastructure and API/website here:

https://github.com/simonmackinnon/cloudguruchallenge-2020-10

8. Clean up resources

When dealing with Machine Learning, instances and SageMaker endpoints can bear large costs very quickly. An important thing to check is the “Endpoints” in the Sagemaker console. I’ve added code at the end of the Notebook to delete the endpoints.

   sagemaker.Session().delete_endpoint(kmeans_predictor.endpoint)
   

However, if there is an error earlier on, it’s worth manually checking that the endpoints really are deleted!

The Notebook instance sise is also considerable. It’s REALLY worth stopping it when not in use. Or, you can run it on your local machine if it has the required memory resources (my MacBook Pro has 16GB RAM, which is more than enough for this exercise). If you’re planning on doing that, make sure that the configured AWS user that you use on your machine has the ability to assume into the SageMaker execution role (the jobs require you to pass it in as a variable)

9. Impovements

At the start of this blog I included an architecture diagram for the whole solution. I also proposed a second version of the application, which would allow users to log into the website, select movies they had previously watched (stored in DynamoDB) and filter those movies out of the recommended results. Here’s an example of how this would work:

Another improvement I’d make would be to add CodeBuild jobs for automated deployments. I didn’t set up a CI/CD pipeline for anything, so this will definitely be part of V2!

I wanted to include Movie posters in the recommendations and title searches. While this information is obtainable via web-scraping of IMDB, or 3rd-party API calls, tying these calls into the API for title recommendations really slowed the site down. I have some ideas for how this would work, namely storing the images in S3 for all titles, iterating over the records in the database using step functions.

Finally, a major imporovement I’d make would be to the clustered data. I think using some Natural Languange Processing to group movie titles based on plot text would be a fantastic way to approach this. Another way would be to get user rating and viewing data and perform collaborative clustering.

If You’re on the Machine-Learning Journey, Take The Train

I’m really a Machine Leanring and Data Science beginner. That being said, the documentation and (especially) the out-of-the-box tools that AWS SageMaker provides for performing Machine Learning are REALLY awesome!

I had a fantastic time learning about what’s required to get data ready for training, what the outputs of ML jobs means, and particularly, validating how good my model is.

There’s a lot more involved in getting this all working, so please reach out if there’s anything in the code that you want me to explain, or provide references for!

References:

• https://github.com/aws/amazon-sagemaker-examples/tree/master/introduction_to_applying_machine_learning/US-census_population_segmentation_PCA_Kmeans
• https://longjp.github.io/statcomp/projects/clusteringimdb.pdf
• https://www.imdb.com/interfaces/
• https://learn.acloud.guru/course/introduction-to-jupyter-notebooks
• https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.get_dummies.html

I gave the #CloudGuruChallenge by Forrest Brazeal (A Cloud Guru) for September a go! I started 1 day before the deadline, so rushed to finish a bit. The next challenge will be worked on straight away, and the Readme files will definitely have more than just titles. “Working software over comprehensive documentation” right?

Here’s some of the stuff that I did! Read the challenge details here: https://acloudguru.com/blog/engineering/cloudguruchallenge-python-aws-etl

Here’s the basic architecture of the solution:

See my code here: https://github.com/simonmackinnon/cloudguruchallenge/tree/main/2020-09

ETL Job using Python

The job runs automatically using a CloudWatch scheduled rule, once per day. Setting this up was pretty straightforward. One gotcha for people is that a CloudWatch rule needs permission to invoke Lambda functions.

It then loads data and puts it into DynamoDB table. I had a pretty fun time coding this. I hadn’t used the Pandas Python library before, so it was good to see the power of it. Some time (what feels like almost a lifetime) ago, I learnt to use R to do data science. This was a pretty similar experience (although zero-indexing helps!). Some of the nuances of the challenge were around only loading the most recent day’s data, so some smarts had to be built into it for that to work. The merge functionality helped to spead up the work

It send SNS notifications for different status updates. I set this up as per the brief, although I was pushing to my topic for every row that couldn’t be read correctly, which proved to be a little verbose (at one point I was sending hundreds of error notifications due to a bug in my code). Pretty fun and easy to set up. Then getting CloudFormation passing the output ARN to the environment variables of the Lambda function kept this relatively re-deployable.

For Reporting, I used API Gateway to expose the DynamoDB table data, and then consumed it in JavaScript. Some of the gotchas in this (a) while the API request type for performing Scan operations is GET, the HTTP method for DynamoDB service calls is always POST (b) I found getting the Integration Request Mapping Template and the Integration Repsonse Mapping Template are right for this is a little difficult (c) ensuring the calls to the API had the correct headers to avoid a CORS / Preflight error is always difficult (and something I should spend some time learning about, it always trips me up). I built a simple vanilla JS demo site (due to time constraints) to retrieve (and sort) the data, then display using Chart.js

Anyway, you access the data URL here:

https://2tp0wsvdr2.execute-api.ap-southeast-2.amazonaws.com/live/cumulativedata

And you can see the graph output here:

http://simonmackinnon.com/cloudguruchallenge-2020-09.html

Infrastructure as Code

Everything is defined in CloudFormation (except uploading function package to S3 and publishing new versions) Some of this was HAAARD… especially setting up API Gateway to expose the DynamoDB data without using Lambda. This is relatively easy in the console, but I found some of the settings difficult in YAML/CloudFormation. As mentioned above setting the Mapping Templates for the request/response continuoulsy lead to formatting issues… until it didn’t.

TBD:

• Lambda layers: the package built was little big, and some of that could be reduced by using layers, especially for the Pandas library
• VPC infrastructure was a little overboard for single lambda
• CodePipeline to test and publish ETL job function package
• CodePipeline to update infrastructure on update
• Build React site to display more interactive/multiple graphs
• API Keys / Security

This one only slightly annoyed me, but still thought it was worth mentioning.

Some things are really easy to in the console vs. via API calls. AWS Lambda “triggers” is a perfect example of this. The general steps are: create a Lambda function, open the Lambda function configuration in the console, click “Add Trigger”, select source service and configure. Done!

Being able to set up lambda triggers for a multitude of triggers from the Lambda console is a really nice (read: simple) way of configuring, and importantly, viewing, what services should be doing so, without having to navigate to each of the respective services’ consoles themselves. When you add Lambda triggers in this way, you can see a visual list of all of the triggers as one of the first things in the Lambda console. Great, really nice UI/UX!

So, now we want to replicate that experience using CloudFormation or API/CLI commands. You would be clever in thinking you can do all of this using the Lambda CLI, given you can do all this in the Lambda Console. And you’d also be wrong. The API call to produce this (listening) trigger is the CreateEventSourceMapping, and the respective CLI command create-event-source-mapping. If you look at this documentation, you’ll see that the only services for which you can create such a mapping, like you can in the console, is DynamoDB, Kinesis and SQS. Only those three… This is because Lambda service can essentially “read” events from these services, rather than be asyncronously or synchronously invoked by the triggering service.

aws lambda create-event-source-mapping \
    --function-name CodeCommitLambda-lambdacodecommit-OT2Z33UZKD9O \
    --batch-size 5 \
    --starting-position LATEST \
    --event-source-arn arn:aws:dynamodb:ap-southeast-2:366389342275:table/TestTable/stream/2020-06-20T04:50:40.178

And, of course, you can set up triggers for each service respectively from the API calls for those services, but it only creates the one-way mapping. The Lambda function(s), in this case, have no knowledge or ownership of the triggers set up, for example, from CodeCommit.

aws codecommit put-repository-triggers \
    --repository-name my-webpage \
    --triggers name=MyLambdaTrigger,destinationArn="arn:aws:lambda:ap-southeast-2:123456789012:function:CodeCommitLambda-lambdacodecommit-OT2Z33UZKD9O",customData="",branches=master,events=all

Given this, it’s disappointing that the Lambda console repsects the mapping for invoke-type triggers, but there’s no way of even listing these kind “mappings” if you’re doing function creation programatically.

As I have written previously, I’ve just committed myself to achieving the AWS DevOps Professional certification. As part of my study, I’m attempting to work through a hands-on online course. I’ve also comitted to doing all demos using only the AWS CLI, SDK or CloudFormation. To force myself to to this, I’ve only granted programatic to my IAM user in my training account. The rationale is this: the console makes deploying things easy, and sets most default values for required fields in API calls appropriately. To get a better understanding of the services being used, provisioning in an automated way ensures these values need to be understood.

As I said, I started this course, primed to only work using scripts and Infrastructure as Code (with the aim of using CloudFormation primarily to ensure easy removal of deployed resources). The very first part of the very first demo: create an IAM User and create HTTPS CodeCommit Security Credentials for it. Easy, right? This is a two second job in the console.

And, while there exists an API for this, CreateServiceSpecificCredential, CloudFormation doesn’t support this IAM feature. Enter, CloudFormation Custom Resources!

The steps needed for this resource could have been really simple, as the API call only requires an existing IAM user’s username, and the endpoint of the AWS service to create the credentials for. I wanted to create a simple automation sequence to allow multiple users to be created with this stack.

I don’t have a lot of experience writing Lambda code for CFN Custom Resources, so I used crhelper to help build out the function scaffolding. This library does a crazy amount of the undifferentiated heavy lifting. All that was required was to pass through the username to the create and reset credentials API calls (I used the Python SDK for this).

The code was really simple:

from crhelper import CfnResource
import boto3, json

helper = CfnResource()
iamclient = boto3.client('iam')

@helper.create
def create_https_credentials(event, _):
    user = event['ResourceProperties']['user']

    response = iamclient.create_service_specific_credential(
        UserName=user,
        ServiceName='codecommit.amazonaws.com'
    )

    helper.Data['ServiceUserName'] = response['ServiceSpecificCredential']['ServiceUserName']
    helper.Data['ServicePassword'] = response['ServiceSpecificCredential']['ServicePassword']

@helper.update
def reset_https_credentials(event, _):
    user = event['ResourceProperties']['user']
    
    response = iamclient.reset_service_specific_credential(
        UserName=user,
        ServiceName='codecommit.amazonaws.com'
    )

    helper.Data['ServiceUserName'] = response['ServiceSpecificCredential']['ServiceUserName']
    helper.Data['ServicePassword'] = response['ServiceSpecificCredential']['ServicePassword']

@helper.delete
def no_op(_, __):
    pass

def handler(event, context):
    print("Started execution of HTTPS Credentials Creator Lambda...")
    print("Function ARN %s" % context.invoked_function_arn)
    print("Incoming Event %s " % json.dumps(event))
    
    helper(event, context)

You can check out (and use) the code for this here: https://github.com/simonmackinnon/codecommit-httpscreds-cloudformation. This repo has CloudFormation templates to deploy single-time resources, as well as to create an IAM user and output the corresponding Access Keys and the CodeCommit HTTPS Security Credentials. Feedback super welcome.

Anyway, at this rate, the 20-hour long course will probably take me about a year to complete, ha ha ha!

Course URL: https://learn.acloud.guru/course/aws-advanced-cloudformation/dashboard

TL;DR

Do this course! Awesome and fun content using practical templates provided and evolved to match the skills being taught. Perfect course introducing some complex and advances topics for AWS CloudFormation. Thanks Adrian!

Long Version

After completing the AWS Associate Certification trifecta late last year, and Azure Fundamentals earlier this year, I took a break from study to figure out what path of learning I wanted to do next. Given I work as an AWS Cloud Engineer, I thought the AWS DevOps Professional certification would be highly relevant as well as an awesome opportunity to learn some new concepts and technology.

This blog is a really good starting point (I think) to what needs to be learnt/studied for this certification. I love Infrastructure as Code, and this post recommended doing the A Cloud Guru - Advanced AWS CloudFormation course to brush up on CloudFormation skills.

I loved this course. It posed some business challenge case studies, in two fictitious companies. This made the learning much more realistic.

The course content provides the templates to be deployed. For the first case-study, I re-wrote this, iterating on it as the course progressed. This meant that I got hands-on experience writing the CFN templates, and importantly experienced all of the troubleshooting that comes along with doing so.

For those who are unfamiliar, Infrastructure as Code is a way of declaring in a text file (of some kind), the infrastructure resources, as well their configuration, that you desire to be created. There are many different libraries, frameworks and services to do so. For AWS, CloudFormation is the native service that they provide to manage this. Some of the advantages of this service over its competitors is the easy integration into your AWS account, ease of learning/setup, as well as a slight security win (looking at you Terraform with your plain-text state-files!).

Some really cool concepts are taught in this, one of my favourites is how cfn-hup is explained, as this is something that always seems confusing to me. Being able to have EC2 resources detect changes in its own meta-data and run some specified commands is really cool. Tying this to re-implement the cfn-init process after a change is detected is a powerful mechanism for triggering reloading of instance setup command when a stack is updated.

The course was, I believe, recorded around 2017/18, so some of the screens in the console are a little out-of-date, although had changed dramatically since then. At one point, we are required to create some Google web authentication credentials to use in an app we create. The steps around this had changes slightly, but the accompanying instructions from ACG helped to navigate these changes.

Another area of learning in this course, that piqued my interest, was CloudFormation custom resources using Lambda. I’ve known about this feature of CFN for some time, and the idea had always interested me. Adrian teaches this content in a very simple manner, especially how the resource lifecycle works using the resource properties/attributes and what the functions’ responses need to contain for it to all work. From these small and simple demos, we automatically allocated CIDR ranges for a multi-environment application within a VPC, a task that normally would require networking knowledge and manual entry. Through this example, Adrian showed the awesome power of extending CloudFormation using Lambda-based custom resources.

Overall, the design/architecture pattern implemented could be used as the foundations for your own projects, etc. even in a work/production setting. Definitely templates that I’ll be hanging onto for some time!!!

Amazon Linux 1 AMI Usage and Upgrade Issue:

• Only one real issue (other than superficial issues related to POC nature of apps/environments). The EC2 instances used in the templates were based off of the Amazon Linux 1 AMI. Given this image type is flagged for End-Of-Life at the end of 2020 this is somewhat problematic. For the first case-study, I updated the template(s) to use Amazon Linux 2, which proved difficult. The cfn-init config packages command has difficulty installing an appropriate version of PHP for WordPress to run when the yum ‘php’ package is used. If the default packages are used, the following error occurs in WordPress:
  
  “Your server is running PHP version 5.4.16 but WordPress 5.2 requires at least 5.6.20.”
  
  To overcome this, we need to install PHP > v7.2 using the amazon-linux-extras. Unfortunately, this isn’t available in the cfn-init configuration packages section. To get this to install, I had to the following command to my install_wordpress configuration: \

commands:
    enable_php:
        cwd: "~"
        command: "amazon-linux-extras install php7.2"

In any case, that seemed to be one of the only issues when upgrading the instance to Amazon Linux 2.

Overall

Pretty stoked to get through this. As with any Infrastructure course, the time taken to get through the content if you do the demos yourself is always a lot longer than the course length, with lots of waiting for stacks to provision/update/delete. Great starting point to move to automation in an AWS native way!