Historic Notes

  • This page was originally based on notes written in 2018, so at least some of it will be out of date.

General Intro

  • Amazon S3 - simple secure storage, used for storing flat files
  • At Samba we kept app definitions in S3 for source control of configs, so we could do rollbacks

Configuring S3 buckets

  • If you click through from S3 to an individual file, you’ll see the url for that file.
    1. You will also see Properties and Permissions, to see how these are configured for other files.

S3 Logs

  • This is for looking at logs which have already been archived into S3 (happens once an hour).
  • !! Don’t configure S3 logging without doing a reboot immediately afterwards !!
    • Otherwise in my experience it brought sites down
  • It can be hard to find the right logs in S3 buckets. 
    • Every time a new server is spun up by the load balancer, we got a new instance Id and therefore a new folder for logs.
    • To get a feel for what instance Id you might be interested in, go to EBS and look at Events on the left hand side – you will see details of recent instances that have been created
    • CAUTION
      • !! In my experience for each instance Id, the logs were not listed sequentially. This means that when you are on page 1 of the results, you will not see all logs for a particular date. You have to CHECK EVERY PAGE OF RESULTS if you want logs for a particular time.
  • How do you know whether logs are being recorded in S3?
    • To find out / change whether logs are being recorded in S3 for a particular environment, go here:
    • EBS | Select application | select environment | click Configuration on the left | look under Software Configuration, where it says Log Options - you can “enable log file rotation to Amazon S3”.

Fetch log files from an S3 docker container

  • To get into a docker container from outside:
    • This: sudo docker ps
    • Then this (fill in id): sudo docker exec –it [id] /bin/bash
  • Then cd logs
  • Current log output will be in [log-name].log, but it is regularly emptied out into zip files in the rotated sub-folder, so sometimes it is empty.
    • Anything in the rotated folder should have already been copied into the S3 bucket
  • If you want to view contents, you can just do cat [log-name].log
  • If you want to copy log file contents to your local machine:
    • Exit docker container: exit
    • Then run this command to copy log out of container and into host: sudo docker cp 470bdbc4d391:/opt/myfolder/logs/mything.log ./
      • You’ll need the correct file name
      • You’ll also need the correct docker container id
    • Then exit instance: logout
    • Then back on local machine, run this command to fetch log and copy to local: scp -i “/Volumes/GoogleDrive/My Drive/go-server.pem” ec2-user@[ip-address]:/home/ec2-user/mything.log “/Volumes/GoogleDrive/My Drive/”
      • You’ll need the correct file name
      • You’ll also need the correct user name / IP address