S3 Log Analysis

Thu 21 May 2015 by Patrick Pierson

I am all for using ELK to analyze logs however sometimes you just want a quick answer like IP addresses of the systems that hit your site. Using the following two commands you can quickly get a view of the IPs that have hit your site. FYI you will need to turn on S3 bucket logging for your static site.

aws s3 cp --recursive s3://logs-bucket ./
cat */* | grep WEBSITE.GET.OBJECT | awk '{print $5}' | sort | uniq -c

Output should look like this:

      1 1.2.3.4
      2 100.101.102.103
     10 1.9.1.68
      1 2.95.43.19
      1 146.228.166.242
     58 9.25.136.225
     16 94.31.214.220

Elasticsearch, Logstash, and Kibana to visualize logs

Tue 16 September 2014 by Patrick Pierson

ELK to visualize logs

read more