: How can I parse Amazon S3 log files? What are the best options for parsing Amazon S3 (Simple Storage) log files? I've turned on logging and now I have log files that look like this: 858e709ba90996df37d6f5152650086acb6db14

@Welton855

We are about to launch Qloudstat which will give you analytics for your S3 account as a service.

Vote Up Vote Down

@Goswami781

Amazon recommends using AWStats and includes instructions on how to set up an AWStats configuration file. You will also need to set up some method to automate downloading the logs to where your AWStats installation is.

Personally I use cron to set off a Python script that uses boto to grab the files.

"""s3logDL.py

Downloads S3 logs and deletes old logs

Usage: python s3logDL.py"""

import sys
import os
from boto.s3 import Connection
import boto.s3

########################


########################

def main(argv):
s3 = Connection('ACCESS_ID', 'PASSWORD')
logsBucket = s3.get_bucket('logbucket')

for b in logsBucket:
#Download log
print b.name
key = logsBucket.get_key(b.name)
fp = open("/PATH/TO/SAVE/POINT/" + b.name, "w")
key.get_file(fp)

#Delete log
key.delete()

exit(0)


########################

if __name__ == '__main__':
main(sys.argv)

Vote Up Vote Down