Automate JSON File Processing From S3 Bucket And Push In DynamoDB Using Lambda

Create Role For Lambda

1. Create policy mentioned below.
   
   COPY

   COPY

   {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "dynamodb:CreateTable",
                "s3:PutObject",
                "s3:GetObject",
                "dynamodb:PutItem",
                "dynamodb:UpdateItem",
                "dynamodb:UpdateTable"
                "logs:CreateLogDelivery",
                "logs:PutMetricFilter",
                "logs:CreateLogStream",
                "logs:GetLogRecord",
                "logs:DeleteLogGroup",
                "logs:GetLogEvents",
                "logs:FilterLogEvents",
                "logs:GetLogGroupFields",
                "logs:CreateLogGroup",
                "logs:DeleteLogStream",
                "logs:GetLogDelivery",
                "logs:PutLogEvents"
            ],
            "Resource": "*"
        }
    ]
   }
   

2. Now create new role for lambda and attach this policy to the role.

Create S3 Bucket And Attach Tags

Creates a new S3 bucket. To create a bucket, you must register with Amazon S3 and have a valid Amazon Web Services Access Key ID to authenticate requests. Anonymous requests are never allowed to create buckets. By creating the bucket, you become the bucket owner.

1. Lets import boto3 module
   
   COPY

   COPY

   import boto3
   

2. We will invoke the client for S3
   
   COPY

   COPY

   client = boto3.client('s3')
   

3. Now we will use input() to take bucket name to be create as user input and will store in variable “bucket_name“.
   Note:- Make sure to check the bucket naming rules here
   
   COPY

   COPY

   bucket_name=str(input('Please input bucket name to be created: '))
   

4. Goto link where you will find all arguments list. Based on your requirement you can put this arguments to list your S3 buckets. This document also mentions datatype of the parameter.
   Note:-Bucket Name argument is mandatory and bucket name should be unique
   
   COPY

   COPY

   response1 = client.create_bucket(
    ACL='public-read-write',
    Bucket=bucket_name
    )
   

5. Now we will use input() to confirm if user wants to go ahead with bucket tagging via user input and will store it in variable “tag_resp“.
   
   COPY

   COPY

   tag_resp=str(input('Press "y" if you want to tag your bucket?: '))
   

6. Now we will use if condition and take user input for tags which needs to be defined for bucket.
   We will store tag key in variable “tag_key” and tag value in “tag_value“. To add tag to bucket we are going to use put_bucket_tagging() method, make sure to check official documentation here In method parameters we are passing variable as “bucket_name“,”tag_key“,”tag_value“.
   
   COPY

   COPY

   if tag_resp == 'y':
   tag_key=str(input("Please enter key for the tag: "))
   tag_value = str(input("Please enter value for the tag: "))
   response2 = client.put_bucket_tagging(
   Bucket=bucket_name,
   Tagging={
       'TagSet': [
           {
               'Key': tag_key,
               'Value': tag_value
           }
       ]
   })
   

   To view entire github code please click here

Create DynamoDB Table

1. Python code in one module gains access to the code in another module by the process of importing it. The import statement combines two operations it searches for the named module, then it binds the results of that search to a name in the local scope.
   
   COPY

   COPY

   import boto3
   

2. We will invoke the resource for DyanamoDB.
   
   COPY

   COPY

   dynamodb = boto3.resource('dynamodb')
   

3. We will use create_table() function to create table in Dynamo DB with following arguments listed below. Here we will see 2 examples one with “primary keys only” and another with “primary key and sort key”. You can find official documentation here.
   Example1:- Below code is to create table with primary key only
   
   COPY

   COPY

   table = dynamodb.create_table(
    TableName='user',
    KeySchema=[
        {
            'AttributeName': 'id',
            'KeyType': 'HASH'    #Partition Key Only
        }
    ],
    AttributeDefinitions=[
        {
            'AttributeName': 'id',
            'AttributeType': 'N'
        }
    ],
    ProvisionedThroughput={
        'ReadCapacityUnits': 1,
        'WriteCapacityUnits': 1
    },
   )
   

   You can find working code for example in Git Repo here