Access GitHub Repo file & Store it in AWS S3 bucket using AWS Lambda Function

Objective

• Access GitHub public repo file & store it in AWS S3 bucket using AWS Lambda Function

Resources / Tech Involved & its use

• GitHub
• @octokit/core library  - to access the GitHub repo via REST/GraphqQL call.
• AWS 
• S3 bucket - to store the files 
• Lambda Function - to read the GitHub repo file via OctoKit and store it in S3
• API contract 
• REST 

Overview

• In Lambda function, we access the GitHub repo using @octokit/code -> REST call to access the file and then store it in the S3 bucket

Workflow Diagram

Lambda Function Code

// repoToS3FileStorage -> lambda function

const AWS = require('aws-sdk');

const octoKit = require('@octokit/core');

const S3 = new AWS.S3();

/**

 * Handler function to read the GitHub repo file & store it in S3 bucket

 */

exports.repoToS3FileStorage = async () => {

  const octokit = new octoKit.Octokit();

  const response = await octokit.request('GET /repos/{owner}/{repo}/contents/{path}', {

    owner: 'vivekVells',

    repo: 'tech_challenge',

    path: 'songData.json'

  });

  // octokit responds the content in 'base64 encoding', decoding this to 'base64' here

  const bodyContent = Buffer.from(response.data.content.toString(), 'base64');

  const params = {

    Body: Buffer.from(bodyContent, 'binary'), 

    Bucket: "iheartsongsbucket", 

    Key: "songData.json"

  };

  await S3.putObject(params).promise();

};

Code Breakdown

Octokit

• make GET REST call with parameters: owner, repository and path 

 const response = await octokit.request('GET /repos/{owner}/{repo}/contents/{path}', {

    owner: 'vivekVells',

    repo: 'tech_challenge',

    path: 'songData.json'

  });

• response would be base64 encoded that can be accessed as 'response.data.content'

S3 

• convert received base64 encoded data to binary format

  const bodyContent = Buffer.from(response.data.content.toString(), 'base64');

• use putObject() from AWS.S3 instance

  const params = {

    Body: Buffer.from(bodyContent, 'binary'), 

    Bucket: "iheartsongsbucket", 

    Key: "songData.json"

  };

  await S3.putObject(params).promise();

• Bucket refers to bucket name and key refers to file name

Reference

• Encoding and Decoding Base64 Strings in Node.js (stackabuse.com) - to understand about nodeJs Buffer and importance of using base64 encoding