cv2 image data upload to s3 python

Paradigm processing with Lambda + OpenCV (gray epitome cosmos)

This commodity is a hands-on article for reviewing and fixing the knowledge gained past developing Serverless Web App Mosaic. It is one of w2or3w / items / 87b57dfdbcf218de91e2).

It would be prissy to read this article afterwards looking at the following.

• Uploading files to S3 and calling Lambda using AWS Dilate

introduction

Implement Lambda office in Python 3.6 that performs paradigm processing on the image triggered by uploading the image to S3. OpenCV is used for paradigm processing.

content

Create a repository for Lambda function projects on Github

Create a repository for your project on GitHub. The repository proper name is sample_lambda_py_project.

Clone repository to Cloud9

          $ git clone https://github.com/{YourGithubID}/sample_lambda_py_project.git                  

Creating a elementary plan

Create an appropriate binder (here, source) in sample_vue_project, and create a lambda_function.py file in it. For the fourth dimension being, allow'south presume that the code is implemented inline in Previous article. (The content of the log message will be changed a little.)

lambda_function.py

                      from urllib.parse import unquote_plus import logging logger = logging.getLogger() logger.setLevel(logging.INFO)  def lambda_handler(result, context):     bucket = event['Records'][0]['s3']['bucket']['proper name']     cardinal = unquote_plus(event['Records'][0]['s3']['object']['key'], encoding='utf-eight')     logger.info("Office Starting time (deploy from S3) : Bucket={0}, Key={one}" .format(saucepan, key))                  

I think the Cloud9 environment looks like this.

Aught the function lawmaking and deploy it to Lambda via S3

Information technology is not necessary for simple code as it is now, just if you employ a library other than the standard library such equally OpenCV, you need to upload the library every bit a package in a zilch file.

So, first, let's zip it without using the library and upload it to Lambda via S3.

Zip compression

Compress nether the directory where lambda_function.py is located with the name lambda-parcel.goose egg. Execute the following control in the directory where lambda_function.py is located.

          $ aught -r ../lambda-package.nada *                  

Upload to S3

Upload the created lambda-packet.zip to S3. Execute the following command while keeping the directory where lambda_function.py is located.

          $ aws s3 cp ../lambda-package.naught s3://sample-vue-projection-saucepan-work/deploy/lambda-package.zippo                  

Deploy in Lambda panel

AWS Console> Lambda> Functions> S3Trigger ********-work To admission. Prepare the function code as follows. Code entry type: File upload from Amazon S3 Runtime: Python 3.6 Handler: lambda_function.lambda_handler Amazon S3 link URL: The link URL of lambda-package.nothing uploaded earlier Click the save push at the summit correct of the screen.

Operation check

After uploading the file from the sample_vue_project web app, AWS Panel> CloudWatch> Log Group> aws / lambda / S3Trigger ********-work Please visit to encounter the latest log stream. Function Outset (deploy from S3) : Bucket=sample-vue-projection-bucket-piece of work, Primal=public/191231063457-16f5aaa24b1/faceA.jpeg Information technology is OK if an INFO log like this is output.

OpenCV installation

In the source directory where lambda_function.py is located, run the post-obit command.

• Brand sure that the Python referenced by pip is 3.half dozen.

          $ pip --version pip 9.0.three from /usr/lib/python3.6/dist-packages (python three.6) $ pip install opencv-python -t .                  

The Lambda function needs to include the library in the zip package to upload, and so -t . Specify this option and install information technology in the current directory.

Gray epitome converter

Update lambda_function.py.

lambda_function.py

                      # coding: UTF-viii import boto3 import os from urllib.parse import unquote_plus import numpy as np import cv2 import logging logger = logging.getLogger() logger.setLevel(logging.INFO) s3 = boto3.customer("s3")  def lambda_handler(event, context):     bucket = result['Records'][0]['s3']['saucepan']['name']     central = unquote_plus(event['Records'][0]['s3']['object']['primal'], encoding='utf-8')     logger.info("Function Kickoff (deploy from S3) : Saucepan={0}, Cardinal={1}" .format(saucepan, key))      fileName = bone.path.basename(key)     orgFilePath = u'/tmp/' + fileName     processedFilePath = u'/tmp/processed-' + fileName      if (not fundamental.startswith("public")):         logger.info("not start with public")         return      keyOut = cardinal.replace("public", "processed", 1)     logger.info("Output local path = {0}".format(processedFilePath))      attempt:         s3.download_file(Bucket=bucket, Key=fundamental, Filename=orgFilePath)          orgImage = cv2.imread(orgFilePath)         grayImage = cv2.cvtColor(orgImage, cv2.COLOR_RGB2GRAY)         cv2.imwrite(processedFilePath, grayImage)          s3.upload_file(Filename=processedFilePath, Saucepan=bucket, Key=keyOut)         logger.info("Function Completed : processed central = {0}".format(keyOut))      except Exception as e:         logger.exception(e)         raise due east              finally:         if os.path.exists(orgFilePath):             os.remove(orgFilePath)         if os.path.exists(processedFilePath):             os.remove(processedFilePath)                  

(* I won't bear on on the basic syntax of Python hither. If you demand information technology, learn somewhere.)

Uploaded Convert a file similar public / 191231112635-16f5bb5247f / lenna.png to a gray epitome, Uploading to a path like processed / 191231112635-16f5bb5247f / lenna.png.

Lambda bones settings

You can specify the retentiveness and timeout in the Lambda basic settings. By default, memory = 128MB, timeout 25 seconds. Photos taken with recent smartphones are big in size, and then if you keep the default settings, you may run out of retentiveness and an exception may occur. When you lot increment the processing performed by Lambda in the future, yous may run out of time and fourth dimension out. Adjust your Lambda preferences equally needed. (* Regarding the size of the epitome, I recall information technology is ameliorate to compress it on the client side earlier uploading.)

Deployment and operation bank check

Zip it up, upload information technology to S3, deploy it to Lambda, and so upload the paradigm from your web app. It is OK if the monochrome image is uploaded beneath the S3 saucepan. Check the CloudWatch log as well. (Lena is cute)

Afterword

When you operate from Lambda faction to S3 which is the trigger source like this sample program, the Lambda function is called by that functioning. If yous are not careful, you lot will end up with an infinite loop of Lambda, so be careful.

If you lot inadvertently autumn into a Lambda infinite loop, you lot should discover information technology right away, but if you don't notice it right away, you don't desire to think most it. The automatic billing addition machine volition continue to practice its chore without hesitation.

The fashion to stop the space loop Lambda is to either remove the function or upload the function code that raises the exception. I recall it's a good thought to upload the function code that raises the exception, considering deleting a function can be cumbersome to create and configure again, simply is there whatever other fashion? Please let me know.

In whatsoever case, you will have to operate it with your trembling hands. While sweating strangely.

Too, I oasis't debugged the Python code called by Lambda yet. It seems that you can debug with Cloud9, so in one case you learn it, yous volition be able to keep with development more efficiently. Once you've mastered it, try to write an article.

wolfealoost.blogspot.com

Source: https://linuxtut.com/en/e6feac50c751bbc64697/

Share this post