debian-mirror-gitlab/doc/user/project/clusters/serverless/aws.md

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

508 lines
16 KiB
Markdown
Raw Normal View History

2020-05-24 23:13:21 +05:30
---
stage: Configure
group: Configure
2021-02-22 17:27:13 +05:30
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
2020-05-24 23:13:21 +05:30
---
2021-03-11 19:13:27 +05:30
# Deploying AWS Lambda function using GitLab CI/CD **(FREE)**
2019-12-04 20:38:33 +05:30
GitLab allows users to easily deploy AWS Lambda functions and create rich serverless applications.
2020-04-08 14:13:33 +05:30
GitLab supports deployment of AWS Lambda functions through GitLab CI/CD using the following Serverless frameworks:
2019-12-04 20:38:33 +05:30
2020-04-08 14:13:33 +05:30
- [Serverless Framework with AWS](#serverless-framework)
- [AWS' Serverless Application Model (SAM)](#aws-serverless-application-model)
## Serverless Framework
2020-06-23 00:09:42 +05:30
The [Serverless Framework can deploy to AWS](https://www.serverless.com/framework/docs/providers/aws/).
2019-12-04 20:38:33 +05:30
2020-01-01 13:55:28 +05:30
We have prepared an example with a step-by-step guide to create a simple function and deploy it on AWS.
2020-04-08 14:13:33 +05:30
Additionally, in the [How To section](#how-to), you can read about different use cases like:
2020-01-01 13:55:28 +05:30
- Running a function locally.
- Working with secrets.
- Setting up CORS.
2021-03-11 19:13:27 +05:30
Alternatively, you can quickly [create a new project with a template](../../working_with_projects.md#create-a-project). The [`Serverless Framework/JS` template](https://gitlab.com/gitlab-org/project-templates/serverless-framework/) already includes all parts described below.
2020-01-01 13:55:28 +05:30
2020-04-08 14:13:33 +05:30
### Example
2019-12-04 20:38:33 +05:30
2021-02-22 17:27:13 +05:30
This example shows you how to:
2019-12-04 20:38:33 +05:30
1. Create a basic AWS Lambda Node.js function.
1. Link the function to an API Gateway `GET` endpoint.
2020-04-08 14:13:33 +05:30
#### Steps
2019-12-04 20:38:33 +05:30
The example consists of the following steps:
2020-04-08 14:13:33 +05:30
1. Creating a Lambda handler function.
1. Creating a `serverless.yml` file.
1. Crafting the `.gitlab-ci.yml` file.
1. Setting up your AWS credentials with your GitLab account.
1. Deploying your function.
1. Testing the deployed function.
2019-12-04 20:38:33 +05:30
Lets take it step by step.
2020-04-08 14:13:33 +05:30
#### Creating a Lambda handler function
2019-12-04 20:38:33 +05:30
2021-02-22 17:27:13 +05:30
Your Lambda function is the primary handler of requests. In this case, create a very simple Node.js `hello` function:
2019-12-04 20:38:33 +05:30
```javascript
'use strict';
module.exports.hello = async event => {
return {
statusCode: 200,
body: JSON.stringify(
{
message: 'Your function executed successfully!'
},
null,
2
),
};
};
```
Place this code in the file `src/handler.js`.
`src` is the standard location for serverless functions, but is customizable should you desire that.
2021-02-22 17:27:13 +05:30
In our case, `module.exports.hello` defines the `hello` handler to reference later in the `serverless.yml`.
2019-12-04 20:38:33 +05:30
2021-03-11 19:13:27 +05:30
You can learn more about the [AWS Lambda Node.js function handler](https://docs.aws.amazon.com/lambda/latest/dg/nodejs-prog-model-handler.html) and all its various options in its documentation.
2019-12-04 20:38:33 +05:30
2020-04-08 14:13:33 +05:30
#### Creating a `serverless.yml` file
2019-12-04 20:38:33 +05:30
2021-02-22 17:27:13 +05:30
In the root of your project, create a `serverless.yml` file containing configuration specifics for the Serverless Framework.
2019-12-04 20:38:33 +05:30
Put the following code in the file:
```yaml
service: gitlab-example
provider:
name: aws
2021-06-08 01:23:25 +05:30
runtime: nodejs14.x
2020-03-13 15:44:24 +05:30
2019-12-04 20:38:33 +05:30
functions:
hello:
handler: src/handler.hello
events:
- http: GET hello
```
Our function contains a handler and a event.
2021-02-22 17:27:13 +05:30
The handler definition provisions the Lambda function using the source code located `src/handler.hello`.
2019-12-04 20:38:33 +05:30
2021-02-22 17:27:13 +05:30
The `events` declaration creates an AWS API Gateway `GET` endpoint to receive external requests and hand them over to the Lambda function via a service integration.
2019-12-04 20:38:33 +05:30
2020-06-23 00:09:42 +05:30
You can read more about the [available properties and additional configuration possibilities](https://www.serverless.com/framework/docs/providers/aws/guide/serverless.yml/) of the Serverless Framework.
2019-12-04 20:38:33 +05:30
2020-04-08 14:13:33 +05:30
#### Crafting the `.gitlab-ci.yml` file
2019-12-04 20:38:33 +05:30
2020-01-01 13:55:28 +05:30
In a `.gitlab-ci.yml` file in the root of your project, place the following code:
2019-12-04 20:38:33 +05:30
```yaml
image: node:latest
stages:
- deploy
production:
stage: deploy
before_script:
- npm config set prefix /usr/local
- npm install -g serverless
script:
- serverless deploy --stage production --verbose
environment: production
```
This example code does the following:
2020-04-22 19:07:51 +05:30
1. Uses the `node:latest` image for all GitLab CI/CD builds
2019-12-04 20:38:33 +05:30
1. The `deploy` stage:
2020-01-01 13:55:28 +05:30
- Installs the Serverless Framework.
- Deploys the serverless function to your AWS account using the AWS credentials
defined above.
2019-12-04 20:38:33 +05:30
2020-04-08 14:13:33 +05:30
#### Setting up your AWS credentials with your GitLab account
2019-12-04 20:38:33 +05:30
2020-01-01 13:55:28 +05:30
In order to interact with your AWS account, the GitLab CI/CD pipelines require both `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` to be defined in your GitLab settings under **Settings > CI/CD > Variables**.
2021-09-30 23:02:18 +05:30
For more information please see [Create a custom variable in the UI](../../../../ci/variables/index.md#custom-variables-validated-by-gitlab).
2019-12-04 20:38:33 +05:30
2021-01-03 14:25:43 +05:30
The AWS credentials you provide must include IAM policies that provision correct
access control to AWS Lambda, API Gateway, CloudFormation, and IAM resources.
2019-12-04 20:38:33 +05:30
2020-04-08 14:13:33 +05:30
#### Deploying your function
2019-12-04 20:38:33 +05:30
2021-02-22 17:27:13 +05:30
`git push` the changes to your GitLab repository and the GitLab build pipeline deploys your function.
2019-12-04 20:38:33 +05:30
2021-02-22 17:27:13 +05:30
Your GitLab deploy stage log contains output containing your AWS Lambda endpoint URL,
with log lines similar to this:
2019-12-04 20:38:33 +05:30
2020-04-22 19:07:51 +05:30
```plaintext
2019-12-04 20:38:33 +05:30
endpoints:
GET - https://u768nzby1j.execute-api.us-east-1.amazonaws.com/production/hello
```
2020-04-08 14:13:33 +05:30
#### Manually testing your function
2019-12-04 20:38:33 +05:30
Running the following `curl` command should trigger your function.
2021-01-03 14:25:43 +05:30
Your URL should be the one retrieved from the GitLab deploy stage log:
2019-12-04 20:38:33 +05:30
2020-03-13 15:44:24 +05:30
```shell
2021-02-22 17:27:13 +05:30
curl "https://u768nzby1j.execute-api.us-east-1.amazonaws.com/production/hello"
2019-12-04 20:38:33 +05:30
```
2020-01-01 13:55:28 +05:30
That should output:
2019-12-04 20:38:33 +05:30
```json
{
"message": "Your function executed successfully!"
}
```
2020-04-22 19:07:51 +05:30
Hooray! You now have a AWS Lambda function deployed via GitLab CI/CD.
2019-12-04 20:38:33 +05:30
Nice work!
2020-04-08 14:13:33 +05:30
### How To
2020-01-01 13:55:28 +05:30
In this section, we show you how to build on the basic example to:
- Run the function locally.
- Set up secret variables.
- Set up CORS.
2020-04-08 14:13:33 +05:30
#### Running function locally
2020-01-01 13:55:28 +05:30
The `serverless-offline` plugin allows to run your code locally. To run your code locally:
1. Add the following to your `serverless.yml`:
```yaml
plugins:
- serverless-offline
```
1. Start the service by running the following command:
```shell
serverless offline
```
Running the following `curl` command should trigger your function.
2020-03-13 15:44:24 +05:30
```shell
2021-02-22 17:27:13 +05:30
curl "http://localhost:3000/hello"
2020-01-01 13:55:28 +05:30
```
It should output:
```json
{
"message": "Your function executed successfully!"
}
```
2020-04-08 14:13:33 +05:30
#### Secret variables
2020-01-01 13:55:28 +05:30
Secrets are injected into your functions using environment variables.
By defining variables in the provider section of the `serverless.yml`, you add them to
the environment of the deployed function:
```yaml
provider:
2021-01-03 14:25:43 +05:30
# Other configuration omitted
# ...
2020-01-01 13:55:28 +05:30
environment:
A_VARIABLE: ${env:A_VARIABLE}
```
From there, you can reference them in your functions as well.
2021-02-22 17:27:13 +05:30
Remember to add `A_VARIABLE` to your GitLab CI/CD variables under **Settings > CI/CD > Variables** to be picked up and deployed with your function.
2020-01-01 13:55:28 +05:30
2021-02-22 17:27:13 +05:30
NOTE:
2020-03-13 15:44:24 +05:30
Anyone with access to the AWS environment may be able to see the values of those
2020-01-01 13:55:28 +05:30
variables persisted in the lambda definition.
2020-04-08 14:13:33 +05:30
#### Setting up CORS
2020-01-01 13:55:28 +05:30
If you want to set up a web page that makes calls to your function, like we have done in the [template](https://gitlab.com/gitlab-org/project-templates/serverless-framework/), you need to deal with the Cross-Origin Resource Sharing (CORS).
The quick way to do that is to add the `cors: true` flag to the HTTP endpoint in your `serverless.yml`:
```yaml
functions:
hello:
handler: src/handler.hello
events:
2021-01-03 14:25:43 +05:30
- http: # Rewrite this part to enable CORS
2020-01-01 13:55:28 +05:30
path: hello
method: get
2021-01-03 14:25:43 +05:30
cors: true # <-- CORS here
2020-01-01 13:55:28 +05:30
```
You also need to return CORS specific headers in your function response:
```javascript
'use strict';
module.exports.hello = async event => {
return {
statusCode: 200,
headers: {
// Uncomment the line below if you need access to cookies or authentication
// 'Access-Control-Allow-Credentials': true,
'Access-Control-Allow-Origin': '*'
},
body: JSON.stringify(
{
message: 'Your function executed successfully!'
},
null,
2
),
};
};
```
2020-06-23 00:09:42 +05:30
For more information, see the [Your CORS and API Gateway survival guide](https://www.serverless.com/blog/cors-api-gateway-survival-guide/)
2020-01-01 13:55:28 +05:30
blog post written by the Serverless Framework team.
2020-04-08 14:13:33 +05:30
#### Writing automated tests
2020-01-01 13:55:28 +05:30
The [Serverless Framework](https://gitlab.com/gitlab-org/project-templates/serverless-framework/)
example project shows how to use Jest, Axios, and `serverless-offline` plugin to do
automated testing of both local and deployed serverless function.
2020-04-08 14:13:33 +05:30
### Examples and template
2020-01-01 13:55:28 +05:30
The example code is available:
2019-12-04 20:38:33 +05:30
2020-06-23 00:09:42 +05:30
- As a [clonable repository](https://gitlab.com/gitlab-org/serverless/examples/serverless-framework-js).
2020-01-01 13:55:28 +05:30
- In a version with [tests and secret variables](https://gitlab.com/gitlab-org/project-templates/serverless-framework/).
2019-12-04 20:38:33 +05:30
2022-01-26 12:08:38 +05:30
You can also use a [template](../../working_with_projects.md#create-a-project)
2020-01-01 13:55:28 +05:30
(based on the version with tests and secret variables) from within the GitLab UI (see
the `Serverless Framework/JS` template).
2020-04-08 14:13:33 +05:30
## AWS Serverless Application Model
AWS Serverless Application Model is an open source framework for building serverless
applications. It makes it easier to build and deploy serverless applications. For more
details, please take a look at AWS documentation on [AWS Serverless Application Model](https://docs.aws.amazon.com/serverless-application-model/).
### Deploying AWS Lambda function using AWS SAM and GitLab CI/CD
GitLab allows developers to build and deploy serverless applications using the combination of:
- [AWS Serverless Application Model (AWS SAM)](https://aws.amazon.com/serverless/sam/).
- GitLab CI/CD.
### Example
2021-02-22 17:27:13 +05:30
This example shows you how to:
2020-04-08 14:13:33 +05:30
- Install SAM CLI.
- Create a sample SAM application including a Lambda function and API Gateway.
- Build and deploy the application to your AWS account using GitLab CI/CD.
### Steps
The example consists of the following steps:
1. Installing SAM CLI.
1. Creating an AWS SAM application using SAM CLI.
1. Crafting the `.gitlab-ci.yml` file.
1. Setting up your AWS credentials with your GitLab account.
1. Deploying your application.
1. Testing the deployed function.
### Installing SAM CLI
AWS SAM provides a CLI called AWS SAM CLI to make it easier to create and manage
applications.
Some steps in this documentation use SAM CLI. Follow the instructions for
[installing SAM CLI](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-install.html)
to install and configure SAM CLI.
2021-01-29 00:20:46 +05:30
If you use [AWS Cloud9](https://aws.amazon.com/cloud9/) as your integrated development
2020-04-08 14:13:33 +05:30
environment (IDE), the following are installed for you:
- [AWS Command Line Interface](https://docs.aws.amazon.com/en_pv/cli/latest/userguide/cli-chap-install.html)
- [SAM CLI](https://docs.aws.amazon.com/en_pv/serverless-application-model/latest/developerguide/serverless-sam-cli-install.html)
- [Docker](https://docs.docker.com/install/) and necessary Docker images.
### Creating an AWS SAM application using SAM CLI
To create a new AWS SAM application:
1. Create a new GitLab project.
1. `git clone` the project into your local environment.
1. Change to the newly cloned project and create a new SAM app using the following command:
```shell
sam init -r python3.8 -n gitlabpoc --app-template "hello-world"
```
1. `git push` the application back to the GitLab project.
This creates a SAM app named `gitlabpoc` using the default configuration, a single
2021-01-29 00:20:46 +05:30
Python 3.8 function invoked by an [Amazon API Gateway](https://aws.amazon.com/api-gateway/)
2020-04-08 14:13:33 +05:30
endpoint. To see additional runtimes supported by SAM and options for `sam init`, run:
```shell
sam init -h
```
### Setting up your AWS credentials with your GitLab account
In order to interact with your AWS account, the GitLab CI/CD pipelines require both
2021-04-17 20:07:23 +05:30
`AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` to be set in the project's CI/CD variables.
2020-04-08 14:13:33 +05:30
To set these:
2021-04-17 20:07:23 +05:30
1. Navigate to the project's **Settings > CI/CD**.
2021-01-29 00:20:46 +05:30
1. Expand the **Variables** section and create entries for `AWS_ACCESS_KEY_ID` and
2020-04-08 14:13:33 +05:30
`AWS_SECRET_ACCESS_KEY`.
1. Mask the credentials so they do not show in logs using the **Masked** toggle.
The AWS credentials you provide must include IAM policies that provision correct access
control to AWS Lambda, API Gateway, CloudFormation, and IAM resources.
### Crafting the `.gitlab-ci.yml` file
2021-09-30 23:02:18 +05:30
In a [`.gitlab-ci.yml`](../../../../ci/yaml/index.md) file in the root of your project,
2020-04-22 19:07:51 +05:30
add the following and replace `<S3_bucket_name>` with the name of the S3 bucket where you
2020-04-08 14:13:33 +05:30
want to store your package:
```yaml
image: python:latest
stages:
- deploy
production:
stage: deploy
before_script:
2022-03-02 08:16:31 +05:30
- apt-get update
- apt-get install -y python3-pip
2020-04-08 14:13:33 +05:30
- pip3 install awscli --upgrade
- pip3 install aws-sam-cli --upgrade
script:
- sam build
- sam package --output-template-file packaged.yaml --s3-bucket <S3_bucket_name>
- sam deploy --template-file packaged.yaml --stack-name gitlabpoc --s3-bucket <S3_bucket_name> --capabilities CAPABILITY_IAM --region us-east-1
environment: production
2020-07-28 23:09:34 +05:30
```
2020-04-08 14:13:33 +05:30
2021-04-29 21:17:54 +05:30
Let's examine the configuration file more closely:
2020-04-08 14:13:33 +05:30
- `image` specifies the Docker image to use for this build. This is the latest Python
image since the sample application is written in Python.
- AWS CLI and AWS SAM CLI are installed in the `before_script` section.
- SAM build, package, and deploy commands are used to build, package, and deploy the
application.
### Deploying your application
2021-02-22 17:27:13 +05:30
Push changes to your GitLab repository and the GitLab build pipeline
deploys your application. If your:
2020-04-08 14:13:33 +05:30
- Build and deploy are successful, [test your deployed application](#testing-the-deployed-application).
- Build fails, look at the build log to see why the build failed. Some common reasons
the build might fail are:
- Incompatible versions of software. For example, Python runtime version might be
different from the Python on the build machine. Address this by installing the
required versions of the software.
2021-04-17 20:07:23 +05:30
- You may not be able to access your AWS account from GitLab. Check the CI/CD variables
you set up with AWS credentials.
2020-04-08 14:13:33 +05:30
- You may not have permission to deploy a serverless application. Make sure you
provide all required permissions to deploy a serverless application.
### Testing the deployed application
To test the application you deployed, please go to the build log and follow the following steps:
2021-04-29 21:17:54 +05:30
1. Click on "Show complete raw" on the upper right-hand corner:
2020-04-08 14:13:33 +05:30
2021-12-11 22:18:48 +05:30
![SAM complete raw](img/sam-complete-raw.png)
2020-04-08 14:13:33 +05:30
1. Look for HelloWorldApi API Gateway endpoint similar to shown below:
2021-12-11 22:18:48 +05:30
![SAM API endpoint](img/sam-api-endpoint.png)
2020-04-08 14:13:33 +05:30
1. Use curl to test the API. For example:
```shell
2021-02-22 17:27:13 +05:30
curl "https://py4rg7qtlg.execute-api.us-east-1.amazonaws.com/Prod/hello/"
2020-04-08 14:13:33 +05:30
```
Output should be:
```json
{"message": "hello world"}
```
### Testing Locally
AWS SAM provides functionality to test your applications locally. You must have AWS SAM
CLI installed locally for you to test locally.
First, test the function.
2021-01-29 00:20:46 +05:30
SAM provides a default event in `events/event.json` that includes a message body of:
2020-04-08 14:13:33 +05:30
2020-04-22 19:07:51 +05:30
```plaintext
{\"message\": \"hello world\"}
2020-04-08 14:13:33 +05:30
```
If you pass that event into the `HelloWorldFunction`, it should respond with the same
body.
Invoke the function by running:
```shell
sam local invoke HelloWorldFunction -e events/event.json
```
Output should be:
```json
{"message": "hello world"}
```
After you confirm that Lambda function is working as expected, test the API Gateway
using following steps.
Start the API locally by running:
```shell
sam local start-api
```
SAM again launches a Docker container, this time with a mocked Amazon API Gateway
2021-01-29 00:20:46 +05:30
listening on `localhost:3000`.
2020-04-08 14:13:33 +05:30
Call the `hello` API by running:
```shell
2021-02-22 17:27:13 +05:30
curl "http://127.0.0.1:3000/hello"
2020-04-08 14:13:33 +05:30
```
Output again should be:
```json
{"message": "hello world"}
```