Thursday, 2 December 2021

Troubleshooting, Tips and Tricks

 

1. How to downgrade/upgrade Jenkins

To Downgrade/upgrade Jenkins on ubuntu from eg version 2.320 to version 2.318 use the below command


sudo apt install jenkins=2.318

sudo systemctl restart jenkins



2. My Instance is Slow after installing Sonarqube

a. increase the storage volume 

b. upgrade your instace type from t2micro to like t2medium

c. try this steps to Extend the OS file

     i. log into the instance and enter the below commands:

      sudo growpart /dev/xvda 1

      sudo resize2fs /dev/xvda1

source:https://blog.shikisoft.com/increase-root-volume-of-ebs-backed-ec2/


3. How to change ansible Tower password(If you forgot)

Log into the Tower Console

Enter the following commands

docker ps
docker exec -it awx_web bash
awx-manage changepassword admin

http://vcloud-lab.com/entries/devops/reset-ansible-awx-tower-admin-password

4. Sonarqube is under maintenance




    1. Now browse http://publicip:9000/setup to upgrade your database
    2. After upgrade the database, browse your sonarqube server

     

  • 5.  Your jenkins pipeline couldnt find a program, eg. kubectl: like below

  • + kubectl get nodes
    /var/lib/jenkins/workspace/test@tmp/durable-fd334313/script.sh: line 1: kubectl: command not found

Login to the machine and copy the program into /usr/local/bin
cp kubectl /usr/local/bin



Sunday, 21 November 2021

Serverless websites on AWS using Lambda and API Gateway


My first thought was: "If there are more than 200+ AWS webservices available as of today, from the statistical standpoint it is very probable that such architecture could be constructed - either by using 1 or combination of more of the remaining 195 services."


The following are said to be required for Serverless process to be feasible:

Req. 1: We need a "Place" in which website components can be stored (storage)

Req. 2: We need an "Address" that can be entered into the browser's location bar allowing users access website (URL endpoint)
Req. 3: We need a "Service" that can listen on "the Address" to the incoming HTTPS requests issued by user's browser and in real-time serve website content stored in "the Place" (webserver app)

Having defined these, the selection of the suitable AWS webservices for the AWS webhosting architecture including creating a corresponding architecture turned out to be an easy task.

Serverless Architecture

AWS Lambda and Amazon API Gateway - two AWS webservices which, when used together as depicted in the following scheme, can fullfil all of our requirements.

AWS Lambda and Amazon API Gateway serverless architecture for hosting static websites

Let's host a serverless website using Lambda and API Gateway

Note: In our example we are going to use AWS Region EU-CENTRAL-1 (Frankfurt). Please feel free to use any other region that suits you most, just make sure that region supports AWS Lambda and Amazon API Gateway. Also if you choose to use different region, be aware that URLs of AWS services mentioned in this article need to be updated as well to fit the new region.

Before we start "playing" with these 2 AWS webservices, we will need a webapp.

1) Sample company Nodejs app

To not overcomplicate things, for the purposes of this article, I created a really very simple Nodejs app.

This website consists of the following website components:

  • 2 HTML files (one representing the index page and the other one the contact page)
  • 1 CSS file (providing formatting and styling - sample fonts and colors)
  • 1 JavaScript file (providing sample animation - snowing effect)
  • 1 PNG image (sample company logo)
  • 1 PDF file (representing sample binary file for download - pricelist)

Before proceeding further, it's necessary that you download the sample Nodejs app I prepared - myLambdaWebsite.zip and store it locally on your computer.




2) AWS Lambda setup

Now, once you have myLambdaWebsite.zip, let's upload it into AWS Lambda.

Here are the steps that need to be taken:

1. Navigate to the awslambda....search for lambda on your aws console


2. Click on the orange button "Create function" at the top right part of the screen



2.1. As "Function name" enter: myLambdaWebsiteas "Runtime" make sure the following is selected: Node.js 14.x

2.2. Click on the orange button "Create function" at the bottom right part of the screen

3. Within the "Code Source" box click on the "Upload From" and choose: " .zip file"


3.1. Click on the "Upload" button and choose the myLambdaWebsite.zip file from the location where you stored it

3.2. Click on the orange button "Save"

This is how your screen should look like if you did it right:

Besides the above mentioned files which were described as being the website components, you can notice that there is 1 additional file that we haven't mentioned yet. It is located in the root directory and it is called "index.js"
What is "index.js" for?

Let me explain..

AWS Lambda is not a webserver to which you just upload your website component files and that's it. AWS Lambda is a serverless compute service that lets you run your code. So that's why we need to upload a special code to the Lambda so that Lambda function knows what to do with the static website component files that we uploaded there as well.

exports.handler = async (event) => {

  try {
    switch(event['path']) { 
      case "/": case "/html": case "/html/": case "/index.html": { 
        event['path'] = "/html/index.html"; 
      } 
    }

    var b = false, 
        s = 200, 
        f = require('fs'), 
        d = f.readFileSync("."+event['path'],"binary"), 
        path = event['path'].split("/"), 
        m = new Map([
            ["htm","text/html"],
            ["html","text/html"],
            ["css","text/css"],
            ["ico","image/x-icon"],
            ["js","text/javascript"],
            ["doc","application/msword"],
            ["docx","application/vnd.openxmlformats-officedocument.wordprocessingml.document"],
            ["gif", "image/gif"],
            ["jpg","image/jpeg"],
            ["jpeg","image/jpeg"],
            ["pdf","application/pdf"],
            ["png","image/png"],
            ["ppt","application/vnd.ms-powerpoint"], 
            ["pptx","application/vnd.openxmlformats-officedocument.presentationml.presentation"],
            ["svg","image/svg+xml"],
            ["txt","text/plain"],
            ["xls","application/vnd.ms-excel"],
            ["zip","application/zip"],
            ["xlsx","application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"]]),
         c = m.get(path[2].split(".")[1]);

    switch(path[1]) { 
      case "img": case "download": { 
        b = true; 
        d = Buffer.from(d,'binary').toString('base64'); 
      } 
    }

  }
  catch (e) { 
    s = 404; 
    c = "text/html", 
    d = f.readFileSync("./html/404.html","binary"); 
  }

  return { 
    statusCode: s, 
    isBase64Encoded: b, 
    headers: {"Content-Type": c}, 
    body: d
  };
};

Everytime this Lambda function is invoked, index.js is executed. Based on the parameters it receives (API Gateway provides to Lambda the URL path), index.js decides which website component file it is going to serve. After it manages to retrieve the website component file from the Lambda function internal local storage, it sends out to the output: HTTP status code, file's Content-Type and the contents of the file itself.

In other words, our index.js file is a simple webserver application written in Node.js that serves static content stored as individual files in the Lambda function itself.

Amazon API Gateway logo

3. Amazon API Gateway setup

Why do we need to configure additional AWS webservice? Isn't AWS Lambda enough?

No it isn't, because AWS Lambda functions are by default not accessible from the outside world (there is no URL which you can type into your browser to access them).

In order to have one, what we need to do, is to setup a custom REST endpoint in the AWS Cloud and "wire" this endpoint to the Lambda function. Technically said, using Amazon API Gateway we have to map access to a specific resource (URL) with the HTTPS method to the invocation of a Lambda function.

Once we do it, we will be able to visit our website by putting this specific endpoint URL as address into the web browser.

And this the reason why we need the Amazon API Gateway.

Here are the steps that need to be taken:

1. Navigate to the Amazon Api Gateway...search for Api Gateway in your console


2. Click on the orange button "Create API" located at the top right part of the screen

3. Within the "REST API" box click on the orange button "Build" 

3.1. As "API name" enter: My Lambda Website 3.2. Click on the blue button "Create API" 

4. From the left sidebar under the "API: My Lambda Website" choose "Settings" (note: there are 2 "Settings" in the left sidebar, choose the indented one with the smaller font)

4.1. Scroll to the bottom of the screen 4.2. Under the "Binary Media Types" heading click on the "+ Add Binary Media Type" and into the input field enter: */*4.3. Scroll to the bottom of the screen and click on the blue button "Save Changes" 


5. From the left sidebar under the "API: My Lambda Website" choose "Resources"5.1. From the "Actions drop-down" choose: Create Method



5.2. Choose "ANY" from the drop-down and click on the grey tick icon


5.3. Make sure the checkbox is checked for the option: "Use Lambda Proxy Integration"
5.4. As "Lambda function" enter the name of your Lambda function: myLambdaWebsite

5.5 Click on the blue button "Save" and then click on the blue button "OK" 6. From the "Actions drop-down" choose: "Create Resource"



6.1. Make sure the checkbox is checked for the option: "Configure as proxy resource" 6.2. Click on the blue button "Create Resource" 


6.3. As "Lambda function" enter the name of your Lambda function: myLambdaWebsite
6.4. Click on the blue button "Save" and then click on the blue button "OK" 

7.  From the "Actions drop-down" choose: Deploy API

7.1. As "Deployment stage" choose: [New Stage], as "Stage name" enter: prod , you can enter any environment name you choose


7.2. Click on the blue button "Deploy" 

8. Copy the "Invoke URL" into the clipboard 

4. Last step

Open your web browser, paste from the clipboard the "Invoke URL" that you just copied, press Enter and the website will load.If you have done everything right according to the steps provided above, you should see on your browser screen this:


Frequently asked questions

1. Are there any limits to the size of the static website which I can upload/host using AWS Lambda?

Yes, there are. You can only upload to each Lambda function ZIP archive of max. size of 50MBs (unzipped content may consume up to 250 MBs though).

2. I uploaded larger website as a ZIP archive into AWS Lambda - the operation was a success, but suddenly I can't use the Lambda online editor anymore. Why is it so?

This is because AWS Lambda console editor has a limit that prevents it from working when the size of your Lambda function exceeds 3MBs.

3. Updating of my static website stored in the local AWS Lambda function storage is not very comfortable for me, especially when I want to add new or update existing images or other binary files. Aren't there any simpler options than including all files in ZIP archive and uploading them back to AWS Lambda?

Yes, there are. But you need to modify index.js file so that it doesn't retrieve files from your AWS Lambda function storage but from some other location (e.g. database). I am about to write new article about this possibility and will reference it from this article once it becomes published.

4. I followed the steps described in this article, but my browser doesn't display images properly. They are rendered as broken images. What am I doing wrong?

This is most probably because you have either skipped step no.4 in the above mentioned Amazon API Gateway instructions or didn't follow this step properly. Please check and repeat this step again, I recommend that you also redeploy your API afterwards.

5. Is AWS Lamba and Amazon API Gateway part of the Free Tier programme? How much do I need to pay for using these webservices?

Great news is that Lambda comes with unlimited Free Tier in which 1 million of Lambda requests and 400,000 GB-seconds of compute time per month are offered to you completely for free. If you go over, you have to pay. See Lambda pricing

As for API Gateway, its Free Tier is more limited and valid only for the 1-year of your AWS account ownership, within this period of time you can make 1 million of API calls, receive 1 million messages and have 750,000 connection minutes per month for free. Fore more info about API gateway pricing, visit this page.

6. Is there a way how to optimize number of API requests and Lambda invocations when hosting website on Lambda and API Gateway?

Yes. There is. One of the ways is to transform your website into the Single Page Application (SPA), the other way is to start embedding content directly in the HTML source of your pages - instead of referencing image files, start embedding them in SVG format, instead of using external CSS and JavaScript files, embed their source into the HTML directly.

7. Quite frankly, the URL of my website is looking quite ugly (%some-random-string%.execute-api.eu-central-1.amazonaws.com/web/). Is there anything I could do about it, can I host my website on traditional domain name like address?

Yes. Just follow the instructions provided by Amazon for setting up custom domain.

Final notes

In the example provided above, I showed you how you can host an entire static website in AWS Cloud using just AWS Lambda and Amazon API Gateway.

Please consider it more like a Proof of Concept which confirms that hosting of static websites using these 2 AWS webservices is technically possible.

There are far much more effective, less complicated and more powerful solutions existing in AWS for static website hosting, 

AWS Lambda and Amazon API Gateway are though very effective for creating serverless backends (microservices).

You can also use them for dynamic website hosting and if this is your case I recommend you to explore the following solutions:

  • AWS Serverless Express - NodeJS-based API framework mimicking routing capabilities of ExpressJS framework inside Lambda functions
  • Bref - an open source project that brings full support for PHP and its frameworks to AWS Lambda
  • ClaudiaJS - Deploying Node.js projects
  • Sparta - framework that transforms a go application into a self-deploying AWS Lambda powered service
  • Up - Deploying Node.js, Golang, Python, Java, Crystal, Clojure and static sites
  • Zappa - Deploying event-driven Python applications (incl. WSGI web apps)

How to upgrade Maven

  java.lang.IllegalStateException I had installed maven in my ubuntu using command  apt install maven This installed maven in path /usr/shar...